Source code Inverse Perspective Mapping C++, OpenCV

Hi all,

Today I bring a very simple code that might be of interest for some of you. It is a C++ class to compute Inverse Perspective Mappings (IPM), or sometimes called bird’s-eye views of a planar surface.

The link to the code: https://sourceforge.net/projects/ipmapping/files/

(I am providing here a CMakeLists.txt and some c++ files, so you need to create a solution with CMake, and the a C++ compiler).

sample

It is nothing else than a plane-to-plane homography, but in my experience it is not that easy to compute. I am using here OpenCV remap functions.

In the example I have hard-coded the necessary four-point correspondences required to compute the homography, but you can compute then as you want. You might use vanishing points, camera calibration information, or any other source.

Of course this type of classes can get really useful when you add them information about the camera calibration.

Hope it is useful!

Kind regards,

This entry was posted in ADAS, Computer vision, OpenCV and tagged , , , , , . Bookmark the permalink.

41 Responses to Source code Inverse Perspective Mapping C++, OpenCV

  1. This tutorial was really helpful. Thanks.
    I have a question though. How can I estimate the distance to the car ahead using IPM information?

    • Hi!
      Thanks. This tutorial shows only how to create warped images. To compute distances you need to calibrate the camera and determine its R and t with respecto to the plane; or perhaps you can just get metric references in your scene and apply scale factor in x and y to the birds-eye views.
      Regards!

  2. Edson Valle says:

    Hello Marco
    OpenCV has also a funcion called warpPerspective to do this perspective transformation.
    Do you know which is faster? I mean, I would like to implement webcam capture, but with warpPerspective, I get no more than 8 fps in my computer..
    Regards

    • Hi Edson,
      The example I provide uses the OpenCV’s remap function, which is way faster than warpPerspective since it pre-computes all the projections that fill the destination image and store the indexes in a x-map and a y-map.
      Kind regards,
      Marcos

      • bad_keypoints says:

        Hi, I want to know if in your case your homography never changes, and that’s why you’re using remap, instead of warpPerspective. Please invest 2 minutes of your life, and have a look at my problem http://stackoverflow.com/questions/32880842/opencv-warpperspective-slow-performance

        Basically, I’m working on an Android app, in which when an object is detected, a video will be played over it, no matter how the camera moves. That being said, the major problem is thus caused by lengthy times of warpPerspective warping the video frames I overlay on the found object in every frame from camera. Like, ~300ms per frame, on a Qualcomm Snapdragon 800 quad core 2.3 GHz processor Android phone with 2GB of RAM.

        And from what I gathered from here: http://romsteady.blogspot.in/2015/07/calculate-opencv-warpperspective-map.html, remap is useful only if homography is constant. Please prove me wrong, with some explanation, so that I can speed up my warp times.

      • Hi!
        This is a very good question.
        The OpenCV warpPerspective function is a function that creates a warp from the scratch, converting each position of a plane into another position using the homography information. Of course, this is very intense computation for images, since each pixel need to be transformed.
        The remap function computes a look-up table, that basically takes the same time as warpPerspective, but whose result can be stored for reuse in subsequent images. You gain time because access to memory is much faster than the projective equations.
        However, of course, if your homography is changing, you can not keep the result of the remap function! Because you can not take advantage of precomputations done in the past.

        In your application, it is absolutely required to map into the object, therefore you need to go through this step. You have maybe two main options:
        1)Using OpenGL instead of OpenCV, because it is specially designed to render 3D entities and creating warps. I am unsure if your HW does have support for OpenGL or has GPU capabilities, but definitely it is worth to investigate the question.
        2)If you have good programming skills, you can go for optimize the code, unwrap loops, use pointer arithmetic and use carefully the registers of your machine by dissasembling the code.

        Good luck with your app!
        Regards,

        Marcos

  3. Jas says:

    Hello,
    Thank you for sharing the source code. I am pretty new to opencv (Actually I just started with Opencvsharp). However, the Opencvsharp library is not as rich as OpenCV itself, so in this case I couldn’t find some methods you have used in this code (for instance method void push-back…).
    Have you already try to implement it in C# ?
    I was wondering if you could help me with this case or if you have the code already so I can check my progress.
    Regards

    • Hi!
      You have actually brought to my attention that there is a C# version of OpenCV. In my research I directly work in C++ (no Matlab nor Python), which in some cases is troublesome, but at the end allow me to have quick prototypes ready for the customers. Using C++ with Qt can possibly be a good alternative to C#.
      Kind regards,
      Marcos

      • Jas says:

        Thank you for your reply. yes there are some wrappers but not as complete as Opencv itself! At the moment I am restricted to working in C# (other parts of the project has been done in C# so far..). So I am guessing that you are not interested in trying the algorithm in Opencvsharp!
        Regards

  4. Zamani says:

    Hi Macro,
    I just want to know what type of camera that you used to capture the video and the images?

    Thanks

  5. Gaurav says:

    Hi Macro,
    My objective is to get a birds eye view of the image and using this, detect lane markings and vehicles and calculate distance from detected objects. Warp perspective is giving me a lot of issues. Also you have mentioned that remap is faster.
    Now I am using as input to opencv a video captured from a camera placed in a moving car. I only have the video. So will this birds eye view be of any help to me? is it even possible to get it?? I do not have camera parameters or any other details. Calculating distance is also imperative. I am using VC++ 2010
    Please shed some light on this. And also please give me some pointers to research. I have a very limited timeline.

    • Hi,
      If your target application can not have a calibration step, you have two options: (a) don’t use bird’s-eye view and explore the entire image searching for vehicles and lanes without info about the perspective; or (b) autocalibrate the scene using the computation of the vanishing points or something equivalent.
      Kind regards,
      Marcos

  6. Gaurav says:

    Also to add to the above post, I am using this code to get birds eye from an image similar to the one you have used. But the output is not even close to birds eye. Not clear about the co-ordinate geometry.Please elucidate.

    int main(void)
    {
    Mat input,dst;
    Point2f inputQuad[4];
    // Output Quadilateral or World plane coordinates
    Point2f outputQuad[4];

    // Lambda Matrix
    Mat lambda( 2, 4, CV_32FC1 );
    //Input and Output Image;

    //Load the image
    input = imread( “C:/176955961.jpg”, 1 );
    imshow(“original”, input);//300×168
    // Set the lambda matrix the same type and size as input
    // Set the lambda matrix the same type and size as input
    lambda = Mat::zeros( input.rows, input.cols, input.type() );

    // The 4 points that select quadilateral on the input , from top-left in clockwise order
    // These four pts are the sides of the rect box used as input
    inputQuad[0] = Point2f( 200,50 );
    inputQuad[1] = Point2f( 300,50);
    inputQuad[2] = Point2f( 0,350 );
    inputQuad[3] = Point2f( input.cols,350);
    //
    // The 4 points where the mapping is to be done , from top-left in clockwise order
    outputQuad[0] = Point2f( 0,0 );
    outputQuad[1] = Point2f( input.cols,0);
    outputQuad[2] = Point2f( 100,input.rows);
    outputQuad[3] = Point2f( input.cols-100,input.rows);

    // Get the Perspective Transform Matrix i.e. lambda
    lambda = getPerspectiveTransform( inputQuad, outputQuad );
    // Apply the Perspective Transform just found to the src image
    warpPerspective(input,dst,lambda,dst.size() );
    imshow(“cpp”,dst);
    return 0;
    }

    • Hi,
      Well, obviously, you are using the default 4-point correspondences I used for my case (which were hardcoded to keep the sample code easy and simple). For your case “C:/176955961.jpg”, you will need to determine how these correspondences are.
      The IPM code assumes you’ve got this info. Methods for getting this can include vanishing point estimation, which is another one main topic in the blog.
      Kind regards,
      Marcos

  7. Gaurav says:

    Also I need to use opencv functions that have minimum or absolutely no system calls because my final intention is to port this code to an embedded platform without OS support.
    objectives: lane marking detection, vehicle detection and distance calculation.
    Also read in a paper that we can use birds eye in combination with a few other algorithms to even detect objects.
    Thanks in advance. Appreciate your time and effort.
    Regards,
    Gaurav

    • Hi,
      Your goals are in line with much research ongoing in the community, so you can find pretty a lot of literature about vehicle detection and calibration (for distance calculation).
      Bird’s-eye view is a nice technique, it can provide some help, but be careful, because computing it has some costs, and needs to be done quite accurately, otherwise you can get undesired additional distortions in your models.
      Kind regards,
      Marcos

  8. Zamani says:

    Hi Marco,
    Another question that I really want to ask regarding your project at youtube, how the output can be flawlessly display the video without seeing the delay causing the output to show frame by frame. What is the method had been used? I had been doing until the Hough Transform process and my output will not display as what you got … I can see the delay between frames or the frames getting slower compare to the original video..

    thanks for the response

  9. Jérôme says:

    Hi,
    Thank you for sharing your code.
    There is a little typo error in:
    void IPM::applyHomographyInv(const Mat& _inputImg, Mat& _dstImg, int _borderMode)
    {
    // Generate IPM image from src
    remap(_inputImg, _dstImg, m_mapX, m_mapY, INTER_LINEAR, _borderMode);
    // it should be remap(_inputImg, _dstImg, m_invMapX, m_invMapY, INTER_LINEAR, _borderMode);
    }
    No questions here, just a thank you ;)
    Best regards,
    Jérôme.

    • Hi Jérôme,

      Thanks a lot for your contribution!
      I believe this bug was corrected in the trunk of the project, so probably you have a non-updated version?
      Regards!

      Marcos

      • Jérôme says:

        yes It is corrected on the trunk.
        I used the direct download link (ipm_code.rar) which last modification date is 2014-02-22.

  10. Neal says:

    Hi,Marcos.I build the project as the source code you provided.But when I run it with the command ipm road.avi.It always error with the information :Error opening file<../../modules/highgui/src/cap_ffmpeg_impl.hpp:537).It seems not find the road.avi file,but the file belongs to the same file folder.And I have also tried to add the full path of the file.But it doesn't work still.Could you tell me what's wrong.Thank you.
    Best regards

    • Hi,
      If you have typed correctly the path or placed the video in the folder where the exe resides, then, it is possible that you do not have the codecs linked to Direct Show. OpenCV uses ffmpeg and it gives this kind of errors when it is not able to open the video file. Try installing XVID (I believe the video is encoded like that although I don’t really remember now), or install the ffdshow codecs (http://sourceforge.net/projects/ffdshow/).
      Regards,
      Marcos

  11. srezarn says:

    Hi Macro,
    Thanks for your code and sharing
    I want know how we can back-projection from IPM to world?

    • Hi,
      You can map from the IPM domain (the bird’s-eye view image) to the world plane by using known scale factors for the X and Y coordinates. Obviously, this need to be provided as input calibration information, or estimated through the observation of known-sized objects in the image.
      This topic (calibration) is challenging, specially in uncontrolled environments like the road and used single camera setups.

      You can take a look to calibration-related projects elsewhere. Also I did discuss about it in my PhD thesis: https://marcosnietoblog.wordpress.com/about/my-phd/

      Regards,
      Marcos

  12. Hello Marcos,
    I’m using you IPM library to estimate the distance of a vehicle in front a facing forward camera. I have the camera calibration parameters and camera’s pitch angle is estimated using the vanish point.

    The problem is that my IPM image is bouncing a lot, making the distance estimation very imprecise. I don’t know if this problem is related to the pitch estimation or anything else. Can you point me a solution?

    Thanks in advance.

  13. Hello Marcos
    I’m currently using your IPM library to estimate the distance of a vehicle seen by a facing forward camera. I have the intrinsic camera calibration parameters and the camera’s pitch angle is computed using vanish point estimation.

    The IPM works well. But my IPM image is strangely bouncing and I don’t know why. This leads me to estimate an imprecise distance to the front vehicle, because the same seems to goes foward and backward as the camera moves. Is this related to the pitch angle estimate? Can you point me a solution?
    Thanks in advance.

    • Hi!
      This is a well-known problem of IPM. It is assuming that the road is a plane. Obviously, real roads are not. They have vertical curvature, therefore the plane-to-plane mapping introduce distortion. Additionally, the camera is never moving perfectly straight and parallel to the road plane. There are vibrations from the car, suspension, and also acceleration and deceleration effects. In summary, the IPM can only hope to have an approximate parallel view of the road, but an arbitrary amount of distortion and variability.
      The distance to the other vehicles must be computed using filtering techniques, such as Kalman filter, that absorb (part of) this variability and produce smooth results. This is a challenge and not yet solved by the scientific community in an universal manner.
      Regards,
      Marcos

      • Hello Marcos,
        Thank you for your quick reply. It was very helpful. I have applied a KF to my distance estimations and it seems better now.
        Thanks,

  14. Bob says:

    Hi Marcos,
    I looked at your IPM code, it requires four points at the source image and four corresponding points to do the job. But my project requires to using camera intrinsic matrix and transformation matrix. How can I do it? overload the getPerspectiveTransform() with my code for calculating homography matrix?
    Regards
    Bob

    • Hi,
      If you do have full calibration, you’ve got all the information you need to project from 3D to 2D. Also, you can create bird’s-eye views of the road, if this is your goal.
      Instead of chosing 4 points manually, you can select them on the 3D world, and project them into the image using your projection matrix. Then, you can create the warp.
      Regards,
      Marcos

  15. Tin Duong says:

    Hi Macro,
    Thanks for your code and sharing.
    Is there a way to use your codes to back-projectio fromn IPM image to normal image ?
    Thank you in advance.

  16. maithamdib says:

    HI Marcos thanks for your hard work. Im carrying out a research project regarding lane detection on smartphone cameras. I would like to extract an IPM of just the Lane in front of the camera including 25% either side of the lane. I have my lane detection working but I am trying to implement inverse perspective mapping. I have my camera projection matrix how do I use this to generate the correct 4 point correspondences and the correct homography matrix to get a birds eye view of the road?

    • Hi,
      The inverse perspective mapping can be thought as an homography between 4 points in the image, and 4 points in the world plane. Therefore, if you are able to locate a vanishing point, you can determine 4 points in the image that corresponds to a 4-point rectangular shape in the world (road) plane.
      If you want to extend horizontally your view of the road, you only need to extend the 4-points in the road, and get the corresponding 4 points in the image by using the homography.
      Regards,
      Marcos

  17. maithamdib says:

    Hi Marcos I was also reading your report regarding Robust vanishing point detection, you mention that you can work out the pitch and yaw angles of the camera through vanishing point estimation, using the angular apertures of the camera. But to my understanding there is no vertical and horizontal aperture. Do you mean the vertical and horizontal Field of View?

    • Hi,
      Yes, the field of view. The vanishing point is directly related to a direction in 3D. Therefore, given the instrinsic parameters, you can known a 3D vector from a 2D point or viceversa.
      Regards,
      Marcos

  18. Ridan says:

    Hello,

    I work on the ADAS Systems and I would like to have some Algorithms (code C++ or Matlab) about ADAS example (camera, line tracker…). Could you show me some programme to help me to start my programmation on ADAS.

    Thanks,

    • Hi,
      Unfortunately, I haven’t published code of entire ADAS applications. Basically, I don’t have enough free time to do this because I would need to prepare the code to be readable, and prepare kind of a tutorial. I work for Vicomtech-IK4, who retain all the IPR.
      Nevertheless, you can probably put an eye to OpenCV and the contrib modules, which look to start having ADAS examples.
      Regards,
      Marcos

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s