Vanishing point detection C++ source code

Hi again!

Finally I could spend some time to renew the methods I developed for vanishing point detection during my PhD. Now I have written it in C++ and using OpenCV 2.3 capabilities.

The result is a class called MSAC (M-estimator SAmpling and Consensus), which is able to detect multiple finite and infinite vanishing points given an input set of line segments, that can be computed using the PPHT of OpenCV 2.3.

QR code with vertical and horizontal vanishing points detected

QR code with vertical and horizontal vanishing points detected

QR code with perspective distortion, and finite vanishing points detected

QR code with perspective distortion, and finite vanishing points detected

I have created a full sample which is called vanishingPoint that can read images or video sequences, call the MSAC class, and visualise the result.

Follow the link to the source code host, examples of use and some instructions.

Hope you find it useful! In case so, you might think of referencing my paper about the topic, or my PhD.

  • M. Nieto and L. Salgado, “Real-time robust estimation of vanishing points through nonlinear optimization,” in IS&T/SPIE Int. Conf. on Real-Time Image and Video Processing, SPIE vol. 7724, 772402, 2010. (DOI 10.1117/12.854592)
  • M. Nieto, “Detection and tracking of vanishing points in dynamic environments,” PhD Thesis, Universidad Politécnica de Madrid, 2010.

Thanks!

Marcos

This entry was posted in Computer vision, OpenCV and tagged , , , . Bookmark the permalink.

26 Responses to Vanishing point detection C++ source code

  1. Pingback: Lane markings detection and vanishing point detection with OpenCV | Marcos Nieto's Blog

  2. Link to source code is dead. Could you reupload it?

  3. zakky says:

    how to include and using msac in opencv?

  4. chaupm says:

    when i’m debugging code in VS C++ 2010, it show vanishingpoint\msac.cpp(641): error C2065: ‘lm_printout_std’ : undeclared identifier. Plz tell me how to fix it.
    Thanks so much.

    • You need to include the Levenberg-Marquard library that is a dependency of my code. It is named lmfit and you can download it from http://apps.jcns.fz-juelich.de/doku/sc/lmfit

      • chaupm says:

        in the lmfit 5.0, the “lm_printout_std” is not definition, so I downloaded lmfit 3.2 to replace it, But when i generating code, it show :
        1>MSAC.obj : error LNK2019: unresolved external symbol _lmmin referenced in function “private: void __thiscall MSAC::estimateNIETO(class cv::Mat &,class cv::Mat &,class cv::Mat &,class std::vector<int,class std::allocator > &,int,class cv::Mat &)” (?estimateNIETO@MSAC@@AAEXAAVMat@cv@@00AAV?$vector@HV?$allocator@H@std@@@std@@H0@Z)
        1>MSAC.obj : error LNK2019: unresolved external symbol _lm_printout_std referenced in function “private: void __thiscall MSAC::estimateNIETO(class cv::Mat &,class cv::Mat &,class cv::Mat &,class std::vector<int,class std::allocator > &,int,class cv::Mat &)” (?estimateNIETO@MSAC@@AAEXAAVMat@cv@@00AAV?$vector@HV?$allocator@H@std@@@std@@H0@Z)
        1>MSAC.obj : error LNK2001: unresolved external symbol _lm_control_double.
        Thanks for your help and Please excuse my bad English.

      • Hi!
        Thx for your comments. I really need to check back the compatibility of my sw with lmfit.
        Meanwhile, just try to make sure you are compiling the correct files because a linker error means that you are not linking the objects that contain the correct symbols.
        Another quick solution is to define yourself that variables.
        Good luck!

      • chaupm says:

        When i’m run vp.exe with help in example it show error(please see details in the image attach)
        http://postimg.org/image/bpct7dj7d/
        i’m using opencv2.3, vs C++ express 2010, windows 7 64 bit.
        i don’t understand what i do wrong
        Thanks for your help and Please excuse my bad English.

      • RAY says:

        HI Marcos
        Would you tell me the Vanishing project dependent lmfit version?it is 3.2 or 5.1?
        I tried to search the lmfit download, many download was invalid.
        Would you tell me the valid download with lmfit and version?
        thank you very much

      • Hi Ray,

        Yes, I am still using the old 3.2 version. I know this is a problem since it is hard to find. It looks Joachim has moved into source forge and shares the newest 5.1 version.

        I have had no time to update my code to use this 5.1 version, but it should be quite straightforward to modify it to read the new structs in lmmin.c

        Thanks for making me aware of this, I will try to update my blog soon.

        Kind regards,

        Marcos

  5. dada says:

    Make sure you are using OpenCV 2.3 (or 2.2) and just clean your project prior build, that should fix that error, however I’ve got another one…
    I couldn’t get that vanisihngPoint project to recognize opencv 2.3 dir straight out of the box (or zip) and I used 2.2 first so CMake-gui could configure and build with it (it threw opencv_dir-not_found prior to having rebuilt the OpenCV 2.2, but for that though I had to change it from VS2012 – I have VC11 only – to VC++ 2010 Express)…
    The app built with opencv 2.2 seemed to work and build with lmfit 3.3, but it throws an exception just prior to completing, but I couldn’t find where, since the errors appeared after the last line – probably during some deinits – and couldn’t be debugged in code but only seen in the output like so:

    Debug Assertion Failed!

    Program: C:\TEMP\vp\Debug\vanishingPoint.exe
    File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
    Line: 1322

    Expression: _CrtIsValidHeapPointer(pUserData)

    For information on how your program can cause an assertion
    failure, see the Visual C++ documentation on asserts.

    (Press Retry to debug the application)
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\uxtheme.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Program Files (x86)\Common Files\Microsoft Shared\Ink\tiptsf.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\oleaut32.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\dwmapi.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\kernel.appcore.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Program Files\Lenovo\Bluetooth Software\syswow64\BtMmHook.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\psapi.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\oleacc.dll’. Cannot find or open the PDB file.
    ‘vanishingPoint.exe’ (Win32): Loaded ‘C:\Windows\SysWOW64\clbcatq.dll’. Cannot find or open the PDB file.
    HEAP[vanishingPoint.exe]: Invalid address specified to RtlFreeHeap( 00F20000, 01118EB8 )
    vanishingPoint.exe has triggered a breakpoint.

    The program ‘[8992] vanishingPoint.exe’ has exited with code 0 (0x0).

    Any chance you could possibly give some insight into which areas of 2.3 are a must have. so it could maybe be easier to see if and how it could be upgraded to work with current 2.4.6 and VS2012/13? I’m guessing something over the window showing the output in 2.2 is not compatible with the your code or limfit 3.3, but I can’t find that line where you’re calling the output for (or could get my added code line imsave(“C:\TEMP\test.jpg”, im) to the back of the processImage to save the given file (but that might be due to the exception). Thanks a bunch!

  6. dada says:

    The very same moment I finished posting and debuggin in VS2010 I found the imshow call which I’m guessing was breaking in VS2012, built alongside OpenCV 2.2 (since I couldn’t get OpenCV 2.3-2.4.3 to build with VS2012/13), so I’ll try to take out that line, only if I could get the imsave(“C:\TEMP\test.jpg”, im) placed at the end of the MSAC::drawCS to work first.
    I’m a newbie to OpenCV and a lot rusty in C++, I might be doing/reading it wrong, could the expected filename in imsave call had to be escaped or something first … and shouldn’t it throw an exception if it’s not .. this way it’s just not doing anything yet??? TIA

  7. dada says:

    Turns out that my C sucks!
    I’ve grown too accustomed to C#’s string literals that I\ve forgotten I should be using / in path strings instead of \:(and probably OpenCV just catches my exception error due to the wrongly written filename path in imsave call)….
    Just one question, since I’ve just started reading your code please, could you please tell me what’s the third value of the VP, I get the first two are X and Y coordinates in still images from the top left to down and right…but that last one is ???

  8. karelknoest says:

    With help of the excellent feedback in your blog, I’ve got the vanishing point detector working :).
    I’m curious about the comparison with results in your paper “Real-time robust estimation of vanishing points through nonlinear optimization”. Comparing the York urban dataset results from the freshly build vp detector with the results in the paper, I notice differences in the results (images in paper generally more accurate).
    Should I look further into the libraries I’m using, or are these different results expected?

    • Hi!
      Thanks for your comments!
      The results are definitely different since the code I am sharing corresponds to simplified methods implemented this way to run faster than the methods I used in my PhD which were by the way done mostly in Matlab.
      Kind regards!

  9. karelknoest says:

    Hi Marcos, thank you for the fast feedback!

  10. Chaupm says:

    HI Marcos,
    Would you can tell me steps by steps of MSAC to detect vanishing point?

  11. Clemens Schartmüller says:

    Hey,
    I’ve just gone through it and tried to implement it in Android / JNI C++ wrapper.
    It generally compiles but in your class MSAC the method fillDataContainers causes a SIGSEV when trying to initialize the Mats __Li, __Mi, __Lenghts, did you already fix this in your personal projects? If so, how?

    If it’s an issue on my side, I’ll make sure to tell you when I found a fix.

  12. Jyuza says:

    Hey, I was looking at your MSAC. MSAC doesn’t work alone. So I found the needed files, lmmin.c, lmin.h, lmstruct.h in “lmfit-6.1.tgz”. But the codes won’t work. First, line 634 and 636 in MSAC.cpp, printflags is member of “Im_control_struct”. Second, line 640 in MSAC.cpp, “lm_printout_std” is undefined and the lmmin() can’t take so many arguments.

    • Hi!
      Yes, you are right. The MSAC code was created with that dependency. The lmfit library has evolved since, and there are some incompatibilities with old code.
      I have committed the lmfit version that works with MSAC so you don’t need to seek for it. Just SVN Update the solution and you will have the correct files to build without errors.
      Regards,
      Marcos

      • Jyuza says:

        Thank you for replying. I solved the problem mentioned before. Right now, I am looking at your several paper and trying to implemented it by myself. I am stuck at the Rotation Matrix R that you used in the paper. I don’t know if I am using the correct rotation Matrix. My R is like this [cosr, 0, sinr; sinθsinr, cosθ, -sinθcosr; -cosθsinr, sinθ, cosθcosr ] and r, θ w.r.t pitch, yaw in your paper. Am I right?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s