Real-time vehicle detection and lane detection for ADAS

This time I bring a new result we have achieved in Vicomtech-IK4: an improved real-time vehicle detection and lane tracking system for ADAS (Advanced Driver Assitance Systems).

It follows the history of developments we’ve been doing in this line, evolving previous approaches (including migration into iOS) into a more reliable and fast solution.

Links to previous videos:

The new developments include real-time performance in Android devices, much more reliable detection and tracking of vehicles (using a multi-thread approach combining different detectors and tracking approaches), and also a holistic model of lanes, which are filtered with a multi-Kalman filter.

Hope you like this video! If you are curious about it or have technical doubts, please, write an entry in this post. Sometimes I am too busy to response quickly, but I try to come back to the blog as often as I can.

See you!

This entry was posted in ADAS, Computer vision, Viulib and tagged , , , . Bookmark the permalink.

19 Responses to Real-time vehicle detection and lane detection for ADAS

  1. Liwei says:

    I am new to ADAS. Is the frame rate of the camera 30fps for ADAS application. To reduce the motion between two consecutive frames, Is much faster frame rate used?

    • Hi,
      Normally, commercial cameras come with 30 fps and 60 fps video modes. Some may offer 120 fps with reduced frame size. Nevertheless, the higher the fps, the less light that the sensor can capture, and then the higher the noise.
      Depending on the application, 30 fps may be enough (e.g. traffic sign recognition).

      • Liwei says:

        Thanks for your reply. Is traffic sign recognition using one image sequence or one single image for ADAS application? Does using one image sequence bring any advantage?

  2. Archer says:

    Hi Macros, what hardware did you use for this project? Is it possible to run this program on a raspberry possible ?

    Thank you :)

  3. Archer says:

    Hi Marcos. I’m a CS student. What you have done here is absolutely awesome. I just have a question. What hardware did you used to run this program? Is the computing power on a raspberry pi sufficient for such program?

    Thank you

    • Hi,
      Thanks for the comment.
      Indeed! It is possible to embed these algorithms. It is not easy, of course. First, we have not gone into low-level code optimization (we program in C++, but we could go down to C or Assembly), but algorithm optimization. This means we take care of analysing the bottlenecks; what information is required at what time and with what accuracy. This way we can remove all unnecessary computation and keep having good results.
      Besides that, for instance, if you use OpenCV, try to compile it with all the flags that better exploit the platform. I guess NEON for the ARM processor, and in case there is a graphics card, try enabling OpenCL.
      Multi-threading is a must. With C++11 it is even easier, with less external dependencies. For instance, you will probably need to use one thread for vehicle detection (using your cascades) another for the tracking, other for capturing, etc.
      At the end, it is like digging a hole in the sand. You start on the top and narrow down finding where you can dig without breaking it all.

      Good luck!

  4. VisionLee says:

    Hi Marcos,
    Amazing work and truly impressed!

    Is there any plan in near future to open source the ADAS application? Or at least share some codes with regard to the core functionalities like Vehicle detection/Multi-lane curvature etc. ?

    Also I couldn’t find any publications describing the algorithms or process pipelines?

    Get some demos to work e.g. using OpenCV is easy. Real time processing under real conditions are much more difficult and challenging so really well done!!

    Kind regards,

  5. VisionLee says:

    Hi Marcos,
    Really nice work, Im impressed!

    Just wondering is there any plan to open-source the codes in near future? Or at least sharing some core functionalities like Vehicle detection/Multi-lane curvature etc?

    I’m currently preparing a potential project on auto lane & vehicle detection but struggling how to ‘connect’ the broken lane marks? Some lane marks are really short in the first place!

    I couldn’t find any publications or papers describing your method or process pipeline?

    Get some demos to work e.g. using OpenCV is easy. Real time processing under real conditions are much more difficult and challenging so really well done!!

    Kind regards,

    • Hi,
      Thanks for your comments!
      You hit the point: real-time video processing is a challenge. Is not only about the algorithm to use, it is the initialization, the auto-assessment, the input & output interfaces, the migration into specific HW with low computational power (e.g. mobile phones), and all of it working well in an environment never seen before, where your system is going to pass a stress-test to check if it is reliable.
      Most of the videos I show are result of work team in the UPM (where I worked until 2010) or Vicomtech-IK4 (where I still work), so I cannot release any code, unless I create something on my own. I could do it, but definitely have no free time. I am sorry for that. Actually, the idea of having this blog is to share with you all what I learn from others or my own experience, but each day I have less time to work on this.
      Hope you progress in your research. Don’t give up, and build things step by step.

  6. Shaoyin Zheng says:

    Can you tell me how to locate/detect the vehicles?what algorithms you use to detect the vehicles?Thank you!

  7. Fahad says:

    Hi Marcos,
    I have need some guidance regarding lane detection and lateral deviation of the car, how can i take the approach step by step. For lane detection first i have to do camera calibration?once i get the intrinsic parameter & extrinsic parameters what is the next step..? like i have to do filter chain processing for instance image segmentation and defining region of interest but how can i then with least square method make a curve of the road and track it and check the deviation of the car so that i know it is deviating from lane and indicate to keep the lane..please i need the guidance for my studies thesis.

    • Hi Fahad,
      You are right, a reasonable approach would be:
      1) Calibrate your camera.
      2) Compute the homography between image plane and road plane (this is, using extrinsics).
      3) Segment the image to locate pixels belonging to lane markings.
      4) Filter our outliers, group pixels, make your detection robust.
      5) Fit model (e.g. parabola), with least squares or RANSAC.
      6) Apply time tracking (e.g. Kalman filter, or particle filter)



  8. Eugene says:

    Hi Marcos,
    I really like the work you guys have accomplished so far. It seems the tracking boxes are more accurate than some other vehicle tracking projects has produced.
    I am currently trying to work on something similar but simpler, like just be able to detect and draw boxes on cars from on-board view. Would you mind share some suggestions on algorithms part? I was simply using contour detection and condition-based filtering to determine which area contains vehicle on pictures. However that might not work so well when analyzing videos.
    Thanks and regards!


    • Hi Eugene,
      Thanks for the comments!
      For vehicle detection, you definitely want to go for a machine learning approach. Traditionally (2000-2010) classifiers such as Adaboost or SVM have been used with Haar, LBP or HOG features (also others, like Gabor, or combinations or several of them). Good results were achieved, although with limited applicability to difficult scenarios. In recent years, deep learning have emerged as the solution (if your system can afford a GPU-enabled platform). NVIDIA Jetson TX1 (and now TX2) is bringing down the prices, and many researchers are sharing their progress via github using these platforms.
      Now the difficulty is to find an appropriate dataset to train your model. You can start with those open (e.g. KITTI – object). And start getting familiar with NVIDIA DIGITS, parametrize your classifier, etc.
      Things are changing for computer vision scientists. We are becoming machine learning scientist : )
      Good luck!


  9. Ben says:

    Hi Marcos,
    I am trying to achieve a similar project for a course. I have most of it but I still have some trouble with the Kalman filter part. Could you give me some advice or hint on how to achieve it? My lane is defined as a polynomial so I tried to track each coefficient as a state but that did not work so well!

    Thanks and regards

    • Hi Ben,
      Kalman filter in this context can be used to smooth noisy signals, under the assumption the noise is Gaussian. Somehow, the noise is modeled as two different variables: on the one hand, the measurement noise, which is what amount of noise you do expect on your measurements (e.g. at pixel level); on the other hand, the system noise, this is, how confident you are that your system behaves linearly.
      The insights of Kalman go beyond that simplicity into the realm of probability and Bayesian networks. For a full understanding on how to tune it, you could devote some of your time studying it from a maths point of view.

      For tracking lanes modeled as parabolic curves, I won’t track the coefficients separately because they reflect linear and quadratic magnitudes, and that is difficult to model. I would suggest to track intersection points: the point where your parabola intersects the bottom line of the image, and another point at some row below the horizon. To avoid perspective problems, tracking these points in a bird’s-eye view simplifies the selection of the noise parameters.

      Good luck!

  10. Jeff Barghout says:

    Great work on the lane/road/object detection code. I’d like to learn more about your approach. Can you send me an email so we can follow-up.

    • Hi Jeff,
      Thanks for writing. I normally can’t devote too much time to the blog, I have very few spare time. If you have comments or doubts, please continue using the forum, I came to the blog from time to time and answer specific questions, so everybody can read them and also maybe participate.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s