Intelligent Transportation Systems Overview

ML ∩ CV ∩ ITS
Traffic Optimization
Conclusions

Machine Learning for Intelligent Transportation

Systems

Patrick Emami (CISE), Anand Rangarajan (CISE), Sanjay

Ranka (CISE), Lily Elefteriadou (CE)

MALT Lab, UFTI

September 6, 2018

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
What is ITS?
Traffic Optimization
Conclusions

ITS - A Broad Perspective

Working definition
Utilizing cutting-edge, synergistic technologies to develop and
improve transportation systems of all kinds

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
What is ITS?
Traffic Optimization
Conclusions

ITS - A More Narrow Perspective

ITS for improved urban mobility

Source: https://www.arch2o.com/future-urban-mobility/
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
What is ITS?
Traffic Optimization
Conclusions

ITS for Urban Mobility - Autonomous Vehicles

Source: http://www.vtpi.org/avip.pdf
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
What is ITS?
Traffic Optimization
Conclusions

ITS for Urban Mobility - Traffic Surveillance

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
What is ITS?
Traffic Optimization
Conclusions

ITS for Urban Mobility - Traffic Optimization

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Machine Learning

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Machine Learning

Working definition
Extracting patterns and abstractions from datasets to make
intelligent decisions on previously unseen data

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Other “Intelligent” Tools

Machine learning is rarely used in isolation, and often overlaps with

the following fields:
1 Discrete and continuous optimization
2 Signal processing
3 Distributed systems
4 Control theory
5 And more...!

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Machine Learning for ITS

Deep neural networks trained on massive datasets are at the cutting-edge

in terms of performance. The theory is lagging behind!

Source: http://yann.lecun.com/exdb/lenet/
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Deep Learning

Source: Andrew Ng: https://www.slideshare.net/ExtractConf

Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

ML ∩ Computer Vision

A primary use of ML in ITS is for intelligent perception

Some key tasks

1 Object detection
2 Multi-object tracking
3 Activity recognition

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Autonomous Vehicles

Source: https://www.wired.com/story/waymo-launches-self-
driving-minivans-fiat-chrysler/,
http://sitn.hms.harvard.edu/flash/2017/self-driving-cars-
technology-risks-possibilities/
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Autonomous Vehicles

Source: https://www.wired.com/story/waymo-launches-self-
driving-minivans-fiat-chrysler/,
http://sitn.hms.harvard.edu/flash/2017/self-driving-cars-
technology-risks-possibilities/
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Traffic Surveillance

Use Computer Vision to try to answer these questions:

How many vehicles?

Any driving the wrong way?
Are pedestrians crossing?

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Object detection

It can explicitly/implicitly answer the following questions

1 Where are the interesting objects within my field of view?
2 What are the object classes (pedestrian, bicyclist, sedan, ...)?
3 How many objects are there?

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Object detection

It can explicitly/implicitly answer the following questions

1 Where are the interesting objects within my field of view?
2 What are the object classes (pedestrian, bicyclist, sedan, ...)?
3 How many objects are there?

For simplicity, we’re lumping localization (where in the image are

the objects) and classification (what class) into detection.

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Object Detection with Deep Learning

Real world challenges

The current best way to handle variations in lighting, orientation,
and scale when deploying is data augmentation.
Source: http://cs231n.github.io/convolutional-networks/
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Multi-object Tracking

Goal is to estimate the trajectories of all objects in a dynamic scene

MOT from a stationary traffic cam MOT using LiDAR from an AV

Source: Luo, et. al. ”Fast and Furious: Real Time End-to-End
3D Detection, Tracking and Motion Forecasting With a Single
Convolutional Net.” CVPR 2018.
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Obstacles to solving MOT

1 Object detectors don’t handle partial/full occlusion or drastic

variations in lighting, color, orientation very well
2 Stitching detections together over time into tracks is a hard
discrete optimization (or inference) problem
3 Sensors are unreliable/noisy
4 MOT systems are typically overly-complex and contain lots of
hand-tuned problem-specific parameters

Source: Emami, Patrick, et al. ”Machine Learning Methods for

Solving Assignment Problems in Multi-Target Tracking.” arXiv
preprint arXiv:1802.06897 (2018).
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Obstacles to solving MOT

1 Object detectors don’t handle partial/full occlusion or drastic

variations in lighting, color, orientation very well
2 Stitching detections together over time into tracks is a hard
discrete optimization (or inference) problem
3 Sensors are unreliable/noisy
4 MOT systems are typically overly-complex and contain lots of
hand-tuned problem-specific parameters

Interesting research question keeping me up at night

Is there a principled way to learn the concept of object permanence
within an MOT system?
Source: Emami, Patrick, et al. ”Machine Learning Methods for
Solving Assignment Problems in Multi-Target Tracking.” arXiv
preprint arXiv:1802.06897 (2018).
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Activity Recognition

Using object detections and trajectories, can we then extract

patterns at the level of behaviors?
1 Pedestrian safety; ID’ing whether a person is walking/about
to walk into the street
2 Vehicle collision prediction
3 Multi-agent modeling at traffic intersections and merging
zones for AVs

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview Overview
ML ∩ CV ∩ ITS Deep Learning
Traffic Optimization Key applications
Conclusions Computer Vision Tasks

Collision Prediction

Source: Xiaohui Huang, Sanjay Ranka and Anand Rangarajan.

Real-time Multi-Object Tracking and Road Traffic Safety
Measurement. In preparation.
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview
Overview
ML ∩ CV ∩ ITS
Traffic Flow Prediction
Traffic Optimization
Traffic Intersections
Conclusions

Traffic Optimization

Guiding question
Using sensors and edge computing, can we maximize the efficiency
of traffic flow through a road network in real-time?

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
Overview
ML ∩ CV ∩ ITS
Traffic Flow Prediction
Traffic Optimization
Traffic Intersections
Conclusions

Traffic Sensors

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
Overview
ML ∩ CV ∩ ITS
Traffic Flow Prediction
Traffic Optimization
Traffic Intersections
Conclusions

Short-term Traffic Flow Prediction

Accurate forecasting of congestion levels enables real-time traffic

planning

Train a model (e.g., deep network or Random Forest) to predict

next 15-30 minutes of traffic flow.
Source: Polson, Nicholas G., and Vadim O. Sokolov. ”Deep
learning for short-term traffic flow prediction.” Transportation
Research Part C: Emerging Technologies 79 (2017): 1-17.
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview
Overview
ML ∩ CV ∩ ITS
Traffic Flow Prediction
Traffic Optimization
Traffic Intersections
Conclusions

Traffic Intersection Optimization

Source: Pourmehrab, M., Elefteriadou, L., Ranka, S., &

Martin-Gasulla, M. ”Optimizing Signalized Intersections
Performance under Conventional and Automated Vehicles
Traffic.” arXiv:1707.01748 (2017)
Emami, et al. ML for ITS
 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
Traffic Optimization
Conclusions

Conclusion

Plenty of challenges when applying ML to ITS

1 Collecting, cleaning, and labeling large-scale datasets
2 Law-makers and policy has to keep up with the tech
3 Brittle models that break when applied to new domains
4 Security and privacy

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
Traffic Optimization
Conclusions

Conclusion

Plenty of challenges when applying ML to ITS

1 Collecting, cleaning, and labeling large-scale datasets
2 Law-makers and policy has to keep up with the tech
3 Brittle models that break when applied to new domains
4 Security and privacy
But we’ve made great progress!

Emami, et al. ML for ITS

 Intelligent Transportation Systems Overview
ML ∩ CV ∩ ITS
Traffic Optimization
Conclusions

Thank you!

Questions?

Twitter: @patrickomid, email: [email protected]

Slides available at: https://pemami4911.github.io

Emami, et al. ML for ITS