Sensor fusion frameworks for objects classification, terrain classification and localization


To develop objects detection, classification and tracking as well as terrain classification and localisation algorithm based on sensor fusion frameworks for an unmanned vehicle.


The research team is on course to develop a novel image terrain classifier as well as object detection algorithm, by which ground slope, road boundaries as well as moving object (vehicles and pedestrians) are able to be determined using multi-sensor information. (i.e. lidar and camera). In order to realise perceptual localisation, multi-sensor information is compared to an existing map to determine the pose of the vehicle with adequate reliability.  The subsequent effort will be to continue developing a multi-sensor multi-target filter, which is able to work optimally in cases where there is limited sensor data, poor environment conditions (at night, in dust and in rain), or high operating speeds. The environment of interest is urban, off-road and highway.

                                              Sensor fusion framework interfaced with the respective algorithms


                                   Terrain Mapping                                                            Object Tracking

Demonstration Video


  1. Dang Kang, Ankit Goyal, Michael Hoy, Junsong Yuan, Justin Dauwels (2016). Efficient Terrain Classification and Mapping with LIDAR and Visual Information. Autonomous Robots and Multirobot Systems (ARMS) Workshop at AAMAS 2016.
  2. Michael Hoy, Chaoqun Weng, Junsong Yuan and Justin Dauwels (2016). Object Detection and Tracking with Multiple Sensor Modalities using Random Finite Set Filtering. Autonomous Robots and Multirobot Systems (ARMS) Workshop at AAMAS 2016.
  3. Michael Hoy, Chaoqun Wang, Junsong Yuan and Justin Dauwels @2016) Bayesian Tracking of Multiple Objects with Vision and Radar. International Conference on Automation, Robotics, Control and Vision.

Principal Investigators

Associate Prof. Justin Dauwels (NTU)

Telephone: 6790 5410
Office: S2.2-B2-15
Mr. Paul Tan (STE)

Telephone: 6660 1052
Office: S1-B4a-03

Co-Principal Investigators

Associate Prof. Yuan Junsong (NTU)

Telephone: 6790 4016
Office: S1-B1b-41