JRMOT[1] is a novel 3D MOT system that integrates information from 2D RGB images and 3D point clouds into a real-time performing framework. There are some details of reading and implementing it.

Contents

• Paper & Code & note
• Paper understanding
• Code understanding
• Note
• References

Paper & Code & note

Paper: JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset(arXiv 2020 paper)
Code: [Pytorch][Updating]
Note: JRMOT

Paper

Abstract

1. JRMOT is a novel 3D MOT system that integrates information from 2D RGB images and 3D point clouds into a real-time performing framework.
2. They also released the JRDB dataset, which is a novel large scale 2D+3D dataset.
3. It demonstrates state-of-the-art performance against competing methods on the popular 2D tracking KITTI benchmark and serves as a competitive 3D tracking baseline for our dataset and benchmark.

Problem Description

MOT: the agent needs to perceive the motion of the multiple dynamic objects and other agents.
Reccent approaches:: perceive 2D motion from RGB video streams.

Problem Solution

1. It integrates 2D detection from Mask R-CNN and 3D information from F-PointNet.
2. It fuses the 3D shape descriptor with a 2D RGB descriptor through Aligned-ReID.
3. It uses optimal joint probabilistic data association step.
4. A novel multi-modal recursive Kalman filter was proposed.

Conceptual Understanding

It shows all components of the JRMOT, and workflow as below.

Experiments

It results on online KITTI Car and Pedestrian Tracking, and gets SOTA performance.

Code

[Updating]

Note

It shows data collection platform and sample visualization of the dataset.

References

1. Shenoi A, Patel M, Gwak J Y, et al. JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset[J]. arXiv preprint arXiv:2002.08397, 2020.