DeepSORT[1] integrates appearance information to improve the performance of SORT, learned a deep association metric. There are some details of reading and implementing it.

Contents

• Paper & Code & note
• Paper understanding
• Code understanding
• Note
• References

Paper & Code & note

Paper: Simple Online and Realtime Tracking with a Deep Association Metric(ICIP 2017 paper)
Code: PyTorch, TensorFlow
Note: Mendeley

Paper

Abstract

1. SORT[2] is a pragmatic approach to multiple object tracking.
2. In this paper, appearance information was integrated to improve the performance of SORT for tackling the long-term occlusions.
3. They place offline pre-traning with a learned deep association metirc on person re-id dataset, while establish measurement-to-track associations using nearest neighbor queries during online application.
4. It reduces the number of identity switches by 45%.

Problem Description

1. Traditional methods are not applicable in online scenarios and the performance of these methods comes at increased computational and implementation complexity.
2. SORT returns a relatively high number of identity switches.

Problem Solution

They overcome this issue by replacing the association metric with a more informed metric that combines motion and appearance information.

Core Conception

State Estimation

Assignment Problem

1. Motion information: Mahalanobis distance $d^{(1)}(i,j)=(d_{j}-y_{i})^{T}S_{i}^{-1}(d_{j}-y_{i})$.
2. Appearance information: smallest cosine distance $d^{(2)}(i,j)=min \lbrace 1-r_{j}^{T}r_{k}^{(i)} |r_{k}^{(i)}\in{R_{i}} \rbrace$.
3. In combination for occlusions: weighted sum $c_{i,j}=\lambda d^{(1)}(i,j)+(1-\lambda)d^{(2)}(i,j)$.

Matching Cascade

Deep Appearance Descriptor

Experiments

Code

1. The complete code can be found in deep_sort[3].
2. Another tensorflow implementation can be found in deep_sort_yolov3[4].

[Updating]

Note

More details about the whole algorithm and its implementation can be found in [5].

References

[1] Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with a deep association metric[C]//2017 IEEE international conference on image processing (ICIP). IEEE, 2017: 3645-3649.
[2] Gojay. “SORT.” https://gojay.top/2020/06/14/SORT/
[3] deep_sort. https://github.com/nwojke/deep_sort
[4] deep_sort_yolov3. https://github.com/Qidian213/deep_sort_yolov3
[5] pprp. “Anasis for Deep SORT.” https://zhuanlan.zhihu.com/p/133678626