Some classification based trackers need robust classifiers [1-3] to deal with mutation noise, and it is difficult to balance the efficiency and robustness of these trackers. In recent years, the rapid development of [4 – 6] detectors makes it possible to design automatic multi-target tracker, which is proved by the successful practical application of "detection and tracking" [7] tracking multiple targets at the same time. The framework of tracking by detection consists of detection and tracking. In the detection part, the detector is used to detect the target in each frame, and the target observation result is output as its output. Observations include their own characteristics, such as location, size and color. In the tracking part, the observation results obtained from the detector need to be selected and connected to form the motion path of the target, which is called data association problem. When solving the problem of data association, the error of test results should be considered. Liu et al. [8] The detection tracking is expressed as a structural prediction problem to consider the unreliability of detection. Mei et al. [9] The robustness of the feature model is improved by using multi view features. Cai et al. [10] proposed a biologically inspired model to deal with change