Abstract: This paper describes the detailed process and the principles for tracking and classifying targets in video streams of the Vision-Based Traffic Monitoring System. Based on object detection, it brings forward a double-difference algorithm, the principle of temporal consistency and maximum likelihood estimation are employed to target classification, and the tracking process combines the template with the current motion regions so as to obtain correlation matching. The experimental results identify that this process can robustly track and classify targets of interest, reject background clutter, and continually track objects over large distances despite occlusions, appearance change, and cessation of the target motion.