Abstract: In this paper, a multi-mode image tracking system is designed, which uses the centralized structure to fuse the tracking result of every mode. Because of the noise in image, the presence of occlusion, and the deformation of tracked target, some tracking algorithms may fail. So it is the main idea in this paper to merge the advantages from all of the tracking algorithms, and then have a robust behavior. There are two tracking models applied in this system, which are the modes based on region and distinguished points. Then these tracking modes track respectively, and the occlusion may be decided by the correlation coefficient, which determines those results that will be sent to fusion center. The experiences show that this system could improve the tracking precision, and have robust behavior.