Abstract: To solve two challenging problems in similarity-based video retrieval:similarity measurement and fast search,a novel similarity-based video approach is proposed.A compact video signature is computed based on the statistics of spatial-temporal features of video according to visual similarity.The video similarity is measured by the computation distance of video signature.For the scalable computing requirement,a search method via clustering index table is presented.The results from the query tests in large database show that this similarity retrieval method is fast and efficient.