Abstract: A human action recognition algorithm based on continuous-space relevance model (CRM) is proposed. Firstly, the algorithm automatically extracts human joint position. The human silhouette is characterized by the skeleton graph which is matched by dynamic time wrapping (DTW). The similar skeleton graph is used to find the template silhouette similar to the silhouette of the target, and the joint point position of the template silhouette is used to estimate the joint point position of the target. Finally, human actions are characterized by human joint point trajectories and recognized by CRM. This algorithm has been trained and tested on KTH human motion data set, Weizmann human action data set and Ballet data set. The results show that the recognition accuracy of this algorithm is either comparable to or much better than other sophisticated probabilistic models.