Abstract: In order to deal with the problems that the gait recognition algorithms are complicated, cost a lot of time and can't be used in real-time condition in extracting the feature of the gait silhouette, a method based on sparse representation and segmented frame difference energy image is proposed. Firstly, the improved segmented frame difference energy image is established as a feature image of gait. Then a dictionary is established based on it and an improved orthogonal matching pursuit algorithm is used for the fast coefficient decomposition. Finally, hidden Markov model is applied to establishing the gait recognition model based on the improved partial frame difference energy image. Selecting 90° views, an experient is done on CASIA_B database. Experimental results show that this methods get better recognition rate and can be used in real-time system.