Abstract: This paper analyses the dynamics of rotary inverted pendulum system and linearizes the pendulum system at the unstable equilibrium point at the "inverted" state. The linear quadratic optimal control strategy is adopted to design the controller and the simulation experiment is done. The controller designed is successfully applied to the actual rotary inverted pendulum system which makes it be steadily stabilized at the "inverted" state. Such application and actual experiment is the first in China.