Abstract: To address delayed obstacle avoidance, trajectory oscillation, and unstable policy updates caused by limited local observations, insufficient temporal information, and sparse rewards in complex dynamic environments, this paper proposes LP-TD3, a reinforcement learning navigation method for automated guided vehicles based on spatiotemporal policy learning and stable experience replay. First, the automated guided vehicle navigation task is formulated as a partially observable Markov decision process, and a composite reward function comprising goal-approach, motion-smoothness, and artificial-potential-field safety-constraint terms is constructed to provide continuous learning feedback for long-horizon navigation. Second, a dual-stream spatiotemporal feature fusion network is designed, in which a fully connected branch extracts current local geometric features and a long short-term memory branch models dynamic evolution information from historical observation sequences, thereby enhancing the agent’s ability to perceive the motion trends of dynamic obstacles. Finally, an annealed prioritized experience replay mechanism is introduced, which gradually increases the correction strength of importance sampling during training to balance critical-sample utilization efficiency and policy-convergence stability. Static and dynamic simulation results on the IRSIM platform show that LP-TD3 achieves navigation success rates of 95.4% and 97.8% and single-step inference times of 14.74 ms and 15.04 ms in static and dynamic test scenarios, respectively, improving obstacle-avoidance reliability and trajectory smoothness while satisfying real-time requirements. The results demonstrate that the proposed method provides an effective solution for mapless autonomous navigation of automated guided vehicles in complex dynamic environments.