Abstract: Because of the sensitivity to the local neighbor structure of image, convolutional neural network has a wide and profound application in the field of computer vision. However, for the ordinary matrix processing without local structure, convolutional neural network is usually not effective for it. For this specific situation, a network framework based on a novel dual linearized connecting is proposed to deal with the ordinary matrix sequence prediction task. By processing of ordinary matrix sequence through dual linearized connecting, the network owns the ability to learn inherent feature and global dependence from data, and better prediction results are obtained. The experimental results show that the proposed algorithm owns robust feature extraction ability for the ordinary matrix sequence without local structure, and can generate better prediction results than mainstream video prediction methods.