Abstract: Shadow reduces the robustness of computer vision. To eliminate this interference, we propose a method to remove shadow from an image automatically. We first use an orthogonal decomposition model to decompose a color image into an illumination invariant image and an illumination image. Then, considering the characteristic that image of the same object shares the same illumination invariant value and have different illumination values in and out of shadow, we generate a shadow-free image by automatically restoring the shadow area. In comparison with existing shadow removal algorithms, our method can generate a shadow-free image without shadow detection. It also has good real-time performance and robustness. It can effectively remove light or fine shadows. For a series of images taken under different illuminations in a scene, our results have better illumination constancy. We verify the real-time property, robustness, and advancement of our method by comparing with advanced shadow removal algorithms.