Abstract: A novel image similarity called implicit linesegment similarity (ILS), and a registration algorithm of infrared and visible images based on ILS, are proposed. Essentially, the algorithm achieves image registration by aligning the corresponding line segment features in two images. First, line segment features are extraced and their coordinate positions in one of the images are recorded. These line segment features are mapped into the second image, in which the geometric mapping relationship are caculated by maximizing the degree of similarity between the line segment features and correspondence regions in the second image. The advantage of this method is that it eliminates the need to directly measure the grey similarity between the two images.A multiresolution analysis method is used to calculate the model parameters from coarse to fine on Gaussian scale space. The geometric transformation parameters are finally obtained by the improved Powell algorithm. Comparative experiments demonstrate that the proposed algorithm can effectively achieve the automatic registration for infrared and visible images. It significantly improves computational efficiency and anti-noise ability beyond previously proposed algorithms.