Abstract: Terahertz (THz) time-domain spectral technology can effectively obtain time and frequency domain information of internal IC structure and detect internal defects. However, it cannot be directly applied in industries because the single frequency images lack the ability of feature expression. Making full use of different characteristics of the chip contained in terahertz spectral data, we propose a fusion algorithm based on sparse representation and multi-scale decomposition that can fuse the low and all-pass frequency component separated from the multi-scale transform. Our findings show that the proposed algorithm effectively enhances the characteristics and improves the resolution of THz images. Moreover, we construct a semiconductor IC dataset and proposes lateral inhibition in a convolutional neural network (LiCNN) according to the IC characteristics of sparsity, non-obvious defect, and low defect probability. The optimized LiCNN achieves simplified parameters, and the validity of defect detection using LiCNN is verified in a small sample dataset. Our proposed method provides IC defect detection techniques for nondestructive testing and quality control.