Image sensor is the main device in camera production and its quality influences the imaging quality directly. Existing detection methods can not accurately detect the defects of image sensors. To address these problems, a two-branch network based on transfer learning is proposed to solve the problem of image sensor defect detection. Firstly, model parameters and detection speed were improved by replacing the backbone feature extraction network of YOLOV with a lightweight network. Secondly, anchor box optimization and focal loss were used to solve the problem of accurate positioning and unbalanced sample classification. Finally, with the cosine annealing scheduler and fixed learning rate, the network's convergence rate and overall performance are improved. The proposed lightweight network algorithm achieves a better effect than the original algorithm. The proposed algorithm has higher detection speed and smaller parameter size than other detection methods under the same size.