Solar cells are a critical component of renewable energy systems and are becoming increasingly important as society moves towards decarbonization. To assess their economic viability and sustainability, developing solar technologies with desired performance levels is an important consideration. Among various material parameters, carrier lifetime is one of the key material properties which influences the performance metrics of solar cells significantly. Measurement of carrier lifetime using experimental methods is a complex and tedious procedure, hence machine learning (ML) methods can be used to accelerate the prediction of carrier lifetime from the Current density-voltage (J-V) characteristics. However, ML methods generally require a large amount of data to achieve accurate predictions. To this end, looking for an efficient solution to learn transferable knowledge for the carrier lifetime prediction of different solar cell materials with less data is of utmost significance. In this paper, we propose a transfer learning (TL) framework to learn common knowledge between Silicon (Si), Gallium arsenide (GaAs), and perovskite solar cells (PSC) for carrier lifetime prediction. Our TL framework uses deep neural network (DNN) architecture with two methods, i) pre-training and fine-tuning entire layers, and ii) pre-training and fine-tuning regressor layer alone. Hence, these methods facilitate data-efficient and parameter-efficient learning. In this way, one can improve the accuracy of carrier lifetime prediction by learning with the source material data and fine-tuning with less data for the material of interest (target material data). The experimental results on simulated data and experimental data indicate that TL has autonomously identified nontrivial transferability across different materials. This leads to faster convergence and more robustness while training a DNN model and higher accuracy for carrier lifetime prediction.