To increase the measurement accuracy of optical systems, which are implemented in various applications, an improvement of the optical measurement technique is required. This paper proposes an image-to-image wavefront sensing approach using a deep neural network that directly predicts the phase image from the corresponding interference fringe image instead of reconstruction by the Zernike coefficients. The model is based on a conditional generative adversarial network (CGAN). To train the model, we used the formula-based ideal interference fringe images as the inputs of the CGAN, to conditionally predict the corresponding phase images as the output. We numerically investigated the performance by calculating the similarity between the ideal phase image and model output. In addition, with reference to a previous study, it was determined whether the model can extract more features from the interferogram for the prediction of Zernike coefficients. Moreover, an optical simulation software was introduced to provide an increased number of actual interferograms, to verify the proposed method. Based on the results, the proposed system can obtain the phase image directly and reduce the error, thus improving the measurement accuracy of the interference fringe.