In this study, the heat transfer characteristics within the heat exchanger using water-based GO nanofluids were comprehensively assessed. An apparatus was constructed by scaling down an industrial heat exchanger. The nanofluid’s thermal conductivity, specific heat capacity, viscosity, density, Prandtl number, and Nusselt number were examined at varying temperatures and GO nanoparticle concentrations. The results revealed that the thermal conductivity of the nanofluid increased with both temperature and nanoparticle concentration, reaching its peak value of 0.380 W.m− 1. K− 1 at 85°C and 0.1%wt, leading to enhanced heat transfer rates through conduction and convection mechanisms. The specific heat capacity increased with temperature but decreased with higher GO nanoparticle contents, with a maximum value of 3403.821 J.kg− 1. K− 1 was recorded at 40°C and 0.01%wt. The viscosity of the nanofluid increased with higher concentrations of GO nanoparticles, and the minimum value of 0.83 mpa.s was observed at 85°C and 0.01%wt. The Prandtl number decreased with temperature but increased with increasing GO nanoparticle concentration, suggesting a transition from convective to conductive heat transfer. A newly derived correlation equation for the Nusselt number, Nu = 0.0059(1 + 7.62φ0.6886)Pe0.001Re0.9238Pr0.4, allows predicting heat transfer enhancement in nanofluids. The findings emphasize the potential of nanofluids for improving heat exchanger performance and offer valuable insights for optimizing nanofluid applications in thermal systems.