Today’s world demands for maximum bandwidth utilization in the area of optical fiber network to achieve serious progress due to data consumption has been grown by > 25% as compared to last year. 5G system is one of the promising technology cater for high bandwidth utilization between multiple transceivers and have the advantages to operate in the milli-meter wave (mm-wave) of 60 GHz. There is much interest of photonic technique such as optical heterodyne coherent detection to produce mm-wave frequencies. We propose a 60 GHz radio over fiber (RoF) link using 168 Gb/s high capacity input data rate based 64-Quadrature amplitude modulation (64-QAM), coherent detection using optical heterodyning and advanced digital signal processing (ADSP) based system. An optimized version of optical heterodyne technique is used to derive 60 GHz radio frequency (RF) signal for connecting radio nodes over wireless link. In order to compensate the irregularities effect of standard signal mode fiber (SSMF), a novel ADSP technique is used which allows good improvement in the spectral signal efficiency and achieved longer distance communication. The ADSP technique plays a key role to achieve high capacity data rate requirements for forthcoming 5G technologies. The proposed RoF system shows that an error free transmission is accomplished over 170 km SSMF and achieved BER is 2.7×10-3 against 7.015 % error vector magnitude (EVM) with 30 dB Signal to Noise ratio (SNR).