In remote sensing due to several constraints of optical imaging sensors high spectral and high spatial features are not present in single image, so high frequency component of panchromatic(PAN) image needs to be restored at the spatial resolution of corresponding multispectral (MS) image. Misbalancing between spatial and spectral details extraction and injection leads to several issues like spectral distortion, intensity variation. Direct fusion methods often generate resultant image with poor contrast and several undesirable effects, while multi resolution analysis (MRA) techniques have achieved success in many fusion algorithms, these methods also suffer from ringing artifact due to strong edge decomposition. In this paper, pansharpening (PS) method is based on total variation-Hilbert − 1(TV-H− 1) model for decomposing MS and PAN image in appropriate cartoon and texture components. To refine the detail map and better quality of fused image adaptive weights are calculated using particle swarm optimization (PSO). For experimental analysis, four different sensors datasets with different regions like city area, hilly area and vegetation area are used. Resultant image and image quality analysis parameters confirm that the proposed method succeed in spectral detail preservation compared with other widely used pansharpening methods.