The JPEG standard allows the use of a customized quantization table; however, it is still challenging to find an optimal quantization table timely. This work aims to solve the dilemma of balancing computational cost and image-specific optimality by introducing a new concept of texture mosaic images. Instead of optimizing a single image or a collection of representative images, the conventional JPEG optimization techniques can be applied to the texture mosaic image to obtain an optimal quantization table for each texture category. We use the simulated annealing technique as an example to validate our framework. To effectively learn the visual features of textures, we use the ImageNet pre-trained MobileNetV2 model to train and predict the new image's texture distribution, then fuse optimal texture tables to come out with an image-specific optimal quantization table. Our experiment demonstrates around 30% size reduction with a slight decrease of FSIM quality but visually indistinguishable on the evaluation datasets. Moreover, our rate-distortion curve shows superior and competitive performance against other prior works under a high-quality setting. The proposed method, denoted as JQF, achieves per image optimality for JPEG encoding with less than one second additional timing cost.