Enhancing the longevity of fruits not only mitigates food waste but also amplifies market sales revenue. The attainment of an extended shelf life can be realized by implementing an efficient fruit-ripeness categorization methodology during the packaging phase. The current classification approaches, comprising three categories of "good," "average," and "poor," primarily evaluate fruit quality rather than maturity, which is a crucial factor specifically for mangoes. In contrast, our proposed approach defines four ripeness categories: "Under-ripe," "Over-ripe," "Ripe (Good)," and "Damaged." As a case study, we focus on the Pakistani Chaunsa Mango variety, acquiring 2000 images directly from the farms prior to packaging. Subsequently, we enlist the expertise of professionals to classify these images into the four predefined categories. Following this, we introduce an automated mango categorization method utilizing the Support Vector Machine (SVM) machine learning model. To train the SVM model, we employ a fusion technique that integrates deep features extracted from three convolutional neural network (CNN) models, namely VGG13, VGG16, and VGG19. Our proposed model accepts RGB images as input and generates the corresponding ripeness category as output. To validate the effectiveness of our approach, we conduct comprehensive experiments. The results exhibit an impressive accuracy of 87.78% on the Chaunsa Mango dataset, surpassing the current state-of-the-art methods. Moreover, we plan to make our dataset publicly available, enabling the scientific community to conduct further experimentation and research in this domain.