Accurate grading of papaya based on ripeness is essential for food and pharmaceutical industries. Manual grading of large quantity of papaya is cumbersome and laborious. Thus, developing an automated solution for addressing fruit grading is a need of the hour. Traditional image processing that uses the color measurement to estimate the ripeness of papaya is less accurate as the light variations can significantly affect the performance. This paper proposes a methodology that employs a lightweight custom Convolutional Neural Network (CNN) for classifying papayas into mature, immature, and partially mature categories with high accuracy. The custom CNN model is trained using RGB images to extract relevant features and perform classification. The model is trained for three different epoch settings 4, 8, and 16 with learning rates of 0.01 and 0.001. Based on the validation results, the proposed CNN model achieves a classification accuracy of 97.6%. The developed model eliminates the need for manual intervention in fruit grading and can be easily integrated into automated systems.