Reinforcement Learning Guided Hierarchical Multimodal Feature Representation for Oral Cancer Diagnosis via Histopathological Image Analysis
International Journal of Medical Toxicology and Forensic Medicine,
Vol. 16 (2026),
1 Dey 2026
,
Page 1-7
https://doi.org/10.22037/ijmtfm.v16.52672
Abstract
Background: Oral cancer (OC) is a major public health concern due to its high mortality rate. Early detection is crucial to improving the survival chances of affected individuals. Conventionally, histopathological examination is used for the diagnosis of OC. However, existing approaches still encounter challenges in capturing complex histopathological patterns and achieving consistent performance across varying image conditions.
Methods: This study proposes a Reinforcement Learning-Based Multimodal Feature Extraction Approach for Early Oral Cancer Detection (RLMFE-OCD) model. The proposed method integrates multiple deep learning models, including GoogLeNet, ResNet-50, and SE-VGG16, to extract diverse and discriminative feature representations. A Deep Q-Network is employed for classification, while the Butterfly Optimization Algorithm is used for hyperparameter optimization.
Results: Experimental results on benchmark datasets establish that the RLMFE-OCD model achieves higher performance, with an accuracy of 94.44% compared with existing methods.
Conclusion: The RLMFE-OCD framework effectively supports early oral cancer detection through multimodal feature extraction and reinforcement learning. It shows potential as a reliable computer-aided diagnostic approach.
- Oral cancer, Histopathological images, Multimodal feature extraction, Deep learning, Deep Q-Network, Artificial intelligence
How to Cite
References
[1] Panigrahi S, Swarnkar T. Machine learning techniques used for the histopathological image analysis of oral cancer: a review. Open Bioinform J. 2020;13(1):106-18. [DOI: 10.2174/1875036202013010106]
[2] Kim DW, Lee S, Kwon S, Nam W, Cha IH, Kim HJ. Deep learning-based survival prediction of oral cancer patients. Sci Rep. 2019;9(1):6994. [DOI: 10.1038/s41598-019-43372-7]
[3] Appavu N. Oral cancer histopathological detection and diagnosis using hybrid AI deep learning methods. In: 2025 3rd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT). IEEE; 2025 Feb. p. 2061-7. [DOI: 10.1109/IDCIOT64235.2025.10915019]
[4] Gab Allah AM, Gaballa M, Elshennawy NM, Elkholy A. Edge-supervised convolutional neural network for histopathological classification of oral cancer images. Neural Comput Appl. 2025;1-20. [DOI: 10.1007/s00521-025-11457-2]
[5] Dhal P, Mishra D, Pradhan B. A deep ensemble-based framework for the prediction of oral cancer through histopathological images. Appl Soft Comput. 2025;113258. [DOI: 10.1016/j.asoc.2025.113258]
[6] Dharani R, Danesh K. Optimised deep learning ensemble for accurate oral cancer detection using CNNs and metaheuristic tuning. Intell Based Med. 2025;100258. [DOI: 10.1016/j.ibmed.2025.100258]
[7] Shukla R, Ajwani B, Sharma S, Das D. Identifying oral carcinoma from histopathological images using unsupervised nuclear segmentation. In: 2024 IEEE 9th International Conference for Convergence in Technology (I2CT). IEEE; 2024 Apr. p. 1-6. [DOI: 10.1109/I2CT61223.2024.10543340]
[8] Santhiya M, Sindhuja M, Jegatha R, Manikandan J. A practical automated framework for oral cancer detection by enhanced convolutional neural networks. In: 2023 12th International Conference on Advanced Computing (ICoAC). IEEE; 2023 Aug. p. 1-7. [DOI: 10.1109/ICoAC59537.2023.10249983]
[9] Song S, Ren X, He J, Gao M, Wang JN, Wang B. An optimal hierarchical approach for oral cancer diagnosis using rough set theory and an amended version of the competitive search algorithm. Diagnostics. 2023;13(14):2454. [DOI: 10.3390/diagnostics13142454]
[10] Marzouk R, Alabdulkreem E, Dhahbi S, Nour MK, Duhayyim MA, Othman M, et al. Deep transfer learning driven oral cancer detection and classification model. Comput Mater Contin. 2022;73(2). [DOI: 10.32604/cmc.2022.029326]
[11] Marasović T, Papić V. Few-shot breast cancer diagnosis using a Siamese neural network framework and triplet-based loss. Algorithms. 2025;18(9):567. [DOI: 10.3390/a18090567]
[12] Li F, Weng C, Liang Y. SE-WiGR: a WiFi gesture recognition approach incorporating the squeeze-excitation mechanism and VGG16. Appl Sci. 2025;15(11):6346. [DOI: 10.3390/app15116346]
[13] Kurek P. Risk-aware deep Q-networks for volatility-driven execution in cryptocurrency markets [dissertation]. Warwick: Warwick Business School, University of Warwick; 2025. [Link]
[14] Ji Q, Li R, Jing S. Uncertainty-based design optimization framework based on an improved chicken swarm algorithm and Bayesian optimization neural network. Appl Sci. 2025;15(17):9671. [DOI: 10.3390/app15179671]
[15] Mendeley Data. Oral cancer histopathological dataset [Internet]. Mendeley Data; 2024. [Link]
[16] Liu P, Bagi K. A tailored deep learning approach for early detection of oral cancer using a 19-layer CNN on clinical lip and tongue images. Sci Rep. 2025;15(1):23851. [DOI: 10.1038/s41598-025-07957-9]
[17] Parola M, Galatolo FA, La Mantia G, Cimino MG, Campisi G, Di Fede O. Towards explainable oral cancer recognition: screening on imperfect images via informed deep learning and case-based reasoning. Comput Med Imaging Graph. 2024;117:102433. [DOI: 10.1016/j.compmedimag.2024.102433]
- Abstract Viewed: 23 times
- PDF Downloaded: 15 times