Skin Disease Prediction and Medicine Recommendation Using Hybrid CNN-RBM with Doctor Verification

Most automated tools for skin diagnosis look at a photograph and stop there. Symptom data — how long the rash has been present, whether it itches, where on the body it appears — is routinely discarded, even though no practising dermatologist would ignore it. On top of that, AI-generated outputs typically reach patients without any medical review at all. This paper describes a system built to correct both problems. Patients upload a lesion photograph and fill out a twenty-item symptom questionnaire. The image is processed by an EfficientNet-B3 convolutional network [16], which produces a 512-dimensional feature vector. A Restricted Boltzmann Machine condenses the symptom responses into a 100-dimensional latent vector. The two feature vectors are fused into a 612-dimensional representation and passed to a softmax classifier. A dermatologist, after consulting the Doctor Portal, is the only one able to share the output with the patient — this step cannot be skipped.
The platform was built using React for the front-end, Flask for the server, and MongoDB for data storage. The system authenticates and encrypts data using JWT tokens, bcrypt, and AES-256.
Our CNN+RBM model outperforms four baseline models — CNN-only, Matrix Factorisation (MF), SVD, and Weighted SVD (WSVD) — across every metric measured. By epoch 14, the model achieves 72.58% accuracy, 74.1% precision, 71.8% recall, and a 72.9% F1 score, with a false positive rate of only 4.2%.

skin disease classification, multimodal deep learning, convolutional neural networks, Restricted Boltzmann Machine, tele-dermatology, doctor verification, DermNet, medicine recommendation

1. World Health Organization. (2021). Skin conditions: Burden of skin disease. WHO Global Report, Geneva. [Google Scholar] [Crossref]

2. Ponnam, L. T., & Punyasamudram, S. D. (2016). Healthcare recommender system using item-based collaborative filtering. Proceedings of ICETETS, 1, 56–60. [Google Scholar] [Crossref]

3. Portugal, I., Alencar, P., & Cowan, D. (2018). The use of machine learning algorithms in recommender systems: A systematic review. Expert Systems with Applications, 97, 205–227. [Google Scholar] [Crossref]

4. Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115–118. [Google Scholar] [Crossref]

5. Tschandl, P., Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, Article 180161. [Google Scholar] [Crossref]

6. Hameed, M., Gupta, R., & Raj, A. (2021). Deep learning for skin disease diagnosis: CNN models comparison. IEEE Access, 9, 123456–123468. [Google Scholar] [Crossref]

7. Kassem, A., Hosny, K., Fouad, M., & Foaud, M. M. (2021). Machine learning and deep learning methods for skin lesion classification and diagnosis: A systematic review. Diagnostics, 11(8), Article 1390. [Google Scholar] [Crossref]

8. Kaur, A., & Singh, M. (2024). A multimodal approach to detection and classification of skin diseases (arXiv:2411.13855) [Preprint]. arXiv. [Google Scholar] [Crossref]

9. Chanda, T., Hauser, K., Hobelsberger, S., et al. (2024). Dermatologist-like explainable AI enhances trust and clinical utility in skin disease diagnosis. Nature Communications, 15, Article 78. [Google Scholar] [Crossref]

10. Chinnasamy, P., Wong, W. K., Raja, A. A., Khalaf, O. I., Kiran, A., & Babu, J. C. (2023). Health recommendation system using deep learning based collaborative filtering. Heliyon, 9, Article e22844. [Google Scholar] [Crossref]

11. Khan, S., Naseem, M., & Hussain, A. (2022). Hybrid RBM–CNN model for improved image recognition. International Journal of Advanced Computer Science and Applications, 13(2). [Google Scholar] [Crossref]

12. Aujla, G. S., Chaudhary, R., Kumar, N., et al. (2019). DLRS: Deep learning-based recommender system for smart healthcare ecosystem. Proceedings of IEEE ICC, Shanghai, China, 1–6. [Google Scholar] [Crossref]

13. Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25, 44–56. [Google Scholar] [Crossref]

14. Obermeyer, Z., & Emanuel, E. J. (2016). Predicting the future — Big data, machine learning, and clinical medicine. New England Journal of Medicine, 375, 1216–1219. [Google Scholar] [Crossref]

15. Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28, 31–38. [Google Scholar] [Crossref]

16. Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, 6105–6114. [Google Scholar] [Crossref]

17. Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554. [Google Scholar] [Crossref]

18. Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018). Deep learning for healthcare: Review, opportunities and challenges. Briefings in Bioinformatics, 19(6), 1236–1246. [Google Scholar] [Crossref]

19. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Proceedings of MICCAI, Munich, Germany, 234–241. [Google Scholar] [Crossref]

20. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. [Google Scholar] [Crossref]

21. This article is published under the Creative Commons Attribution License (CC BY 4.0), permitting unrestricted use, sharing, and reproduction of the work with proper attribution. [Google Scholar] [Crossref]