Pemisahan Objek Sel Tumpang Tindih pada Citra Pap Smear dengan Metode Deep Learning dan Watershed

  • Muh. Jamil
    Universitas Nusa Mandiri, Jakarta

  • Dwiza Riana
    Universitas Nusa Mandiri, Jakarta

Keywords: Object Separation, Cervical Cancer, Pap Smear Image, Overlapping Cells, Segmentation


The object observed in the Pap Smear image is Cervical Cancer which forms overlapping cells. This cancer must be observed early because it is a disease that causes the death of thousands of women worldwide every year. The death rate from this disease is the fourth highest among women. One way to be aware of this disease is to do an early check on the Pap Smear test image. This cell separation process uses the image segmentation method. This method is one of the important steps to be able to identify existing cell objects. This study proposes a segmentation method to separate 2 overlapping cells in the RepomedUNM dataset. The dataset is engineered in the manufacture of synthetic Pap Smear images. The segmentation method proposed is a Deep Learning-based method so that it can identify 2 overlapping cells in one area. The level of accuracy of the test with an average score of Intersection over Union (IoU) is 0.9003. And the results of segmentation with Deep Learning can be divided into all areas using the Watershed segmentation method. So that this research becomes a reference in the early identification of Cervical Cancer.


Rio, S., & Suci, E. S. T. (2017). Persepsi tentang kanker serviks dan upaya prevensinya pada perempuan yang memiliki keluarga dengan riwayat kanker. Jurnal Kesehatan Reproduksi, 4(3), 159-169. doi: 10.22146/jkr.36511

Wantini, N. A., & Indrayani, N. (2019). Deteksi dini kanker serviks dengan inspeksi visual asam asetat (IVA). Jurnal Ners dan Kebidanan (Journal of Ners and Midwifery), 6(1), 027-034. doi: 10.26699/jnk.v6i1.ART.p027

Riana, D., & Hidayanto, A. N. (2017, August). Integration of Bagging and greedy forward selection on image Pap Smear classification using Naïve Bayes. In 2017 5th International Conference on Cyber and IT Service Management (CITSM) (pp. 1-7). IEEE. doi: 10.1109/CITSM.2017.8089320

Hidayatulloh, T., Herliana, A., & Arifin, T. (2016). Klasifikasi sel tunggal Pap Smear berdasarkan analisis fitur berbasis naive bayes classifier dan particle swarm optimization. Swabumi, 4(2), 186-193.

Pasrun, Y. P., Fatichah, C., & Suciati, N. (2016). Penggabungan Fitur Bentuk dan Fitur Tekstur yang Invariant terhadap Rotasi untuk Klasifikasi Citra Pap Smear. Jurnal Buana Informatika, 7(1). doi: 10.24002/jbi.v7i1.479

Husain, N. P., & Fatichah, C. (2017). Segmentasi Citra Sel Tunggal Smear Serviks Menggunakan Radiating Component Normalized Generalized GVFS. Jurnal Nasional Teknik Elektro dan Teknologi Informasi (JNTETI), 6(1), 107-114.

Zhao, M., Wang, H., Han, Y., Wang, X., Dai, H. N., Sun, X., ... & Pedersen, M. (2021). Seens: Nuclei segmentation in Pap Smear images with selective edge enhancement. Future Generation Computer Systems, 114, 185-194. doi: 10.1016/j.future.2020.07.045

Braz, E. F., & Lotufo, R. D. A. (2017). Nuclei detection using deep learning. In Proc. Simpósio Brasileiro Telecomunicações Processamento Sinais (pp. 1059-1063). doi: 10.14209/sbrt.2017.48

Gautam, S., Bhavsar, A., Sao, A. K., & Harinarayan, K. K. (2018, March). CNN based segmentation of nuclei in PAP-smear images with selective pre-processing. In Medical Imaging 2018: Digital Pathology (Vol. 10581, pp. 246-254). SPIE. doi: 10.1117/12.2293526

Harangi, B., Toth, J., Bogacsovics, G., Kupas, D., Kovacs, L., & Hajdu, A. (2019, September). Cell detection on digitized Pap Smear images using ensemble of conventional image processing and deep learning techniques. In 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA) (pp. 38-42). IEEE. doi: 10.1109/ISPA.2019.8868683

Araújo, F. H., Silva, R. R., Ushizima, D. M., Rezende, M. T., Carneiro, C. M., Bianchi, A. G. C., & Medeiros, F. N. (2019). Deep learning for cell image segmentation and ranking. Computerized Medical Imaging and Graphics, 72, 13-21. doi: 10.1016/j.compmedimag.2019.01.003

Mahyari, T. L., & Dansereau, R. M. (2021, January). Deep Learning Methods for Image Decomposition of Cervical Cells. In 2020 28th European Signal Processing Conference (EUSIPCO) (pp. 1110-1114). IEEE. doi: 10.23919/Eusipco47968.2020.9287435

Kupas, D., & Harangi, B. (2021, November). Solving the problem of imbalanced dataset with synthetic image generation for cell classification using deep learning. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (pp. 2981-2984). IEEE. doi: 10.1109/EMBC46164.2021.9631065

Ibtehaz, N., & Rahman, M. S. (2020). MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural networks, 121, 74-87. doi: 10.1016/j.neunet.2019.08.025

Iglovikov, V., & Shvets, A. (2018). Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv:1801.05746. [Online]. Available:

Cheng, D., & Lam, E. Y. (2021). Transfer learning U-Net deep learning for lung ultrasound segmentation. arXiv preprint arXiv:2110.02196. [Online]. Available:

Pravitasari, A. A., Iriawan, N., Almuhayar, M., Azmi, T., Irhamah, I., Fithriasari, K., ... & Ferriastuti, W. (2020). UNet-VGG16 with transfer learning for MRI-based brain tumor segmentation. TELKOMNIKA (Telecommunication Computing Electronics and Control), 18(3), 1310-1318. doi: 10.12928/TELKOMNIKA.v18i3.14753

Cui, B., Chen, X., & Lu, Y. (2020). Semantic segmentation of remote sensing images using transfer learning and deep convolutional neural network with dense connection. IEEE Access, 8, 116744-116755. doi: 10.1109/ACCESS.2020.3003914

Branch, M. V., & Carvalho, A. S. (2021). Polyp Segmentation in Colonoscopy Images using U-Net-MobileNetV2. arXiv preprint arXiv:2103.15715.

Li, X., Chen, H., Qi, X., Dou, Q., Fu, C. W., & Heng, P. A. (2018). H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE transactions on medical imaging, 37(12), 2663-2674. doi: 10.1109/TMI.2018.2845918

Wei, Z., Song, H., Chen, L., Li, Q., & Han, G. (2019). Attention-based DenseUnet network with adversarial training for skin lesion segmentation. IEEE Access, 7, 136616-136629. oi: 10.1109/ACCESS.2019.2940794

Wang, W., Taft, D. A., Chen, Y. J., Zhang, J., Wallace, C. T., Xu, M., ... & Xing, J. (2019). Learn to segment single cells with deep distance estimator and deep cell detector. Computers in biology and medicine, 108, 133-141. doi: 10.1016/j.compbiomed.2019.04.006

Riana, D., Rahayu, S., Hadianti, S., Frieyadie, F., Hasan, M., Karimah, I. N., & Pratama, R. (2022). Identifikasi Citra Pap Smear RepoMedUNM dengan Menggunakan K-Means Clustering dan GLCM. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi), 6(1), 1-8. doi: 10.29207/resti.v6i1.3495

Riana, D., Hadianti, S., Rahayu, S., Hasan, M., Karimah, I. N., & Pratama, R. (2021, December). RepoMedUNM: A New Dataset for Feature Extraction and Training of Deep Learning Network for Classification of Pap Smear Images. In International Conference on Neural Information Processing (pp. 317-325). Springer, Cham. doi: 10.1007/978-3-030-92307-5_37

Zhang, J., Hu, Z., Han, G., & He, X. (2016). Segmentation of overlapping cells in cervical smears based on spatial relationship and overlapping translucency light transmission model. Pattern recognition, 60, 286-295. doi: 10.1016/j.patcog.2016.04.021

Lu, Z., Carneiro, G., Bradley, A. P., Ushizima, D., Nosrati, M. S., Bianchi, A. G., ... & Hamarneh, G. (2016). Evaluation of three algorithms for the segmentation of overlapping cervical cells. IEEE journal of biomedical and health informatics, 21(2), 441-450. doi: 10.1109/JBHI.2016.2519686

Siddique, N., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021). U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access, 9, 82031-82057. doi: 10.1109/ACCESS.2021.3086020

Du, G., Cao, X., Liang, J., Chen, X., & Zhan, Y. (2020). Medical image segmentation based on u-net: A review. Journal of Imaging Science and Technology, 64, 1-12. oi: 10.2352/J.ImagingSci.Technol.2020.64.2.020508

Adiba, A., Hajji, H., & Maatouk, M. (2019, March). Transfer learning and U-Net for buildings segmentation. In Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society (pp. 1-6). doi: 10.1145/3314074.3314088

Karthick, R. (2018). Deep Learning For Age Group Classification System. International Journal Of Advances In Signal And Image Sciences, 4(2), 16-22. doi: 10.29284/ijasis.4.2.2018.16-22

McAuliffe, M. J., Lalonde, F. M., McGarry, D., Gandler, W., Csaky, K., & Trus, B. L. (2001, July). Medical image processing, analysis and visualization in clinical research. In Proceedings 14th IEEE Symposium on Computer-Based Medical Systems. CBMS 2001 (pp. 381-386). IEEE. doi: 10.1109/CBMS.2001.941749

Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., & Savarese, S. (2019). Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 658-666). doi: 10.1109/CVPR.2019.00075

Arora, R., Saini, I., & Sood, N. (2021). Multi-label segmentation and detection of COVID-19 abnormalities from chest radiographs using deep learning. Optik, 246, 167780. doi: 10.1016/j.ijleo.2021.167780

Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of big data, 6(1), 1-48. doi: 10.1186/s40537-019-0197-0

How to Cite
Jamil, M., & Riana, D. (2022). Pemisahan Objek Sel Tumpang Tindih pada Citra Pap Smear dengan Metode Deep Learning dan Watershed. Jurnal Informasi Dan Teknologi, 4(4), 253-259.