SKIN LESION CLASSIFICATION BASED ON DEEP ENSEMBLE CONVOLUTIONAL NEURAL NETWORK

The Intelligent Systems Research Group (ISysRG) has completed research on the classification of skin lesions. One of the potentially cancerous skin lesions is melanoma (Mayo Foundation for Medical Education and Research (MFMER), 2017). Melanoma is formed in melanocyte cells (Hartman & Lin, 2019). Today, melanoma cancer causes many deaths in the world (American Cancer Society. Atlanta, 2019). Melanoma examination in the early stages is very important so that patient survival can be improved (Balch et al., 2009). One way to detect melanoma cancer is to use digital imaging called dermoscopy. Medical diagnosis using a dermoscopy with a computerized system is needed at this time.

We have implemented the classification of skin lesions in the HAM10000 dataset. This dataset consists of 10015 images of skin lesions with 7 diagnosis categories. This dataset can be accessed publicly at https://doi.org/10.7910/DVN/DBW86T. Samples of dermoscopy image data in each diagnostic category can be seen in Figure 1

Figure 1. Dermoscopy Image Samples on HAM10000 Dataset

Deep learning (DL) has been used to process dermoscopy image data in several studies (Esteva et al. 2017) (Romero Lopez et al. 2017) (Yamashita et al. 2018) (Mahmud et al. 2018) (Hosny, Kassem, and Foaud 2019) (Shahin et al., 2019). The DL method applied for the classification of skin lesions is Convolutional Neural Networks (CNN). CNN is generally used to process 2D data, for example, a color images consisting of 2D arrays (LeCun et al., 2015). CNN is composed of convolutional layers, pooling layers, and fully connected layers (Zhao & Kumar, 2017). The ilustration of CNN architecture for skin lesion classification can be seen in Figure 2.

Figure 2. CNN Architecture

Our research proposes an ensemble model approach by combining three deep CNNs architectures namely Inception V3, Inception ResNet V2, and DenseNet 201 which can be seen in Figure 3. Our system can classify 7 classes of skin lesions with excellent results. This research achieved average accuracy, average precision, average sensitivity, average specificity, and average F1 scores are 97.23%, 90.12%, 97.73%, 82.01%, and 85.01% respectively

Figure 3. Proposed CNNs with Ensemble Learning

Real time processing of Fetal Echocardiography using Deep Learning For Congenital Heart Defect

Fetal echocardiography (fetal echo) is an ultrasonic procedure conducted during pregnancy to measure the unborn baby’s heart. Referral for fetal echo typically occurs between 18 and 22 weeks gestational age. With advances in ultrasound technology, which has allowed for the significant improvements in high‐resolution imaging necessary to visualize the condition of congenital heart disease (CHD) (Pike et al.,2014). CHD diagnoses in the first trimester were first reported in the early 1990s, using transvaginal ultrasound (Bronshtein et al., 1991). CHD is a general term for a structural or functional defect of the heart that is present at birth. The heart is a complex organ formed from cells derived from at least four distinct progenitor cell types (Smith et al., 2020).

The Intelligent Systems Research Group (ISysRG) has implemented real time processing of Fetal Echocardiography using Deep Learning for Congenital Heart Defect.  YOLO v4 and the Faster R-CNNs architecture are implemented for object detection algorithm. This research was supported by public video of fetal heart (.mp4 format). Based on YOLO v4 architecture, we detected chambers of fetal heart, i.e., left atrial, right atrial, left ventricle, right ventricle, and aorta. Meanwhile, the Faster-RCNN algorithm is performed to detect four chambers, Right Ventricular Outflow Tract (RVOT), Light Ventricular Outflow Tract (LVOT), and 3 vessels.

Video 1

Video 2

YOLO v4: Real-Time Object Detection

Figure 1. Faster-RCNN algorithm is performed to detect four chambers, Right Ventricular Outflow Tract (RVOT), Light Ventricular Outflow Tract (LVOT), and 3 vessels.

Electrocardiogam Research

Based on The Indonesia Sample Registration System Survey (SRSS), among all types of Non-Communicable Disease (NCD), cardiovascular disease (CVD) was the highest cause of death at 12.9 percent at all ages (Riskesdas, 2013). Unhealthy lifestyle behavior is also a factor in the increasing prevalence of CVD. The standard of modernization refers to the lifestyle of urban communities. The thinner boundary between urban and rural areas causes a lifestyle in a rural similar to that of an urban area. So that, in handling the CVD in rural and urban, there is no significant difference. However, infrastructure development in rural areas is still slow, so access to health services is still uneven and limited (Riskesdas, 2013). Otherwise, socio-economic problems and the level of knowledge of rural communities about health are relatively lower than urban communities.

Nowadays, the use of electronic technologies for patient health medical information has produced a large amount of data from various populations, cell types, and disorders. Due to the increasing amount of data along with time and energy limitation of clinical experts in analyzing large amounts of data, a system is needed to help this process. Artificial Intelligence (AI) can be a significant element for developing a rapidly system towards CVD. AI has also been recognized as an initial step toward precision medicine. Based on AI approachment, it can be used to produce models or concepts from integration and analysis among data so that the information provided is more accurate and can help clinical experts in making decisions for CVD interpretation.

Among the patient health medical information related to CVD, electrocardiogram (ECG) is a key component of clinical diagnosis and management of inpatients and outpatients that can provide important information about the cardiac disease (Goldberger et al., 2017). Some cardiac disease can be recognized only through an ECG. ECG records electrical signals related to heart activity and producing a voltage-chart cardiac rate and being a cardiological test that has been used in the past 100 years (Khan, 2008).

To minimize the limited access to rural health care in handling cardiac abnormalities, a cheap ECG simulation system is needed as a simple test tool to measure and record the electrical activity of cardiac. A cheap ECG simulation system will be the initial detector for classifying heart disease, such as cardiomyopathy, heart failure, myocardial infarction, myocarditis, bundle branch block, dysrhythmia, etc. The interpretation of CVD from ECG is a difficult task. It takes an experienced cardiologist to be able to interpret the condition of the human cardiac. Therefore, an ECG simulation system for classifying cardiac abnormalities is made based on Artificial Intelligence (AI).

Figure 1. ECG Processing

The Intelligent Systems Research Group (ISysRG) has published some articles that implemented ECG for diagnosing some cardiac abnormalities using the Physionet dataset (https://physionet.org/). The PhysioNet Resource’s original and ongoing missions were to conduct and catalyze for biomedical research and education.

The lists of the published articles in the past two years (2019 – 2020):

  1. Deep Learning with a Recurrent Network Structure in the Sequence Modeling of Imbalanced Data for ECG-Rhythm Classifier (link: https://www.mdpi.com/1999-4893/12/6/118)
    Publisher: Algorithm, MDPI, 2019

    The interpretation of Myocardial Infarction (MI) via electrocardiogram (ECG) signal is a challenging task. ECG signals’ morphological view show significant variation in different patients under different physical conditions. Several learning algorithms have been studied to interpret MI. However, the drawback of machine learning is the use of heuristic features with shallow feature learning architectures. To overcome this problem, a deep learning approach is used for learning features automatically, without conventional handcrafted features. This paper presents sequence modeling based on deep learning with recurrent network for ECG-rhythm signal classification. The recurrent network architecture such as a Recurrent Neural Network (RNN) is proposed to automatically interpret MI via ECG signal. The performance of the proposed method is compared to the other recurrent network classifiers such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). The objective is to obtain the best sequence model for ECG signal processing. This paper also aims to study a proper data partitioning ratio for the training and testing sets of imbalanced data. The large imbalanced data are obtained from MI and healthy control of PhysioNet: The PTB Diagnostic ECG Database 15-lead ECG signals. According to the comparison result, the LSTM architecture shows better performance than standard RNN and GRU architecture with identical hyper-parameters. The LSTM architecture also shows better classification compared to standard recurrent networks and GRU with sensitivity, specificity, precision, F1-score, BACC, and MCC is 98.49%, 97.97%, 95.67%, 96.32%, 97.56%, and 95.32%, respectively. Apparently, deep learning with the LSTM technique is a potential method for classifying sequential data that implements time steps in the ECG signal.

  2. Deep classifier on the electrocardiogram interpretation system (link: https://iopscience.iop.org/article/10.1088/1742-6596/1246/1/012030)
    Publisher: IOP, 2019

    Electrocardiogram (ECG) is a primary diagnostic tool for cardiovascular diseases. A higher accuracy of heart diseases needs an automatic classification for intelligent interpretation of cardiac arrhythmia. The classification process consists of following stages: detection of QRS complex in ECG signal, feature extraction from detected QRS using R-R interval, segmentation of rhythms using extracted feature set, learning system by using Deep Neural Networks (DNNs). The performance is analyzed as a rhythm of arrhythmia classifier and MIT-BIH arrhythmia database uses to validate the method. To benchmark, the performance of DNNs algorithm is compared to, MLP and SVM algorithm in terms of accuracy. The result obtained show that the proposed method provides good accuracy about 97.7 % with less expert interaction.

  3. An Automated ECG Beat Classification System Using Deep Neural Networks with an Unsupervised Feature Extraction Technique (link: https://www.mdpi.com/2076-3417/9/14/2921)
    Publisher: Applied Science, MDPI, 2019

    An automated classification system based on a Deep Learning (DL) technique for Cardiac Disease (CD) monitoring and detection is proposed in this paper. The proposed DL architecture is divided into Deep Auto-Encoders (DAEs) as an unsupervised form of feature learning and Deep Neural Networks (DNNs) as a classifier. The objective of this study is to improve on the previous machine learning technique that consists of several data processing steps such as feature extraction and feature selection or feature reduction. It is also noticed that the previously used machine learning technique required human interference and expertise in determining robust features, yet was time-consuming in the labeling and data processing steps. In contrast, DL enables an embedded feature extraction and feature selection in DAEs pre-training and DNNs fine-tuning process directly from raw data. Hence, DAEs is able to extract high-level of features not only from the training data but also from unseen data. The proposed model uses 10 classes of imbalanced data from ECG signals. Since it is related to the cardiac region, abnormality is usually considered for an early diagnosis of CD. In order to validate the result, the proposed model is compared with the shallow models and DL approaches. Results found that the proposed method achieved a promising performance with 99.73% accuracy, 91.20% sensitivity, 93.60% precision, 99.80% specificity, and a 91.80% F1-Score. Moreover, both the Receiver Operating Characteristic (ROC) curve and the Precision-Recall (PR) curve from the confusion matrix showed that the developed model is a good classifier. The developed model based on unsupervised feature extraction and deep neural network is ready to be used on a large population before its installation for clinical usage.

  4. Deep Learning with Long Short-Term Memory for Enhancement Myocardial Infarction Classification (link: https://ieeexplore.ieee.org/abstract/document/8916683/)
    Publisher: IEEE, 2019 (2019 6th International Conference on Instrumentation, Control, and Automation (ICA))

    Myocardial infarction (MI) may be a minor event in a type of chronic disease, even undetectable. However, it can also be a major disaster that causes sudden death. The multivariance in ECG signals for different patients causes the interpretation of existence MI is a difficult task. The various conventional method is proposed to diagnose MI of ECG signals. The conventional classifier algorithm uses a shallow feature learning architecture based on the hand-crafted feature. This paper is only a preliminary study so that this paper contains only brief analysis and plan. However, it can present other point-of-view to process cardiac rhythm that associated in timesteps based on deep learning approach. Basically, a shallow feature learns as well as deep learning. However, the advantage and characteristics of deep learning will make classifier learn automatically without having to involve human intervention. Long short-term memory (LSTM) as deep learning classifier is proposed to the binary classification of MI and healthy control patients. The public ECG signals dataset of Physionet is used to support our research. In the evaluation of binary classification, balanced accuracy (BAcc) and Matthew’s Correlation Coefficient (MCC) metrics are used to analyze imbalance sequential data of 4.57 Imbalance Ratio (IR). The overall, 3 hidden LSTM layers as classifier show good performance in imbalanced data to classify MI with precision, sensitivity, F1 score, BAcc, and MCC is 0.91, 0.91, 0.90, 0.83 and 0.75 respectively.

  5. Deep Learning-Based Stacked Denoising and Autoencoder for ECG Heartbeat Classification (link: https://www.mdpi.com/2079-9292/9/1/135)
    Publisher: Electronics, MDPI, 2019

    The electrocardiogram (ECG) is a widely used, noninvasive test for analyzing arrhythmia. However, the ECG signal is prone to contamination by different kinds of noise. Such noise may cause deformation on the ECG heartbeat waveform, leading to cardiologists’ mislabeling or misinterpreting heartbeats due to varying types of artifacts and interference. To address this problem, some previous studies propose a computerized technique based on machine learning (ML) to distinguish between normal and abnormal heartbeats. Unfortunately, ML works on a handcrafted, feature-based approach and lacks feature representation. To overcome such drawbacks, deep learning (DL) is proposed in the pre-training and fine-tuning phases to produce an automated feature representation for multi-class classification of arrhythmia conditions. In the pre-training phase, stacked denoising autoencoders (DAEs) and autoencoders (AEs) are used for feature learning; in the fine-tuning phase, deep neural networks (DNNs) are implemented as a classifier. To the best of our knowledge, this research is the first to implement stacked autoencoders by using DAEs and AEs for feature learning in DL. Physionet’s well-known MIT-BIH Arrhythmia Database, as well as the MIT-BIH Noise Stress Test Database (NSTDB). Only four records are used from the NSTDB dataset: 118 24 dB, 118 −6 dB, 119 24 dB, and 119 −6 dB, with two levels of signal-to-noise ratio (SNRs) at 24 dB and −6 dB. In the validation process, six models are compared to select the best DL model. For all fine-tuned hyperparameters, the best model of ECG heartbeat classification achieves an accuracy, sensitivity, specificity, precision, and F1-score of 99.34%, 93.83%, 99.57%, 89.81%, and 91.44%, respectively. As the results demonstrate, the proposed DL model can extract high-level features not only from the training data but also from unseen data. Such a model has good application prospects in clinical practice.

Independent Research: ISysRG Fights COVID-19, For World, For Indonesia

“Faster R-CNNs: Automated Detection of COVID-19 on Chest Computed Tomography (CT) Images”

Contributor

This research is contributed by the members of the Intelligent System Research Group (ISysRG) Faculty of Computer Science, Universitas Sriwijaya.

Head of ISysRG

Prof. Dr. Ir. Siti Nurmaini M.T

Physician Experts

  • dr. Alexander Edo Tondas
  • Dr. dr. Radiyati Umi Partan
  • dr. Rachmat Hidayat

Research Assistants

  • Muhammad Naufal Rachmatullah, M.T
  • Annisa Darmawahyuni, M.Kom
  • Firdaus, S.T., M.Kom.

External Contributors

  • Wahyu Caesarendra, S.T., M.Eng., Ph.D. (Universiti of Brunei Darussalam)
  • Timotius Wira Yudha (Medical Students of Universitas Sriwijaya)

Students

Ade Iriani Sapitri, Adithia Jovandy, Jannes Effendi, M. El Qiliqsandy, Putri Wulandari, Lia Anggraini, Abdullah Farhan, Ahmad Fansyuri, Ghina Auliya, Bima Kurniawan, Ribowo Agusti Sunoki, Arjuno Gusendi, Pininggit Harun Kusuma Nasution, M. Alfin Sukma Wardani, Ria Esafri, Ryan Darmawan Siregar, Aldi Predyansyah, Xosya Salassa, Rafi Audi Prayoga, Armanda Sanjaya, Liya Anggraini, Alna Yopa Khotimah, Suci Dwi Lestari, Annisa Karima R. Harahap, Irvan Fahreza, Bima Pratama Anom, Tri Agung Hermawan, Tiara Annisa Dina, Nadhya Hassni, M. Divo Trinanda, Leni Estiyani, Rizqi Abraqa Ramadhan, Alif Muhamad Hafidz, Tomi Mandala Putra, Sinta Bella, Azis Mulki Rafani, Muhammad Rizky Aditya Utama, RM. Ardiansyah, M. Adhitya Reski Pratama. R, Irawan, Kms. Irwan Gunawan, Faisal Baja Esa Putra, Ikhsan Nuh Atthalla, Wais Al Qarni.

.

Introduction

Coronavirus disease 2019 (COVID-19) outbreak, was firstly reported in Huanan Seafood Wholesale Market, Wuhan, China, which sells live animals. The current research briefly summarized the characteristics of patients who have been infected with COVID-19, MERS-CoV, and SARS-CoV. The incubation period infected by COVID-19 is generally 3 to 7 days and up to 14 days. Fever, fatigue, and dry cough are the main manifestations at the prodromal phase. Other researchers reported a familial cluster of pneumonia associated with the COVID-19, indicating person-to-person transmission. In Indonesia, a positive COVID-19 has 4241 patients confirmed and 373 deaths, 8.79% of contaminated on 12 April 2020 based on Indonesia National Disaster Management Authority.

Currently, high-resolution computed tomography (HRCT) is of outstanding importance as it is the main tool for screening, primary diagnosis, and evaluation of disease severity. Chest CT examination is very important in the initial diagnosis of the new type of pneumonia, and CT changes are variable. By this independent research funding, Intelligent System Research Group (ISysRG) is supported by the COVID-19 pneumonia CT images from Radiopaedia (https://radiopaedia.org), case courtesy of Dr. Bahman Rasuli (rID: 74576). The evaluation of CT Images is validated by medician and radiologists. The total number of CT images raw data are 1714; 419 COVID-19 pneumonia, 200 normal lung CT, and 24 non-pneumonic lung disease infected by virus and 1071 by bacteria.

Deep learning, as the outstanding algorithm, can be implemented in medical applications, specifically in image processing for CT. To investigate diagnostic performance by using a deep learning method with a Faster-R Convolutional Neural Networks (Faster R-CNNs) for the object detection of acquired pneumonia and other non-pneumonic lung disease using chest CT.

.

Sample of CT Images Raw Data

Figure 1. Raw Data Samples

Pre-processing Data

All data around 419 images infected by COVID-19 with pixel size ranging from 256 x 256 to 1500 x 1800. The amount of chest CT images is divided by 80% for the training set and 20% for the testing set. The number of 335 and 84 chest CT images, respectively.

Figure 2. CT Scan Image Preparation
Figure 3. The process of COVID-19 region detection

Faster R-CNNs

This research conducts the architecture that consists of the region proposal network (RPN) as a region algorithm and the Faster R-CNNs as a detector network. The VGG19 and ResNet50 architecture is designed to deliver high area detection accuracy. The Faster R-CNN detector also consists of a backbone of the CNNs, a pooling layer region of interest (ROI) and entirely linked layers followed by two sibling branches for classification and bounding box regression. Backbone modeling with hyperparameters for the VGG16 as RPNs, 50 to 200 epochs, 1 batch size, 0.0001 learning rate, 0.9 momentum, and Stochastic Gradient Descent optimizer for the pre-trained VGG16 model used to learn 2697 from 275 images.

Figure 4. The Faster R-CNNs Architecture for COVID-19 Detection

Progress 1 (Initial Results)

By using CNNs-based object detection approach with Faster-RCNN algorithm, we achieved overall accuracy of 98%. In addition, the high performance of the CNNs-based Faster-RCNN model that we built in this study was checked with 79.3% accuracy using external samples. More specifically, our model achieved relative high sensitivity as a screening tool, 0.88 and 0.83 on the internal and the external dataset, respectively. Besides, the research has compared to the diagnosis of RT-PCR based nucleic acid detection, which recent data have suggested that the accuracy of nucleic acid testing is only about 50-90%. In our initial research using lung CT imaging feature extraction, the research can achieve 98% accuracy. The results outperformed nucleic acid testing.

Figure 5. The result of COVID-19 object detection based on proposed Faster R-CNNs algorithm (Progress 1)

Progress 2 (Final Results)

In the next phase, the pre-tuning and fine-tuning hyperparameters is used for getting the best model. For the object detection of COVID-19 by Faster R-CNNs algorithm, this research achieved 99% accuracy, 98% sensitivity, 100% specificity, and 100% precision on CT chest radiography. The result of CT images can detect two conditions including COVID-19 infected and healthy. Besides, the proposed model with unseen data achieves accuracy, sensitivity, specificity, precion and F1-score are 100%, respectively.

Figure 6. The result of COVID-19 object detection based on proposed Faster R-CNNs algorithm (Progress 2)