COVID-19 originated in Wuhan, China, in December 2019 and turned into a worldwide pandemic. As of December 2022, the number of people infected with this disease reached approximately 650 million, of whom more than 6.64 million had lost their lives. Although the number of new infections appeared to have abated in the spring of 2023, the past explosion of infections strained medical care systems in several countries. To cope with this pressure, these countries have changed their responses toward infected patients based on the severity of their illness. In Japan, as defined in Table 1, which shows the Ministry of Health, Labour and Welfare guidelines on the severity of COVID-19 [1], responses were divided into 4 categories of severity, ranging from mild to serious illness. Mild illness is defined as “an oxygen saturation of 96% or more, or a clinical condition of no respiratory symptoms or coughing without shortness of breath (SoB) but no evidence of pneumonia in either case,” and moderate illness I is defined as “an oxygen saturation of greater than 93% but less than 96%, or a clinical condition of shortness of breath or pneumonia.” Moderate illness II is defined as “oxygen saturation of 93% or less, or oxygen administration is required.” The target population for this study was individuals who were recovering at home or in recuperation facilities. Therefore, they were theoretically patients with mild illness. Still, due to worsening conditions or shortcomings of medical services, this population included patients with moderate illness I who should have been treated in a hospital. Therefore, accurately classifying these 2 adjacent severity categories (mild illness and moderate illness I) is essential in determining appropriate measures, such as early hospitalization, by detecting worsening conditions in patients with mild illness.

Oxygen saturation (SpO₂) measurements using a pulse oximeter were crucial for assessing the severity of illness. Daily measurements of SpO₂ and body temperature, along with the assessment of physical conditions, were essential for monitoring disease progression from mild illness to moderate illness I over approximately 1 week or more during the recuperation period. However, the explosive increase in the number of COVID-19 patients made it difficult to distribute pulse oximeters to all patients with mild illness, especially young patients who were forced to recuperate at their homes rather than in health care facilities. This unexpected shortage of pulse oximeters has motivated us to devise alternative and cost-effective ways to monitor for worsening medical condition in persons exhibiting mild illness.

Previous research on Parkinson disease, Alzheimer disease, depression, and other psychiatric disorders such as stress [2-7] has shown that voice biomarkers can be leveraged to noninvasively and cost-effectively identify the presence or absence of diseases, classify symptoms, and monitor conditions. Voice biomarkers could also be an alternative method to detect changes in disease severity from mild illness to moderate illness I in COVID-19, which is a respiratory disease and has been reported to cause acoustic changes in the voice due to inflammation of the pharynx in the vocal tract, vocal cords, or both, as well as a lower expiratory volume due to pneumonia [8]. Moreover, significant differences in jitter (fluctuation of the fundamental frequency on the time axis), shimmer (fluctuation of the amplitude on the power axis), and harmonic-to-noise ratio (HNR) were reported between healthy subjects and those with COVID-19 [8-11]. There are also reports that COVID-19 can be detected from acoustic data obtained from a patient’s cough [12].

Dynamic time warping (DTW) is an effective algorithm for measuring the similarity between 2 patterns. The DTW distance, which is a computational result obtained by the DTW algorithm using 2 waveform features, progressively approaches zero as the features become more similar, whereas it increases as the features become less similar (Figure 1).

This metric has been widely used since the 1980s in fields such as motion recognition, speech recognition, and time-series data analysis [13-15]. For example, it was reported that DTW could differentiate between healthy people and people with walking disabilities with high accuracy by processing differences in the gait patterns acquired by accelerometer sensors on smartphones [13]. The effectiveness of an automated scoring system applied in conjunction with the DTW algorithm for evaluating the progress of speech audiometric rehabilitation was reported to be similar to that of conventional manual scoring methods [14]. It has also been reported that complementing the mel-frequency cepstral coefficient (MFCC) algorithm with the DTW algorithm improved voice recognition performance. The DTW algorithm has been introduced as a feature-matching technique for voice recognition [15]. The feature-matching performance of the DTW algorithm (ie, the scoring method for 2D feature similarity) may function effectively for the desired classification of long vowel samples.

This feasibility study aimed to test whether DTW-based voice biomarkers can be used to achieve a binary classification of mild illness and moderate illness I for COVID-19 at a significant level.

We conducted a cross-sectional study using the voice samples of COVID-19 patients.

This study recruited participants through a brochure that was distributed exclusively to COVID-19 patients who were aged 20 years and older, were positive for SARS-CoV-2 in PCR testing, and recuperated at designated facilities or at home in Kanagawa prefecture, Japan, between June 2021 and March 2022. Patients who consented to the study’s objectives were requested to register for participation using the QR code on the brochure through their smartphones. The participants were asked to provide their daily vital signs data, including voice recordings during the recuperation period, but were also given the option to withdraw from the study (opt out) at any time of their own accord. A ¥1000 (US $6.68) Amazon gift card was given to participants as compensation. Because this study only included patients with mild illness or moderate illness I, who did not require hospitalization, patients with moderate illness II were not included. The participants were divided into 2 groups, the mild group and moderate I group, according to the definitions given in Table 1.

Those who agreed to participate in the study were asked to install a smartphone app and enter their basic information, vital signs data (temperature and SpO₂), symptom scores, and voice recordings on the first day of recuperation. From the second day onward, the participants were required to enter their vital signs data, symptom scores, and voice recordings daily until the last day of recuperation. Voice data were stored together with text data indicating the symptoms and vital signs on a dedicated server with high security. The voice recordings were in the WAV format with a sampling rate of 48 kHz and a bit depth of 16 bits using the 3 long vowels /a/, /e/, and /u/. Participants were asked to explain their reasoning if they wished to withdraw from the study. Table 2 shows the timing and data entry items of the participants.

To calculate the DTW distance, a 10-cycle waveform sample was extracted for each date from the participants’ long-vowel recordings in the WAV format using Audacity (version 3.1.3; Audacity Team) at a sampling rate of 48 kHz (Figure 2). Then, standardization was achieved along the power axis within the range of –1 to +1 as the maximum amplitude, and the time axis involved 1000 data points that were multiplied by 1/48,000 seconds, considering the length of a 10-cycle waveform. To read and standardize the WAV data, R (version 4.4.2; R Core Team) with the tuneR package (version 1.4.0) was used.

After standardizing the power and time of the 110 waveform samples, the DTW distance was calculated for each sample paired with those of the remaining 109 samples. The DTW distances that were obtained were divided into 2 categories based on 2 kinds of labels for the 109 waveform samples. Therefore, each sample was assigned 2 variables for DTW distance. In the mild group, these were 61 or 60 DTW distances, and in the moderate I group, these were 48 or 49 DTW distances. The average and variance of the DTW distances were calculated for each group. For the mild group, the average index (ie, the mild-group filtering [MiF] average) and variance index (ie, MiF variance) of the DTW distance were obtained, whereas for the moderate I group, the average index (ie, moderate-group filtering [MoF] average) and variance index (MoF variance) of the DTW distance were obtained. These 4 indices were obtained from a single waveform sample. The indices for the 3 vowels, /a/, /e/, and /u/, were prepared and are represented as shown in Table 3. Thus, 12 indices were used in the subsequent analyses. The average values and variances of the DTW distances for the 2 groups were statistically investigated to determine whether they exhibited significant values for the 2-group classification scheme.

The study was conducted in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of Kanagawa University of Human Services (SHI 3-001, dated May 27, 2021, and SHI 26, dated November 25, 2021). Informed consent was obtained from all participants involved in the study.

In June 2021 and March 2022, approximately 540,000 COVID-19 patients were recorded in Kanagawa prefecture. After requesting approximately 10,000 people to participate in the study, our study recruited 295 participants in the same period, of whom 291 were eligible to participate because participants who did not meet the inclusion criteria, such as minors, those with invalid data registration, or those who withdrew midway through evaluation, were excluded.

Seventy-four participants who reported no symptoms of coughing, throat pain, chest pain, or SoB during recuperation were assigned to the mild group. Of the 217 participants who reported any symptoms, 68 of them with symptoms of SoB were assigned to the moderate I group. Of the 149 participants who reported symptoms other than SoB, 6 of them with SpO₂ values less than 96% were assigned to the moderate I group, and 143 participants who reported SpO₂ values of no less than 96% were assigned to the mild group. The 291 participants were classified into 2 groups: 217 as mild and 74 as moderate I. Figure 3 shows a flowchart of study participation.

The primary periods of infection in Japan were during the Delta period, from July to December 2021, and the Omicron period, from January to June 2022. According to previous reports [16,17], COVID-19 exhibits varying levels of infectivity, severity, and symptoms, depending on the type of mutant strain present. We identified the time of infection in Japan and carefully matched the 291 study participants who had already been labeled into 2 groups. Table 4 shows the attribution matrix for the participants by the time of infection, sex, and severity. Table 5 shows the attribution matrix for the same sample based on the time of infection, sex, and age group. Finally, 110 participants (61 with mild illness and 49 with moderate illness I) were included in the study.

This feasibility study demonstrated that the DTW distance–based voice biomarkers generated by the GLM had a balanced accuracy ranging from 80.2% to 88% and high model performance, indicated by the AUC ranging from 86.5% to 96.5%, for 3 vowels when classifying between mild illness and moderate illness I in COVID-19 patients.

Our previous pilot study using the DTW algorithm to differentiate between the voices of subjects with and without COVID-19 investigated the use of different waveforms with 1, 3, 5, 10, 20, 30, and 50 cycles. Among these waveform cycles, 10 and 20 cycles resulted in reasonable discrimination between the subjects with and without COVID-19 for all 3 subjects tested. (A summary of the test results is shown in Multimedia Appendix 2.) We selected a 10-cycle waveform, rather than a 20-cycle waveform, in order to reduce as much as possible the total computing cost incurred when using the DTW algorithm (see the section “Computing Cost”).

Medical treatments for COVID-19 vary depending on the severity of the illness. Patients with mild illness may need only to recuperate at home or a designated facility, whereas patients with moderate illness I may need to be hospitalized. In this study, the DTW distance–based voice biomarker was tested for distinguishing between mild and moderate illness I. A balanced accuracy ranging from 80.2% to 88% was achieved, and the model performance indicated by the AUC ranged from 86.5% to 96.5% for the vowels /a/, /e/, and /u/. This voice biomarker system can be used in case of an unexpected shortage of pulse oximeters as an alternative and cost-effective method for monitoring worsening medical conditions in patients with mild illness that are recuperating at home or a medical facility.