Digital Phenotyping of Voice Biomarkers for Psychological States in College Students: Longitudinal Mobile App Study

Introduction

The transition to college represents a critical developmental period characterized by increased independence, shifting social networks, and heightened academic and psychosocial demands. Thus, college transitions are inherently stressful [1-3]. As a result of uncertainty and other stressors, college students carry a substantial mental health burden. Recent data indicate 38% of students screen positive for moderate or severe depression and 34% for anxiety, with 13% reporting suicidal ideation in the past year [4]. In addition to these more severe diagnosable mental health problems, many students experience high levels of negative stress, loneliness, not feeling accepted, and negative affect that may impair functioning and progress to clinical disorders [4,5].

Early detection of negative mental health states represents a critical opportunity for timely, targeted, early intervention that can improve both mental health and academic outcomes [5,6]. Indeed, social isolation and loneliness increase the risk of depression and anxiety up to 9 years downstream [7], and college students who report early and often that they do not belong tend to have higher depression at the end of a semester [8]. Early detection efforts can be particularly impactful in college populations because of stressful transitions these young people undergo during a developmental period when social hierarchies are unstable and affective states are more dynamic and context-dependent relative to older adults [6].

Even though there is a great need for detecting risk for mental health problems in college students, traditional assessments are poorly suited to capturing early signals. Subpar detection can be traced to an overreliance on episodic, in-person evaluations when problems have already escalated to the point of concern and/or retrospective self-report measures administered infrequently and with questionable precision. Additionally, both standard approaches are limited by biases associated with subjective self-reported measurement [6]. Ecological momentary assessment (EMA) research has demonstrated that momentary data collection substantially improves accuracy and ecological validity compared to retrospective reporting, but typical EMA methods disrupt daily routines by “pinging” individuals with prompts to complete self-reports [6]. Thus, “light touch” digital approaches that greatly reduce burdens on college students, but can also collect data in brief time windows, may be more acceptable and engaging to young people, a perspective supported by qualitative evidence [9,10].

Digital mobile devices may provide a low-burden, scalable means for assessing, and ultimately improving, college students’ mental health [11]. In addition to providing access to self-report questionnaires, mobile phones enable digital phenotyping through collection of multimodal data streams, including voice recordings through voice journaling. Voice-derived biomarkers such as prosodic, spectral, and temporal speech features have demonstrated utility in detecting and monitoring depression and anxiety [12,13]. More recent studies using mobile-collected speech data further support the feasibility of predicting mental health symptom severity in real-world populations to detect clinical conditions and estimate severity [13]. These findings suggest that vocal biomarkers may provide an objective and scalable marker of psychopathology. However, existing research has focused primarily on identifying or diagnosing clinical conditions, with limited attention to negative affective states or to young adult populations. This blind spot is particularly notable given that strong and volatile negative emotions predict the onset of mood disorders [8]. From a psychopathology prevention perspective, it is critical to identify the risk markers of a pending depressive or anxious episode.

Although prior work has examined the use of voice biomarkers in relation to individual mental health constructs, no study, to our knowledge, has evaluated their ability to predict multiple psychological state changes (eg, perceived stress, loneliness, perceived acceptance, and core affect) simultaneously within a college population. Also, while much research on core affect focuses on specific emotions as predictors of downstream outcomes [14], advances in emotion science have emphasized understanding more basic underlying affective processes that contribute to the construction of emotional experiences [15]. That is, researchers frequently emphasize core affect—or low-level, consciously accessible affective feelings and neurophysiological responses [16]—states of valence (positivity vs negativity) and activation (high vs low) as proximal predictors of health-related outcomes [17]. The current work advances research at the nexus of mobile sensing, emotion, and affective states by focusing on identifying biomarkers associated with core affect as assessed via emoji usage.

To advance knowledge in identifying affective risk factors, the research reported here leveraged a mobile health platform, Meet Pandora (developed by Taehyung Cho), to collect high-frequency, multimodal data including ecologically relevant self-report and voice-derived features from college students in their daily environments. We evaluated whether these signals predict psychological state changes in stress, loneliness, acceptance, and core affect, with the goal of enabling earlier detection of mental health risk in this high-need population.

Methods

Participants

Participants were undergraduate students enrolled at a highly selective private Northeast US university and recruited through a university study portal. All research was approved by the site’s institutional review board, along with informed consent obtained from all participants. Two cohorts of students were recruited: (1) spring 2024 semester (Cohort 1, n=169) and (2) spring 2025 semester (Cohort 2, n=120). Data from both cohorts were integrated for analysis, with most self-report surveys and voice prompts being the same in both studies. Participants in Cohort 2 were offered a stress survey that was not available to the participants in Cohort 1, and Cohort 2 also had additional voice prompts that students were able to respond to (Multimedia Appendix 1).

The Meet Pandora App

The Meet Pandora smartphone app, available on Android and Apple iOS operating systems, was used to collect data. After signing informed consent, per protocol, students were onboarded onto the app and were asked to review and accept Pandora Bio’s privacy policy. Additionally, before submitting any data on the app, they were prompted to enable the app to use the device microphone and were asked to opt in (check) other permissions for location data collection and use of their smartphone camera. Voice recordings were not collected passively and required the students to actively engage (click a button) to record. They were told that all data would be deidentified and that their identifiable data would not be shared. Meet Pandora facilitates frequent, low-burden self-report using formats such as voice journals, self-reports, and emoji-based representations of core affect. The tool collects data both actively through participant self-reports (validated instruments such as the Patient Health Questionnaire 8-item (PHQ-8), the Generalized Anxiety Disorder 7-item scale (GAD-7), and the Flourishing and Stress scales, as well as brief daily, weekly, and monthly self-reports), and passively through derived voice biomarkers, GPS data, and phone usage metrics (Table 1).

Self-Reported Measures

Acceptance and Loneliness

Questions on acceptance and loneliness were administered to participants weekly. To measure acceptance, participants were asked, “How accepted do you feel by your friend group?” (1=not at all accepted and 5=strongly accepted). To measure loneliness, they were asked, “How often have you felt lonely?” (1=very slightly or not at all and 5=extremely). A high or low acceptance and loneliness dichotomous label was derived from participants’ responses, where a score of 3 (the midpoint) or higher was labeled as “high” acceptance or loneliness, and scores of 1 or 2 were labeled as “low.”

Stress

As part of the weekly survey, participants reported their stress by responding to the question “How much stress are you experiencing in your life right now? (1=none, 4=a moderate amount, and 7=an extreme amount).” An ordinal stress label was derived from participants’ responses by categorizing scores into 3 ranked groups: low (scores 1‐3), medium (score 4), and high (scores 5‐7). Notably, this stress measure was only included in the second study (n=120).

Self-Regard

Another essential psychological state that was measured and derived is the participant’s self-regard [18]. As part of the weekly self-reports, participants were asked these four questions: “How accepted do you feel by your friend group?” “How often have you felt lonely?” “To what extent did you go out of your way to avoid one or more difficult situations?” and “Overall, how good or bad did you feel about yourself this week? (1=Extremely good to 7=Extremely bad).” A composite weekly self-regard score was derived in 2 ways. First, a weekly continuous self-regard score was derived by averaging z-score-scaled responses to the 4 questions (excluding observations for which at least one question was unanswered). Second, for interpretability, self-regard weekly state was derived as high or low self-regard using principal component analysis (PCA) and K-means clustering.

Daily Affective States

Participants also provided their affective states via emojis during daily prompts. Participants chose up to 3 out of 16 emojis to best represent how they felt in the moment. Available options (and their activation and valence categorization) were: Accepted (low activation and positive), Angry (high activation and negative), Annoyed (high activation and negative), Anxious (high activation and negative), Confused (low activation and negative), Contented (low activation and positive), Excited (high activation and positive), Fearful (high activation and negative), Happy (high activation and positive), Hopeful (high activation and positive), Lonely (low activation and negative), Proud (high activation and positive), Refreshed (low activation and positive), Relaxed (low activation and positive), Sad (low activation and negative), and Tired (low activation and negative; Figure 1). Because every participant’s daily entry included at least 1 and up to 3 emojis, the first emoji was treated as the primary indicator of their core affect and used to derive all subsequent daily affect labels. To evaluate broad daily affect states, a daily ordinal label was calculated to capture the net polarity of all emojis selected during a given self-report. This was computed by calculating the ratio of positive emojis to the total number of emojis entered, which was then mapped onto a standardized ordinal scale.

We also calculated valence by arousal interactions. The first emoji selected by participants was divided into 4 categories: high activation and positive, high activation and negative, low activation and positive, and low activation and negative. This classification system was created by intersecting the binary valence dimension (positive vs negative affect) with the binary arousal dimension (high vs low activation).

‎

Passive Measures

Voice biomarkers were derived from participant speech recordings to capture daily variation in vocal and linguistic behavior. We derived features that capture both how they speak and what they say. During the study, participants were prompted to provide a voice journal 3 times per week by responding to 1 of 50 available questions, such as “Describe the highs and lows of your day.” Questions were either positive or neutral, designed to prompt responses between 30 seconds and 2 minutes. Participants could choose a different question to respond to if they did not want to respond to the one initially surfaced to them. From the participants’ audio responses, features from 2 general categories were derived and measured: speech and acoustic. The description of all derived speech and acoustic features is shown in Table 2.

Data Analysis Plan

Joining Voice Entries and Weekly or Daily Self-Report

Voice data entries were joined to each participant’s corresponding responses to weekly prompts to integrate the biomarker data with psychological labels (loneliness, acceptance, stress, and self-regard). To ensure temporal relevance leading up to the self-reported psychological states, a 7-day lookback window was applied. Daily voice biomarker data were linked to a weekly survey label only if the audio entry was recorded on the same day or within the 7 days immediately preceding that specific self-report. For analysis of voice and daily self-report, the voice and daily affect labels were joined if both data points were recorded on the exact same calendar day for each participant. This day-level alignment ensured that the vocal expressions being analyzed had direct correspondence to the participant’s self-reported affective state at that specific time.

Feature Processing, Biomarker Outliers, and Data Cleaning

To understand and remove outliers in the derived voice biomarkers, a large language model (LLM; Google Gemini 2.0 Flash) was used to examine every instance to identify physically impossible observations [24]. Based on the model’s output, 424 outliers were identified due to the presence of background noise. Multimedia Appendix 2 includes chat history and prompts. Table 3 summarizes the removal criteria, which were implemented using R (R Foundation for Statistical Computing) code based on the criteria below.

In the multivariable mixed-effects modeling of voice biomarkers associated with loneliness, acceptance, stress, and daily emojis, a target feature removal was implemented to maximize data retention. For instance, if the mean pitch for a specific voice sample was physically impossible, only the mean pitch for that instance was removed, and the rest of the features for that sample were retained.

From participants’ daily voice journals, 16 features were extracted and derived. Prior to statistical modeling, a correlation analysis was conducted to assess voice feature multicollinearity. Magnitude variance, which measures the variance in volume, had a Pearson correlation of 0.92 with magnitude mean, and average spectral centroid, which measures the brightness and sharpness of the audio, had a Pearson correlation of 0.94 with zero crossing rate. Consequently, these 2 features were excluded from subsequent analysis. The feature correlation matrix is given in Multimedia Appendix 3. Prior to modeling, voice biomarkers were standardized using the built-in scale() function in R to ensure a mean of 0 and a SD of 1, thereby ensuring the direct comparison of effect sizes across different features.

Modeling Process

Because each voice sample was elicited by a prompt that varied in emotional sentiment (positive or neutral), the sentiment of the prompt is a potential source of variation in the voice biomarkers. For example, a positive prompt may elicit a faster speech than a neutral one, introducing variation in the voice biomarker that reflects the eliciting condition rather than the student’s underlying state. Therefore, we evaluated whether prompt sentiment was associated with voice biomarkers, and where it was associated, we adjusted for prompt sentiment to reduce measurement-related variability. Specifically, for each voice biomarker (feature), the model treated prompt sentiment as a binary value (positive or neutral) and was parameterized as follows:

${\displaystyle {\displaystyle logit\left(P\left(PromptSentimen{t}_{ij}=1\right)\right)={\beta }_0+{\beta }_{1}Featur{e}_{ij}+u_j+{\epsilon }_{ij}}}$

where Prompt_Sentiment_ij denotes prompt sentiment for the i^th observation from the j^th student, Feature_ij represents the voice feature for observation i from participant j, u_j is the student-level random intercept, and ${\epsilon }_{ij}$ is the error term.

Voice biomarker features were also divided into 2 distinct groups: speech and acoustic features. Speech features are features that describe the temporal characteristics. Acoustic features describe the underlying physical properties of the voice, such as pitch mean, root-mean-square (RMS) energy, and magnitude. To control for the inflation of Type I errors from multiple comparisons, P values were adjusted using the Benjamini-Hochberg False Discovery Rate method within each feature group (speech and acoustic; Multimedia Appendix 4).

Outcome Variable Coding

Weekly psychological states (acceptance and loneliness) were treated as binary (high vs low). Self-regard was modeled 2 ways: as a binary outcome (1=low self-regard and 0=high self-regard) and as a continuous z-score composite, on which higher scores indicate lower self-regard. Daily affect sentiment was treated as a 5-point affect scale, and stress was treated on a 3-point scale with higher numbers associated with higher stress. Valence and arousal derived from emojis as described earlier in the methods were coded as follows: valence was treated as 1=negative and 0=positive, while arousal was treated as 1=high activation and 0=low activation. The combination of valence and arousal was treated as 1=negative and high activation, with 0 used to denote 3 other categories (positive and high activation, positive and low activation, and negative and low activation).

Analysis Modeling of Weekly and Daily Outcomes and Voice Biomarkers

Modeling the association between voice biomarkers and daily and weekly outcomes accounted for the repeated outcome measures over days or weeks, repeated voice biomarkers over weeks, the measurement-related variability introduced by the prompt sentiment, and potential confounders (ie, person-level features that are associated with both the outcome of interest and with the voice biomarkers). Each voice biomarker was first standardized and then represented in the model as person-level mean (between-person effect) and person-mean centered by subtracting the person-mean from the value at the given timepoint (within-person effect). Potential confounders that were included in all models include gender identity (male, female, or other), mental health diagnosis (yes or no), and foreign national status (yes or no). In addition to gender identity, mental health diagnosis, and foreign national status fixed effects, additional fixed effects included day number and voice biomarker (feature). Random effects for student (intercept) were also included to account for repeated measures. Consistent with the analysis of prompt sentiment, voice biomarkers were divided into two groups: speech and acoustic, and P value adjustments were made within each feature group.

In each model, the psychological state (acceptance, loneliness, stress, self-regard, or daily affect) served as the dependent variable, and a single voice biomarker served as the primary predictor, entered as both its person-level mean (between-student effect) and its person-mean-centered deviation (within-student effect). One model was fit per voice biomarker, each adjusted for the covariates described above and including a random intercept for student.

The model family was determined by the measurement scale of the outcome. Dichotomous outcomes (acceptance, loneliness, and the binary self-regard label) were fit as logistic mixed-effects models using glmer() in the lme4 package. Ordinal outcomes (weekly stress on a 3-point scale and daily affect on a 5-point scale) were fit as cumulative-link, proportional-odds mixed models using clmm() in the ordinal package. Continuous weekly self-regard was fit as a linear mixed-effects model using lmer() in the lme4 and lmerTest packages, with degrees of freedom and P values derived from the Satterthwaite approximation. All models were fit with the bobyqa optimizer, and 95% CIs were obtained from the fixed-effects estimates. Associations are reported as odds ratios (OR) for the logistic and ordinal models and as unstandardized regression coefficients for the linear model, in which the outcome is on a standardized (z-score) scale. The logistic parameterization is as follows and the parameterization for the ordinal stress and continuous self-regard outcome model is similar.

${\displaystyle {\displaystyle \begin{array}{ll}\mathrm{logit}\left(P\left(Outcom{e}_{ij}=1\right)\right)& ={\beta }_0+{\beta }_{1}FeatureMea{n}_j+{\beta }_2{FeatureDiff}_{ij}\\ & +{\beta }_{3}Gende{r}_j+{\beta }_{4}Diagnose{s}_j+{\beta }_{5}ForeignNationa{l}_j\\ & +{\beta }_{6}PromptSentimen{t}_{ij}+{\beta }_{7}DayNumbe{r}_{ij}+u_{0j}+{\epsilon }_{ij}\end{array}}}$

where Outcome_ij denotes the binary outcome variable for the i^th observation from the j^th student, FeatureMean_j represents the mean of the feature for the jth student, and FeatureDiff_ij represents the feature difference of the ith observation from the mean for the jth student. Gender_j represents the gender, Diagnoses_j represents the diagnosis status of the j^th student, ForeignNational_j represents the foreign national status of the j^th student, PromptSentiment_ij represents the prompt the student j responded to at observation, i, DayNumber_ij represents the response day i of the jth student, and u_0j is the student-level random intercept and ε_ij is the error term.

Ethical Considerations

The study was conducted under an institutional review board–approved study protocol at the University of Rochester. The University of Rochester Research Subjects Review Board (RSRB) provided approval for all data collection and data treatment procedures prior to the collection of any data with the RSRB approval number of 00011296. All participants signed informed consent and were enrolled in-person into the study. Participants were informed by the University of Rochester study coordinator that their participation was voluntary and that they could withdraw at any time. All participants were offered the opportunity to ask questions regarding informed consent. Before using the Meet Pandora app, at the time of onboarding, and following informed consent, the Meet Pandora privacy policy and terms and conditions of use were surfaced in the app, and students were asked to read and accept the terms and conditions. The privacy policy states that their data will not be identifiable and will not be shared beyond deidentified data used for research purposes. All participants were required to be 18 years or older at the time of enrollment. Age and gender characteristics are summarized in the paper.

Results

Cohort Characteristics

Students (N=289) participated for an average of 49.40 days (SD 20.78, range 1‐101 days). The age and prevalence of mental health diagnoses are representative of a typical college-age population. The cohort is diverse with respect to sexual orientation, race, religion, and first-generation and international students and is skewed towards females (229/289, 79%). Table 4 contains demographic information.

In total, 281 of 289 (97.2%) students responded to 1 or more voice prompts throughout the study. A total of 2036 weekly self-reports were available for analysis. After accounting for missing values, 1330 (65.6%) of entries reported low loneliness, and 696 (34.4%) reported high loneliness. For acceptance, 1893 (93.7%) of entries reported high acceptance, and 128 (6.3%) reported low acceptance. Because the stress measure was only administered to the second cohort, 802 weekly self-reports included stress data. After accounting for missing values, 271 (34.7%) reported low stress, 189 (24.2%) reported medium stress, and 322 (41.2%) reported high stress. For weekly continuous self-regard measurements, 273 students had 1308 entries with a valid derivation. For the dichotomous label of weekly self-regard, 703 (53.8%) indicated high self-regard and 605 (46.2%) indicated low self-regard.

There were significantly more data for daily entries. For daily ordinal affect, there were 5687 entries with 1906 (33.5%) negative, 489 (8.6%) somewhat negative, 530 (9.3%) neutral, 620 (10.9%) somewhat positive, and 2142 (37.7%) positive affect. When categorized by the interaction of valence and arousal, 1810 (31.8%) of these daily entries reflected low-activation and negative states, 1506 (26.5%) low-activation and positive, 1462 (25.7%) high-activation and positive, and 909 (16%) high-activation and negative.

Prompt Sentiment and Voice Biomarkers

Five speech features returned statistically significant associations (adjusted P<.05) with positive prompt sentiment, suggesting that whether the prompt was positive or neutral introduced variation in speech features. Conversely, no acoustic features returned statistically significant results. Because prompt sentiment was shown to affect participants’ speech features, it was added as a controlling covariate in subsequent multivariable analyses. All subsequent models associating voice biomarkers with outcomes converged. Tables 5 and 6 contain speech and acoustic features and their analysis results.

Acceptance, Loneliness, Stress, and Self-Regard

Self-regard was modeled in 2 ways, as a continuous outcome and as a dichotomous outcome. Inspection of the distribution of responses across Likert-scale responses for each of the outcomes indicated that dichotomization was necessary for loneliness and acceptance, and that collapsing to 3 categories was required for stress in order to estimate within-student and between-student associations (Multimedia Appendix 5). While a sufficient number of state changes were not observed for modeling the association of voice biomarkers with a change in a student’s state of acceptance (results were not statistically significant), the majority of models converged in order to estimate the association of voice biomarkers with changes in a student’s feelings of loneliness and stress. State changes for each outcome are depicted in Multimedia Appendix 6.

Neither speech nor acoustic features were associated with acceptance or loneliness. Among students who used more positive language than their peers on average, stress was lower (OR 0.68, 95% CI 0.50‐0.92; adjusted P=.049) and their self-regard was higher (adjusted P=.04). The dichotomous self-regard analysis also showed the finding of higher word count associated with lower within-person self-regard. Table 7 contains the acceptance, loneliness, stress, and dichotomous self-regard results. Table 8 contains the continuous self-regard results.

Daily Affect

All models evaluating the within-student and between-student associations of voice biomarkers with daily affect, measured by daily mood score, valence, arousal, and the combination of valence and arousal converged. See Table 9 for the daily affect results.

Speech features were associated with within-student negative daily affect, where a higher daily affect score was more negative. Specifically, a higher number of total utterances than usual was associated with negative daily affect (OR 1.16, 95% CI 1.09, 1.23; adjusted P<.001). Additionally, a negative sentiment, measured by the Valence Aware Dictionary and Sentiment Reasoner (VADER) sentiment score, was associated with negative daily affect both within-student (OR 0.92, 95% CI 0.87-0.97; adjusted P=.007) and between students (OR 0.87, 95% CI 0.78-0.98; adjusted P=.04).

Acoustic features also correlated with within-student negative daily affect. Decreased volume and physical energy than usual were associated with negative daily affect: RMS energy (OR 0.90, 95% CI 0.85-0.95; adjusted P<.001); magnitude mean (OR 0.83, 95% CI 0.78-0.88; adjusted P<.001). Furthermore, lower average pitch (OR 0.80, 95% CI 0.73-0.86; adjusted P<.001), pitch variance (OR 0.90, 95% CI 0.83-0.97; adjusted P=.01), and harmonics-to-noise ratios (OR 0.91, 95% CI 0.85-0.97; adjusted P=.005) were significantly associated with lower daily affect.

Affect Valence, Arousal, and Interaction

Models fit to the 2 dimensions of core affect (valence and arousal) [16] demonstrated significant within-student associations with both speech and acoustic features, predominantly for valence but not for arousal. Consistent with the daily affect analysis, negative valence (1 vs 0 for positive valence) was significantly associated with students voicing a higher number of utterances (OR 1.16, 95% CI 1.08-1.24; adjusted P=.003), a higher rate of utterances per minute than usual (OR 1.13, 95% CI 1.05-1.21; adjusted P=.01), and by lower VADER sentiment scores than usual (OR 0.89, 95% CI 0.83-0.95; adjusted P=.007).

Within-student acoustic features demonstrated strong negative associations with valence. Consistent with the ordinal daily affect findings, when in a positive valence state, students voiced higher volume measured by magnitude mean (OR 0.83, 95% CI 0.77-0.89; adjusted P<.001), pitch, measured by pitch mean (OR 0.78, 95% CI 0.71-0.86; adjusted P<.001), and energy, measured by RMS energy (OR 0.91, 95% CI 0.85-0.97; adjusted P=.03), and harmonics-to-noise ratio (OR 0.89, 95% CI 0.82-0.96; adjusted P=.03) compared to their usual. No voice features were associated with arousal states.

Treating high arousal negative valence versus the 3 other states (high arousal and positive valence, low arousal and positive valence, and low arousal and negative valence), only between-student utterances per minute were associated.

Discussion

Principal Results

This study evaluated the utility of smartphone-captured speech and acoustic features as objective indicators of psychological states in college students. Principal findings indicate that voice biomarkers were associated with daily affect states, self-regard, and stress, but no reliable associations with loneliness or acceptance. These associations were predominantly within-student, describing how a student’s voice shifted as their own state changed over time. For self-regard and stress, only speech and linguistic features tracked these within-student shifts. Participants’ affect, derived via an ordinal label on a 5-point scale and via a binary valence label, exhibited consistent results, revealing that positive and negative core affects were characterized by a distinct speech and acoustic profile, dominated by the acoustic elements. Behaviorally, positive daily affect was associated with fewer total utterances, a lower rate of utterances per minute, and higher positive sentiment in the transcribed text. The arousal component of affect did not yield any reliable associations, while between-student utterances per min were associated with higher odds of reporting a high-activation negative state, the only reliable association observed for the valence-by-arousal interaction.

The observed discrepancy between the predictive ability of daily versus weekly models aligns with established literature on EMA and recall bias [6]. However, this difference is also driven by differences in statistical power: weekly models have fewer observations, hence fewer state changes with which to model within-student associations. When recalling contextualized emotion states such as loneliness, acceptance, stress, and self-regard over the course of a week, participants mentally aggregate and average their emotions over several days, introducing cognitive biases and inconsistencies to their responses. For instance, a high-intensity emotional experience on one day might have an outsized impact on a weekly recall in addition to temporal misalignment between the assessments. Because participants’ core affects naturally fluctuate throughout the week, the effect captured by a voice biomarker may not align with the retrospective psychological state reported each week. In contrast, daily emoji self-reports capture acute, in-the-moment affective states, and core affect in particular, which not only physically manifest in neurophysiological responses but also in the vocal biomarkers on the same calendar day of recording. This reinforces the foundational premise of EMA: capturing data in the moment reduces the cognitive biases inherent in retrospective surveys and reveals critical affective fluctuations that episodic reporting misses [6]. By pairing high-frequency voice biomarkers with high-frequency, low-burden self-reports like emojis, this study successfully uses EMA principles for modern digital phenotyping.

Comparison With Prior Work

Our acoustic findings for negative valence are consistent with prior clinical research on vocal biomarkers. Negative daily affect was characterized by lower magnitude mean, lower pitch mean, reduced RMS energy, and lower harmonics-to-noise ratios. These features mirror lower, less energetic voice profiles classically associated with clinical depression and negative affect in previous research [12,13].

While several studies have demonstrated that the affective dimension of arousal is strongly encoded in acoustic features such as pitch and intensity [29,30], our study identified within-person associations for valence but not arousal. There are 2 explanations for this observation. First, valence and arousal were measured by a pictorial (eg, emoji-based) single momentary assessment, similar to the nonverbal self-report pictorial technique used in another study which found that arousal was more difficult to report and more error-prone [31]. Second, the evidence that arousal is strongly encoded in voice largely derives from posed, acted, or exaggerated portrayals and stimuli, which may be attenuated in a more natural setting as was studied here. Therefore, the null associations we observed for arousal may reflect both the measurement method and the natural setting rather than an absence of vocal arousal encoding.

The relationship between higher word count and low self-regard provides a unique behavioral phenotype. While previous studies [12,13] have linked clinical depression to decreased speech quantity, our findings suggest that in a college population, low self-regard may manifest behaviorally as over-explaining, or increased verbal load and processing during open-ended voice prompts, driving higher word count. This finding coheres well with research conducted on social media platforms that observes over-disclosure is tied to increased risk for mental health problems [32].

Our finding that vocal biomarkers track within-person changes in daily affect has clinical relevance based on prior literature. Daily negative affect and affect dynamics have been shown to predict depression and anxiety symptoms in young adults [33], and heightened affective instability has been shown to distinguish individuals with current depressive and anxiety disorders from those without [34]. Prior work has demonstrated that passive and active data collection using smartphones can predict clinically relevant states such as depressed mood among undergraduate college students with clinical depression [35]. Furthermore, deep learning models have been applied to longitudinal smartphone data to predict future anxiety symptoms [36]. These results position voice as a candidate signal within a broader set of biomarkers to passively assess mental health in young adults using smartphones.

Limitations and Future Directions

Several limitations should be considered when interpreting results from this study. First off, using smartphones to collect data introduces variability and noise in derived voice biomarkers due to hardware and contextual variability. Related to voice journaling used in this study, the sample size and design did not allow us to more directly assess the impact of cultural-linguistic diversity. Second, to promote engagement, abbreviated or derived psychological measures (such as the self-regard label and single-item emoji selections) were used rather than full-length, clinically validated diagnostic interviews. This approach assesses psychological states quickly without disrupting the participants’ day, which allows for a wealth of data that is less impacted by missingness, but lacks the diagnostic depth of traditional clinical assessments. Next, this study used an observational, correlational design. While mixed-effects modeling effectively isolated the impact of within-person fluctuations and controlled for individual baselines, and care was taken to control for multiple comparisons, direct causal inferences are not possible without accompanying experimental data. Lastly, in our conservative analytical approach, several biomarkers approached but did not meet statistical significance for their association with loneliness and acceptance. Future research would benefit from more tightly aligning the time scale between voice biomarkers and psychological constructs. A larger study with a longer follow-up period would increase statistical power and allow for the application of deep-learning models. Such a study could uncover additional novel associations, including the association between affective states, psychological states (self-regard, loneliness, acceptance, and stress), and mental health diagnoses.

Conclusions

This study demonstrates that smartphone-captured voice biomarkers can be an effective tool for monitoring changes in affective states. While voice biomarkers did not predict loneliness or acceptance evaluated at the weekly level, they were highly sensitive to weekly assessments of stress and daily fluctuations in affect and valence. The majority of associated voice features reflect how someone speaks (pitch and intensity, etc), not what they say (VADER sentiment). This study suggests that college student self-monitoring built on prosody rather than language could inform and nudge students to take the next step based on their own within-person affective changes while being less privacy-invasive than a tool that analyzes what was said. Furthermore, these voice features leverage an individual student’s changes from their own speaking rather than a comparison to other students, which furthers their perception of privacy when using such a tool. These findings underscore the importance of using high-frequency and low-friction measures such as emojis, short surveys, and voice journaling when conducting digital phenotyping, which confirms existing research [37]. Ultimately, this approach paves the way for the development of scalable, targeted interventions that act on precise mechanisms and are capable of reaching college students before symptoms escalate into clinical disorders.