The study was conducted in accordance with the Declaration of Helsinki, and approved by the Ethics Committee of the Technical University of Darmstadt (approval EK 11/2023; March 20, 2023). In the first session, all participants were informed about the specific purpose of the study. We informed them that all collected data are confidential and solely used in anonymized form. To ensure anonymity, each participant was assigned a pseudonym. Participants were informed about the risks and their right to terminate the experiment at any point without the need for an explanation. Afterward, participants provided written informed consent, completed a demographics questionnaire, and responded to inquiries regarding their physical activity.

The study used a repeated measures design with each participant completing 2 sessions on separate days, with a minimum resting period of 48 hours in between. Before undergoing the tests, participants were advised to refrain from consuming alcohol or any other substances that could potentially influence their respiratory system and HR. This precautionary measure aimed to ensure accurate readings and mitigate the risk of any potential false results during the testing procedure. The initial session was conducted in a controlled laboratory setting to establish a reference value for VO₂max. The subsequent session took place on the university’s stadium track field, using the Apple Watch Series 7 to obtain an estimated VO₂max value. Following the completion of both sessions, the VO₂max values obtained from the 2 methods were compared against each other for analysis.

The performance test in the laboratory setting was assessed through an endurance test using a cycle ergometer. Such tests are widely used in sports science to measure VO₂max, serving as a crucial indicator of aerobic endurance performance [43]. Due to the lack of medical expertise to conduct a maximal exercise test, we alternatively conducted a graded exercise test until subjective exhaustion. This decision was influenced by our ability to adhere to a rigorous protocol within the controlled environment of the laboratory, as well as the availability of the necessary equipment to monitor respiratory parameters and promptly terminate the session if the participant’s safety was compromised. Submaximal exercise prediction was also used in the field test using the Apple Watch, which facilitates comparison of the values derived from sessions 1 and 2.

Accordingly, the reference VO₂max value was determined through a graded exercise test conducted on a cycle ergometer, using the portable metabolic gas analyzer (VO2 Master Health Sensors Inc [44]). Evidence of the measurement accuracy of the hardware used can be found in references [45,46]. The gas analyzer was calibrated prior to each test (ie, for each participant), using a 3-L syringe for both flow and gas calibration. Furthermore, the supervisor entered the participants’ age, sex, height, and weight in the VO2 Master Manager app (installed on an iPhone 13 Mini), which was paired with the gas analyzer. After the calibration, participants put on the electrocardiogram chest strap (Polar H10 Heart Rate Sensor [47]) and the gas analyzer while the supervisor (SY) checked the plausibility of the system (ie, both sensors connected to the smartphone via Bluetooth and transmitting the data via VO2 Master Manager app). Once participants successfully put on the equipment, they were instructed to sit on the cycle ergometer (ERGO-FIT Cycle 4073 [48]) after adjusting the seat height according to their height.

Once the setup was completed, the endurance test was conducted. The laboratory protocol was equal for male and female participants. Throughout the test, vital parameters (ie, the HR and breathing) and the participant’s current state were continuously monitored. Participants started with a 3-minute warm-up phase, riding on the cycle ergometer at a workload of 50 W at a speed of 60 rotations per minute. Afterward, the ergometer’s resistance was increased by 50 W every 2 minutes until one of the termination criteria was met (based on the criteria by Klingenheben et al [49]):

We intentionally used the Fairbarn equation to predict the maximum HR, instead of using the Fox equation HR_maxFox=220–age [51], which is only dependent on age. According to the analysis by Cleary et al [52], the Fairbarn equation, which considers the age and sex of the participants, is more accurate.

At the end of the session, protocol outcomes were saved for each participant. In addition to VO₂max, the gas analyzer provided the following parameters in real time:

Within 1 week after the initial laboratory session, participants were provided with an iPhone SE 2020 and an Apple Watch (Series 7, 41 mm). The Apple Watch was paired with an iPhone that had been reset to factory setting to ensure data privacy. To complete the setup of the Apple Watch, the supervisor (SY) ensured that participants entered their age, sex, height, and weight in the iPhone.

Participants were instructed to wear the Apple Watch continuously, including during sleep and showers, for at least 48 hours prior to the second session. This prolonged wearing duration was essential as the Apple Watch required at least 24 hours of continuous wear time to reliably estimate VO₂max. The precise algorithm for VO₂max estimation is not publicly disclosed; however, discussions with Apple technical support revealed that it incorporates resting HR measurements, exercise HR measurements, and GPS-derived velocity data from outdoor runs. To ensure a valid VO₂max from the Apple Watch, we consulted with the manufacturer and adhered to the following procedure: participants needed to complete at least 1 training prior to the track field test, that is, an outdoor walk for 15-20 minutes. They needed to manually measure the HR every hour (using the preinstalled Health app), in addition to the passive measurements of the Apple Watch itself. Throughout the process, participants needed to ensure that the Apple Watch was always connected to the iPhone, which maintained an internet connection.

Only participants who followed the instructions and completed the outdoor walk were permitted to proceed with the run test. The run test was conducted at the university stadium at the Technical University of Darmstadt. Consistent with our laboratory protocol, we used a submaximal exercise test to mitigate the risk of injury; however, in this session, the test was conducted outdoors. The outdoor setting was necessary to ensure a sufficient GPS signal.

Before the run test, participants were given brief instructions. Particularly, they were instructed to activate the outdoor running app on their Apple Watch prior to starting the track run. To minimize the risk of injury, the protocol included a 5-minute warm-up phase, during which participants ran at a moderate pace. Following the warm-up, participants continued at a self-selected running pace, ensuring a minimum duration of 15 minutes. Once participants completed the run and returned to the starting point, they stopped the recording on their Apple Watch and proceeded with a cool-down phase. Subsequently, the supervisor accessed relevant metrics from the Health app on the paired iPhone, specifically the estimated VO₂max in the cardio fitness section.

Participants were recruited among students and employees of the Technical University of Darmstadt through the Discord server from the IT department and the university’s mailing list. To ensure a diverse range of fitness levels, we also recruited members of a local fitness studio. Eligibility criteria required participants to be older than 18 years and in good health. To streamline the selection process, the Physical Activity Readiness Questionnaire [53] was administered. As a result, individuals with any preexisting heart disease, cardiovascular conditions, orthopedic injuries, or current use of medication were deemed ineligible for participation.

To determine the required sample size, we conducted a priori power analysis using G*Power (version 3.1; Heinrich-Heine-Universität Düsseldorf) [54] with a power of 0.8, a significance level of 0.05, and a medium effect size of 0.5. This analysis indicated a minimum sample size of 27 participants. Therefore, considering expected dropouts, we initially aimed for a larger sample size of at least 30 participants. Recruitment took place over a 4-week period in the spring of 2023.

All data were analyzed using MATLAB (MathWorks, Inc), including external code [55,56].

We first assessed the limit of agreement between the values obtained from laboratory measurements and those provided by the Apple Watch using the Bland-Altman plot. The Bland-Altman plot enables us to evaluate if the 2 methods of measurement show a sufficient level of agreement [57]. It displays the limits of agreement by using the mean and SD of the differences between the 2 methods. As recommended by the authors themselves, 95% of the data points should lie within ±2 SD of the mean difference [57,58]. Additionally, the plot also allows us to spot outliers and to see whether there is any trend in overestimating or underestimating.

Second, in addition to the Bland-Altman plots, we calculated the ICC(2,1) to test for bias and absolute agreement in VO₂max estimation. ICC is different from correlations such as Pearson or Spearman correlation. Calculating correlation is not appropriate to evaluate the measure of agreement, especially as the correlation coefficient depends on both the variation between individuals (ie, between the true values) and the variation within individuals (measurement error) [57]. ICC is suitable for reliability analyses, where a value less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 indicate poor, moderate, good, and excellent reliability, respectively [59].

Third, similar to other validation studies, we used the MAPE and RMSE to calculate the overall measurement error between the VO₂max value derived from the Apple Watch and the metabolic gas analyzer. MAPE was calculated as the average absolute difference between the actual and the predicted measure divided by the actual measure and multiplied by 100 [60]. Furthermore, RMSE was calculated as the square root of the average of the squared differences between predicted and observed values [61].

Finally, to determine any significant differences between the predicted and measured VO₂max, we used statistical tests, specifically the paired 1-tailed t test. We tested the assumption of normally distributed data using the Anderson-Darling test (P=.65). Furthermore, we calculate the effect size using Hedges g, taking the sample size into account [62], with a value of 0.2 representing a small, 0.5 a medium, and 0.8 a large effect size [63].

In the first step, we analyzed the entire data set to assess the overall performance of the Apple Watch. Additionally, we aimed to get better insights regarding its performance across varying user fitness levels. To achieve this, participants were categorized into 3 groups based on their reference VO₂max obtained from the laboratory setting. Hence, based on the fitness categories outlined by the Fitness Registry and the Importance of Exercise National Database [64], participants were divided into poor, good, and excellent fitness levels, allowing us a more nuanced investigation of the Apple Watch’s estimations.

Out of the 30 (14 female and 16 male) initially recruited participants, 6 participants withdrew from the study before the first session due to health and personal reasons. Additionally, after the initial session, 4 participants were deemed ineligible for the study due to health concerns and recommendations from their respective health care providers, and 1 participant did not attend the second session due to personal reasons.

A total of 19 participants successfully completed the initial session in the laboratory setting, which involved a cycle test until subjective exhaustion and metabolic gas analysis, followed by the second session including an outdoor running test. Among the participants, 7 participants were female (mean age 28.86, SD 10.48 years; mean BMI 23.09, SD 2.31 kg/m²) and 12 participants were male (mean age 28.17, SD 12.40 years; mean BMI 23.76, SD 3.99 kg/m²). Participant characteristics are further detailed in Table 1.

The detailed results are presented in Table 2. The mean VO₂max determined in the laboratory setting was 45.88 (SD 9.42) mL/kg/minute, ranging from 32 to 64 mL/kg/minute. Furthermore, the mean estimated VO₂max from the Apple Watch was 41.37 (SD 6.50) mL/kg/minute, ranging from 29 to 52 mL/kg/minute. Our analysis revealed that the measured VO₂max is significantly higher than the predicted value from the Apple Watch (t₁₈=2.51; P=.01) with a medium effect size (Hedges g=0.53). These findings are consistent with observations from the Bland-Altman plot (Figure 1A), showing an overall underestimation of VO₂max by the Apple Watch. Specifically, the mean difference (bias) between the laboratory value and the estimated VO₂max value from the Apple Watch is –4.51 (SD 7.82) mL/kg/minute. Although all data points fall within the limits of agreement, indicating “good agreement” between the 2 methods, the ICC(2,1) of 0.47 (95% CI 0.06-0.75) suggests only poor to moderate reliability.

We furthermore analyzed the limit of agreement for participants with lower and higher fitness levels. When the participants were split into groups of poor (n=3), good (n=11), and excellent (n=5) fitness levels, the smartwatch showed a bias of mean 3.80 (SD 5.05) mL/kg/minute, mean –3.37 (SD 6.69) mL/kg/minute, and mean –12.00 (SD 4.98) mL/kg/minute, respectively. As depicted in Figures 1B-1D, the Apple Watch tends to overestimate VO₂max for participants with a poor fitness level while underestimating it for those with a higher fitness level. Moreover, the ICC for poor and excellent fitness levels was 0.14 and 0.23, respectively, indicating poor reliability. Only for participants with good (n=11) fitness levels, an ICC(2,1) of 0.60 indicates moderate reliability. However, it is important to highlight the limitations associated with interpreting the results for subgroups due to the small sample size.

The MAPE in the cohort of all participants (n=19) was 15.78%, with an RMSE of 8.85 mL/kg/minute. Upon dividing the VO₂max values into categories based on poor, good, and excellent fitness levels, the smartwatch showed MAPEs of 10.71%, 14.59%, and 21.47%, respectively. Regarding RMSE, the smartwatch showed values of 5.61, 7.22, and 12.80 mL/kg/minute for participants with poor, good, and excellent fitness levels, respectively. However, as already mentioned before, it is important to emphasize the limitation in interpreting results for subgroups due to the limited sample size.

The purpose of this study was to assess the accuracy of the VO₂max estimation of the Apple Watch Series 7. Other validation studies using the Apple Watch focused on evaluating the accuracy of measuring oxygen consumption reserve [41], HR [38,39], HR variability [40], or energy expenditure [37]. To the best of our knowledge, this is the first study validating the VO₂max using the Apple Watch, aside from Apple’s validation study [42].

Overall, our findings reveal a significant underestimation of the estimated VO₂max value from the Apple Watch (t₁₈=2.51; P=.01; bias: mean –4.51, SD 7.82 mL/kg/minute; Hedges g=0.53). These results deviate from the original validation study by Apple [42], which reported a smaller bias of mean 1.2 (SD 4.4) mL/kg/minute and mean 1.4 (SD 4.7) mL/kg/minute for the design and validation groups, respectively. However, it is important to acknowledge that our VO₂max value from the Apple Watch was obtained after only 1 outdoor walking and running session. According to Apple’s explanation, increasing the number of outdoor workouts enhances the accuracy of the VO₂max estimate [42]. In contrast to our study, Apple’s validation study was designed as a longitudinal study, extending over an average of 441 days for the design group and 390 days for the validation group. The researchers computed the mean and SD for differences between the last estimated VO₂max from the Apple Watch and the mean VO₂max value determined in up to 6 maximal or submaximal cardiopulmonary exercise tests while wearing the Apple Watch Series 4. However, it remains unclear how exactly the cardiopulmonary exercise test was conducted. Therefore, a direct comparison of our results with theirs is not feasible as they estimated VO₂max from multiple workouts. It is plausible that our results would show also a smaller error if the participants in our study wore the watch for a longer duration. Apple’s statement that the VO₂max estimation by the Apple Watch is accurate and reliable compared to conventional methods of VO₂max measurement [42] can therefore not be contradicted on the basis of the available findings.

Our findings regarding intraclass correlation reveal that ICC(2,1)=0.47, indicating relatively poor reliability, as outlined in reference [59]. Upon excluding participants with poor and excellent fitness levels and focusing solely on those with good fitness levels, we observed an improved ICC(2,1) value of 0.60, suggesting moderate reliability. These results underscore the influence of fitness levels on the reliability of VO₂max estimation through the Apple Watch. The validation study conducted by Apple calculating ICC(A,1), yielded values of 0.89 and 0.86 for the design and validation groups, respectively, indicative of good reliability [42]. Notably, Apple’s evaluation involved assessing absolute agreement per participant by comparing the last valid VO₂max estimate with the value estimated at least 28 days prior. This methodology differs from our approach, where we aimed to evaluate the reliability between laboratory-measured values and Apple Watch estimates without a significant time gap.

There is no standardized threshold for high or low MAPE, but we consider an error below 5% to be a good indicator for an accurate measurement. Regarding our results from the Apple Watch, we can conclude that regardless of the fitness level of the participants, the MAPE exceeded 10%. Unfortunately, related studies do not consistently report MAPE values. Nevertheless, 1 study using Polar [30] showed MAPE values above 10% (specifically 13.2%). In addition, studies with Fitbit devices showed MAPE around 10% [27,65]. Conversely, studies on Garmin devices [25,30,33,35], using algorithms developed by Firstbeat Technologies [15], consistently reported MAPE values well below 10%, highlighting their superior accuracy compared to other smartwatches.

We furthermore attempted to compare our results on ICC with those of other studies. Since not all studies provided comprehensive information regarding ICC forms used, making direct comparisons proved to be challenging. Nevertheless, studies on the Garmin Watch have indicated high reliability, with ICC(2,1)=0.87 [29] or ICC(3,1)=0.94 [35], although it is important to note that the latter study validated the estimation of VO₂ peak rather than VO₂max.

In terms of fitness levels, this study aligns with findings from related research using various smartwatches. Consistent with observations from references [30,31,33-35,65], our results suggest a tendency for the Apple Watch to overestimate VO₂max values among users with poor fitness levels (mean 3.80, SD 5.05 mL/kg/minute) and underestimate them among those with higher fitness levels (mean –3.37, SD 6.69 mL/kg/minute and mean –12.00, SD 4.98 mL/kg/minute for good and excellent fitness levels, respectively). However, it should be noted that this study involved a relatively small sample size, and classifying participants based on their fitness levels further reduced the sample size in each group (n=3 for participants with lower fitness, n=11 for those with good fitness, and n=5 for those with excellent fitness). Despite this limitation, our findings suggest that the Apple Watch may provide more accurate VO₂max estimates for users with poor or good fitness levels. This conclusion is further supported by MAPE, which shows a smaller error for users with poorer fitness levels while the error increases in participants with higher fitness levels (see also Table 2). This could be attributed to the potential influence of fitness levels on the accuracy of physiological measurements obtained through wearable devices. Nonetheless, further research with larger sample sizes is necessary to validate and elucidate these observations. Such investigations could shed light on the factors influencing the performance of wearable devices in estimating VO₂max across various fitness levels, thereby enhancing our understanding of their use in health and fitness monitoring.

The major limitation of this study is the small sample size. Although we aimed to recruit at least 30 participants, we ultimately obtained complete data from only 19 participants. To address this limitation, we reported effect sizes alongside our statistical tests, ensuring that our results remain reliable despite the smaller sample size. Nevertheless, further studies with larger and more varied populations are recommended to build on these findings and enhance the statistical power of the conclusions. It would also be beneficial to extend the duration during which participants consistently wear a smartwatch, as we believe that longer wear periods may enhance the accuracy of VO₂max estimation by the Apple Watch.

Although VO₂max measurement is considered the gold standard among sports medicine professionals for determining an individual’s fitness level, prior research has suggested that VO₂max is constrained by the variability in an individual’s effort and is highly reliant on VO₂max extent to which participants are properly motivated to achieve their true maximum [66]. Furthermore, as VO₂max criteria are not standardized, there is some uncertainty regarding whether the true VO₂max has actually been attained and if a maximum effort has been exerted [67]. To address these concerns, Edvardsen et al [68] proposed revised termination criteria for VO₂max tests that consider sex and age. Furthermore, as the true VO₂max value can differ, depending on whether the cardiopulmonary exercise testing was done on a treadmill or cycle ergometer, it would be important to use both tests independently to achieve optimal fitness assessment [69]. Nevertheless, varying termination criteria, testing methodologies, and participant populations across studies continue to pose challenges [67]. Despite these challenges, our aim involved making selective comparisons between our study and related research, diligently acknowledging the notable differences between the studies.

Another limitation we encountered was related to calibration error. Our attempt to compare the approximate prediction method of the Apple Watch with a gas analyzer was conducted using a graded exercise test until subjective exhaustion, potentially leading to an underestimation of the true VO₂max value. Noonan and Dean [70] outlined the advantages of submaximal exercise tests over maximal exercise tests, citing factors such as requirements for trained personnel and safety concerns. They conclude that submaximal exercise tests are reliable if an appropriate protocol is selected and the protocol is followed. However, it is crucial to note the potential influence of different protocols or increased participant motivation, as these factors could impact the measured VO₂max.

An additional limitation of our study is the lack of medical equipment. Ideally, we would have conducted periodic blood samples to measure the lactate threshold, allowing us to detect the point when the participant’s respiratory system attained its maximum capacity. The lactate concentration in blood is a valuable metric to monitor because an increase in blood lactate indicates a transition from aerobic to anaerobic exercise, suggesting that the body has surpassed its capacity for oxygen uptake to supply the muscles adequately [71]. Unfortunately, due to the unavailability of suitable equipment and the lack of medical professionals capable of carrying out such data collection, we were unable to include blood lactate as a termination criterion in our study. Additionally, it would have been ideal to monitor the volume of carbon dioxide produced; however, this capability is not provided by the VO2 Master Analyzer.

Overall, the Apple Watch Series 7 underestimated VO₂max compared to the values obtained using the gold standard assessment methods within a laboratory setting. This underestimation was even pronounced in participants with very high fitness levels. On the contrary, VO₂max values were overestimated by the Apple watch in participants with comparably low fitness levels. These findings highlight the importance of calibrating consumer-grade fitness trackers for greater accuracy across a diverse range of fitness levels. As consumer-grade technology continues to evolve, there is an opportunity for ongoing research and development to close the gap between the accuracy of portable devices and laboratory-grade equipment. This would not only enhance individual training and health monitoring but could also expand the use of such wearables in professional sports and clinical settings.