We compared five separate training approaches: four subsets of MediaPipe hand landmarks fed into an LSTM and fine-tuned MobileNetV2 convolutional features fed into an LSTM. The hand landmark approaches provided an exploration of activity detection on privacy-preserved feature representations. Because we strived to use machine learning classifiers in low-resource settings such as mobile devices, we additionally aimed to make our models and feature representations as light as possible.

We used the SSBD [49] for training and testing of our models. To the best of our knowledge, SSBD is the only publicly available data set of self-stimulatory behaviors containing examples of head banging, hand flapping, and spinning. SSBD includes the URLs of 75 YouTube videos, and for each video, annotations of the time periods (eg, second 1 to second 35) when each self-stimulatory behavior was performed. Multiple videos contain multiple time periods for the same behavior (eg, seconds 1-3 and 5-9 both contain hand flapping) as well as multiple behaviors (eg, seconds 1-3 show head banging and seconds 5-9 show hand flapping). We only used the hand flapping annotations.

To obtain control videos absent of hand flapping displays, we first downloaded all YouTube videos in SSBD that contained sections of hand flapping. Each section in a video exhibiting hand flapping was extracted to create a new clip. The parts of the video without hand flapping (ie, with no annotations) were isolated to create control clips. This data curation process is illustrated in Figure 1.

After extracting all positive and control clips from the downloaded videos, we aimed to maximize the amount of training data in each class. Because a hand flapping event occurs within a couple of seconds, we split any clips longer than 2 seconds into smaller clips. We manually deleted any videos that were qualitatively shaky or of low quality. In total, we extracted 50 video clips displaying hand flapping and 50 control videos.

We evaluated five separate feature extraction methods. For four of them, we used the numerical coordinates of the detected hand landmarks concatenated into a 1-dimensional vector as the primary feature representation. For the remaining model, we fine-tuned a mobile-optimized CNN, MobileNetV2 [51], to learn features derived from raw image sequences. We noted that the landmark-based feature representations are privacy-preserved, as they do not require the face of the participant to be shown in the given data for adequate classification.

To extract the hand coordinates, we used MediaPipe, a framework hosted by Google that detects the landmarks on a person’s face, hands, and body [52]. MediaPipe’s hand landmark detection model provides the (x, y, z) coordinates of each of the 21 landmarks it detects on each hand. The x coordinate and y coordinate describe how far the landmark is on the horizontal and vertical dimensions, respectively. The z coordinate provides an estimation of how far the landmark is from the camera. When MediaPipe does not detect a landmark, the (x, y, z) coordinates are all set to 0 for that landmark.

The first landmark-based feature representation approach we tried used all 21 landmarks on each hand provided by MediaPipe to create the location vector fed into the LSTM. SSBD’s videos mostly contain children whose detected hand landmarks are closer together due to smaller hands. This could be a problem when generalizing to older individuals with wider gaps between hand landmarks. To help the model generalize beyond hand shape, one possible solution is to use a curated subset of landmarks.

To eliminate hand shape all together, one could use only one landmark. We tried this method by using a single landmark at the base of the hand. However, because the videos in SSBD may be shaky, reliance on MediaPipe being able to detect this landmark may have led to empty features for some frames. One way to circumvent this problem is to take the mean of all the (x, y, z) coordinates of detected landmarks and use the average coordinate for each hand. We call this method the “mean landmark” approach.

We took the first 90 frames of a video and for each frame, we concatenated the feature vectors and used them as input for each timestep of an LSTM model (Figure 2). We experimented with subsets of landmarks provided by MediaPipe; we tried using all 21 landmarks, 6 landmarks (5 at each fingertip and 1 at the base of the hand), and with single landmarks. We note that the concatenated coordinates of landmarks will always form a vector that is 6 times larger than the number of landmarks used because there are 3 coordinates for a single landmark and 2 hands for which each landmark can be detected.

The neural network architecture we used for all experiments consisted of an LSTM layer with a 64-dimensional output. The output of the LSTM was passed into a fully connected layer with sigmoid activation to obtain a binary prediction. To minimize overfitting, we also inserted a dropout layer between the LSTM and the dense layer with a dropout rate of 30%. The landmark-based models contained nearly 3 million parameters. (Table 1). We note that the number of parameters depends on the feature approach; Table 1 shows the number of parameters based on our heaviest feature approach of using all 21 landmarks.

We experimented with other model architectures before selecting this model. We found that adding more than one LSTM or fully connected layer did not cause any notable difference in performance; thus, we removed these layers to minimize the model’s capacity for overfitting. We also experimented with the output dimensionality of the LSTM; we tried 8, 16, 32, and 64. We found that using 32 and 64 performed similarly, with 64 usually performing slightly better.

We trained all models with binary cross-entropy loss using Adam optimization [53]. We tried learning rates of 0.0005, 0.0001, 0.0005, 0.001, and 0.1, and found that in almost all cases 0.01 worked best. All models and augmentations were written using Keras [54] with a TensorFlow [55] back end run on Jupyter. No GPUs or specialized hardware were required due to the low-dimensional feature representation, and training a single model took a few minutes on a CPU with 32GB of RAM.

For all models, we trained the model until there was consistent convergence for 10 or more epochs. This resulted in 75 epochs of training across all models. After training, we reverted the model’s weights to its weights for which it performed best. We used this strategy for all feature approaches.

We used 5-fold cross validation to evaluate each model’s average accuracy, precision, recall, and F1 score across all folds for training and testing. However, because of our small data set, the particular arrangement of the videos in each fold substantially affected the model’s performance. To minimize this effect, we ran the 5-fold cross-validation procedure 100 times, each with a different random seed, resulting in a total of 500 distinct folds. We further ensured that each fold was completely balanced in both the training and testing set (50% head banging and 50% not head banging). In all folds, there were 10 videos displaying hand flapping and 10 videos displaying head banging.

We report the mean and SD of each metric across all 500 folds as well as the area under receiver operating characteristics (AUROC). For all feature approaches, we also show the average receiver operating characteristics (ROC) curve across all folds.

This approach used all 21 landmarks on both hands for a total of 42 unique landmarks. We show the results of this approach in Table 2. In Figure 3, we show the ROC curves of the model with and without augmentations.

When using all the landmarks, we used graphical interpolation to fill in the coordinates of missing landmarks to help reduce the effects of camera instability. However, when we tried this, we found that it often decreased accuracy and resulted in higher SDs. We therefore decided to discontinue using interpolation when evaluating the approaches described in the next section. We conjecture that the inability of MediaPipe to detect hand key points could be a salient feature for hand flapping detection, and this feature becomes obfuscated once key points are interpolated.

Here, we describe the mean and one landmark approaches, both of which relied on a single landmark on each hand as the feature representation. We show the results of both approaches, with and without augmentations, in Table 3. In Figure 4, we show the average ROC curve for both approaches.

We used the six landmarks on the edges of the hands to create the location frames. We achieved an F1 score and classification accuracy of about 72.3% (Table 4). We also achieved an AUROC of 0.76 (Figure 5).

Of all of the landmark-based approaches, the six landmarks approach yielded optimal results. All of the validation metrics were higher with this approach than those previously discussed.

In the approaches discussed so far, MediaPipe was consistently used as a feature extractor to bring each video frame into a lower-dimensional vector representation. Here, we substituted the MediaPipe feature extractor with MobileNetV2’s [51] convolutional layers (pretrained on ImageNet [56] and fine-tuned on SSBD) as a feature extractor. As with the landmark-based approaches, this extracted vector was fed into an LSTM network to obtain a prediction for whether hand flapping was present in the video. We evaluated this model on the same 100 data sets (500 total folds), as we used for all other approaches. The ROC curve of this model is shown in Figure 6, and the metrics are detailed in Table 5.

The MobileNetV2 model achieved an accuracy and F1 score both around 85%, surpassing the performance of all the landmark-based approaches. The MobileNetV2 models also had a higher capacity to overfit, achieving near perfect accuracies in training (>99.999%), whereas all landmark-based approaches never surpassed 90% for any of the training metrics. We conjecture that this is because the MobileNet V2 model has learned both the feature extraction and discriminative steps of the supervised learning process.

We conducted a 2-sided t test to determine whether the differences we observed for each approach (including the MobileNetV2 method) were statistically significant. We applied Bonferroni correction across the comparisons, deeming a P value <.005 as statistically significant. We show the P values from comparing all the approaches with each other on the 4 aforementioned metrics in Table 6.

Most of the comparisons between approaches were statistically significant after Bonferroni correction. The two single landmark approaches (mean and one landmark) were not statistically significant for any of the metrics.

We explored several feature representations for lightweight hand flapping classifiers that achieved respectable performance on the SSBD. The highest-performing model used MobileNetV2 to extract features and achieved a test F1 score of 84 (SD 3.7). A model trained with all hand landmarks reached an F1 score of 66.6 (SD 3.35). A model trained with a select 6 landmarks reached an F1 score of 68.3 (SD 3.6). A model trained using a single landmark at the base of the hands reached an F1 score of 64.9 (SD 6.5).

One point of interest in this study is the trade-off between privacy-preserved solutions and performance in diagnostic machine learning tasks. While the MobileNetV2 model outperformed all the MediaPipe classifiers, the MobileNetV2 model lacks the capability to preserve the privacy of the participants, as the participants’ faces were ultimately used in the data needed for classification. We expect this to be a difficulty for future research in the behavioral diagnostic space.

The primary limitation of this approach is that without further class labels across a variety of hand-related activities and data sets, there is a probable lack of specificity in this model when generalizing to other data sets beyond the SSBD. Hands can move but not display hand flapping or self-stimulatory movement. Furthermore, stereotypic use of hands may occur in the absence of a formal autism diagnosis. Multi-class models that can distinguish hand movement patterns are required for this degree of precision. Such models cannot be built without corresponding labeled data sets, and we therefore highlight the need for the curation of data sets displaying behaviors related to developmental health care.

For this study to truly generalize, further validation is required on data sets beyond the SSBD. While the SSBD was curated with autism diagnosis in mind, the paper describing the original data set does not necessarily include children with confirmed autism diagnoses. Existing mobile therapies that collect structured videos of children with autism [16-18,40] can be used to acquire data sets to train more advanced models, and these updated models can be integrated back into the digital therapy to provide real-time feedback and adaptive experiences.

There are myriad challenges and opportunities for computer vision recognition of complex social human behaviors [57], including socially motivated hand mannerisms. Additional prospects for future work include alternative feature representation and incorporation of modern architectures such as transformers and other attention-based models.

The hand movement classifier we describe here is one of a potential cocktail of classifiers that could be used in conjunction not only to extract features relevant to an autism diagnosis but also to provide insight into which particular symptoms of autism a child is exhibiting. The primary benefit of this approach is for greater explainability in medical diagnoses and a strive toward specificity in automated diagnostic efforts.