How are Mobility Lab's algorithms validated?

APDM’s algorithm verification and validation is one of our most important and focused efforts. The development process follows these general steps:

• Determine which metrics we want to provide to our customers. This is typically performed through customer interaction, extensive literature review, and input from our scientific fellow, Fay Horak.
• Research the best approach for estimating these metrics with our body worn sensors. The goal is to provide the most accurate metrics available through body worn inertial sensors. In some cases, as with our Sway metrics, algorithms had already been developed and validated that could be re-implemented in our system. In other cases, as with our Gait metrics, we used state of the art methods to implement our own algorithms.
• Verification that our algorithms work as they were designed to
  • Intermediate assertions are used extensively throughout our code. These result in run-time errors if certain logical tests aren’t satisfied.
  • Extensive visualization of intermediate processing steps. An internal verification report for a single Walk trial, for example, can exceed 100 pages, with extensive analysis and visualizations of all the processing steps. These are not intended for customers, but rather for our internal algorithm developers. They can provide insight into early or intermediate steps within the algorithm when developing or tuning.
  • A unit test framework. These exercise every algorithm using known input recordings to ensure that processing completes successfully and that all metric values are generated. Metric output files are compared against known, good values to ensure that no output values change due to unintended side effects of other development work.
  • A large and growing data set of recordings that cover a wide range of ages and levels of mobility. When changes are made to our algorithms, they are run against this growing body of recordings and any significant changes to the output are inspected. This is critical to ensure that the algorithms generalize to a wide variety of human subjects with varying mobility and that there are no unintended side effects resulting from modifications to the algorithm (improving one aspect of an algorithm may degrade its performance in other aspects).

• Validation that our algorithms are measuring what they are supposed to measure
  • Internal Validation: APDM goes through a rigorous process to validate our algorithms internally, before they are released to our customers. The general approach is to compare the results to the industry “gold standard” if such a standard exists. When one doesn’t exist, we have to determine an alternate form of validation. For example, the “gold standard” for stride length measurement is a gait mat, which accurately records exactly where your feet are placed while walking. For sway metrics, force plates are the “gold standard” for measuring a subject’s center of pressure. For range of motion metrics, optical motion capture is often the standard.
  • External Validation: This form of validation largely takes on two forms: 1) explicit 3rd party validation studies similar to our internal validation process and, 2) the use of our algorithms in published research. Due to APDMs established history in this industry, the Mobility Lab System is included in numerous unsolicited, 3rd party studies and publications. 

Below is a list of the metrics provided in Mobility Lab, along with the predominant form of validation used for the given metric. The metric units and definitions can be found here.

Internal Validation Notes:

[1] For these gait metrics, a gait mat and/or optical motion capture (MoCap) was used as a “gold standard”. The validation study included 22 subjects divided among healthy controls and three disease populations (7 healthy controls, 5 with Parkinson's, 5 with MS, 5 with stroke). These results were used internally and not yet published.

[2] For these sway metrics, the algorithm we adopted for use in Mobility Lab had been previously validated using a force plate. These results were published in the paper “ISway: a sensitive, valid and reliable measure of postural control”, by Martina Mancini (2012). We worked directly with the algorithm developer to ensure that we faithfully implemented the algorithm used in this paper.

[3] These angular sway metrics are a transformation of the acceleration based sway metrics reported in [2], with the goal of making them more understandable to our end users. A correlation analysis was performed to confirm the outputs are perfectly correlated, with only a scaling difference between the two. Mobility Lab provides both the acceleration and angle based sway metrics.

[4] The algorithm we adopted had been previously validated using a force plate. These results were published in the paper “Anticipatory Postural Adjustments Prior To Step Initiation Are Hypometric In Untreated Parkinson’s Disease: An Accelerometer-based Approach”, by Martina Mancini (2009). We worked directly with the algorithm developer to ensure that we faithfully implemented the algorithm used in this paper.

[5] For this validation, videos were recorded and manually annotated by experts. 18 recordings were used and each video was annotated by two independent experts to look for inter-rater variability and accuracy. These results were compared to the metrics that are automatically computed by Mobility Lab. These results were used internally and were not published.

[6] For the turn algorithm, we collected data with healthy and Parkinson’s subjects following a prescribed path with a variety of turn angles. We compared our results with both the prescribed protocol and optical motion capture data. We also collected data with simultaneous video and compared our analysis results with expert annotation of the events in the video. These results were published in the paper “Continuous Monitoring of Turning in Patients with Movement Disability”, by Mahmoud ElGohary (2014).

External Validation Notes:

[7] As previously indicated, there have been numerous unsolicited, 3rd party publications that further validate our algorithms. Due to the complexity of accurately quantifying gait with inertial sensors, we will highlight a single publication, titled “Validity and repeatability of inertial measurement units for measuring gait parameters”, by Edward P. Washabaugh (2017) that compared our system (using foot based inertial sensors) with the results from an instrumented treadmill and other standard clinical procedures. To quote the conclusion of this paper:

“The IMU system used in this study appears to be both accurate and repeatable for measuring spatiotemporal gait parameters in healthy young adults, particularly when using the foot configuration. This held true for both treadmill and overground conditions regardless of walking asymmetries. The ICC and MDC values observed for the system are also comparable to the existing gold standard gait evaluation techniques. These findings have meaningful implications for clinicians and researchers who use IMUs for evaluating and studying gait deficits.”