How are Mobility Lab's algorithms validated?

APDM’s algorithm verification and validation is one of our most important and focused efforts. The development process follows these general steps:

  • Determine which metrics we want to provide to our customers. This is typically performed through customer interaction, extensive literature review, and input from our scientific fellow, Fay Horak.
  • Research the best approach for estimating these metrics with our body worn sensors. The goal is to provide the most accurate metrics available through body worn inertial sensors. In some cases, as with our Sway metrics, algorithms had already been developed and validated that could be re-implemented in our system. In other cases, as with our Gait metrics, we used state of the art methods to implement our own algorithms.
  • Verification that our algorithms work as they were designed to
    • Intermediate assertions are used extensively throughout our code. These result in run-time errors if certain logical tests aren’t satisfied.
    • Extensive visualization of intermediate processing steps. An internal verification report for a single Walk trial, for example, can exceed 100 pages, with extensive analysis and visualizations of all the processing steps. These are not intended for customers, but rather for our internal algorithm developers. They can provide insight into early or intermediate steps within the algorithm when developing or tuning.
    • A unit test framework. These exercise every algorithm using known input recordings to ensure that processing completes successfully and that all metric values are generated. Metric output files are compared against known, good values to ensure that no output values change due to unintended side effects of other development work.
    • A large and growing data set of recordings that cover a wide range of ages and levels of mobility. When changes are made to our algorithms, they are run against this growing body of recordings and any significant changes to the output are inspected. This is critical to ensure that the algorithms generalize to a wide variety of human subjects with varying mobility and that there are no unintended side effects resulting from modifications to the algorithm (improving one aspect of an algorithm may degrade its performance in other aspects).
  • Validation that our algorithms are measuring what they are supposed to measure
    • Internal Validation: APDM goes through a rigorous process to validate our algorithms internally, before they are released to our customers. The general approach is to compare the results to the industry “gold standard” if such a standard exists. When one doesn’t exist, we have to determine an alternate form of validation. For example, the “gold standard” for stride length measurement is a gait mat, which accurately records exactly where your feet are placed while walking. For sway metrics, force plates are the “gold standard” for measuring a subject’s center of pressure. For range of motion metrics, optical motion capture is often the standard.
    • External Validation: This form of validation largely takes on two forms: 1) explicit 3rd party validation studies similar to our internal validation process and, 2) the use of our algorithms in published research. Due to APDMs established history in this industry, the Mobility Lab System is included in numerous unsolicited, 3rd party studies and publications. 

Below is a list of the metrics provided in Mobility Lab, along with the predominant form of validation used for the given metric. The metric units and definitions are included in a separate table at the end of this form.

Metric Class

Name

Validation Method

GAIT

   

Gait/Lower Limb

Cadence

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Gait Cycle Duration

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Gait Speed

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Double Support

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Terminal Double Support

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Elevation at Midswing

 

Gait/Lower Limb

Lateral Step Variability

GaitRite Mat,Optical MoCap[1]

Gait/Lower Limb

Lateral Swing Max

Optical MoCap[1]

Gait/Lower Limb

Pitch at Initial Contact

 

Gait/Lower Limb

Pitch at Toe Off

 

Gait/Lower Limb

Stance

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Step Duration

GaitRite Mat,Optical MoCap[1]

Gait/Lower Limb

Stride Length

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Swing

GaitRite Mat,Optical MoCap[1][7]

Gait/Lower Limb

Toe Out Angle

GaitRite Mat,Optical MoCap[1]

Gait/Lower Limb

Initial Contact

GaitRite Mat,Optical MoCap[1]

Gait/Lumbar

Coronal Range of Motion

 

Gait/Lumbar

Sagittal Range of Motion

 

Gait/Lumbar

Transverse Range of Motion

 

Gait/Trunk

Coronal Range of Motion

 

Gait/Trunk

Sagittal Range of Motion

 

Gait/Trunk

Transverse Range of Motion

 

Gait/Upper Limb

Maximum Velocity

Optical MoCap[1]

Gait/Upper Limb

Range of Motion

Optical MoCap[1]

SWAY

   

Postural Sway/Acc

95% Ellipse Sway Area

Force Plate[2]

Postural Sway/Acc

RMS Sway

Force Plate[2]

Postural Sway/Acc

RMS Sway (Coronal)

Force Plate[2]

Postural Sway/Acc

RMS Sway (Sagittal)

Force Plate[2]

Postural Sway/Acc

Centroidal Frequency

Force Plate[2]

Postural Sway/Acc

Centroidal Frequency (Coronal)

Force Plate[2]

Postural Sway/Acc

Centroidal Frequency (Sagittal)

Force Plate[2]

Postural Sway/Acc

Frequency Dispersion

Force Plate[2]

Postural Sway/Acc

Frequency Dispersion (Coronal)

Force Plate[2]

Postural Sway/Acc

Frequency Dispersion (Sagittal)

Force Plate[2]

Postural Sway/Acc

Jerk

Force Plate[2]

Postural Sway/Acc

Jerk (Coronal)

Force Plate[2]

Postural Sway/Acc

Jerk (Sagittal)

Force Plate[2]

Postural Sway/Acc

Mean Velocity

Force Plate[2]

Postural Sway/Acc

Mean Velocity (Coronal)

Force Plate[2]

Postural Sway/Acc

Mean Velocity (Sagittal)

Force Plate[2]

Postural Sway/Acc

Path Length

Force Plate[2]

Postural Sway/Acc

Path Length (Coronal)

Force Plate[2]

Postural Sway/Acc

Path Length (Sagittal)

Force Plate[2]

Postural Sway/Acc

Range

Force Plate[2]

Postural Sway/Acc

Range (Coronal)

Force Plate[2]

Postural Sway/Acc

Range (Sagittal)

Force Plate[2]

Postural Sway/Angles

Duration

Force Plate[2]

Postural Sway/Angles

95% Ellipse Sway Area

Correlation Analysis[3]

Postural Sway/Angles

RMS Sway

Correlation Analysis[3]

Postural Sway/Angles

RMS Sway (Coronal)

Correlation Analysis[3]

Postural Sway/Angles

RMS Sway (Sagittal)

Correlation Analysis[3]

POSTURAL TRANSITION

   

Anticipatory Postural Adjustment

APA Duration

Force Plate[4]

Anticipatory Postural Adjustment

First Step Duration

Force Plate[4]

Anticipatory Postural Adjustment

First Step Range of Motion

Force Plate[4]

Anticipatory Postural Adjustment

Maximum AP Acceleration

Force Plate[4]

Anticipatory Postural Adjustment

Maximum ML Acceleration

Force Plate[4]

Sit to Stand

Duration

Video annotation[5]

Sit to Stand

Start

Video annotation[5]

Stand to Sit

Duration

Video annotation[5]

Stand to Sit

Start

Video annotation[5]

Turns

Angle

Optical MoCap[6]

Turns

Duration

Optical MoCap[6]

Turns

Peak Velocity

Optical MoCap[6]

Turns

Start

Optical MoCap[6]

Turns

Steps

Optical MoCap[6]

 

Internal Validation Notes:

[1] For these gait metrics, a gait mat and/or optical motion capture (MoCap) was used as a “gold standard”. The validation study included 22 subjects divided among healthy controls and three disease populations (7 healthy controls, 5 with Parkinson's, 5 with MS, 5 with stroke). These results were used internally and not yet published.

[2] For these sway metrics, the algorithm we adopted for use in Mobility Lab had been previously validated using a force plate. These results were published in the paper “ISway: a sensitive, valid and reliable measure of postural control”, by Martina Mancini (2012). We worked directly with the algorithm developer to ensure that we faithfully implemented the algorithm used in this paper.

[3] These angular sway metrics are a transformation of the acceleration based sway metrics reported in [2], with the goal of making them more understandable to our end users. A correlation analysis was performed to confirm the outputs are perfectly correlated, with only a scaling difference between the two. Mobility Lab provides both the acceleration and angle based sway metrics.

[4] The algorithm we adopted had been previously validated using a force plate. These results were published in the paper “Anticipatory Postural Adjustments Prior To Step Initiation Are Hypometric In Untreated Parkinson’s Disease: An Accelerometer-based Approach”, by Martina Mancini (2009). We worked directly with the algorithm developer to ensure that we faithfully implemented the algorithm used in this paper.

[5] For this validation, videos were recorded and manually annotated by experts. 18 recordings were used and each video was annotated by two independent experts to look for inter-rater variability and accuracy. These results were compared to the metrics that are automatically computed by Mobility Lab. These results were used internally and were not published.

[6] For the turn algorithm, we collected data with healthy and Parkinson’s subjects following a prescribed path with a variety of turn angles. We compared our results with both the prescribed protocol and optical motion capture data. We also collected data with simultaneous video and compared our analysis results with expert annotation of the events in the video. These results were published in the paper “Continuous Monitoring of Turning in Patients with Movement Disability”, by Mahmoud ElGohary (2014).

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.