Dispelling Four Challenges in Inertial Motion Tracking with One Recurrent Inertial Graph-based Estimator (RING)
Simon Bachhuber Ive Weygers Thomas Seel



 

First]S. Bachhuber 
First]I. Weygers 
Second]T. Seel

[First]Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg, 91052 Erlangen, Germany (e-mail: first.last@fau.de)
[Second]Institute of Mechatronic Systems, Leibniz Universität Hannover, 30167 Hannover, Germany (e-mail: first.last@imes.uni-hannover.de)

§ ABSTRACT                 
In this paper, we extend the Recurrent Inertial Graph-based Estimator (RING), a novel neural-network-based solution for Inertial Motion Tracking (IMT), to generalize across a large range of sampling rates, and we demonstrate that it can overcome four real-world challenges: inhomogeneous magnetic fields, sensor-to-segment misalignment, sparse sensor setups, and nonrigid sensor attachment. RING can estimate the rotational state of a three-segment kinematic chain with double hinge joints from inertial data, and achieves an experimental mean-absolute-(tracking)-error of 8.10± 1.19 degrees if all four challenges are present simultaneously.
The network is trained on simulated data yet evaluated on experimental data, highlighting its remarkable ability to zero-shot generalize from simulation to experiment.
We conduct an ablation study to analyze the impact of each of the four challenges on RING's performance, we showcase its robustness to varying sampling rates, and we demonstrate that RING is capable of real-time operation.
This research not only advances IMT technology by making it more accessible and versatile but also enhances its potential for new application domains including non-expert use of sparse IMT with nonrigid sensor attachments in unconstrained environments.



Recurrent Neural Networks, Inertial Measurement Units, Orientation Estimation, Sparse Sensing, Magnetometer-free, Sensor-to-Segment Alignment






§ INTRODUCTION

Numerous recent developments in biomedical engineering applications require precise estimation of the motion of articulated bodies in space. Some prominent examples include unobtrusive human motion tracking outside the lab <cit.>, and realizing intelligent symbiosis between humans and robots that enter immersive environments <cit.>. Inertial Measurement Units (IMUs) are used in all these systems because of their unique ability to track movements of articulating rigid bodies of Kinematic Chains (KCs), in a cheaper and more reliable way than State-Of-The-Art (SOTA) multi-camera systems that require continuous line of sight.

All IMU-based motion tracking use cases heavily rely on Inertial Motion Tracking (IMT) algorithms that fuse different measurement signals to estimate motion parameters. This, however, is inherently limited by the following four key challenges <cit.>: 

(1) Inhomogeneous magnetic fields indoors and near ferromagnetic materials or electric devices;

(2) Sensor-to-segment alignment that involves determining joint positions and axis directions in local sensor coordinates;

(3) Solving sparse problems where some segments of the KC are not equipped with a sensor;

(4) Addressing real-world disturbances due to nonrigid sensor attachment and caused by large acceleration signals from impacts and soft tissue artifacts.


In recent years, many highly specialized methods have been proposed to address these challenges. We will provide a brief overview of the latest and most notable developments accompanied by recent comprehensive methodological overviews. First, a multitude of different kinematic constraints are proposed to replace missing magnetometer heading information, as reviewed in <cit.>. Furthermore, recent general-purpose <cit.> magnetometer-free attitude estimators by <cit.> achieved remarkable accuracy improvements in comparison with, e.g., the widely used filters from <cit.> and <cit.>. Second, several algorithms have been developed to achieve automatic sensor-to-segment alignment, as outlined by <cit.> for specific joints with full sensor setups <cit.>.
Third, a recent trend in sparse sensor setups is visible with methods that either use a limited number of sensors but include magnetometer measurements <cit.> or are magnetometer-free <cit.>. Finally, the literature on IMT methods to overcome real-world disturbances is limited and focuses on late interception by outlier rejection techniques (<cit.>) or further advances in connection constraints <cit.>.

Real-world IMT applications typically present multifaceted challenges, requiring data-driven state observers like the Recurrent Neural Network-based Observer (RNNO) (<cit.>) that can effectively address the increasing complexity. To overcome a redundant implementation task in retraining RNNOs for every combination in a large grid of IMT challenges, we proposed the Recurrent Inertial Graph-based Estimator (RING) (<cit.>) as a pluripotent approach that solves IMT Problems (IMTPs) of tree-structured systems.

Despite RING's ability to provide a solution to a variety IMT challenges, its real-world applicability for a combination of all the aforementioned IMT challenges has not been investigated and their individual impact are unknown. Furthermore, while RING is aimed to be applicable in a plug-and-play fashion, it still requires a specific fixed sampling rate, which vastly limits its applicability in practice. Moreover, the real-time capability in inference has not been explored. In this work, we enhance and validate RING's usability with the following contributions:


    
  * Extending RING's usability by enabling applicability to data from a broad range of sampling rates.
	
    
  * Solve for the first time the four IMT challenges at once.
	
    
  * Show zero-shot experimental generalizability in an extensive ablation study to gain insights on the performance of RING on individual IMT challenges. 
	
    
  * Analyze the real-time capability of RING.





§ PROBLEM FORMULATION

Consider a KC with three segments that are connected by hinge joints with arbitrary and unknown joint axes directions.
Only the outer bodies are equipped with nonrigidly attached IMUs.
A KC is a rigid-body system and it consists of multiple rigid objects (segments) that are rigidly attached to coordinate systems (bodies).
In general, the topology of such a rigid-body system can be represented by a Connectivity Graph (CG) <cit.> where nodes represent bodies and edges represent degrees of freedom in the system.
Here, for each segment there is one body with one segment attached to it, such that there is a one-to-one correspondence between segments and bodies.
After the N bodies have been numbered, the CG can be encoded via a parent array λN where λi is the body number of the parent of body i.
For a three-segment KC, this graph representation and the one parent array utilized in this work is shown in Figure <ref>.

In this work, the goal is to estimate the complete rotational state of the KC up to a global heading offset <cit.>. We approach this through a filtering problem formulation,  where an estimate of the complete rotational state x(t) is obtained at every time instant t from the current and all previous IMU measurements y(t' ≤ t) that are combined into one measurement signal y(t) ∈ℝ^12 defined as

    y(t) = (
        ω_1(t)^⊺, 
        ρ_1(t)^⊺, 
        ω_3(t)^⊺, ρ_3(t)^⊺)^⊺  ∀ t

where ω_i(t), ρ_i(t) denote gyroscope and accelerometer measurements at time t of the IMU that is nonrigidly attached to body 𝒮_i ∈{1, 3}.
The rotational state x(t) ∈ℍ^3 of the KC is straightforwardly defined by

    x(t) = (10(t)^⊺, 21(t)^⊺, 32(t)^⊺)^⊺  ∀ t

where ij(t) denotes the orientation from body 𝒮_i to body 𝒮_j at time t and where body 0 denotes the earth frame.
Note that the 10 can only be estimated up to a heading offset from 6D measurements.

Real-world applicability requires solving all of the following challenges of the IMTP that is said to

    
  * be magnetometer-free (or 6D in contrast to 9D) if the IMUs measure only three-dimensional angular rates and specific forces, and not provide magnetometer readings.
    
  * require sensor-to-segment alignment when hinge joint axes directions are unknown.
    
  * be sparse if not every segment that constitute the KC has an IMU attached. Here,the middle segment does not have an IMU attached. An IMTP that has an IMU attached to each body is said to have a full IMU setup.
    
  * suffer from motion artifacts if the IMUs are not rigidly attached to the respective bodies, such that there can occur transnational and rotational motions between the segment and IMU. An IMTP without motion artifacts assumes that there cannot exist any relative motion between segment and IMU.

From this, it follows that the IMTP considered here is magnetometer-free, requires sensor-to-segment alignment, sparse, and suffers from motion artifacts.



§ METHODS

In this work, we extend the Neural Network-based (NN-based) multiple-IMU sensor fusion algorithm from <cit.>.
We address all four key IMT challenges outlined in Section <ref>, while enabling sampling rate robustness and showcase that RING is real-time capable. This is achieved by both adapting the training procedure (Section: <ref>) and the NN-based multiple-IMU sensor fusion algorithm (Section: <ref>).




 §.§ Simulated Training Data at Various Sampling Rates

RING <cit.> is trained on large amounts of simulated input-output data at various sampling rates. The procedure that generates the data for the training of RING is called the Random Chain Motion Generator (RCMG) <cit.>.
It generates extensively augmented random motions of KCs with:

    
  * different number of segments,
    
  * randomized segment lengths,
    
  * randomized IMU placement,
    
  * randomized joint axes directions, and
    
  * rigidly or nonrigidly attached IMUs (by simulating spring-damper-systems with randomized damping and stiffness parameters) <cit.>.

From these random KC motions we compute IMU and orientation measurements, but in this work at various sampling rates. These form the input-output pairs for training RING.

RCMG can be summarized as a function that only from PseudoRNG returns the training pair

    
  * XTN10, where Xi6 is the 6D IMU data for body i (if it is not dropped out), and where Xi69 is the joint axis direction of the hinge joint between body i and its parent (if it is not dropped out, and if the parent is not the base), and where X[, i, 10] is the inverse sampling rate 1/F, and
    
  * YTN where Yi is the orientation from body i to its parent λ[i], and

where T is the number of timesteps, and N is the number of bodies in the KC (here N=3).

To achieve a wide coverage, training data is generated for sampling rates drawn from 

    F ∈{40, 60, 80, 100, 120, 140, 160, 180, 200}
 
and to allow for efficient training data batching, the sequence duration is adjusted based on the sampling rate to achieve a common number of timesteps of T=6000. A training batch is then built up by stacking 512 sequences, and additional details regarding the RCMG can be found in <cit.>.



 §.§ Neural Network Architecture: RING with Sampling Rate Input

We use a NN trained on data generated using the procedure described in Section <ref>.
The network architecture is based on RING <cit.>, a powerful multiple-IMU sensor fusion algorithm that is composed of a decentralized network of message-passing Recurrent NNs (RNNs). Most notably, RING's parameters are defined on a per-node level and shared across all nodes in the graph.
This design enables RING to be applied to broad range of IMTPs with a single set of parameters and enables its exceptional pluripotency.

In this work, the architecture of RING is extended to additionally accept a sampling rate input, such that the dimensionality of RING's per-timestep and per-node input increases by one.
To summarize, RING can be viewed as the following step function that maps the previous state of RING ξ_t-1N2H and network input X_t N10 at time t to the next RING state ξ_t and output Ŷ_t ∈ℍ^N, i.e.,

    ξ_t, Ŷ_t = ring(ξ_t-1, X_t, λ)
      ∀ t

where H is the hidden state dimensionality, ξ_0 = 0, and where the vector λ is the parent array, defined in Section <ref>.
Internally, RING has the parameters of

    
  * the Message-MLP-network f_θ: ℝ^H →ℝ^M, and
    
  * the Stacked-GRUCell-network g_θ: ℝ^2H×ℝ^2M+10→ℝ^2H which consists of the sequence of Gated-Recurrent-Unit(GRU)Cell, LayerNorm, GRUCell <cit.>, and
    
  * the Quaternion-MLP that combines a Layernorm, and a MLP-network h_θ: ℝ^H →ℝ^4, and

where M is the dimensionality of the messages that are passed along the edges of the graph.
Note that the two GRUCells each have a hidden state dimensionality of H, thus the hidden state of RING is of dimensionality of 2H.
Then, equation (<ref>) consists of several consecutive steps, for all N bodies:

    
  * Messages M_t NM are computed using f_θ.
    
  * Messages are passed along the edges of the graph.
    
  * The hidden state is updated using g_θ.
    
  * The unnormalized output ỸN4 is computed using the Quaternion-MLP.
    
  * The output is normalized to allow interpretation as unit quaternions. The final RING output is one unit quaternion per node Ŷ_t ∈ℍ^N.


RING is trained by comparing Ŷ_t to the ground truth Y_t which is provided by the RCMG, and by minimizing the mean-squared orientation error.

Additional details regarding the the RING architecture and its optimization strategy can be found and is exactly the same as in <cit.>.

A software implementation of the RCMG and RING, and the source code for creating the results presented in Section <ref> are hosted on GitHub[<https://github.com/SimiPixel/ring>].



§ RESULTS AND DISCUSSION

In this section, we evaluate the accuracy of RING's orientation estimation, trained in simulation only, when applied to experimental data (see Section <ref>) of the problem specified in Section <ref>.
The performance of RING is compared to (SOTA) methods (see Section <ref>).


It is remarkable, that RING can solve an experimental IMTP that combines all four challenges in IMT (magnetometer-free, unknown sensor-to-segment alignment, sparse sensor setup, and nonrigid sensor attachment) simultaneously, despite being trained on simulated data only.






 §.§ Experimental Setup and Data Acquisition

We utilize a five-segment KC to record the experimental data but only the data for the given IMTP relevant parts of the KC are used for evaluation. The five-segment KC includes a singular spherical joint followed by three hinge joints, each oriented along the x, y, and z axes, respectively. Each segment of the KC was equipped with two IMUs: one rigidly attached to the segment and another nonrigidly attached using foam padding, as depicted in Figure <ref>.

Two distinct trials were conducted, involving random movements of the five-segment KC. Here, two three-segment KCs are thus extracted, one with joint axes direction x and y, and one with y and z.
During evaluation, the first trial spans a duration of 66 and features a diverse range of motions.
Additionally, the second trial, with a length of 68, includes periods where the entire KC remained stationary.
Overall, this results in four trials in total.

We refer to <cit.> for additional details regarding the experimental setup and preprocessing.







 §.§ Evaluation Metrics and Baselines

The ground truth orientations for the experimental trials (see Section <ref>) were recorded using optical motion capture.
Orientation estimation accuracy is quantified using the Mean-Absolute-(tracking)-Error (MAE) in degrees.
Here, the mean calculation reduces the dimensions of the different trials, time, and three orientations (including inclination and two relative orientations).
In the time dimension, initial 5 of each trial were deliberately excluded from the MAE calculations. This decision was made to ensure that the recorded errors accurately reflected the system's performance post-convergence.

To the best of the authors' knowledge, there exists no alternative method that can be applied to the IMTP as described in Section <ref>.
However, two SOTA baseline methods can be identified after simplifying the IMTP so that it does not contain all four challenges simultaneously.
The first baseline is obtained by using conventional IMT methods, that is, using a full 9D IMU setup and tracking each segment independently.
The SOTA method for such single-IMU sensor fusion is VQF <cit.>.
The second baseline is obtained after eliminating the challenge of anatomical calibration.
Under the assumption of known joint axes direction, RNNO can be applied <cit.>.
Note that since two KCs with different directions of the joint axes are used for experimental validation (see Section <ref>), two trained RNNO networks are required.
To compensate for the violation of the rigid-IMU-attachment assumption, both baselines additionally utilize a low-pass filter.
The cutoff frequency was grid searched and we report only the best result for each baseline method.



 §.§ Experimental Validation of RING

The trained RING is applied to experimental data from an IMTP that combines the four challenges of nonrigid IMU attachment, misaligned sensors and segments, magnetometer-free measurements, and a sparse sensor setup.
The MAE in the orientation estimate for RING and the two SOTA baseline methods are reported in Table <ref> and confirms that RING outperforms both alternative methods despite solving the more challenging IMTP.
The first 15 seconds of one example sequence are shown in Figure <ref> and demonstrate RING's prediction performance and quick convergence.

In Table <ref>, we conduct an ablation study to analyze the impact of nonrigid IMU attachment, sensor-to-segment alignment, and sparse IMU setup on RING's orientation estimation accuracy.

In Figure <ref>, the experimental data is resampled to a wide range of sampling rates to assess the robustness of RING w.r.t. the sampling rate. RING achieves a nearly constant orientation estimation accuracy which only, unsurprisingly, degrades slightly for low sampling rates.







 §.§ Real-time Applicability of RING

By design, RING can be applied online as it is defined by a step function (see eq. (<ref>)) that, based on the measurements at a certain timestep (see eq. (<ref>)), returns an updated internal state and the rotational state estimate (see eq. <ref>) of the KC.
Therefore, if the step function executes faster than the sampling rate requires, then it is said to be real-time capable.
After compilation, the runtime of the step function of RING is (794±16.7) s on a M2 Macbook Pro.
Thus, RING is real-time capable up to ≈1000.



§ CONCLUSION

In this work, we have extended RING, a powerful IMT method, to generalize across a wide range of sampling rates, and we have showcased that it can simultaneously overcome the four key challenges in IMT: inhomogeneous magnetic fields, sensor-to-segment misalignment, sparse sensor setups, and nonrigid sensor attachment.
With an experimental tracking MAE of 8.10± 1.19 degrees if all four challenges are present simultaneously, RING accurately estimates the rotational state of a three-segment KC from IMU measurements.
RING leverages a decentralized network of message-passing RNNs that is trained on simulated data but is capable of zero-shot generalization to real-world data.
Our evaluations reveal RING's superiority over SOTA methods in terms of accuracy and applicability, additionally, we demonstrate RING's robustness across various sampling rates, and its real-time capability.
By enabling plug-and-play usability and extending the applicability of inertial motion capture technology, RING not only advances the field but also opens new avenues for research and practical applications in environments previously deemed challenging.


The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The hardware is funded by the German Research Foundation (DFG).




