The participant was a 30-year-old female with congenital bilateral limb absence. Both limbs were affected at the wrist disarticulation level. The participant reported use of single degree of freedom, direct control myoelectric prostheses with both limbs for 27 years, as well as right-hand dominance. The study was reviewed and approved by the Institutional Review Board at George Mason University. The participant provided written informed consent to participate in this study, and to include her data and identifiers in publications about the study.
The participant was fitted with thermoplastic test sockets on both residual limbs using supracondylar suspension. The sockets weighed 223 g. A TASKA Hand and Quick Disconnect Wrist (TASKA Prosthetics, Christchurch, New Zealand) and MC Standard Wrist Rotator (Motion Control Inc., Salt Lake City, UT) were mounted to the socket. The left hand was 19.3 cm long and weighed 567 g. The right hand was 20.4 cm long and weighed 680 g. The wrist rotator was 7 cm long and weighed 149 g. The batteries and on/off switch weighed 105 g. The resulting prosthesis was extremely long relative to the participant's height (149.86 cm) and residual limb length (20.5 cm for the right limb and 19.3 cm for the left limb). It was also considerably heavier than the participant's clinically-prescribed myoelectric prostheses, with much of the weight located distally due to the size of the TASKA Hand (Figure 1). The participant reported that the socket alone was well-fitted over the limb, but when the components were attached to the distal end of the socket (i.e., the TASKA Hand, wrist rotator, batteries, and on/off switch), the fit noticeably deteriorated.
A low-profile, high-frequency, linear 16HL7 ultrasound transducer weighing 11 g was mounted on the socket via a custom 3D printed bracket (Figure 1). The transducer was positioned such that it made direct contact with the volar surface of the residual limb when the prosthesis was donned (i.e., over the forearm muscle tissue of the residual limb). We acquired ultrasound images using a clinical ultrasound system (Terason uSmart 3200T, Terason, Burlington, MA) and transferred them to a PC in real-time using a USB-based video grabber (DVI2USB 3.0, Epiphan Systems, Inc., Palo Alto, CA). The video grabber digitized the images at 8 bits/pixel. Using a custom MATLAB script (MathWorks, Natick, MA), we downscaled the ultrasound images to 100 x 140 pixels before processing with a classifier, as described below.
We acquired a set of ultrasound images under various training conditions to be used as training data for a classifier (Table 1). For each acquisition, we instructed the participant to perform a forearm muscle contraction corresponding to a desired hand grasp and maintain this contraction for a specified duration. We also instructed the participant to perform contractions at a comfortable level and allowed her to rest between periods of classifier training to minimize fatigue. Note that the TASKA hand and wrist rotator were not active during classifier training.
We used linear discriminant analysis (LDA) classifiers to predict the user's desired hand grasp from the acquired ultrasound training data. While more complex classification algorithms are possible, LDA classifiers are commonly used for myoelectric control given their simple implementation, strong classification performance, and computational efficiency. As described below, we considered different classifier training strategies to account for changes to arm position and socket loading.
We trained the classifiers to recognize a set of three intended hand grasps: tripod grasp, index finger point, and rest. Any repeatable muscle deformation pattern could be mapped to these intended hand grasps, therefore we asked the participant to choose a set of muscle contractions that would be easiest to perform based on her experience using direct control myoelectric prostheses. The participant had congenital limb absence and could not reliably produce tripod grasp or index finger point, so she instead chose to produce a set of muscle contractions corresponding to wrist flexion, wrist extension, and rest (i.e., a relaxed muscle state). Within the classifiers, we mapped wrist flexion to tripod, wrist extension to point, and rest to rest.
The participant reported a deteriorated socket fit when all the distal components were attached. We therefore conducted classifier training using two separate loading conditions (Figure 1). Under the load A training condition, classifier training was performed with all the distal components attached to the socket (i.e., the TASKA Hand, wrist rotator, batteries, and on/off switch). Under the load B training condition, training was performed without the TASKA hand and wrist rotator attached to the socket. However, in this condition we temporarily attached a weight to the distal end of the socket equal to the weight of the participant's clinically-prescribed prosthetic hand (Transcarpal Hand DMC Plus, Ottobock, Duderstadt, Germany). The attached weight was designed to make the resultant socket length approximately similar to the length of the participant's clinically-prescribed prosthesis. We designed this loading condition to approximate the inertial loading and end-effector distance that the participant normally experiences using her prostheses.
We considered three classifier training strategies to account for changes to arm position during real-world prosthesis use (Figure 2). In the first strategy (i.e., the static training strategy), we recorded training data for the set of hand grasps while the participant held her arm still for 5 seconds in seven different positions. In the second strategy (i.e., the dynamic training strategy), we recorded training data for the set of hand grasps while the participant moved her arm for 5 seconds following four prescribed movement patterns. In the third strategy (i.e., the continuous dynamic training strategy), we recorded training data for a set of hand grasps while the participant moved her arm through all seven positions in a pre-defined 20-sec movement pattern. We chose this approach to minimize the total training time required and to reduce the potential impact of extraneous arm motion between data collection periods.
1) Ipsilateral Countertop: as if reaching for an object located in front of and ipsilateral to the body at waist height
2) Midline Countertop: as if reaching for an object located directly in front of the body at waist height
3) Contralateral Countertop: as if reaching for an object located in front of and contralateral to the body at waist height
4) Ipsilateral Bookshelf: as if reaching for an object located in front of and ipsilateral to the body at head height
5) Midline Bookshelf: as if reaching for an object located directly in front of the body at head height
6) Contralateral Bookshelf: as if reaching for an object located in front of and contralateral to the body at head height
7) Drinking Glass: as if reaching for an object located directly in front of the body at sternum height
We used a naming convention for the seven arm positions and four movement patterns to make it easier for the participant quickly recognize and perform the requested action. The names were selected based on a functional task related to each position or pattern. The seven arm positions for the static training strategy were chosen to cover the majority of the reachable workspace: 
To maintain consistency between trials, the positions were defined relative to the participant's anatomy and displayed on a wall in front of the participant at about an arm's length away (Supplementary Figure S1).
1) Countertop Wipe: moving from Contralateral Countertop to Midline Countertop to Ipsilateral Countertop in one fluid motion
2) Lawnmower: moving from Contralateral Countertop to Drinking Glass to Ipsilateral Bookshelf in one fluid motion
3) Sliding Glass Door: moving from Contralateral Bookshelf to Midline Bookshelf to Ipsilateral Bookshelf in one fluid motion
4) Drawing Blinds: moving from Contralateral Bookshelf to Drinking Glass to Ipsilateral Countertop in one fluid motion
The four prescribed movement patterns for the dynamic training strategy were chosen to cover the seven arm positions: 
The continuous dynamic training strategy was defined as a combination of the four movement patterns from the dynamic training strategy. However, the direction of the Countertop Wipe and Sliding Glass Door movement patterns were reversed so that the combined patterns could be performed continuously.
Before performing any real-time functional testing of the SMG-controlled prosthetic hand, we first examined the offline performance of classifiers using our different training strategies to account for socket loading and changes in arm positions.
For a given set of training strategies, we collected classifier training and testing data during a single session. Two repeated sets of muscle contractions were collected for the set of hand grasps during each arm position or movement pattern. For example, under the load A training condition using the static training strategy, we collected two sets of ultrasound images over two 5-sec collection periods for muscle contractions corresponding to tripod, point, and rest for each of the seven static arm positions. Note that for our offline testing, we only considered the static and dynamic training strategies and did not consider the continuous dynamic training strategy.
The order of the selected hand grasps and arm positions or movement patterns were randomized. To avoid including any transient motion at the start of the recording, we instructed the participant to assume their initial arm position and then provided two audio cues (i.e., beeps). The first beep notified the participant that their prescribed arm motion would start in 3 seconds (i.e., a preparatory period). The second beep occurred 3 seconds later to notify the participant to initiate the prescribed arm motion (e.g., by holding their arm still for static training sessions or by moving their arm for dynamic training sessions). Ultrasound images were recorded at the end of the preparatory period and continued for 5 seconds, during which the participant maintained the requested muscle contraction. After the data collection session, data for two repeated sets of muscle contractions were randomly assigned as either classifier training data or classifier testing data.
For a given set of training strategies, we built a series of LDA classifiers to predict a user's desired hand grasp from the designated training data. For sessions using the static training strategy, we built seven classifiers using training data from each of the seven static positions individually and an eighth classifier from all seven static positions collectively. Similarly, for sessions using the dynamic training strategy, we built four classifiers using training data from each of the four dynamic movement patterns individually and a fifth classifier from all four dynamic patterns collectively.
We then calculated classification accuracy by inputting the designated testing data from each static position or dynamic movement pattern into the relevant classifier. This process generated a predicted grasp for each frame of testing data, which could be compared to the true grasp. Note that because the participant held only one grasp when recording a given dataset, the true grasp was the same for each frame in that dataset. Finally, classification accuracy was calculated by summing the number of correctly-predicted frames and dividing by the number of total frames in the dataset: 
Note that although data was collected for 5 seconds for both the static and dynamic training strategies, the total number of frames fluctuated between 57 or 58 across datasets.
We used the Structural Similarity Index (SSIM) to quantify differences in the ultrasound images acquired during classifier training. SSIM quantifies the similarity between two images by decomposing them into luminance, contrast, and structure components, which are compared separately between the two images. A final value between -1 and 1 is then computed as an index of similarity, where 1 represents perfect similarity. Given two images  and , the index is defined aswhere , c , and  are the luminance, contrast, and structure components. The components are defined as  where , , , , and  are the local means, standard deviations, and cross-variance, and constant . The exponents are used to adjust the relevance of each component and were defined as . To provide context to a calculated SSIM value, we determined the similarity of ultrasound images due to chance was 0.229. We determined the similarity due to chance as the average SSIM between 100 randomly generated ultrasound images (i.e., 4,950 unique comparisons). Each image was generated using random image intensity values from a Rayleigh distribution () matching a representative ultrasound image of forearm muscle tissue.
To understand the consistency of ultrasound images for repeated conditions, we assessed the similarity of the ultrasound images for the two sets of contractions collected using the same classifier training strategies and hand grasp. Similarly, to understand the uniqueness of ultrasound images for differing hand grasps, we assessed the similarity of ultrasound images for two sets of contractions collected using the same classifier training strategies but different hand grasps. For all these comparisons, we calculated the SSIM within each frame of their 5-sec training periods. We then used a t-test to compare the average SSIM values between images using the same hand grasp and images using different hand grasps.
After offline classification testing, we conducted real-time functional testing using SMG to control the TASKA hand. Commands for the predicted hand grasp were delivered to the TASKA hand via Bluetooth. During functional testing, the MC Standard Wrist Rotator was fixed in place such that hand pronation or supination could not be controlled.
We collected a new set of training data to build classifiers for functional testing. We selected classifiers using different training strategies based on our evaluation of offline performance. By considering a smaller set of classifiers during functional testing we hoped to reduce the user's burden of repeated functional testing under separate classifier strategies (e.g., to reduce fatigue). For the functional testing that involved classifiers trained using the static strategy, we included training data from all seven positions collectively. We also used a classifier built from training data collected using the continuous dynamic strategy. The functional tests were conducted immediately after training the relevant classifier (i.e., we did not train all classifiers before beginning the functional testing). Note that the TASKA hand was not attached when collecting training data during the load B training condition. However, the TASKA hand and related components were reattached to complete the functional testing.
We configured the TASKA hand to perform the three grasps executed in classifier training, including tripod, point, and rest (Figure 3). We chose to implement these three grasps because only two of the grasps would be needed to perform the functional tests (tripod and rest) and one of the grasps would not be helpful (point). Each grasp was defined to be a discrete configuration of the thumb and fingers on the TASKA hand, such that a trained classifier initiated the TASKA hand to assume that configuration (i.e., the position and velocity of the thumb and fingers during grasp transitions were set in advance and not controlled by the classifier or participant). Thus, a participant could physically implement a grasp by activating their muscles in a manner that enabled the classifier to identify the desired grasp based on the ultrasound image of muscle deformation. We defined the rest grasp such that the thumb, index, and middle fingers of the TASKA hand were partially extended and the ring and little fingers were in a closed position (Figure 3). We used this definition to prevent the ring and little fingers from inadvertently moving blocks out of their designated rows during the targeted Box and Blocks Test, as described below. For consistency, we used this definition of rest grasp during all three functional tests.
We instructed the participant to perform three functional tests involving grasping and moving one-inch wooden blocks. Each functional test included a quantifiable score related to completion speed. We also recorded the participant's performance with a video camera. Although the primary purpose was to ensure accurate quantification of the test scores, we also used comparisons of the recorded video for a general observational analysis of functional performance.
Additionally, we calculated two outcome measures to characterize the efficiency of grasp selection. First, we counted the "number of transient bouts" during each test. A transient bout was defined as an instance when the classifier predicted a grasp for less than five consecutive frames. Note that a transient bout does not necessarily indicate that a predicted grasp was misclassified, as it is possible for a user to select a grasp for a brief period (i.e., <5 frames) if desired. However, given the relatively slow, pick-and-place nature of each functional test, we consider it unlikely that a user would deliberately choose to switch grasps that quickly. This outcome measure thus serves as an indirect indicator of a classifier's ability to predict a user's intended grasp. Second, we quantified the percent of all frames that were classified as point during each functional test. Although point was not a helpful grasp during functional testing, we included it in the repertoire of available grasps so that we might examine a participant's ability to select a desired grasp from a set of grasps. Because the participant should not be selecting point to accomplish the functional tasks, these instances of point also served as an indirect indicator of a classifier's ability to predict a user's intended grasp. Note that classification of point during a functional test does not necessarily indicate that a predicted grasp was misclassified, but we consider it unlikely for a user to voluntarily select this grasp when attempting to hold a block.
The Box and Blocks Test (BBT) is a common measure of gross manual dexterity. The set-up consists of two 10-inch square compartments separated by a six-inch-tall partition (Figure 4). The compartment on the side of the arm being tested is filled with 150 one-inch wooden blocks, which are mixed such that the blocks rest in many different orientations. The test is scored by the number of blocks transported over the partition in 1 minute. Subjects may transport the blocks in any order as long as their fingertips cross the partition before releasing the block into the opposite compartment. The BBT apparatus was placed on a table set to 10 cm below the participant's right anterior superior iliac spine. The apparatus was positioned 4 cm from the proximal edge of the table with the box partition aligned with the participant's midline.
The Targeted Box and Blocks Test (tBBT) is a modified version of the BBT involving only 16 blocks. The blocks are placed in a four-by-four grid in the compartment on the side of the arm being tested (Figure 4). The blocks are numbered 1 to 16, beginning with the innermost block on the bottom row and moving across each row. Subjects must transport the blocks between the compartments in numbered order. Each block must be placed in its mirrored position in the other compartment. The test is scored by the time required to transport all 16 blocks. The tBBT was performed on the same table as the BBT with the apparatus placed in the same position described previously. Note that we made a minor modification to the tBBT apparatus by removing both compartments' outer walls (Figure 4). Because of the large size of the TASKA Hand, it collided with the walls when the participant attempted to manipulate blocks located near the edge of the compartment. Since this study was intended to demonstrate the feasibility of using prostheses controlled by SMG during functional tasks, we felt it was appropriate to remove the walls so that the tBBT could be completed more easily.
The Rainbow Test was developed for this study to evaluate grasp control over a wider variety of arm positions than is required by the BBT or tBBT (Figure 4, Supplementary Figure S2). A series of 14 squares were marked on a magnetic whiteboard following an approximate arch shape. One-inch magnetized blocks with magnets attached were placed inside each square. We instructed the participant to transport each block from the whiteboard to a collection box placed at waist height along the midline of the body. Blocks were transported in a designated order, beginning with the bottom blocks on the side ipsilateral to the prosthesis and continuing up each column. The test was scored by the time required to transport all blocks.
1) Short-term testing: The participant trained a classifier and performed the set of three functional tests in random order. The participant then repeated the set of three functional tests (presented in a new random order) for a second and third time without retraining the classifier.
2) Three-hour testing: The participant trained a classifier and then wore the socket for 3 hours. Every 30 min, the participant performed the set of three functional tasks in random order. During each break between the functional testing, the TASKA hand was turned off and the participant was able to move freely and perform normal daily activities. Although the TASKA Hand could not be actively controlled during breaks, it could still be used for passive object manipulation. The participant performed pre-defined tasks during each break (Table 2), which were staggered to require increased arm movement and socket loading over time. For this testing scenario, we used a linear regression model to reveal any changes to the outcome measures over this 3-h period.
3) Simulated donning/doffing: The participant trained a classifier and performed the set of three functional tests in random order. Next, the ultrasound transducer was removed and replaced to simulate donning/doffing of the socket. The participant then repeated the set of three functional tests in random order without retraining the classifier.
To assess whether the participant could repeatedly perform functional tasks using an SMG-controlled prosthetic hand, real-time functional performance was assessed for three separate scenarios: 
End-to end system latency was approximated as the delay between the onset of volitional finger flexion in an individual without limb loss and the corresponding onset of finger flexion on the TASKA hand. We manually identified the start of each movement based on the acceleration signals (Supplementary Figure S3), and these time differences were averaged over 5 cycles. The resulting value encompasses latency associated with data acquisition, data transfer, processing, and communication with the TASKA Hand via Bluetooth.
We also calculated the processing latency for the MATLAB-based classification algorithm using all datasets recorded during real-time functional testing (see Section 2.7). The difference in successive timestamps were averaged across all files to compute the processing latency. Lastly, we computed the classification throughput as the number of predictions divided by the total elapsed time, based on the real-time functional testing datasets (see Section 2.7).