Generating Realistic Arm Movements in Reinforcement Learning: A Quantitative Comparison of Reward Terms and Task Requirements

Mimicking of human-like arm movement characteristics involves considering three factors during control policy synthesis: (a) task requirements, (b) noise during movement execution, and (c) optimality principles. Previous studies showed that when these factors (a-c) are considered individually, it is possible to synthesize arm movements that either kinematically match experimental data or reproduce the stereotypical triphasic muscle activation pattern. However, no quantitative comparison has assessed the realism of arm movements generated by each factor, nor has it been determined whether combining these factors results in movements with human-like kinematic characteristics and the triphasic muscle pattern. To investigate this, we used reinforcement learning to learn a control policy for a musculoskeletal arm model, aiming to discern which combination of factors (a-c) results in realistic arm movements according to four frequently reported stereotypical characteristics. Our findings indicate that incorporating velocity and acceleration requirements into the reaching task, employing reward terms that minimize mechanical work, hand jerk, and control effort, along with the inclusion of noise during movement, leads to realistic human arm movements by reinforcement learning. We expect that the gained insights will help in the future to better predict desired arm movements and corrective forces in wearable assistive devices.