Proceedings of the Royal Society B: Biological Sciences
Restricted accessResearch articles

Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

Adam M. Roth

Adam M. Roth

Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA

Contribution: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Jan A. Calalo

Jan A. Calalo

Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA

Contribution: Formal analysis, Validation, Visualization, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Rakshith Lokesh

Rakshith Lokesh

Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA

Contribution: Validation, Visualization, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Seth R. Sullivan

Seth R. Sullivan

Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA

Contribution: Validation, Visualization, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Stephen Grill

Stephen Grill

Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA

Contribution: Supervision, Writing – review & editing

Google Scholar

Find this author on PubMed

,
John J. Jeka

John J. Jeka

Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA

Interdisciplinary Neuroscience Graduate Program, University of Delaware, Newark, DE 19716, USA

Biomechanics and Movement Science Program, University of Delaware, Newark, DE 19716, USA

Contribution: Supervision, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Katinka van der Kooij

Katinka van der Kooij

Faculty of Behavioural and Movement Science, Vrije University Amsterdam, Amsterdam, 1081HV, The Netherlands

Contribution: Supervision, Validation, Writing – review & editing

Google Scholar

Find this author on PubMed

,
Michael J. Carter

Michael J. Carter

Department of Kinesiology, McMaster University, Room 203, Ivor Wynne Centre, Hamilton, L8S 4L8, Ontario, Canada

Contribution: Conceptualization, Methodology, Supervision, Visualization, Writing – original draft, Writing – review & editing

Google Scholar

Find this author on PubMed

and
Joshua G. A. Cashaback

Joshua G. A. Cashaback

Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA

Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA

Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA

Interdisciplinary Neuroscience Graduate Program, University of Delaware, Newark, DE 19716, USA

Biomechanics and Movement Science Program, University of Delaware, Newark, DE 19716, USA

[email protected]

Contribution: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Google Scholar

Find this author on PubMed

From a baby’s babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea—that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.

Footnotes

Co-senior authors.

Current address: Biomedical Engineering, University of Delaware, STAR Campus, Room 201J, Newark, DE 19711, USA.

Electronic supplementary material is available online at https://doi.org/10.6084/m9.figshare.c.6837535.

References