Annual Reviews www.annualreviews.org/aronline Ann. Rev. Neurosci. 1989. 12 : 15743 Copyright © 1989 by Annual Reviews Inc. All rights reserved

LEARNING ARM KINEMATICS AND DYNAMICS Christopher

G. Atkeson

Department of Brain and Cognitive Sciences and the Artificial Intelligence Laboratory, MassachusettsInstitute of Technology, Cambridge, Massachusetts 02139 INTRODUCTION

AND SCOPE

OF

THE

REVIEW

One of the impressive feats of biological motor control systems is their ability to calibrate themselves and improvetheir performance with experience. Man-mademachines are typically calibrated using external measuring devices and references, and often execute the same errors repeatedly. An important step toward understanding biological motor behavior, and also toward building more useful machines, is to understand howmotor learning is achieved. The architecture of the nervous system affects what can be learned and howit might be learned. Hypotheses concerning biological motor control are primarily discussed from the point of view of how well a proposed control scheme might generate desired movements. Few have evaluated hypotheses from a motor learning perspective. This review explores the implications of several proposed motor control schemesfor certain issues of learning, such as generalization, learning efficiency, rate of learning, and the ability to interpret and correct performance errors. Myown interest in motor learning was sparked by discovering that somecontrol schemes, such as the equilibrium trajectory approach (Berkinblit et al 1986, Hogan et al 1987), do not support efficient movementrefinement during practice (C. G. Atkeson, unpublished results). This review focuses on the information processing issues involved in motor learning. Learning is an information processing problem: A learning algorithm modifies present and future commandsby using information from previous experience so as to improve performance. Marr (1982) emphasized the need for a computational approach to information pro157 0147~306X/89/0301~ 157502.00

Annual Reviews www.annualreviews.org/aronline 158

ATKESON

cessing problems in neuroscience. He emphasized separating the different levels of computational theory, algorithm, and implementation. Computational theories describe the information processing problems faced by the brain, whereas algorithms specify how to solve those problems, and implementations specify the mechanismsused to execute algorithms. Recent reviews in this series (Hildreth & Koch 1987; Poggio & Poggio 1984, Robinson 1981, Ullman 1986) and elsewhere (Hildreth & Hollerbach 1985, Hollerbach 1982, Raibert 1986, Saltzman 1979) discuss computational approaches and how these approaches are shaped by and help guide other forms of neuroscience rcsearch. Robotics is an important source of ideas for biologists. The study of robot control and learning contributes to the understanding of biological motor control and learning, and therefore this review also surveys relevant work in robotics. Manyproposed methods for robot control and learning are also applicable for biological control, and vice versa. Although the actuators, sensors, structure, and computationalarchitecture of the control systems of mechanical and biological limbs differ, at an information processing level the control problems are similar. The same laws of physics apply, and the mechanical objectives of movementand manipulation are identical. Trying to program a robot to solve a problem that humans routinely solve leads to greater comprehension and insight into what makes that problem difficult. Robots provide a testbed for developing and evaluating hypothesized control schemes and allow exploration of a wide range of possible solutions, someof which maybe directly applicable to biological motor control. Specific algorithms and the observed behavior of those algorithms on a machinecan guide biological research by suggesting novel experimental tests of hypothesized control schemes. In trying to build a machinethat actually works, theories are forced to he specific and concrete, and difficult issues cannot be ignored. Manycontrol schemeshave been criticized for requiring accurate internal models. A reason for renewedinterest in motor learning is the recent research showing that it is possible to identify accurate models from movementdata. A description of these demonstrations of motor learning in robots is included in this review. This article emphasizes arm control. Twokinds of motor learning are examined. The first involves building and refining internal models of the motor apparatus and the external world, and using those models to generate the appropriate actuator commandsand to guide interpretation and processing of sensory data. The second involves using internal models to transform performance errors into commandcorrections. The particular tasks used to assess motor learning include movingthe hand to a visually observed target and attaining a desired pattern of movementeither with

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

159

or without a load. A recent review of arm motor control psychophysics and neurophysiology is provided by Georgopoulos (1986). Recent reviews of motor learning from a variety of perspectives include Adams(1984, 1987), Annett (1985), Grossberg &Kuperstein (1986), Salmoniet al (1984), Schmidt (1982), and Smyth & Wing (1984). The study of motor learning mayhelp reveal the forms of representation used by the nervous system. Twoalternative representations, structured and tabular, have received a great deal of attention and support different forms of learning. Structured representations can take advantage of prior knowledgeof the constraints on a system, whereas tabular representations are general purpose in that they can represent almost any type of transformation. The review focuses on the transformations involved in translating a behavioral objective into muscle activation, and the implications for motor learning of howthese transformations are represented. The next section discusses these transformations. In this article the discussion of the issues and the review of the literature have been separated. The third section explores important issues in learning these transformations. The final sections survey howparticular proposals have addressed these issues. MOTOR CONTROL TRANSFORMATIONS

INVOLVES

Motor control maybe viewed as a series of transformations from a specified behavioral objective to a plan specifying the desired mechanicaloutput of the motor apparatus and finally to a pattern of activation of the muscles (Hollerbach 1982, Hildreth & Hollerbach 1985). Progress has been made in understanding the planning process (see Hollerbach & Atkeson 1987 for further references). In this section the transformations that maybe involved in executing a movementare discussed. It is assumed that a movementplan already exists. Twotypes of transformations, kinematic and dynamic, are explored. Kinematic Transformations One series of transformations involved in the execution of a movementis kinematic. A limb, as a mechanical device, converts muscle lengths and joint angles to hand positions. This process is referred to as forward kinematics (Figure 1A). In Figure 2A an idealized model of this transformation for movementsin a vertical plane is illustrated. The arm is modeled as two rigid links of length l~ and 12. The links rotate about horizontal parallel axes that are fixed with respect to the links. For this idealized modelthe forward kinematics from joint angles (01, 02) to hand position (x, y) are given

Annual Reviews www.annualreviews.org/aronline 160

ATKESON 1.

\l, sin (O,) +l: sin (O~+O2)

The nervous system is faced with the following problem: Given a desired hand position, choose the appropriate muscle lengths and corresponding joint angles to achieve that position. The transformation from desired hand position to the corresponding joint angles and muscle lengths is knownas the inverse kinematic transformation (Figure 1.4). That we can view a target in space, close our eyes, and moveour hand to that target suggests that we have an internal representation or model of the transformation from desired hand positions to joint angles and muscle lengths. Internal models of arm kinematics are also useful in choosing torques at the joints of the arm to achieve a particular force and torque between the hand and some external object. Learning to moveaccurately may involve building internal models of kinematic transformations. Dynamic Transformations Another set of transformations become important when the pattern of movementis to be controlled. In order to drive the limb along a particular A desired hand position in visual coordinates

~

joint angles inverse kinematics

actual hand position

~1

forward kinematics

(muscle lengths)

B desired pattern of motion

joint torques

¯ ctuaJ pattern of motion forw~r d dyna.miea

inverse dyna~nics ~’~-I

(mtmcle activation)

C desired performance

actual performance

command

~ I inverse

[

~ I forwa,rd

I

Figure1 Example motorcontrol transformations.

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

161

(x,y)

A

X

Figure 2 Idealized model of two-joint arm: (A) Arm kinematics. corresponding inertial parameters.

(B) Arm segments

Annual Reviews www.annualreviews.org/aronline 162

ATKESON

trajectory, the appropriate torques must be applied at the joints. The transformation from joint torques to movementis referred to as forward dynamics (Figure 1/3). The pattern of muscle activation to achieve these torques may be generated in a variety of ways, including pure feedback (closed loop) control, pure feedforward (open loop) control, or combination of feedback and feedforward control (Houk & Rymer 1981). Muscle mechanical properties, such as stiffness and viscosity, also contribute to the applied torques (Bizzi et al 1978). For fast movements,such as a baseball pitch, feedback control can play only a small role in controlling the movement because of information transmission delays. Instead, the nervous system must specify the pattern of muscle activation corresponding to the desired pattern of motion. The transformation from a desired pattern of motion to the actuator commands necessary to achieve that motion is referred to as inverse dynamics(Figure 1B). The inverse dynamics transformation from a desired pattern of motion to muscle activation maybe broken up into a series of transformations: pattern of motion to joint torques, joint torques to muscle forces, and muscle forces to the necessary muscle activations. Equations 2 and 3 give thc joint torques (z~ and 22) for the idealized two-joint arm model as function of the desired joint positions (0~ and 02), joint velocities (01 02), and joint accelerations (0"1 and 0"2): )Z~ ~ m~cx,gCos(O~)--mlCylgsin(Ol

+I, 0",+ m2[g/, cos(0 +m2Cxz[g cos (0~+ 02) + 2l~0"~cos (02) + 2 cos (02) --21,0102sin (02)--l,0~2sin (02)] + m2G2[--9sin (0, + 02) --

2l,

0102 COS (02) -- 10 22 COS (02) ] ~-

12(0", + G)

2.

22 =mzCx219 cos (0~ + 02) l~O’~cos (02) + ll O 2~ sin(02) + rn 22y2[--9sin (0~+ 02)--110"~sin (02) + llO ~ COS (02)] nt- 12(0"1 q- g2).

3,

Each segment of the arm has a mass (m~ and m2), a location of the center of mass relative to the proximal joint [the vectors 21 = (Cx,,cyl) and c2 = (c~2, cy~)], and a momentof inertia for rotations aroundthe joint axis (I~ and I~) (Figure 2B). The parameter 9 is the gravitational constant. inertial parameters of mass, mass moment(the product of the arm segment

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

163

mass and its center of mass location), and momentof inertia appear linearly in the inverse dynamics. Simpler modelsmay be used to approximate the dynamics. Different versions of the equilibrium trajectory approach, for example, either ignore dynamics or use a configuration-independent mass-spring-damper model to approximate the dynamics (Hogan et al 1987, Feldman 1986). In both engineering and biology, control problems can be posed in the following way: The mechanical apparatus to be controlled transforms its inputs (commands)into some outputs (performance). The control system generates the appropriate commandsbased on the desired performance of the motor apparatus. To achieve high levels of performance a control system must implement the inverse of the transformation performed by the motor apparatus (Figure 1C). This is true even when using feedback control, and it is especially true in biological systemswheresignaling delays limit possible feedback gains. Oneview of n~otor learning is that its goal is to build an accurate inverse modelof the motor apparatus. Inverse Transformations Can Be Implemented In Many Ways Inverse models of the motor apparatus can be represented in manyways. The inverse kinematic transformation is used here as an example. The inverse kinematic transformation for the idealized two-joint arm model can be represented mathematically:

02=arcco kO, = arctan(~)-arctan(l,

.)

4.

12sin(O~) + 12 cos (0:)J"

To control a two-joint robot arm, the inverse kinematic transformation could be implementedas a digital computerprogram(Figure 3A), a special purpose analog (or digital) computational circuit that corresponds to the mathematical Expressions 4 and 5 (Figure 3B), or a lookup table (Figure 3C). These are examples of how the same information-processing problem, i.e. computing the inverse kinematic transformation, can be solved by different algorithms and implementations. The nervous system could also use a variety of mechanismsto implement the inverse kinematic transfo~ation. Neural circuits might exist that correspond to or approximate the circuit of Figure 3B. Manyhypothesized brainstem circuits for oculomotorcontrol are of this nature, in that signals are represented by the amountof neural firing, and operations on signals

Annual Reviews www.annualreviews.org/aronline 164 A

ATKESON Digital

computer program

theta2

= arccos( (x*x ÷ y*y - 11"11 - 12"12)/(2.11’12) ~hegal = arctan( y, x ) - arc~an( 12*sin( th~ta2 ). II ÷ 12*eos(th~ta~)

]3

Analog computational circuit y input x input

output

input

input

C Lookuptable

1.0 0.Si(81, 0.6 0.4,

(5~I09)

0.2 0 -0.2 -0.6 -0.8 -1.0 -I.0

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

X Figure 3

Alternative

implementations

of two-joint

inverse kinematics.

0.8

1.0

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

165

are performed by the interaction of signals with operations analogous to addition, multiplication, and integration (Robinson 1981). Other proposed neural representations are similar to the tabular implementationof Figure 3 C, in that a signal is representedby activity at a particular location within a neural structure, and operations on signals are performedusing patterns of connections or mappings between neural structures (Knudson et al 1987). The superior colliculus is an exampleof such a tabular representation, in that activity at a particular location in the superior colliculus corresponds to a particular eye movement. ISSUES

IN

REPRESENTING

TRANSFORMATIONS

This section discusses the issues and implications involved in using different representations of transformations for motor control and learning. Two extreme alternatives are structured and tabular representations. Models such as the mathematical Expressions 4 and 5, the computer program (Figure 3A), and the computationalcircuit (Figure 3B) reflect the structure of a transformation, in that only a few parameters are required to describe the transformation and the modelis specific to a particular transformation. Such models are referred to as structured models. Tables, such as the examplein Figure 3C, are an unstructured methodof representing models, in that they can be applied to a wide variety of transformations and require manyparameters to be specified. Different forms of representations support different forms of learning. The study of motor learning mayhelp reveal the forms of representation used by the nervous system. LOCAL GENERALIZATION The process whereby experience in one activity leads to improvedperformancein another is referred to as generalization. Typically, transformations such as inverse kinematic and dynamic transformations are continuous, in that similar inputs lead to similar outputs. Therefore, experience of a particular posture or movementshould improve the ability to attain similar postures or movements.Tabular representations support this form of local generalization because table entries can be shared between different inputs, and interpolation over several entries can be used to smooth the transformed data. GLOBAL GENERALIZATION Representations that capture the structure of a transformation support a more powerful form of generalization, global generalization. In the example of the inverse kinematic circuit shownin Figure 3B, only the lengths of the links and the appropriate calibration of joint angle measurementsmust be specified for the circuit to represent accurately the entire transformation. The circuit can be calibrated by using

Annual Reviews www.annualreviews.org/aronline 166

ATKESON

only a few sets of inputs and outputs. It is then capable of generalization to all possible inputs and outputs. Global generalization works as long as the structure of the internal modelmatches the actual motor apparatus. If the joint axes moverelative to the links, due to rolling or sliding of the joint surfaces, or the joint axes are not parallel, then the structure of the idealized two joint kinematic model(Eqs. 4 and 5) is incorrect, and no set of parameters (link lengths, joint angle offsets, etc) will allow the circuit of Figure 3B to predict correctly joint angles for all possible hand positions. The point at which such structural modeling errors degrade the internal model so muchthat it is no longer useful is difficult to characterize. Because structured models incorporate knowledgeof the structure of the motor apparatus, they typically are able to improve more quickly and generate better performance than tablebased models for a given amountof experience. However,this is true only if the modelstructure is approximatelycorrect. Learning rate and efficiency are important issues in handling relatively rapid changes in the motor system, as is the case in handling loads and in muscle fatigue. Whena load is lifted, the dynamicsof the arm maychange dramatically. Structured models can be updated quickly because only a few parameters must be estimated, and knowledge about the load can be generalized to all movements.Table-based models cannot use observed data as efficiently because they do not incorporate knowledge about the structure of the system. Becausetables only support local generalization, the effect of the load must be learned again for each new type of movement. Tabular methods can improve their ability to handle loads by haying different tables for different loads together with somemethodfor recognizing loads so that the appropriate table is used whenthat load is present. LEARNING RATE AND EFFICIENCY

an appropriate model structure found? Scientists or engineers might use idealized models from physics, such as Newtonian dynamics. Graphs of the inputs versus the outputs of the transformation may suggest appropriate structures. In general, however, an adaptive machine must simply hypothesize possible modelstructures, estimate the corresponding parameters, and evaluate the modelto see howwell the modelwith its assumedstructure fits the data. Identifying a model structure for its arm kinematics and dynamics may not be necessary when designing an adaptive robot. Knowledge about structure may be preprogrammed, with only the parameters adjusted during the lifctimc of the machine.Similarly, in biological systems the structure of the motorapparatus is essentially fixed during the lifetime of the organism, and only parametric changes occur. WHERE DO MODEL STRUCTURES COME FROM? How is

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

167

SPACEREQU~RE~r~NTS OE a~ABLESProblems arise in applying straightforward tabular representations to complex problems, because the size of a table grows exponentially with the number of indices used. In representing the arm inverse dynamics for a six-jointed arm in a table, for example,six joint angles, six joint angular velocities, and six desired joint accelerations wouldform a set of 18 indices. If each of the 18 dimensions of the table were divided into only 10 parts, there wouldbe 10~s entries in the table, a number larger than the number of neurons or synaptic connections in the brain. Moresophisticated table organizations are necessary to reduce the table size. DECISIONS MADE? Another problem with the straightforward tabular representation presented here is that the functions that store and retrieve data must be chosen before any data is collected. These functions play a very important role in smoothing and temporally filtering the data. In general, the amount of smoothing and temporal filtering required dependson the noise present in the data and the density of nearby data points. Better use of the data could result if these data processing decisions could be delayed until the data are actually used. Simonset al (1982) present a schemefor varying the resolution of the table as required by the pattern of input data. WHEN ARE DATA PROCESSING

DIFFICULTY OFPARAMETER ESTIMATION It is not difficult to initialize and maintain a table based on the observed input and output values of a transformation, as long as there is somemethodfor generating the table before enough data are obtained to represent the transformation accurately. On the other hand, a structured model may present a very difficult parameter estimation problem, depending on its mathematical structure. Parameters that appear linearly in a model can be estimated by using simple regressions, but parameters appearing nonlinearly in a model must typically be estimated iteratively. Nonlinear parameter estimation methodsalso suffer from the problem of finding parameter values that are only locally optimal rather than globally optimal. CHOICE OF COORDINATE SYSTEMS It is not clear what coordinate systems are actually used by the brain. For example, instead of a Cartesian coordinate system, polar coordinates or some non-orthogonal coordinate system could be used to indicate a hand location in visual space (Soechting & Ross 1984). Tabular representations are typically very flexible in terms of what coordinate systems can be used. The linearization of sensory measurements or muscle commandsis not necessary for tabular representations to work. Structures in the brain such as the visual cortex or the superior colliculus, whichuse location of neural activity to represent where

Annual Reviews www.annualreviews.org/aronline 168

ATKESON

a sensory stimulus is located or where a movementshould go, use computational maps rather than sets of coordinates (Knudson et al 1987). Indexing is performed by the pattern of neural connections. The choice of coordinate systems is more important for structured representations. The use of a structured representation based on a particular choice of input and output coordinate systems requires that the input and output coordinates be calibrated according to the coordinate systems used. Other considerations are the time required for a computation and the accuracy required of the computational elements. Implementations with long series of computations or computational elements may require both high-speed and high-accuracy components. Tabular representations use manycomputational elements, but they are typically used in parallel so the demands on computational speed and accuracy of the elements may be reduced. Another problem with both tabular and structured representations is that they are both specialized for a particular transformation direction. A model of the inverse kinematic transformation cannot be used to do the forward kinematic transformation, and vice versa, although the model includes the knowledgenecessary to perform this transformation. The possibility of hybrid or intermediate forms of representation should not be ignored. Computationalcircuits and tabular representations can be combined by replacing an element of the circuit (the arccos or arctan elements of Figure 3B, for example) with a table of the function that the element implements. Replacing functions with tables is often used in computer programs to speed up computation, as well as in modern electrical circttit design, where complexcomputational elements are replaced with tabulated values stored in a memorydevice. USINGSENSORSTO DECOUPLE IDENTIFICATION PROBLEMS System identification, the building of models, can be simplified by using sensors to decouple different parts of the system. For example, sensing of forces and torques by the hand allows estimation of load parameters independently of the arm dynamics. More generally, covering an arm or any effector system with a sensory barrier such as a tactile sensing skin allows the separate identification of internal system dynamics, external system dynamics, and disturbances. Without a sensory barrier the combined system of arm plus external world maybe too complex to identify robustly. Decoupling system identification problems allows different identification procedures to be applied to the different subsystems. In the case of an arm with variable loads, one expects the arm dynamicsto vary slowly, whereas load dynamics change rapidly as loads are picked up or put down. Widely different rates and types of system change call for different systemidenti-

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

169

fication algorithms to track the changes. Another reason for decoupling arm and load identification is that the arm structure is relatively constant whereasthe dynamicsof different loads can have quite different structures. Armidentification can be based on a fixed modelstructure, while a complete load identification system must handle manydifferent types of loads. Similarly, sensing the outputs of the actuators allows internal models of actuators to be identified separately from models of arm dynamics, BUILDING INTERNAL TRANSFORMATIONS

MODELS

OF

This section reviews proposals for how internal models of the motor apparatus and the external world might be built, used, and refined, Table-Based Methods Lookup table-based methods provide a simple yet general approach to representing transformations. Tables also provide an easily modifiable representation appropriate for motor learning. Early work on using tabular or mapping representations to solve motor control problems was inspired by the regular structure of the cerebellum (Grossberg 1969, Marr 1969, Albus 1971, Pellionisz &Llinas 1979, Fujita 1982a,b). Other work attempted to use neuron-like elements or models of neural networks as the components of motor learning systems (Widrow& Smith 1964, Albus 1981, Ersu &Tolle 1984, Barto et al 1983). These schemesstore knowledge as weights of the connections between model neurons, and are related to Perceptrons (Rosenblatt 196l, Mi~’~sky& Papert 1969) and current work on multilayer "connectionist" networks (Rumelhart et al 1986). The pattern of connections in these proposed networks can bc mapped into patterns of table indexing (Michie & Chambers1968, Albus 1975a,b). robot control, tabular representations have been proposed as a way to avoid computational delays during real time control (Albus 1975a,b, Raibert 1977, Raibert & Wimberly1984, Raibert 1986), as underlying mechanisms for adaptive control, and as substrates for motor learning in artificial intelligence (Michie & Chambers1968, Selfridge et al 1985). Albus (1975a,b, 1981) proposed using a tabular representation based on Perceptrons (Rosenblatt 1961, Minsky & Papert 1969) to represent internal models of inverse transformations and also to encode trajectories that would achieve particular task goals. Inverse kinematic transformations would be represented by using table lookup to find joint angles corresponding to a desired hand position, as shownin Figure 3C. Solving the IMPLEMENTINGCONTENT-ADDRESSABLEMEMORIES tiY USING FIASFnNG

Annual Reviews www.annualreviews.org/aronline 170

ATKESON

inverse dynamics would involve looking up joint torques on the basis of desired joint positions, velocities, and accelerations. A table entry is represented in a distributed fashion by a group of weights, and local generalization and smoothing are performed by sharing weights between neighboring entries. The weights in the table are updated on the basis of observed input and output pairs. The table size is reduced by randomly mappingthe originally astronomically large table into a reasonably sized table. This schemetakes advantage of the fact that such a table wouldbe used sparsely, in that only a small numberof different types of movements would be made comparedto all possible movementsthat the original table was capable of representing. Albus’s approach is equivalent to a computer science technique known as hashing, and is one way to implement a contcnt-addressable memoryin software (Kohonen 1980). Miller (1986, Miller et al 1987) has used Albus’s schemeto modelthe inverse kinematic transformation for a robot controllcd by a TV camera and to model the inverse dynamics of a robot. E1-Zorkany et al (1985) have implemented Albus’s schemefor improved trajectory control of a robot. LOCAL LINEARMODELS Pellionisz & Llinas represent a transformation with manylocal linear models in their tensor-based scheme(Pellionisz Llinas 1979, 1980, 1982, 1985). A table-like representation is used to store the different local linear modelscorresponding to different points in the generally nonlinear transformation. Learning is performed by an iterative procedure to find eigenvectors and eigenvalues of the matrix representing the local linear approximation (Pellionisz &Llinas 1985). This approach should work well for transformations that are linear for a given arm configuration, such as the transformation from desired hand velocities to joint velocities (Pellionisz & Llinas 1980) and the transformation from desired joint torques to muscle forces (Gielen & van Zuylen 1986). The approach maywork less well for transformations that are nonlinear for all input variables, such as the transformation from desired hand position to joint angles, or the inverse dynamicstransformation from desired joint positions, velocities, and accelerations to joint torques. In these cases the approach faces a worse problem of storage size than if the transformation is represented directly in a table. In general, the same numberof indices must be used to select the appropriate local transformation matrix as would be needed to select the appropriate output, and the transformation matrices (whose size is the number of inputs multiplied by the number of outputs) must be stored rather than just the outputs. Arbib & Amari(1985) provide a stimulating analysis of the Pellionisz &Llinas approach. MULTILAYER NEURAL NETWORK APPROACHES Initial steps are being made toward using multilayer neural network models to address the issues of

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

171

representing transformations such as inverse kinematic and dynamictransformations (Hinton 1984, Kawato et al 1987, Kuperstein 1987, Jordan 1986). Although successful network learning procedures have been demonstrated, it is not yet clear howwell they will scale up to realistically sized problems, howeffectively they use data to learn, and howquickly they can learn (Hinton 1986). BALANCING AN INVERTEDPENDULUM Dynamic balance is an important issue in legged locomotion, and provides a challenging continuous control problem with which to evaluate motor learning schemes. Many studies have dealt with the problem of balancing an inverted pendulum on a movable cart, muchas a vertical broomstick can be balanced on the open palm of one’s hand by moving the hand (Widrow & Smith 1964, Michie & Chambers 1968, Barto et al 1983, Selfridge et al 1985). A feedback control law was tabulated so that the appropriate movementsof the cart are made in response to measured pendulum angle and angular velocity and cart position and velocity. The learning problem in these studies is filling and updating the control law table. Unlike tabulating arm kinematics and dynamics, the appropriate control law cannot be simply observed from measured commandsand observed performance. The control law can be specified externally from a teacher (Widrow&Smith 1964) or it maybe learned on the basis of control failures occurring whenthe inverted pendulumfalls or the cart movestoo far to the left or right (Michie & Chambers1968, Barto et al 1983, Selfridge et al 1985). USINGTABLESTO CONTROL LOCOMOTION Raibert (1986, Raibert & Wimberly 1984) used a table to control forward running speed while maintaining balance of a one-legged hopping machine. The table represented the dynamicsof the hopper during its stance phase and was used to choose the appropriate foot placement at the beginning of the stance phase to achieve a desired forward velocity, body angle, and body angular rate. The size of the table used was reduced (a) by not using desired performance as indices into the table and looking up the appropriate command directly, but instead by using the table as an aid to searching for the command that wouldlead to the desired performance,(b) by ignoring somestate variables that vary cyclically during the locomotion cycle, and (c) by using polynomials to approximate the contents of the table. The data used in the table were generated by simulating a model of the hopper, but presumably they could also have been generated from actual observations of the dynamics of the hopper. Miura & Shimoyama(1980) have explored the use of tables to control a biped walking machine. ASSOCIATIVE CONTENT-ADDRESSABLE MEMORIES

Recent developments of

Annual Reviews www.annualreviews.org/aronline

172

ATKESON

massively parallel computers such as the Connection Machine(Hillis 1985) raise the possibility of modelingassociative content addrcssable memories (Kohonen 1980). These types of memories obviate the need for complex indexing schemesto find relevant data points in tables. Current models of the Connection Machine have 64,000 processors and the possibility of simulating many more. Each previously observed data point is assigned to a processor, and the multiple processors search through the data in parallel to find data points close to a requested set of indices. Each processor computes the contribution of its data points to a requested prediction on the basis of distance betweenthe sets of indices. The Connection Machine has a global addition operation to sum the manycontributions. It is no longer necessary to allocate storage for all possible combinations of indices, as only data points actually experienced are stored. Having more indices in a data point only adds linearly to storage space size rather than exponentially, since only the size of the data points is increased, not their number. Eventually mechanismsto coalesce and "forget" data points will be needed when the processors or memoryis Fully utilized. A simple solution to this problem is to store the time the data point was collected as part of the data point and discard data points on the basis of age. A more complex scheme could take into account the density of nearby data points, and combineredundant data points. It is also no longer necessary to organize storage for a particular set of indices. Different sets of indices can be selected by the processors from the same data point, allowing both forward and inverse transformations to be done with the same table. Hybrid Approaches Raibert took advantage of the structure of rigid body dynamics to reduce the size of the table needed to represent the inverse dynamics of a robot arm (Raibert 1977, 1978, Raibert & Horn 1978). The inertial forces are proportional to joint accelerations (note that 0"1 and 0"2 appear linearly in Eqs. 2 and 3), and the Coriolis and centripetal forces are quadratic in joint velocities (joint velocities only appear as the products 0102, 012, and 0~ in Eqs. 2 and 3), and therefore both joint accelerations and joint velocities can be removedas indices of the table at the expense of some additional computation. Local generalization and smoothing in the table was performed by using large table entries and interpolating across neighboring entries, and the table entries were updated by using a regression technique. Structured Approaches KINEMATIC MODEL BUILDING In robotics, kinematic calibration has been based on estimating parameters of model structures (see Whitney et al 1986, An et al 1988 for examples of such work and further references).

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

173

Typically, a model structure is chosen based on the assumptions of rigid links with joint rotation axes fixed relative to the links and linear joint sensors with the correct scale factor. Estimated parameters include link lengths, joint sensor offsets, relative orientations of consecutive axes, compliance in the transmission betweenthe joint position sensor and the rigid link, backlash, and gear transmission errors. The information used to calibrate the robot includes the joint sensor readings and measurements of the positions of various parts of the arm or just the tip of the arm in some external coordinate frame, Vision-based measurementsystems have been used as an external measurementsource. Often, the parameter estimation procedures are quite complex due to the nonlinear nature of the problem and the number of parameters. Biological limbs maypresent a more difficult calibration problem. Joints often have instantaneous axes of rotation that moverelative to the adjacent links as the joint surfaces roll or slide on each other (Alexander 1981). Calibrating joint angle or arm configuration measurementsbased on joint and muscle proprioceptive signals may be a problem (Burgess et al 1982, Mathews 1982). Given the complex model structure required to capture these effects, it maybe appropriate to explore table-based kinematic calibration methods (Kuperstein 1987, Jordan 1986). ESTIMATING LOAD PARAMETERS Methods to estimate the inertial parameters (mass, location of the center of mass, and the moments of inertia) of a rigid body load during general movement have been proposed (An et al 1988, Atkeson et al 1986, Olsen & Bekey 1985, Mukerjee 1984, Mukerjee & Ballard 1985). The rigid body dynamics modelstructure is linear in the unknowninertial parameters, as in Eqs. 2 and 3, makingthe parameters straightforward to estimate with regression techniques. Information needed to estimate the inertial parameters include forces and torques exerted by the arm on the load, and arm position, velocity, and acceleration. The estimation equations can be integrated to eliminate the need to measure or calculate arm acceleration. Estimation algorithms have been implementedon robots equipped with a wrist forcetorque sensor (Atkeson et al 1986). Goodestimates were obtained for load mass and center of mass, and the forces and torques due to the movement of the load could be predicted accurately. Load momentsof inertia were more difficult to estimate due to their small effect on the load forces and torques. Becausetheir effect is so small, however, it is not important to estimate these parameters accurately in order to predict the load forces and torques. DYNAMIC MODEL BUILDING:

ESTIMATING ARM PARAMETERS A method very similar to load inertial parameter estimation can be used to estimate the DYNAMIC MODEL BUILDING:

Annual Reviews www.annualreviews.org/aronline 174

ATKESON

inertial parameters of the segments of the arm (An et al 1988, Atkesorr et al 1986, Mukerjee 1984, Mukerjee & Ballard 1985, Olsen & Bekey 1985, Khosla 1986). Essentially, each ~egmentof the arm is treated as a load on the more proximal segments of the arm. The unknowninertial parameters of these arm links are mass, location of the center of mass, and moments of inertia. Again, since the rigid body dynamicsmodelstructure is linear in the unknowninertial parameters, these parameters can be estimated with regression techniques. The information used to estimate the parameters includes the torques at the joints of the arm, and the position, velocity, and acceleration of the arm (the position, velocity, and acceleration of each joint). The estimation equations can be integrated to eliminate the need to estimate arm or joint accelerations. The estimation algorithms have been implemented on a robot, and a good match was obtained between predicted joint torques from the estimated modeland actual joint torques (Atkeson et al 1986). The modelhas also been used for feedforward and computedtorque control of the robot (An et al 1987). CORRECTING

PERFORMANCE

ERRORS

Previous sections have dealt with how the motor apparatus could build models of both kinematic and dynamic transformations. Inevitably, the inverse transformations used to control the motor system will not be entirely correct. Errors in the inverse transformation will lead to incorrect commands, and performance errors. This section describes how performance errors can be used to improve future commands. A Kinematic

Example

Correcting kinematic errors is used as an initial example. Consider a vision system observing our idealized two-joint arm. A visual target is selected and represented in visual coordinates. The inverse kinematic transformation is used to computethe joint angles that will place the hand at the visual target. The arm is movedto those exact joint angles, but due to errors in the inverse kinematic transformation the hand is not exactly at the visual target. The vision system measures this performance error in visual coordinates rather than motor coordinates, and so feedback cannot be used directly to correct the error unless the transformation from visual to motor coordinates is known. The brain must transform the hand position error in visual coordinates (a performanceerror) into corrections the joint angles (a command correction). If a tabular representation of the inverse kinematic transformation is used, then performance errors can be corrected by simply adding the new data point of joint angles and observed hand position to the table. The

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

175

new data point should slightly modify the nearby mapping so that the predicted joint angles for the desired hand position will be changedin the correct direction. The performance error can be reduced without any explicit error detection or correction (Raibert 1978). There arc several possible problemswith the above scenario. If the error is too large, compared to the generalization generated by the smoothingdone by the table, the original transformation of desired hand position to joint angles may not be modified at all. The sameerror will be generated again. This problem is particularly acute whentables are initialized and have little or no correct data. The resolution of the table maynot be adequate to represent the inverse transformation accurately. In addition, the smoothing and temporal averaging done by the table mayreduce the effect of new data and slow downthe reduction of the error. If a structured modelis used to represent the inverse kinematic transformation, new parameter estimates can be made to improve the model for arm configurations similar to this one. The approach of reestimating parameters to correct for errors was proposed by Mukerjee (1984) for improving a model of the inverse dynamics transformation during practice of a movement.The use of data from a small range of inputs or states, however, can cause parameter estimation procedures to distort the parameter estimates in order to compensatefor structural modelingerrors. This can cause predictions for other ranges of data not used in the estimation to deteriorate. Normally,a wide range of inputs and states should be used for parameter estimation of structured models to achieve both a good predictive ability over a correspondingrange of outputs and the ability to generalize betweenquite different inputs. In order to use feedback directly to correct the error, the measurederror (in visual coordinates) must be transformed into the command coordinate system (joint angles). The visual error can be transformed to joint angles by using the same inverse kinematic transformation we used to predict the initial set of joint angles. Although the represented inverse kinematic transformationis inaccurate, it is still the best guess as to howto interpret the performance error. This problem can be mathematically formulated: Initially the desired hand position Xd= (Xd, Yd) is transformed into joint angles 0 = (01, 02) the inverse kinematic transformation IK() given by Eqs. 4 and 5. 0 = IK(xa).

6.

Unfortunately, when the arm is movedto these joint angles the actual hand position x = (x, y) as measuredby the vision system is different from the desired hand position due to inaccuracies in the inverse kinematic transformation IK(). One way to reduce the hand position error is to

Annual Reviews www.annualreviews.org/aronline 176

ATKESON

correct the commandedjoint angles by mapping the perceived hand position error into joint coordinates. The hand position error is mappedby using the inverse kinematic transformation to transform the actual and desired hand positions to joint coordinates and then subtract them to estimate the error in joint coordinates: estimated error in joint coordinates = IK(x)--IK(x~).

7.

This estimated error is then used to correct the commandedjoint angles by simply subtracting the error: 0correcte

d = 0 --

[IK(x)-- IK(xa)].

8.

Since thc inverse kinematic transformation is not entirely accuratc, the error will not be reduced completely. The same procedure can be repeated to reduce the error further. The important point is that in order to use feedback to correct errors, perceived errors must be transformed into the appropriate commandcoordinate system, and the same inverse transformations that were used to generate the initial command can be used to correct that command. Learning

from Practicing

a Movement

The problem of transforming performance errors into commandcorrections is present whena particular movementis learned by practicing that movement.Performanceerrors in terms of incorrect positions, velocities, and accelerations of the arm must be used to correct commandsthat are in terms of joint torques or muscle activations. The dynamic performance errors can be mappedto commandcorrections by using an internal model of the inverse dynamics of the motor apparatus, muchas an internal model of the inverse kinematics was used to correct positioning errors. In robotics, recent work in a number of laboratories has focused on how to refine feedforward commandsfor repetitive movements on the basis of previous movementerrors. Workon repeated trajectory learning includes that of Arimoto et al (1984, 1985), Casalino & Gambardella (1986), Craig (1984), Furuta & Yamakita(1986), Hara et al (1985), kopos (1986), Mita & Kato (1985), Morita (1986), Togai & (1986), Uchiyama (1978), Wang(1984), and Wang& Horowitz (1985). These papers discuss only linear learning operators and emphasize the stability of the proposed algorithms. Little work has emphasized performance,i.e. the convergencerate of the algorithm. Simulations of several of these algorithms have revealed very slow convergence and large sensitivity to disturbances and sensor and actuator noise (C. G. Atkeson, unpublished results). In a paper by Atkeson & Mclntyre (1986a,b), an explicit model of the

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

177

nonlinear robot inverse dynamics was used to transform performance errors into commandcorrections. This technique enabled a robot to improveits ability to follow a desired trajectory by practicing a movement. Figure 4 shows the performance of such a learning scheme when implementedon a three-joint direct drive robot arm. Figure 4A shows the initial feedforwardtorques applied to the robot for each of the three joints. Joint 1 Feedfonvard Torque

Joint2

Joint 3

A. Initial

movementfeedforward torques

C. Feedforward torques after 3 practice movements

05seconds

Joint 1 Veloc~Error

Joint2

Joint3

B. Initial

movementvelocity errors

D. Velocity errors after 3 practice movements

Figure 4 Learning from practice

in a three-joint

robot arm.

Annual Reviews www.annualreviews.org/aronline 178

ATKESON

These torques were calculated by applying a model of the inverse dynamics of the robot 1D() to the desired trajectory joint positions Od(t), joint velocities 0~(t), and joint accelerations 0~j(t): z~(t) = ID[Od(t),Od(O,0"d(t)]

9.

In addition to the feedforward torques, the robot controller used feedback control to generate torques (~) to compensate for observed trajectory errors. The total commanded joint torques were the sum of the feedforward and feedback torques: ¯ (t) = ~f,-(t) + ~b(t).

10.

Figure 4B showsthe resulting velocity errors for each joint in attempting to follow the desired trajectory. The model of the robot inverse dynamics was used to transform the trajectory performance errors into corrections to the torque commandsto the robot, just as kinematic errors were corrected by using the inverse kinematic model of the arm: ¯ f~c......d(t) =,(t) -- {ID[O(t),O(t), O’(t)]- ID[Od(t),}.

11.

Figure 4C shows the corrected feedforward torques for each of the joints after three practice movements,and Figure 4D shows the greatly reduced velocity errors that result. The robot learned to improve its trajectory following performance on this trajectory by practicing a movementand observing and correcting its own errors. One drawback of this approach is that it only produces the appropriate command for a single trajectory. Howto modify that commandfor similar trajectories is currently under investigation. Motor control schemes that do not include an internal model of the dynamics of the motor apparatus, such as the equilibrium trajectory approach (Berkinblit et al 1986, Hoganet al 1987), do not support efficient movement refinement during practice (C. G. Atkeson, unpublished results). There is a great deal of evidence from research on adaptive human manual control that performance improvements in compensatory tracking tasks are due to improvements of an internal model of the dynamics of the controlled system (Sheridan &Ferrell 1974). Additionally, a source great frustration to designers of motor psychophysics experiments is their subjects’ ability to build internal predictive modelsof target movementin pursuit tracking tasks (see Rouse1980 for further references). Recent work in oculomotor control has suggested that an internal model of the motor apparatus is used to maintain accurate performance (Optican & Miles 1985).

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING

179

SUMMARY In this review I have discussed how the form of representation used in internal models of the motor apparatus affects howand what a system can learn. Tabular models and structured models have benefits and drawbacks. Structured modelsincorporate knowledgeof the structure of the controlled motorapparatus. If that knowledgeis correct, or close to the actual system structure, the structured models will support global generalization and rapid, efficient learning. Tabular models can play an important role in learning to control systems wheneither the system structure is not known or only knownapproximately. Tabular models are general and flexible. Techniques for combining these different representations to attain the benefits of both are currently under investigation. In the control of multijoint systems such as the humanarm, internal models of the motor apparatus are necessary to interpret performance errors. In the study of movementsrestricted to one joint, the problem of interpreting performanceerrors is greatly simplified and often overlooked, as performance errors can usually be related to commandcorrections by a single gain. Whenmultijoint movementsof the same motor systems are examined, however, the complex nature of the control and coordination problems faced by the nervous system become evident, as well as the sophistication of the brain’s solutions to these problems. Recent progress in the understanding of adaptive control of eye movementsprovides a good exampleof this (Berthoz &Melvill-Jones 1985). Experimental studies of the psychophysics of motor learning can play an important role in bridging the gap between computational theories of how abstract motor systems might learn and physiological exploration of how actual nervous systems implement learning. Quantitative analyses of the patterns of motor learning of biological systems mayhelp distinguish alternative hypotheses about the representations used for motor control and learning. What a system can and cannot learn, the amountof generalization, and the rate of learning give clues as to the underlying performance architecture. It is also important to knowthe actual performancelevel of the motor system (Loeb 1983). Different proposed control strategies will be able to attain different performance levels, and the use of simplifying control strategies maybe evident in the control and learning performance of motor systems. ACKNOWLEDGMENTS I thank the manycolleagues whocommentedon drafts of this article. This article describes research done at the Center for Biological Information

Annual Reviews www.annualreviews.org/aronline 180 Processing

ATKESON (Whitaker

College)

and the Artificial

Intelligence

Laboratory

at the Massachusetts Institute of Technology. Support for this research is provided by a grant from the Whitaker Health Sciences Fund. Support for the A. I. Laboratory’s research is provided in part by the Office of

Naval Research University ResearchInitiative Programunder Office of Naval Research contract N00014-86-K-0685, and the AdvancedResearch Projects Research

Agency of the Department contract N00014-85-K-0124.

of Defense

under

Office

of Naval

Literature Cited Adams, J. A. 1984. Learning of movement Arimoto, S., Kawamura,S., Miyazaki, F., sequences. PsychoL Bull. 96:3-28 Tamaki, S. 1985. Learning control theory for dynamical systems. Proc. 24th Conf Adams, J. A. 1987. Historical review and appraisal of research on the learning, on Decision Control, Fort Lauderdale, Fla., Dec. 11-13, pp. 1375-80 retention, and transfer of humanmotor skills. Psychol. Bull. 101:41-74 Atkeson, C. G., An, C. H., Hollerbach, J. Albus, J. S. 1971. Atheory of cerebellar funcM. 1986. Estimation of inertial parameters tion. Math. Biosci. 10:25-61 of manipulator loads and links. Int. J. Albus, J. S. 1975a. A new approach of Robot. Res. 5:101 19 manipulator control: The cerebellar mo- Atkeson, C. G., Mclntyre, J. 1986a. Robot trajectory learning through practice. IEEE del articulation controller (CMAC). DynamicS, vst. Measure. Control 97: 220Conf. on Robot. Automat., San Francisco, Calif, April 7-10, pp. 1737-42 27 Albus, J. S. 1975b. Data storage in the cereAtkeson, C. G., McIntyre, J. 1986b. Applibellar model articulation controller cations of adaptive feedforward control in robotics. Proc. 2nd IFAC Worskhop on (CMAC)..I. Dynamic Syst. Measure. Control 97:228 33 Adaptit~e Systems in Control Signal Process., Lund, Sweden, July 1-3, pp. 137-42 Albus, J. S. 1981. Brains, Behavior, and Robotics. Peterborough, NH: Byte Barto, A. G., Sutton, R. $., Anderson, C. Alexander, R. M. 1981. Mechanics of skeleW. 1983. Neuronlike adaptive elements that can solve difficult learning control ton and tendons. In Handbook of Physiology, Section 1: The Nervous System, problems. IEEE Trans. on Systems, Man, Volume 2, Motor Control, Part 2, ed. Cybernet. SMC-13:834~16 V. B. Brooks, pp. 17 42. Bethesda, MD: Berkinblit, M. B., Feldman, A. G., Fukson, Am.Physiol. Sac. O. I. 1986. Adaptability of innate motor An, C. H., Atkeson, C. G., Griffiths, J. D., pattcrns and motor control mechanisms. Hollerbach, J. M. 1987. Experimental Behav. Brain Sci. 9:585-638 evaluation of feedforward and computed Berthoz, A., Melvill Jones, G. 1985. Adaptive torque control. IEEE Conf. on Robot. Mechanisms in Gaze Control. NewYork, Automat., Raleigh, NC, March 30-April NY:Elsevier 3, pp. 165-68 Bizzi, E., Dev, P., Morasso, P., Polit, A. An, C. H., Atkeson, C. G., Hollerbach, J. 1978. Effect of load disturbances during centrally initiated movements.J. NeuroM. 1988. Model-Based Control of a Robot Manipulator. Cambridge, Mass: MIT physiol. 41:542-56 Press Burgess, P. R., Wei, J. Y., Clark, F. J., Annett, J. 1985. Motorlearning: A review. In Simon, J. 1982. Signaling of kinesthetic Motor Behavior: Programming, Control, information by peripheral sensory recepand Acquisition, ed. H. Heuer, U. Kleintors. Ann. Rev. Neurosei. 5:189-218 beck, K. H. Schmidt, pp. 189-212. New Casalino, G., Gambardella, L. 1986. LearnYork: Springer-Verlag ing of movements in robotic manipuArbib, M. A., Amari, S. I. 1985. Sensorilators. See Atkeson & McIntyre 1986a, pp. 572-78 motor transformations in the brain (with a critique of the tensor theory of cerebelCraig, J. J. 1984. Adaptive control of manipulators through repeated trials. lum). J. Theoret. Biol. 112:123-55 Arimoto, S., Kawamura, S., Miyazaki, F. Proc. Am. Control Conf, San Dieffo, June 1984. Bettering operation of robots by 6~8, pp. 1566-74 learning. J. Robot. Syst. l: 123-40 El-Zorkany, H. I., Liscano, R., Tondue, B.

Annual Reviews www.annualreviews.org/aronline MOTOR LEARNING 1985. A sensor based approach for robot programming. Proc. SPIE Conf. on Intelligent Robots and Comput.Vision, pp. 28997 Ersu, E., Tolle, H. 1984. A new concept for learning control inspired by brain theory. IFAC 9th Worm Congr., Budapest, Hungary, July 2~6, pp. 1039-44 Feldman, A. G. 1986. Once more on the equilibrium-point hypothesis (2 model) for motor control. J. Motor Behav. 18: 1754 Fujita, M. 1982a. Adaptive filter model of the cerebellum. Biol. Cybernet. 45: 195206 Fujita, M. 1982b. Simulation of adaptive modification of the vestibulo-ocular reflex with an adaptive filter modelof the cerebellum. Biol. Cybernet. 45:207-14 Furuta, K,, Yamakita, M. 1986. Iterative generation of optimal input of a manipulator. See Atkeson & Mclntyre 1986a, pp. 579-84 Georgopoulos, A. P. 1986. On reaching. Ann. Rev. Neurosci. 9:147-70 Gielen, C. C. A. M., van Zuylen, E. J. 1986. Coordination of arm muscles during flexion and supination: Application of the tensor analysis approach. Neurosci. 17: 527-39 Grossberg, S. 1969. On learning of spatiotemporal patterns by networks with ordered sensory and motor components. 1: Excitatory components of the cerebellum. Studies Applied Math. 48:105 32 Grossberg, S., Kuperstcin, M. 1986. Neural dynamics of adaptive sensotT-motor control. Amsterdam: North-Holland Hara, S., Omata, T., Nakano, M. 1985. Synthesis of repetitive control systems and its apptcation. See Arimoto et al 1985, pp. 1387-92 Harokopos, E. G. 1986, Optimal learning control of mechanical manipulators in repetitive motions. See Atkeson & McIntyre 1986a, pp. 396-401 Hildreth, E. C., Hollerbach, J. M. 1985. The computational approach to vision and motor control. AI Memo846, MIT A.I. Lab., Cambridge, Mass. Hildreth, E. C., Koch, C. 1987. The analysis of visual motion: From computational theory to neuronal mechanisms. Ann. Rev, Neurosci. 10:477-533 Hillis, W. D. 1985. The Connection Machine, Cambridge: MIT Press Hinton, G. E. 1984. Parallel computations for controlling an arm. J. Motor Behav. 16:171-94 Hinton, G. E. 1986. Learning in massively parallel nets. AAAI-86: 5th Natl. Conf on Artificial Intelligence, Philadelphia,August 11-15, p. 1149

181

Hollerbach, J. M. 1982. Computers, brains, and the control of movemcnt. Trends Neurosci. 5:189-92 Hollerbach, J. M., Atkeson, C. G. 1987. Deducing planning variables from experimentalarm trajectories: Pitfalls and possibilities. Biol. Cybernet. 56:279-92 Hogan, N., Bizzi, E., Mussa-Ivaldi, F. A., Flash, T. 1987. Controlling multi-joint motor behavior. Exercise Sport Sci. Rev. 15:153-90 Houk, J. C., Rymer, W. Z. 1981. Neural control of muscle length and tension. See Alexander 1981, pp. 257-324 Jordan, M. I. 1986. Serial order: Aparallel, distributed processing approach. Rep. No. 8604, Inst. Cognitive Sci., Univ. Calif., San Diego Kawato, M., Furukawa, K., Suzuki, R. 1987. A hierarchical neural-network model for control and learning of voluntary movement.Biol. Cybern. 57:169 85 Khosla, P. K. 1986. Estimation of robot dynamics parameters: Theory and application. 2rid Int. lASTEDConf on Appl. Control and Identification, Los Angeles, Dec. Knudsen, E. 1., du Lac, S., Esterly, S. D. 1987. Computational maps in the brain. Ann, Rev, Neurosci. 10:41-65 Kohonen, T. 1980. Content-Addressable Memories. NewYork: Springer-Verlag Kuperstein, M. 1987. Adaptive visual-motor coordination in multijoint robots using parallel architecture. See Anet al 1987, pp. 1595-1602 Loeb, G. E. 1983. Finding commonground between robotics and physiology. Trends Neurosci. 6:203-4 Mart, D. 1969. A theory ofcerebellar cortex. J. Physiol. London 202:437 70 Marr, D. 1982. Vision. San Francisco: Freeman Matthews, P. B. C. 1982. Where does Sherrington’s "muscular sense" originate? Muscles, joints, corollary discharges? Ann. Re~. Neurosci. 5:189-218 Michie, D., Chambers, R. A. 1968. Boxes: An experiment in adaptive control. In Machine Intelligence 2, ed. E. Dale, D. Michie, pp. 137-52. NewYork: Elsevier Miller, W. T. 1986. Anonlinear learning controller for robotic manipulators. SPIE5th Conf on Intelligent Robots and Computer Vision, Cambridge, Mass., Oct. 28-31 Miller, W. T., Glanz, F, H., Kraft, L. G. 1987. Application of a general learning algorithm to the control of robotic manipulators. Int. J. Robot. Res. 6:84-98 Minsky, M., Papert, S. 1969. Perceptrons: An Introduction to Comtmtational Geometry. Cambridge, MIT Press Mira, T., Kato, E. 1985. lterative control

Annual Reviews www.annualreviews.org/aronline 182

ATKESON

and its application to motion control of lar control of balance in a dynamiclegged robot arm--a direct approach to servosystem. IEEE Trans. on Systems, Man, problems. See Arimoto et al 1985, pp. Cybernet. SMC-14:2:334-39 1393-98 Robinson, D. A. 1981. The use of control systems analysis in the neurophysiology of Miura, H., Shimoyama, I. 1980. Computer control of an unstable mechanism (in eye movements. Ann. Rev. Neurosci. 4: Japanese). J. Fac. Eng. Univ. Tokyo 17: 463-503 Rosenblatt, F. 1961. Principles of Neuro12-13 Morita, A. 1986. A study of learning condynamics: Perceptrons and the Theory of trollers for robot manipulators with sparse Brain Mechanisms. Washington, DC: data. MSthesis, Mechanical Engineering, Spartan Cambridge, MIT Rouse, W. B. 1980. Systems Engineering Mukerjee, A. 1984. Adaptation in biological Models qf Human-Machine Interaction. sensory-motor systems: A model for New York: North-Holland robotic control. Vol. 521, Proc., SPIE Rumelhart, D. E., McClelland, J. L., PDP Research Group. 1986. Parallel DisConf. on Intelligent Robots and Computer Vision, SPIE, Cambridge, November tributed Processing: Explorations in the Mukerjee, A., Ballard, D. H. 1985. Self-caliMicrostructure of Cognition, Vol. 1, Founbration in robot manipulators. Proc. dations. Cambridge: MITPress 1EEE Conf. Robot. Automat., St. Louis, Salmoni, A. W., Schmidt, R. A., Walter, C. Mar. 25-28, pp. 105(~57 B. 1984. Knowledgeof results and motor Olsen, H. B., Bekey, G. A. 1985. Identilearning: Areviewand critical reappraisal. fication of parameters in modelsof robots Psychol. Bull. 95:355-86 with rotary joints. See Mukerjee& Ballard Saltzman, E. 1979. Levels of sensorimotor 1985, pp. 1045-50 representation. J. Math. Psychol. 20: 91Optican, L. M., Miles, F. A. 1985. Visually 163 induced adaptive changes in primate sacSchmidt, R. A. 1982. Motor Control and cadic oculomotor control signals. J. Learning. Champaign, II1: HumanKineNeurophysiol, 54:940-58 tics Publ. Pellionisz, A., Llinas, R. 1979. Brain model- Selfridge, O. G., Sutton, R. S., Barto, A. G. ing by tensor network theory and com1985. Training and tracking in robotics. puter simulation. The cerebellu~n: DisProc. IJCALLos Angeles, Aug. 18-23, pp. tributed processor for predictive co670-72 ordination. Neuroscience 4:323~48 Sheridan, T. B., Ferrell, W. R. 1974. ManPellionisz, A., Llinas, R. 1980. Tensorial Machine Systems. Cambridge: MIT Press approach to the geometry of brain funcSimons, J., VanBrussel, H., De Schotter, J., Verhaert, J. 1982. A self-learning autotion: Cercbellar coordination via a metric tensor. Neuroscience 5:1125 36 maton with variable resolution for high Pellionisz, A., Llinas, R. 1982. Space-time precision assembly by industrial robots. representation in the brain: The cereIEEE Trans. on Automatic Control ACbellum as a predictive space-time metric 27:1109 13 tensor. Neuroscience 7:2949 70 Smyth, M. M., Wing, A. W., eds. 1984. The Pellionisz, A., Llinas, R. 1985. Tensor netPsychology of Human Movement. New work theory of the metaorganization of York: Academic functional geometries in the central nerSoechting, J. F., Ross, B. 1984. Psychophysical determination of coordinate repvous system. Neuroscience 16:245-73 Poggio, G. F., Poggio, T. 1984. The analysis resentation of human arm orientation. of stereopsis. Ann. Rev. Neurosci. 7: 379Neuroseienee 13:595~504 412 Togai, M., Yamano,O. 1986. Learning conRaibert, M. H. 1977. Analytical equations trol and its optimality: Analysis ~nd its vs. table look-up for manipulation: A uniapplication to controlling industrial rofying concept. Proc. IEEE Conf Decision bots. See Atkeson & McIntyre 1986a, pp. Control, NewOrleans, Dec., pp. 576-79 248 53 Uchiyama, M. 1978. Formation of highRaibert, M. H. 1978. A model for sensorimotor control and learning. Biol. speed motion pattern of a mechanical arm Cybernet, 29:29-36 by trial. Trans Soc. Instrum. Control Eng. (Jpn.) 19:706-12 Raibert, M. H. 1986. Legged Robots That Balance. Cambridge: MIT Press Ullman, S. 1986. Artificial intelligence and the brain: Computational studies of the Raibert, M. H., Horn, B. K. P. 1978. Manipulator control using the Convisual system. Ann, Rev. Neurosci. 9:1-26 figuration Space Method.lndust. Robot 5: Wang,S. H. 1984. Computedreference error 69-73 adjustment technique (CREATE)for the Raibert, M. H., Wimberly, F. C. 1984. Tabucontrol of robot manipulators. 22nd Ann.

Annual Reviews www.annualreviews.org/aronline

MOTOR LEARNING

183

Allerton Conf. on Commun.,Control, Cornand Information Sciences, ed. J. T. Tou, put&9, Oct. R. H. Wilcox. Washington, DC: Clever Wang, S. H., Horowitz, I. 1985. CREATE-Hume Press a new adaptive technique. Proc. of the 19th Whitney, D. E., Lozinski, C. A., Rourke, J. Ann. Conf. on Inform. Sci. Systems, March M. 1986. Industrial robot forward caliWidrow, B., Smith, F. W. 1964. Pattern bration methodand results. J. Dyn. Syst. recognizing control systems. In Computer Meas. Control 108:1-8