A mechanical failure is defined by the author as a failure in one or more mechanical aspects of the system. This can be a component breaking or yielding (deforming such that it does not return to its original shape), a mechanical jam, or a physical disconnection which is

A computer failure may occur through a software crash, electrical short or disconnection, a software bug or error, memory corruption, or computational insufficiency, as in Figure 9(e).

Ultimately, many failures can be detected and 'handled' in such a way that potentially devastating effects are mitigated. It may be convenient to include as a safety device a redundant low level microprocessor-based 'watchdog' which is independent of the standard system, as in Figure 10. It is connected to all the sensor information and actuators, and has a programmed set of boundaries to maintain. Its job is to oversee all the components of the system and ensure they are functioning. The watchdog and main computer system ping each other frequently in a simple way such as with a pulse train. If there is a failure in either system component (main or watchdog) the system, depending on its function, may be shut down. The watchdog may be connected to the actuators such that it may disable the power output and engage some braking action if desired. It is also generally a high bandwidth system (since it is not performing many computations). If the system is mission-critical, such as in a military application or in a flight control or other situation where it may be dangerous for the system to be suddenly disabled entirely, it may be more effective to determine the safest course of action which still achieves the objective. In the case of a walking robot, where a Case 1-4 failure has occurred in the leg, it may still be possible to walk. Consider humans with an injured joint - they can often offload much of the dynamic load to anther leg or joint. It is simply a matter of an adaptive control algorithm which recomputes a solution for a different system configuration. Consider sending a robot into a situation where it will be gradually melting - a radioactive disaster or extreme conditions. The robot needs to be capable of adapting to loss of actuators, sensors, and some computational capabilities. An algorithm such as discussed

Any of these can lead to the control system not acting as designed.

brake directly.

*Case 3: Mechanical failure*

undesirable, as in Figure 9(d). *Case 4: Computer or electronic failure*

*Achieving the goal regardless*

Case 1.

Fig. 9. (a) A typical engineered control system. It consists of a mechanical structure, sensors, actuators, power input/supply, and some computer system. (b)-(e) Some examples of failures of the typical system, and how they can be classified into one of the four cases (power supply omitted for clarity). (b) Case 1 : sensor failure - a sensor stops functioning properly. (c) Case 2 : actuator failure - in this case an actuator breaks, but it could be that it has an internal electrical failure. (d) Case 3 : mechanical failure - here we see that a main mechanical component has dislocated completely. (e) Case 4 : Computer failure - here it seems that the computer has been destroyed, but this type of failure also can be a software crash.

to function abnormally (a wire breaks, it wears out, shock destroys it, a bad ground, etc). The robot control algorithm assumes it is still being fed sensor information, and acts accordingly. This will likely cause a disastrous acceleration until a boundary is struck with maximum force, and the user may be injured.

In the case of a haptic system, it is better to have a motor turn off rather than accelerate uncontrollably, however how does one integrate available information to react properly? There actually is information available to tell us that there is a hardware failure, even if the sensor failure itself is literally undetectable.

### *Case 2: Actuator failure*

There are a number of ways an actuator can fail (Figure 9(c)). An electrical actuator may fail such that it becomes passive, unresponsive, sluggish, or lacking in power output. It may also lock or jam due to some component failure. The internal circuit of the actuator may fail such that it shorts (which tends to make the actuator act as a viscous brake) or has an open circuit, which leads to a non-responsive, passive actuator. A hydraulic actuator may lock, or it may leak fluid which leads to pressure loss and either stiffness or loss of power, depending on where the leak occurs. It may also suffer a compressor failure (which may lead to a total failure of the actuator) or component wear (which may lead to inefficiency or lack of functionality as well). Pneumatic actuators may experience a lock-up, but due to gas compressibility this is less common. Often the failure of pneumatics lead to rapid functionality loss as the gas 18 TBD

(d) (e)

Fig. 9. (a) A typical engineered control system. It consists of a mechanical structure, sensors, actuators, power input/supply, and some computer system. (b)-(e) Some examples of failures of the typical system, and how they can be classified into one of the four cases (power supply omitted for clarity). (b) Case 1 : sensor failure - a sensor stops functioning properly. (c) Case 2 : actuator failure - in this case an actuator breaks, but it could be that it has an internal electrical failure. (d) Case 3 : mechanical failure - here we see that a main mechanical component has dislocated completely. (e) Case 4 : Computer failure - here it seems that the

to function abnormally (a wire breaks, it wears out, shock destroys it, a bad ground, etc). The robot control algorithm assumes it is still being fed sensor information, and acts accordingly. This will likely cause a disastrous acceleration until a boundary is struck with maximum force,

In the case of a haptic system, it is better to have a motor turn off rather than accelerate uncontrollably, however how does one integrate available information to react properly? There actually is information available to tell us that there is a hardware failure, even if the

There are a number of ways an actuator can fail (Figure 9(c)). An electrical actuator may fail such that it becomes passive, unresponsive, sluggish, or lacking in power output. It may also lock or jam due to some component failure. The internal circuit of the actuator may fail such that it shorts (which tends to make the actuator act as a viscous brake) or has an open circuit, which leads to a non-responsive, passive actuator. A hydraulic actuator may lock, or it may leak fluid which leads to pressure loss and either stiffness or loss of power, depending on where the leak occurs. It may also suffer a compressor failure (which may lead to a total failure of the actuator) or component wear (which may lead to inefficiency or lack of functionality as well). Pneumatic actuators may experience a lock-up, but due to gas compressibility this is less common. Often the failure of pneumatics lead to rapid functionality loss as the gas

computer has been destroyed, but this type of failure also can be a software crash.

and the user may be injured.

*Case 2: Actuator failure*

sensor failure itself is literally undetectable.

(a) (b) (c)

Fig. 10. One possible safety configuration for a watchdog system. In this system there is a simple microprocessor overseeing all safety-critical functions of the main device - sensors, actuators, and boundaries. If anything is incorrect, it can disable the motors and enable a brake directly.

pressure loss may be more rapid than for a fluid-based system. Many pneumatic, electrical, and hydraulic systems include integrated sensors which may fail, as previously discussed in Case 1.

### *Case 3: Mechanical failure*

A mechanical failure is defined by the author as a failure in one or more mechanical aspects of the system. This can be a component breaking or yielding (deforming such that it does not return to its original shape), a mechanical jam, or a physical disconnection which is undesirable, as in Figure 9(d).

### *Case 4: Computer or electronic failure*

A computer failure may occur through a software crash, electrical short or disconnection, a software bug or error, memory corruption, or computational insufficiency, as in Figure 9(e). Any of these can lead to the control system not acting as designed.

### *Achieving the goal regardless*

Ultimately, many failures can be detected and 'handled' in such a way that potentially devastating effects are mitigated. It may be convenient to include as a safety device a redundant low level microprocessor-based 'watchdog' which is independent of the standard system, as in Figure 10. It is connected to all the sensor information and actuators, and has a programmed set of boundaries to maintain. Its job is to oversee all the components of the system and ensure they are functioning. The watchdog and main computer system ping each other frequently in a simple way such as with a pulse train. If there is a failure in either system component (main or watchdog) the system, depending on its function, may be shut down. The watchdog may be connected to the actuators such that it may disable the power output and engage some braking action if desired. It is also generally a high bandwidth system (since it is not performing many computations). If the system is mission-critical, such as in a military application or in a flight control or other situation where it may be dangerous for the system to be suddenly disabled entirely, it may be more effective to determine the safest course of action which still achieves the objective. In the case of a walking robot, where a Case 1-4 failure has occurred in the leg, it may still be possible to walk. Consider humans with an injured joint - they can often offload much of the dynamic load to anther leg or joint. It is simply a matter of an adaptive control algorithm which recomputes a solution for a different system configuration. Consider sending a robot into a situation where it will be gradually melting - a radioactive disaster or extreme conditions. The robot needs to be capable of adapting to loss of actuators, sensors, and some computational capabilities. An algorithm such as discussed

Fig. 12. Animation sequence of three manipulators tracking a rotating stationary trajectory (Frames advance starting from top-left in sequence to the right in the same fashion as normal english text). Note the multiple contact breaks in order to allow the object to be continuously

Real-Time Control in Robotic Systems 229

depend upon others, Figure 11 suggests a sequence which is typically effective. Each component is interdependent with each other component, however an effective starting point is to define the problem, or class of problems first. This gives a sense of the requirements for timing. Then the designer can compute required bandwidth and dimensionality of the system (as well as constraining the design by the available computational power), or generate something like the sensorimotor system of a human, with a high mechanical dimensionality, but limitations on overall bandwidth and dimensionality of control, leading to optimal actions for a particular task. At each stage of the process, safety engineers or the primary designer can design checks such as a secondary safety system and impose safety-based constraints. Real-time control itself, in order to be most effective, should not be decoupled from the design of the actual system. Otherwise the system may be unnecessarily limited due to designed-in bottlenecks. However, being aware of the concepts presented in this chapter, a control engineer can effectively maximize the capabilities of a system which already exists

We begin with a description of the problem, originally described in (Simpkins et al., 2011), and discussed previously in Section 3.4.1. In order to study biological systems, and how they interact with their environment through manipulation and locomotion, it is useful to have a robotic device which mimics certain key aspects of biology (those systems capable of manipulation and dextrous locomotion), such as output-to-actuator backdrivability1, low endpoint inertia, high dynamic capabilities, and sensitivity. After carefully determining the requirements for manipulation and locomotion, the author and collaborators drew upon biological systems in terms of timing and dynamic capability to define requirements. In addition, there was a desire to create a modular system which could ultimately be assembled quickly into a variety of configurations. There are a variety of requirements determined,

1 Application of force at the output can about a change of state at the input of a back-drivable system.

rotated.

and must merely be controlled.

**5. Results and application**

Fig. 11. Suggested flowchart for real-time system design. Note that safety is to be part of each stage of the design process. In addition, bandwidth, dimensionality, and timing are somewhat interdependent, which is the reason that feedback is incorporated in this chart. The first thing to start with as a question is 'what are the dynamics of interest?' This leads to the computations outlined in Section 3.1. Second, one must determine just how many degrees of freedom the system needs to perform all its tasks (Section 3.3), and how complex the control algorithms need to be (and can be, depending upon hardware available). This determines dimensionality, which affects the timing. Bandwidth limitations impose constraints on both dimensionality and timing (slower timing allows for higher dimensionality, and vice versa for faster timing).

in Section 3.4.1 can easily handle these sorts of challenges. Identification of the failure is a challenge. This can be achieved through the integration of the watchdog. A watchdog can be used to compare the measurements with expected outputs - for example a sensor with a zero voltage output which measures velocity of a motor normally, while a motor is being commanded to a high velocity, and is currently not drawing very much current (meaning it probably is not stalled), and has significant back-emf being generated can be a flag for a sensor or actuator failure. All the components of a system work together in unison, and the dynamics of a system express themselves through all the sensors and actuators. Expected values can be integrated into the watchdog in the form of a classifier which can output error states very quickly and efficiently, if well-trained and carefully coded. The watchdog and computer compare error signals, and can eliminate (from the control algorithm), freeze, or otherwise remove from or adjust the dynamics as much as possible to minimize failure impact. Then the control algorithm is simply updated with a new number of actuators or in the case of something similar to locking an injured joint, a new number of degrees of freedom, and it continues to solve the problem in the optimal fashion given the new system. Since everything is solved online this is possible.

## **4. Putting it all together**

Methods have been presented which, when combined, effectively deal with the real-time control problem for complex robotic systems. Since some of the components of the design 20 TBD

Fig. 11. Suggested flowchart for real-time system design. Note that safety is to be part of each

in Section 3.4.1 can easily handle these sorts of challenges. Identification of the failure is a challenge. This can be achieved through the integration of the watchdog. A watchdog can be used to compare the measurements with expected outputs - for example a sensor with a zero voltage output which measures velocity of a motor normally, while a motor is being commanded to a high velocity, and is currently not drawing very much current (meaning it probably is not stalled), and has significant back-emf being generated can be a flag for a sensor or actuator failure. All the components of a system work together in unison, and the dynamics of a system express themselves through all the sensors and actuators. Expected values can be integrated into the watchdog in the form of a classifier which can output error states very quickly and efficiently, if well-trained and carefully coded. The watchdog and computer compare error signals, and can eliminate (from the control algorithm), freeze, or otherwise remove from or adjust the dynamics as much as possible to minimize failure impact. Then the control algorithm is simply updated with a new number of actuators or in the case of something similar to locking an injured joint, a new number of degrees of freedom, and it continues to solve the problem in the optimal fashion given the new system. Since everything

Methods have been presented which, when combined, effectively deal with the real-time control problem for complex robotic systems. Since some of the components of the design

stage of the design process. In addition, bandwidth, dimensionality, and timing are somewhat interdependent, which is the reason that feedback is incorporated in this chart. The first thing to start with as a question is 'what are the dynamics of interest?' This leads to the computations outlined in Section 3.1. Second, one must determine just how many degrees of freedom the system needs to perform all its tasks (Section 3.3), and how complex the control algorithms need to be (and can be, depending upon hardware available). This determines dimensionality, which affects the timing. Bandwidth limitations impose constraints on both dimensionality and timing (slower timing allows for higher

dimensionality, and vice versa for faster timing).

is solved online this is possible.

**4. Putting it all together**

Fig. 12. Animation sequence of three manipulators tracking a rotating stationary trajectory (Frames advance starting from top-left in sequence to the right in the same fashion as normal english text). Note the multiple contact breaks in order to allow the object to be continuously rotated.

depend upon others, Figure 11 suggests a sequence which is typically effective. Each component is interdependent with each other component, however an effective starting point is to define the problem, or class of problems first. This gives a sense of the requirements for timing. Then the designer can compute required bandwidth and dimensionality of the system (as well as constraining the design by the available computational power), or generate something like the sensorimotor system of a human, with a high mechanical dimensionality, but limitations on overall bandwidth and dimensionality of control, leading to optimal actions for a particular task. At each stage of the process, safety engineers or the primary designer can design checks such as a secondary safety system and impose safety-based constraints. Real-time control itself, in order to be most effective, should not be decoupled from the design of the actual system. Otherwise the system may be unnecessarily limited due to designed-in bottlenecks. However, being aware of the concepts presented in this chapter, a control engineer can effectively maximize the capabilities of a system which already exists and must merely be controlled.

## **5. Results and application**

We begin with a description of the problem, originally described in (Simpkins et al., 2011), and discussed previously in Section 3.4.1. In order to study biological systems, and how they interact with their environment through manipulation and locomotion, it is useful to have a robotic device which mimics certain key aspects of biology (those systems capable of manipulation and dextrous locomotion), such as output-to-actuator backdrivability1, low endpoint inertia, high dynamic capabilities, and sensitivity. After carefully determining the requirements for manipulation and locomotion, the author and collaborators drew upon biological systems in terms of timing and dynamic capability to define requirements. In addition, there was a desire to create a modular system which could ultimately be assembled quickly into a variety of configurations. There are a variety of requirements determined,

1 Application of force at the output can about a change of state at the input of a back-drivable system.

Fig. 14. Image sequence of a bio-mimetic robotic finger designed by the author undergoing real-time control as described in this chapter. The finger is tracking a cyclical shape in 3-Dimensional space at approximately a 10Hz frequency. The robot is controlled wirelessly, through bluetooth, and is communicating 9x12bit values for position, velocity, and coil current at 1kHz, and assuming it is receiving the same amount of data as it is sending (as a worst-case), required bandwidth is 9x12x1000x2 = 216,000 bits per second, or 216kbps. The bandwidth of the connection is 2e6bps, or 2Mbps, which is nearly 10 times greater. In this case the algorithm controlling the finger is not computing any contacts, and so executes in an

Real-Time Control in Robotic Systems 231

This simulation runs entirely in real-time, with no offline components. With code executing in Matlab R2008b for Macintosh upon an Intel Core 2 Duo processor at 2.33GHz with 2GB of RAM, each loop takes on average 30msec, as measured from within Matlab (See Matlab help on tic-toc), which is fast enough for real-time manipulation. Human beings have an update rate perceptually of approximately 30Hz, and dynamically of only a few Hz (Nigg & Herzog, 1994). Additionally, it would be straightforward to optimize this code to execute far faster by

The fully implemented control algorithm including hardware executes in approximately the same time, as the hardware is provided the command computed by the simulation, and the actual object states are fed back into the system to adjust for the difference between simulation and reality. Essentially including the hardware in the loop in parallel with the model adds a model reference control element to the hierarchical optimal control strategy. The majority of the processor time is spent on the contact dynamics computations, and could be further improved to minimize some calculations without changing the algorithm significantly. Therefore when some manipulators are not in contact (as in Figure 14), computation time is reduced dramatically. The 30Hz or even a 200Hz sample rate are well within the 1kHz bandwidth of the robot manipulator and wireless communication system. It is notable that, in Figure 13(b), the variability in timing is less than one sample, and so the state space representations described in Section 3.1 hold. In the case that the manipulator is tracking a trajectory, as in Figure 13(a), we see that this hierarchical strategy is capable of smooth rapid control of the manipulator. If the current Matlab implementation were to be used, and the system were required to perform manipulation at a faster rate than 30Hz, the concepts in Section 3.3 could be more explicitly included in the control strategy of Section 3.4.1 by adding a cost on the number of simultaneous manipulator control inputs. A lower dimensionality means the algorithm will execute faster and be capable of a higher bandwidth, while still

average of less than 1msec at the PC side.

implementing it in C, for example.

being able to increase dimensionality on demand.

Fig. 13. (a) The robot, controlled hierarchically from a PC running matlab and communicating over bluetooth, with a local low level PID controller, is capable of tracking a cyclical motion very rapidly. The timing of the loop for this plot is approximately 1msec, with some variability in timing. (b) In this plot, the loop (which here includes contact dynamics and a more complex simulation) was deliberately executed in a non-optimal fashion in order to vary the timing of the sampling and updates, for demonstration purposes. When such variability is handled properly, as described in this chapter, the system can be built to remain stable.

and a summary is a 1kHz maximum bandwidth for the control of five fingers through one bluetooth or USB connection, ability to apply approximately 1N of endpoint force, minimal friction and endpoint inertia, resolution above one-tenth of a degree, and measures of force in addition to position. These requirements pose a challenge as communicating position, force, and current for each joint of each finger for five fingers at 1kHz becomes 9000 variables per second, and at 12-bits of resolution, for five fingers, this becomes 540kbps. If the robot is receiving, for example, no more than three variables for command of either position, velocity, or force output, then the most data must be sent from the low to the higher levels, as the serial interface used can simultaneously send and receive data.

There have been a few biologically-based robots recently developed, however they all suffer from some limitation imposed by a lack of an integrative real-time approach such as discussed in this chapter. It is not unusual for a well-designed hand mechanism to have a maximum bandwidth of one-tenth of a Hz to move from open-to-closed. Consider attempting most daily tasks where every motion could be performed in no shorter time than ten seconds. Dynamic manipulation would indeed be impossible! We omit the discussion of the design itself here, shown in Figure 6(a), instead focusing on how to maximize the real-time capabilities.

Figure 12 demonstrates a simulation based upon the real-time concepts discussed in previous sections. This is an implementation of a real-time hierarchical optimal control of a number of manipulators tracking a reference trajectory, and can be applied to the physical robots. In this case the high level is a simple proportional-derivative controller tracking a 'rotate continuously' command. It is clear that the algorithm is capable of multiple contact breaks required in order to allow continuous rotation.2

2 Further results and animations can be found at the author's home page, http://casimpkinsjr.radiantdolphinpress.com/pages/papersv2.html. In addition, this is the location for all the author's publications, with most available for download.

22 TBD

Time (msec)

communicating over bluetooth, with a local low level PID controller, is capable of tracking a cyclical motion very rapidly. The timing of the loop for this plot is approximately 1msec, with some variability in timing. (b) In this plot, the loop (which here includes contact dynamics and a more complex simulation) was deliberately executed in a non-optimal fashion in order to vary the timing of the sampling and updates, for demonstration purposes. When such variability is handled properly, as described in this chapter, the system can be

and a summary is a 1kHz maximum bandwidth for the control of five fingers through one bluetooth or USB connection, ability to apply approximately 1N of endpoint force, minimal friction and endpoint inertia, resolution above one-tenth of a degree, and measures of force in addition to position. These requirements pose a challenge as communicating position, force, and current for each joint of each finger for five fingers at 1kHz becomes 9000 variables per second, and at 12-bits of resolution, for five fingers, this becomes 540kbps. If the robot is receiving, for example, no more than three variables for command of either position, velocity, or force output, then the most data must be sent from the low to the higher levels, as the serial

There have been a few biologically-based robots recently developed, however they all suffer from some limitation imposed by a lack of an integrative real-time approach such as discussed in this chapter. It is not unusual for a well-designed hand mechanism to have a maximum bandwidth of one-tenth of a Hz to move from open-to-closed. Consider attempting most daily tasks where every motion could be performed in no shorter time than ten seconds. Dynamic manipulation would indeed be impossible! We omit the discussion of the design itself here,

Figure 12 demonstrates a simulation based upon the real-time concepts discussed in previous sections. This is an implementation of a real-time hierarchical optimal control of a number of manipulators tracking a reference trajectory, and can be applied to the physical robots. In this case the high level is a simple proportional-derivative controller tracking a 'rotate continuously' command. It is clear that the algorithm is capable of multiple contact breaks

2 Further results and animations can be found at the author's home page, http://casimpkinsjr.radiantdolphinpress.com/pages/papersv2.html. In addition, this is the location

shown in Figure 6(a), instead focusing on how to maximize the real-time capabilities.

0.025 0.03 0.035 0.04 0.045 0.05 0.055

200 300 400 500 600

Sample

(b)

0 1

(a)

interface used can simultaneously send and receive data.

required in order to allow continuous rotation.2

for all the author's publications, with most available for download.

ϕ2

**Reference Actual**

Fig. 13. (a) The robot, controlled hierarchically from a PC running matlab and

0

built to remain stable.

1

ϕ1

Fig. 14. Image sequence of a bio-mimetic robotic finger designed by the author undergoing real-time control as described in this chapter. The finger is tracking a cyclical shape in 3-Dimensional space at approximately a 10Hz frequency. The robot is controlled wirelessly, through bluetooth, and is communicating 9x12bit values for position, velocity, and coil current at 1kHz, and assuming it is receiving the same amount of data as it is sending (as a worst-case), required bandwidth is 9x12x1000x2 = 216,000 bits per second, or 216kbps. The bandwidth of the connection is 2e6bps, or 2Mbps, which is nearly 10 times greater. In this case the algorithm controlling the finger is not computing any contacts, and so executes in an average of less than 1msec at the PC side.

This simulation runs entirely in real-time, with no offline components. With code executing in Matlab R2008b for Macintosh upon an Intel Core 2 Duo processor at 2.33GHz with 2GB of RAM, each loop takes on average 30msec, as measured from within Matlab (See Matlab help on tic-toc), which is fast enough for real-time manipulation. Human beings have an update rate perceptually of approximately 30Hz, and dynamically of only a few Hz (Nigg & Herzog, 1994). Additionally, it would be straightforward to optimize this code to execute far faster by implementing it in C, for example.

The fully implemented control algorithm including hardware executes in approximately the same time, as the hardware is provided the command computed by the simulation, and the actual object states are fed back into the system to adjust for the difference between simulation and reality. Essentially including the hardware in the loop in parallel with the model adds a model reference control element to the hierarchical optimal control strategy. The majority of the processor time is spent on the contact dynamics computations, and could be further improved to minimize some calculations without changing the algorithm significantly. Therefore when some manipulators are not in contact (as in Figure 14), computation time is reduced dramatically. The 30Hz or even a 200Hz sample rate are well within the 1kHz bandwidth of the robot manipulator and wireless communication system. It is notable that, in Figure 13(b), the variability in timing is less than one sample, and so the state space representations described in Section 3.1 hold. In the case that the manipulator is tracking a trajectory, as in Figure 13(a), we see that this hierarchical strategy is capable of smooth rapid control of the manipulator. If the current Matlab implementation were to be used, and the system were required to perform manipulation at a faster rate than 30Hz, the concepts in Section 3.3 could be more explicitly included in the control strategy of Section 3.4.1 by adding a cost on the number of simultaneous manipulator control inputs. A lower dimensionality means the algorithm will execute faster and be capable of a higher bandwidth, while still being able to increase dimensionality on demand.

Krstic, M., Kanellakopoulos, I. & Kokotovic, P. (1995). *Nonlinear and Adaptive Control Design*,

Real-Time Control in Robotic Systems 233

Li, W. & Todorov, E. (2004). Iterative linear quadratic regulator design for nonlinear bio-logical

Movellan, J., Tanaka, F., Fortenberry, B. & Aisaka, K. (2005). The rubi/qrio project: Origins,

Nilsson, J. (1998). *Real-time control systems with delays*, PhD thesis, Dept. of automatic control,

Puschner, P. & Nossal, R. (1998). Testing the results of static worst-case execution-time

Santello, M., Flanders, M. & Soechting, J. (1998). Postural hand synergies for tool use, *Journal*

Simpkins, A. (2009). *Exploratory studies of sensorimotor control and learning with system*

Simpkins, A., Kelley, M. & Todorov, E. (2011). Modular bio-mimetic robots that can interact

Simpkins, A. & Todorov, E. (2011). Complex object manipulation with hierarchical optimal

Todorov, E. & Ghahramani, Z. (2003). Unsupervised learning of sensory-motor primitives,

Todorov, E. & Ghahramani, Z. (2004). Analysis of synergies underlying complex hand

Todorov, E., Hu, C., Simpkins, A. & Movellan, J. (2010). Identification and control of a

Todorov, E. & Jordan, M. (2002). Optimal feedback control as a theory of motor coordination,

Todorov, E. & Jordan, M. (2003). A minimal intervention principle for coordinated movement,

Vandeweghe, J. M., M.Rogers, M.Weissert & Matsuoka, Y. (2004). The act hand: Design of the

Wilhelm, R., Engblom, J., Ermedahl, A., Holsti, N., Thesing, S., Whalley, D., Bernat, G.,

*Advances in Neural Information Processing Systems* **15**: 27–34.

Stengel, R. (1986). *Stochastic Optimal Control: Theory and Application*, John Wiley and Sons. Todorov, E. (2004). Optimality principles in sensorimotor control, *Nature Neuroscience*

Mardia, K., Kent, J. & Bibby, J. (1992). *Multivariate Analysis*, Academic Press.

Nigg, B. & Herzog, W. (1994). *Biomechanics*, John Wiley and Sons, New York.

movement systems, *Proc. of the 1st International Conference on Informatics in Control,*

principles, and first steps, *IEEE International Conference on Development and Learning*

analysis, *Proceedings of the 19th IEEE Real-Time Systems Symposium (RTSS98) Madrid*.

*identification and stochastic optimal control*, PhD thesis, University of California, San

with the world the way we do, *Proceedings of the IEEE International Conference on*

control, *IEEE Symposium of Adaptive Dynamic Programming and Reinforcement Learning,*

IEEE, *In Proceedings of the 25th Annual International Conference of the IEEE Engineering*

manipulation, IEEE, *Proceedings of the 26th Annual International Conference of the IEEE*

pneumatic robot, *Proc. of the fourth IEEE RAS / EMBS International Conference on*

skeletal structure, *Proc. of the IEEE International Conference on Robotics and Automation*.

Ferdinand, C., Heckmann, R., Mitra, T., Mueller, F., Puaut, I., Puschner, P., Staschulat,

John Wiley and Sons.

**1**: 80 – 86.

*Automation and Robotics* **1**: 222–229.

Lund institute of technology, Sweden.

*of Neuroscience* (18): 10105–15.

Diego, La Jolla, CA.

**7**: 907–915.

*Robotics and Automation* .

*IEEE Computer Society, Paris, France*.

*in Medicine and Biology Society*.

*EMBS, San Francisco, CA, USA*.

*Biomedical Robotics and Biomechatronics* .

*Nature Neuroscience* **5**(11): 1226–1235.

### **6. Conclusion**

Modern robotic systems are becoming more and more complex, and requiring far more sophisticated controllers to be designed and implemented than ever before in order to deal with the demand for them to perform more complex tasks. These changes have raised a new challenge for engineers and roboticists. This chapter has presented a first step toward a unifying theory which combines and expands upon modern real-time design control techniques. It broke the problem down into sub-problems which each have a significant body of literature, while describing the fact that these seemingly separate fields are in fact intertwined, are part of a whole, and complement each other. By being at least aware of this method of organizing the problem, and the potential pitfalls associated with the sub-problems, it is possible to improve the end result of a design by making revisions and requirements which are more appropriate far earlier in the process than otherwise possible.

A number of results were presented for the control strategy discussed, as well as an implementation on a real dynamic bio-mimetic robot. Currently, many of the Matlab functions associated with this robot are being implemented in C and made available as MEX files which Matlab can still make use of. This should improve the execution time significantly of the simulation, as well as all other aspects of the computations. These techniques will be part of the development of new skills for these dynamic manipulators. The two dimensional representation has been extended to three dimensions, and will be part of a series of papers to be released soon by the author.

These new challenges require a different overall strategy for problem solving than ever before in engineered systems, but have the potential to truly revolutionize all aspects of human experience. By developing and refining methodologies, we will transition from individual designs which take years of iterative slow improvements with many dead ends to clear paths with fewer roadblocks, and new tools to illuminate the way.

### **7. References**

Anderson, T. W. (2003). *An Introduction to Multivariate Statistical Analysis*, John Wiley and Sons. Åström, K. & Wittenmark, B. (1990). *Computer controlled systems - theory and design*, 2nd edn, Prentice Hall, Englewood cliffs, NJ.

24 TBD

Modern robotic systems are becoming more and more complex, and requiring far more sophisticated controllers to be designed and implemented than ever before in order to deal with the demand for them to perform more complex tasks. These changes have raised a new challenge for engineers and roboticists. This chapter has presented a first step toward a unifying theory which combines and expands upon modern real-time design control techniques. It broke the problem down into sub-problems which each have a significant body of literature, while describing the fact that these seemingly separate fields are in fact intertwined, are part of a whole, and complement each other. By being at least aware of this method of organizing the problem, and the potential pitfalls associated with the sub-problems, it is possible to improve the end result of a design by making revisions and requirements which are more appropriate far earlier in the process than otherwise possible. A number of results were presented for the control strategy discussed, as well as an implementation on a real dynamic bio-mimetic robot. Currently, many of the Matlab functions associated with this robot are being implemented in C and made available as MEX files which Matlab can still make use of. This should improve the execution time significantly of the simulation, as well as all other aspects of the computations. These techniques will be part of the development of new skills for these dynamic manipulators. The two dimensional representation has been extended to three dimensions, and will be part of a series of papers to

These new challenges require a different overall strategy for problem solving than ever before in engineered systems, but have the potential to truly revolutionize all aspects of human experience. By developing and refining methodologies, we will transition from individual designs which take years of iterative slow improvements with many dead ends to clear paths

Anderson, T. W. (2003). *An Introduction to Multivariate Statistical Analysis*, John Wiley and Sons. Åström, K. & Wittenmark, B. (1990). *Computer controlled systems - theory and design*, 2nd edn,

Butterfab, J. & et al. (2001). Dlr-hand ii: Next generation of a dextrous robot hand, *Proc. of the IEEE International Conference on Robotics and Automation* pp. 109–114. Erez, T., Todorov, E. & Tassa, Y. (2011). Fast model predictive control for complex robotic

Ermedahl, A. (2003). *A modular tool architecture for worst-case execution time analysis*, PhD thesis,

Ferziger, J. (1998). *Numerical Methods for Engineering Application*, 2nd edition edn, John Wiley

Franklin, G. F. & et al. (1994). *Feedback control of dynamic systems*, 3rd edn, Addison Wesley,

Grob, H.-G. (2001). A prediction system for evolutionary testability applied to dynamic execution time analysis, *Information and Software Technology* (43): 855–862. Hansen, J., Hissam, S. & Moreno, G. (2009). Statistical-based wcet estimation and validation, *9th International Workshop on Worst-Case Execution Time (WCET) Analysis* .

Bernstein, N. (1967). *The coordination and regulation of movements*, Pergamon, London.

**6. Conclusion**

be released soon by the author.

**7. References**

with fewer roadblocks, and new tools to illuminate the way.

behavior, *Robotics Science and Systems (RSS'11)* .

Prentice Hall, Englewood cliffs, NJ.

Uppsala University.

Reading, MA.

and Sons, New York, NY.

Erik Billing, Thomas Hellström and Lars-Erik Janlert

**Robot Learning from Demonstration Using**

In this chapter, the prediction algorithm Predictive Sequence Learning (PSL) is presented and evaluated in a robot *Learning from Demonstration* (LFD) setting. PSL generates hypotheses from a sequence of sensory-motor events. Generated hypotheses can be used as a semi-reactive controller for robots. PSL has previously been used as a method for LFD (Billing et al., 2010; 2011) but suffered from combinatorial explosion when applied to data with many dimensions, such as high dimensional sensor and motor data. A new version of PSL, referred to as *Fuzzy Predictive Sequence Learning* (FPSL), is presented and evaluated in this chapter. FPSL is implemented as a Fuzzy Logic rule base and works on a continuous state space, in contrast to the discrete state space used in the original design of PSL. The evaluation of FPSL shows a significant performance improvement in comparison to the discrete version of the algorithm. Applied to an LFD task in a simulated apartment environment, the robot is able to learn to

navigate to a specific location, starting from an unknown position in the apartment.

Learning from Demonstration is a well-established technique for teaching robots new behaviors. One of the greatest challenges in LFD is to implement a learning algorithm that allows the robot pupil to generalize a sequence of actions demonstrated by the teacher such that the robot is able to perform the desired behavior in a dynamic environment. A behavior may be any complex sequence of actions executed in relation to sensor data (Billing

The LFD problem is often formulated as four questions, *what-to-imitate*, *how-to-imitate*, *when-to-imitate* and *who-to-imitate* which leads up to the larger problem of how to evaluate an imitation (Alissandrakis et al., 2002). Large parts of the literature approach the learning problem by trying to find the common features within a set of demonstrations of the same behavior. A skill is generalized by exploiting statistical regularities among the demonstrations (e.g. Calinon, 2009). This is reflected in the *what-to-imitate* question, originally introduced in a

An action or sequence of actions is a successful component of imitation of a particular action if it achieves the same subgoal as that action. An entire sequence of actions is successful if it successively achieves each of a sequence of abstracted

The problem is difficult since a certain behavior can be imitated on many different abstraction levels. Byrne & Russon (1998) identified two levels; the *action-level imitation* copying the surface of the behavior and a *program-level imitation* copying the structure of the behavior. A

classical work by Nehaniv & Dautenhahn (2000) and is in a longer form described as:

**1. Introduction**

& Hellström, 2010).

subgoals.

*Department of Computing Science, Umeå University*

**Predictive Sequence Learning**

*Sweden*

**12**

J. & Stenström, P. S. (2008). The worst-case execution time problem - overview of methods and survey of tools, *ACM Transactions on Embedded Computing Systems* **7**(3).

## **Robot Learning from Demonstration Using Predictive Sequence Learning**

Erik Billing, Thomas Hellström and Lars-Erik Janlert *Department of Computing Science, Umeå University Sweden*

### **1. Introduction**

26 TBD

234 Robotic Systems – Applications, Control and Programming

Wilkinson, D. D., Vandeweghe, J. M. & Matsuoka, Y. (2003). An extensor mechanism for

Wittenmark, B., Nilsson, J. & Torngren, M. (1995). Timing problems in real-time control systems, *In Proceedings of the 1995 American Control Conference. Seattle, WA* . Zhou, K. & Doyle, J. (1998). *Essentials of Robust Control*, Prentice Hall, Upper Saddle River, NJ.

*Automation*, Vol. 1, pp. 238 – 243.

J. & Stenström, P. S. (2008). The worst-case execution time problem - overview of methods and survey of tools, *ACM Transactions on Embedded Computing Systems* **7**(3).

an anatomical robotic hand, *Proc. of the IEEE International Conference on Robotics and*

In this chapter, the prediction algorithm Predictive Sequence Learning (PSL) is presented and evaluated in a robot *Learning from Demonstration* (LFD) setting. PSL generates hypotheses from a sequence of sensory-motor events. Generated hypotheses can be used as a semi-reactive controller for robots. PSL has previously been used as a method for LFD (Billing et al., 2010; 2011) but suffered from combinatorial explosion when applied to data with many dimensions, such as high dimensional sensor and motor data. A new version of PSL, referred to as *Fuzzy Predictive Sequence Learning* (FPSL), is presented and evaluated in this chapter. FPSL is implemented as a Fuzzy Logic rule base and works on a continuous state space, in contrast to the discrete state space used in the original design of PSL. The evaluation of FPSL shows a significant performance improvement in comparison to the discrete version of the algorithm. Applied to an LFD task in a simulated apartment environment, the robot is able to learn to navigate to a specific location, starting from an unknown position in the apartment.

Learning from Demonstration is a well-established technique for teaching robots new behaviors. One of the greatest challenges in LFD is to implement a learning algorithm that allows the robot pupil to generalize a sequence of actions demonstrated by the teacher such that the robot is able to perform the desired behavior in a dynamic environment. A behavior may be any complex sequence of actions executed in relation to sensor data (Billing & Hellström, 2010).

The LFD problem is often formulated as four questions, *what-to-imitate*, *how-to-imitate*, *when-to-imitate* and *who-to-imitate* which leads up to the larger problem of how to evaluate an imitation (Alissandrakis et al., 2002). Large parts of the literature approach the learning problem by trying to find the common features within a set of demonstrations of the same behavior. A skill is generalized by exploiting statistical regularities among the demonstrations (e.g. Calinon, 2009). This is reflected in the *what-to-imitate* question, originally introduced in a classical work by Nehaniv & Dautenhahn (2000) and is in a longer form described as:

An action or sequence of actions is a successful component of imitation of a particular action if it achieves the same subgoal as that action. An entire sequence of actions is successful if it successively achieves each of a sequence of abstracted subgoals.

The problem is difficult since a certain behavior can be imitated on many different abstraction levels. Byrne & Russon (1998) identified two levels; the *action-level imitation* copying the surface of the behavior and a *program-level imitation* copying the structure of the behavior. A

encoded relative object displacement and the relevant aspects of the behavior were in this way extracted as the common features in the demonstration set. Calinon et al. (2007) developed this approach by encoding the demonstration set using a mixture of Gaussian/Bernoulli distributions. The Gaussian Mixture Model approach is attractive since the behavior is divided into several distributions with different covariance, and different metrics can in this way be selected for different parts of the demonstrated behavior. More recently, similar encoding strategies have been evaluated for learning of a robot navigation task (de Rengervé

Robot Learning from Demonstration Using Predictive Sequence Learning 237

Another common approach to LFD is to map the demonstration onto a set of pre-programmed or previously learned primitives controllers (Billing & Hellström, 2010). The approach has strong connections to *behavior-based architectures* (Arkin, 1998; Matari´c, 1997; Matari´c & Marjanovic, 1993) and earlier reactive approaches (e.g. Brooks, 1986; 1991). When introducing behavior primitives, the LFD process can be divided into three tasks (Billing & Hellström,

3. *Behavior coordination*, referring to identification of rules or switching conditions for how the

Behavior segmentation and recognition can be seen as one way to approach the what-to-imitate problem, whereas behavior coordination is part of how-to-imitate. The approach represents one way of introducing good bias in learning and solve the generalization problem by relying on previous behavioral knowledge. While there are many domain specific solutions to these three subproblems, they appear very difficult to solve in the general case. Specifically, behavior recognition poses the problem of mapping a sequence of observations to a set of controllers to which the input is unknown. Again, the need to introduce a metric of

Part of the problem to find a general solution to these problems may lie in a vague definition of *behavior* (Matari´c, 1997). The notion of behavior is strongly connected to the purpose of executed actions and a definition of goal. Nicolescu (2003) identified two major types of goals:

The use of behavior primitives as a basis for imitation has many connections to biology (e.g. Matari´c, 2002) and specifically the mirror system (Brass et al., 2000; Gallese et al., 1996; Rizzolatti et al., 1988; Rizzolatti & Craighero, 2004). While the role of the mirror system is still highly debated, several groups of researchers propose computational models where perception and action are tightly interweaved. Among the most prominent examples are the *HAMMER* architecture (Demiris & Hayes, 2002; Demiris, 1999; Demiris & Johnson, 2003) and the *MOSAIC* architecture (Haruno et al., 2001; Wolpert & Kawato, 1998). Both these architectures implement a set of modules, where each module is an inverse model (controller) paired with a forward model (predictor). The inverse and forward models are trained together such that the forward model can predict sensor data in response to the actions produced by the inverse model. The inverse model is tuned to execute a certain behavior when the forward model produces good predictions. The prediction error is used to compute a bottom-up

**Maintenance goals:** A specific condition has to be maintained for a time interval.

**Achievement goals:** A specific condition has to be reached.

1. *Behavior segmentation* where a demonstration is divided into smaller segments. 2. *Behavior recognition* where each segment is associated with a primitive controller.

et al., 2010).

2010):

imitation appears.

**1.2 Behavior primitives as a basis for imitation**

primitives are to be combined.

third level, the *effect-level imitation*, was introduced by Nehaniv & Dautenhahn (2001) in order to better describe imitation between agents with dissimilar body structures. Demiris & Hayes (1997) proposed three slightly different levels: *1) basic imitation* with strong similarities to the notion of action-level imitation, 2) *functional imitation* that best corresponds to effect-level imitation and 3) *abstract imitation* that represents coordination based on the presumed internal state of the agent rather than the observed behavior. Demiris and Hayes give the example of making a sad face when someone is crying.

The necessity to consider the level of imitation in LFD becomes apparent when considering two demonstrations that look very different considered as sequences of data, but that we as humans still interpret as examples of the same behavior since they achieve similar results on an abstract level. This would correspond to a functional or program-level imitation. In these situations it is very difficult to find similarities between the demonstrations without providing high level knowledge about the behavior, often leading to specialized systems directed to LDF in limited domains.

A related problem is that two demonstrations of the same behavior may not have the same length. If one demonstration takes longer time than another, they can not be directly compared in order to find common features. Researchers have therefore used techniques to determine the temporal alignment of demonstrations. One common technique is *dynamic time warping* (Myers & Rabiner, 1981)*,* that can be used to compensate for temporal differences in the data. Behaviors can be demonstrated to a robot in many different ways. Argall et al. (2009) outline four types of demonstrations: A direct recording of sensor stimuli, joint angles, etc., is referred to as an *identity record mapping.* In this case, the robot is often used during the demonstration and controlled via teleoperation or by physically moving the robot's limbs (kinestetic teaching). An external observation, e.g. a video recording of the teacher, is called a *non-identity record mapping*. This type of demonstrations poses a difficult sensing problem of detecting how the teacher has moved, but also allows much more flexible demonstration setting. The teacher may have a body identical to that of the pupil (*identity embodiment*) or a body with a different structure (*non-identity embodiment*). In the latter case, the demonstration has to be transformed into corresponding actions using the body of the pupil, a difficult problem known as the *correspondence problem* (Nehaniv & Dautenhahn, 2001). In this work we focus on LFD via teleoperation. Sensor data and motor commands are in this setting recorded while a human teacher demonstrates the desired behavior by tele-operating the robot, producing demonstrations with identity in both record mapping and embodiment.

### **1.1 Metric of imitation**

Successful imitation requires that relevant features of the demonstration are selected at a suitable imitation level and processed into a generalized representation of the behavior. The process is difficult to implement in a robot since it is often far from obvious which imitation level that is optimal in a specific situation, and the relevance of features may consequently vary significantly from one learning situation to another. This problem has been formalized as a *metric of imitation*, defined as a weighted sum over all strategy-dependent metrics on all imitation levels (Billard et al., 2003).

The metric of imitation was originally demonstrated on a manipulation task with a humanoid robot (Billard et al., 2003). With focus on the correspondence problem, Alissandrakis et al. (2005) propose a similar approach to imitation of manipulation tasks. The what-to-imitate problem is approached by maximizing trajectory agreements of manipulated objects, using several different metrics. Some metrics encoded absolute trajectories while other metrics encoded relative object displacement and the relevant aspects of the behavior were in this way extracted as the common features in the demonstration set. Calinon et al. (2007) developed this approach by encoding the demonstration set using a mixture of Gaussian/Bernoulli distributions. The Gaussian Mixture Model approach is attractive since the behavior is divided into several distributions with different covariance, and different metrics can in this way be selected for different parts of the demonstrated behavior. More recently, similar encoding strategies have been evaluated for learning of a robot navigation task (de Rengervé et al., 2010).

## **1.2 Behavior primitives as a basis for imitation**

2 Robot Control

third level, the *effect-level imitation*, was introduced by Nehaniv & Dautenhahn (2001) in order to better describe imitation between agents with dissimilar body structures. Demiris & Hayes (1997) proposed three slightly different levels: *1) basic imitation* with strong similarities to the notion of action-level imitation, 2) *functional imitation* that best corresponds to effect-level imitation and 3) *abstract imitation* that represents coordination based on the presumed internal state of the agent rather than the observed behavior. Demiris and Hayes give the example of

The necessity to consider the level of imitation in LFD becomes apparent when considering two demonstrations that look very different considered as sequences of data, but that we as humans still interpret as examples of the same behavior since they achieve similar results on an abstract level. This would correspond to a functional or program-level imitation. In these situations it is very difficult to find similarities between the demonstrations without providing high level knowledge about the behavior, often leading to specialized systems directed to LDF

A related problem is that two demonstrations of the same behavior may not have the same length. If one demonstration takes longer time than another, they can not be directly compared in order to find common features. Researchers have therefore used techniques to determine the temporal alignment of demonstrations. One common technique is *dynamic time warping* (Myers & Rabiner, 1981)*,* that can be used to compensate for temporal differences in the data. Behaviors can be demonstrated to a robot in many different ways. Argall et al. (2009) outline four types of demonstrations: A direct recording of sensor stimuli, joint angles, etc., is referred to as an *identity record mapping.* In this case, the robot is often used during the demonstration and controlled via teleoperation or by physically moving the robot's limbs (kinestetic teaching). An external observation, e.g. a video recording of the teacher, is called a *non-identity record mapping*. This type of demonstrations poses a difficult sensing problem of detecting how the teacher has moved, but also allows much more flexible demonstration setting. The teacher may have a body identical to that of the pupil (*identity embodiment*) or a body with a different structure (*non-identity embodiment*). In the latter case, the demonstration has to be transformed into corresponding actions using the body of the pupil, a difficult problem known as the *correspondence problem* (Nehaniv & Dautenhahn, 2001). In this work we focus on LFD via teleoperation. Sensor data and motor commands are in this setting recorded while a human teacher demonstrates the desired behavior by tele-operating the robot, producing demonstrations with identity in both record mapping and embodiment.

Successful imitation requires that relevant features of the demonstration are selected at a suitable imitation level and processed into a generalized representation of the behavior. The process is difficult to implement in a robot since it is often far from obvious which imitation level that is optimal in a specific situation, and the relevance of features may consequently vary significantly from one learning situation to another. This problem has been formalized as a *metric of imitation*, defined as a weighted sum over all strategy-dependent metrics on all

The metric of imitation was originally demonstrated on a manipulation task with a humanoid robot (Billard et al., 2003). With focus on the correspondence problem, Alissandrakis et al. (2005) propose a similar approach to imitation of manipulation tasks. The what-to-imitate problem is approached by maximizing trajectory agreements of manipulated objects, using several different metrics. Some metrics encoded absolute trajectories while other metrics

making a sad face when someone is crying.

in limited domains.

**1.1 Metric of imitation**

imitation levels (Billard et al., 2003).

Another common approach to LFD is to map the demonstration onto a set of pre-programmed or previously learned primitives controllers (Billing & Hellström, 2010). The approach has strong connections to *behavior-based architectures* (Arkin, 1998; Matari´c, 1997; Matari´c & Marjanovic, 1993) and earlier reactive approaches (e.g. Brooks, 1986; 1991). When introducing behavior primitives, the LFD process can be divided into three tasks (Billing & Hellström, 2010):

Behavior segmentation and recognition can be seen as one way to approach the what-to-imitate problem, whereas behavior coordination is part of how-to-imitate. The approach represents one way of introducing good bias in learning and solve the generalization problem by relying on previous behavioral knowledge. While there are many domain specific solutions to these three subproblems, they appear very difficult to solve in the general case. Specifically, behavior recognition poses the problem of mapping a sequence of observations to a set of controllers to which the input is unknown. Again, the need to introduce a metric of imitation appears.

Part of the problem to find a general solution to these problems may lie in a vague definition of *behavior* (Matari´c, 1997). The notion of behavior is strongly connected to the purpose of executed actions and a definition of goal. Nicolescu (2003) identified two major types of goals:

**Maintenance goals:** A specific condition has to be maintained for a time interval.

**Achievement goals:** A specific condition has to be reached.

The use of behavior primitives as a basis for imitation has many connections to biology (e.g. Matari´c, 2002) and specifically the mirror system (Brass et al., 2000; Gallese et al., 1996; Rizzolatti et al., 1988; Rizzolatti & Craighero, 2004). While the role of the mirror system is still highly debated, several groups of researchers propose computational models where perception and action are tightly interweaved. Among the most prominent examples are the *HAMMER* architecture (Demiris & Hayes, 2002; Demiris, 1999; Demiris & Johnson, 2003) and the *MOSAIC* architecture (Haruno et al., 2001; Wolpert & Kawato, 1998). Both these architectures implement a set of modules, where each module is an inverse model (controller) paired with a forward model (predictor). The inverse and forward models are trained together such that the forward model can predict sensor data in response to the actions produced by the inverse model. The inverse model is tuned to execute a certain behavior when the forward model produces good predictions. The prediction error is used to compute a bottom-up

of BECCA is to capture the discrete episodic nature of many types of human motor behavior, without introducing a priori knowledge into the system. BECCA was presented as a very general reinforcement learning system, applicable to many types of learning and control problems. One of the core elements of BECCA is the temporal difference (TD) algorithm *Sequence Learning* (SL) (Rohrer, 2007). SL builds sequences of passed events which is used to predict future events, and can in contrast to other TD algorithms base its predictions on

Robot Learning from Demonstration Using Predictive Sequence Learning 239

Inspired by BECCA and specifically SL, we developed the PSL algorithm as a method for LFD (Billing et al., 2010; 2011). PSL has many interesting properties seen as a learning algorithm for robots. It is model free, meaning that it introduces very few assumptions into learning and does not need any task specific configuration. PSL can be seen as a variable-order Markov model. Starting out from a reactive (first order) model, PSL estimates transition probabilities between discrete sensor and motor states. For states that do not show Markov property, the order is increased and PSL models the transition probability based on several passed events. In this way, PSL will progressively gain memory for parts of the behavior that cannot be modeled in a reactive way. In theory, there is no limitation to the order of the state and hence the length of the memory, but PSL is in practice unable to capture long temporal dependencies

PSL has been evaluated both as a controller (Billing et al., 2011) and as a method for behavior recognition (Billing et al., 2010). Even though the evaluation overall generated good results, PSL is subject to combinatorial explosion both when the number of sensors and actuators increase, and when the demonstrated behavior requires modeling of long temporal dependencies. PSL can however efficiently model short temporal dependencies in a semi-reactive way and should thus be a good platform for implementing a hierarchical system

In this chapter, we present and evaluate a new version of PSL based on Fuzzy Logic. While keeping the core idea of the original PSL algorithm, the new version can handle continuous and multi dimensional data in a better way. To distinguish between the two, the new fuzzy version of the algorithm is denoted FPSL, whereas the previous discrete version is denoted DPSL. A detailed description of FPSL is given in Section 2. An evaluation with comparisons between the two algorithms is presented in Section 3, followed by a discussion

FPSL builds fuzzy rules, referred to as *hypotheses h*, describing temporal dependencies between a sensory-motor event *et*+1 and a sequence of passed events

Υ*i* is the event variable and *Eh* (*e*) is a fuzzy membership function returning a membership value for a specific *e*. The right hand side *E*¯*h* is a membership function comprising expected events at time *t* + 1. |*h*| denotes the length of *h*, i.e., the number of left-hand-side conditions of the rule. Both *E* and *E*¯ are implemented as standard cone membership functions with base

A set of hypotheses can be used to compute a prediction *e*ˆ*t*+1 given a sequence of passed

|*h*|−2 ∧ ... ∧ Υ*t is Eh*

0 *C*

⇒ Υ*t*+1 *is E*¯ *h* (1)

, defined up until current time *t*.

|*h*|−1 ∧ Υ*t*−|*h*|+2 *is Eh*

many previous states.

due to combinatorial explosion.

and conclusions in section 4.

*et*−|*h*|+1,*et*−|*h*|+2,...,*et*

*h* :

width *ε* (e.g. Klir & Yuan, 1995).

**2. Predictive Sequence Learning**

sensory-motor events *η*, defined up to the current time *t*:

Υ*t*−|*h*|+1 *is Eh*

similar to the HAMMER and MOSAIC architectures.

signal for each module. Based on the bottom-up signal, a top-down responsibility signal or confidence value is computed and propagated to each module. The output of the system is a combination of the actions produced by each inverse model, proportional to their current responsibility. The responsibility signal also controls the learning rate of each module, such that modules are only updated when their responsibility is high. In this way, modules are tuned to a specific behavior or parts of a behavior. Since the prediction error of the forward model is used as a measure of how well the specific module fits present circumstances, it can be seen as a metric of imitation that is learnt together with the controller. The architecture can be composed into a hierarchical system where modules are organized in layers, with the lowest layer interacting with sensors and actuators. The bottom-up signal constitutes sensor input for the layer above and actions produced by higher levels constitutes the top-down responsibility signal.

One motivation for this architecture lies in an efficient division of labor between different parts of the system. Each module can be said to operate with a specific temporal resolution. Modules at the bottom layer are given the highest resolution while modules higher up in the hierarchy have decreasing temporal resolution. State variables that change slowly compared to a specific module's resolution are ignored by that module and are instead handled by modules higher up in the hierarchy. Slowly changing states that lead to high responsibility for the module is referred to as the module's *context*. In a similar fashion, variables that change fast in comparison to the temporal resolution are handled lower in the hierarchy. This allows each module to implement a controller where the behavior depends on relatively recent states. Long temporal dependencies are modeled by switching between modules, which removes the requirement for each model to capture these dependencies. Furthermore, updates of a single behavior or parts of a behavior will only require updates of a few modules and will not propagate changes to other modules. See Billing (2009) for a longer discussion on these aspects of hierarchical architectures.

The HAMMER and MOSAIC architectures make few restrictions on what kind of controllers each module should implement. We argue however, that modules should be *semi-reactive*, meaning that action selection and predictions of sensor events should be based on recent sensor and motor events. Strictly reactive modules are not desirable since each module must be able to model any dependency shorter than the temporal resolution of modules in the layer directly above.

The division of behavior into modules is however also producing a number of drawbacks. The possibility for the system to share knowledge between behaviors is limited. Moreover, the system has to combine actions produced by different modules, which may be difficult in cases when more than one module receives high responsibility.

One architecture with similarities to HAMMER and MOSAIC able to share knowledge between different behaviors is *RNNPB* (Tani et al., 2004). *RNNPB* is a recurrent neural network with parametric bias (PB). Both input and output layer of the network contains sensor and motor nodes as well as nodes with recurrent connections. In addition, the input layer is given a set of extra nodes, representing the PB vector. The network is trained to minimize prediction error, both by back-propagation and by changing the PB vector. The PB vector is however updated slowly, such that it organizes into what could be seen as a context layer for the rest of the network. In addition to giving the network the ability to represent different behaviors that share knowledge, the PB vector can be used for behavior recognition.

Another architecture known as *Brain Emulating Cognition and Control Architecture* (BECCA) (Rohrer & Hulet, 2006) heavily influenced our early work on the PSL algorithm. The focus 4 Robot Control

signal for each module. Based on the bottom-up signal, a top-down responsibility signal or confidence value is computed and propagated to each module. The output of the system is a combination of the actions produced by each inverse model, proportional to their current responsibility. The responsibility signal also controls the learning rate of each module, such that modules are only updated when their responsibility is high. In this way, modules are tuned to a specific behavior or parts of a behavior. Since the prediction error of the forward model is used as a measure of how well the specific module fits present circumstances, it can be seen as a metric of imitation that is learnt together with the controller. The architecture can be composed into a hierarchical system where modules are organized in layers, with the lowest layer interacting with sensors and actuators. The bottom-up signal constitutes sensor input for the layer above and actions produced by higher levels constitutes the top-down

One motivation for this architecture lies in an efficient division of labor between different parts of the system. Each module can be said to operate with a specific temporal resolution. Modules at the bottom layer are given the highest resolution while modules higher up in the hierarchy have decreasing temporal resolution. State variables that change slowly compared to a specific module's resolution are ignored by that module and are instead handled by modules higher up in the hierarchy. Slowly changing states that lead to high responsibility for the module is referred to as the module's *context*. In a similar fashion, variables that change fast in comparison to the temporal resolution are handled lower in the hierarchy. This allows each module to implement a controller where the behavior depends on relatively recent states. Long temporal dependencies are modeled by switching between modules, which removes the requirement for each model to capture these dependencies. Furthermore, updates of a single behavior or parts of a behavior will only require updates of a few modules and will not propagate changes to other modules. See Billing (2009) for a longer discussion on these

The HAMMER and MOSAIC architectures make few restrictions on what kind of controllers each module should implement. We argue however, that modules should be *semi-reactive*, meaning that action selection and predictions of sensor events should be based on recent sensor and motor events. Strictly reactive modules are not desirable since each module must be able to model any dependency shorter than the temporal resolution of modules in the layer

The division of behavior into modules is however also producing a number of drawbacks. The possibility for the system to share knowledge between behaviors is limited. Moreover, the system has to combine actions produced by different modules, which may be difficult in

One architecture with similarities to HAMMER and MOSAIC able to share knowledge between different behaviors is *RNNPB* (Tani et al., 2004). *RNNPB* is a recurrent neural network with parametric bias (PB). Both input and output layer of the network contains sensor and motor nodes as well as nodes with recurrent connections. In addition, the input layer is given a set of extra nodes, representing the PB vector. The network is trained to minimize prediction error, both by back-propagation and by changing the PB vector. The PB vector is however updated slowly, such that it organizes into what could be seen as a context layer for the rest of the network. In addition to giving the network the ability to represent different behaviors

Another architecture known as *Brain Emulating Cognition and Control Architecture* (BECCA) (Rohrer & Hulet, 2006) heavily influenced our early work on the PSL algorithm. The focus

cases when more than one module receives high responsibility.

that share knowledge, the PB vector can be used for behavior recognition.

responsibility signal.

aspects of hierarchical architectures.

directly above.

of BECCA is to capture the discrete episodic nature of many types of human motor behavior, without introducing a priori knowledge into the system. BECCA was presented as a very general reinforcement learning system, applicable to many types of learning and control problems. One of the core elements of BECCA is the temporal difference (TD) algorithm *Sequence Learning* (SL) (Rohrer, 2007). SL builds sequences of passed events which is used to predict future events, and can in contrast to other TD algorithms base its predictions on many previous states.

Inspired by BECCA and specifically SL, we developed the PSL algorithm as a method for LFD (Billing et al., 2010; 2011). PSL has many interesting properties seen as a learning algorithm for robots. It is model free, meaning that it introduces very few assumptions into learning and does not need any task specific configuration. PSL can be seen as a variable-order Markov model. Starting out from a reactive (first order) model, PSL estimates transition probabilities between discrete sensor and motor states. For states that do not show Markov property, the order is increased and PSL models the transition probability based on several passed events. In this way, PSL will progressively gain memory for parts of the behavior that cannot be modeled in a reactive way. In theory, there is no limitation to the order of the state and hence the length of the memory, but PSL is in practice unable to capture long temporal dependencies due to combinatorial explosion.

PSL has been evaluated both as a controller (Billing et al., 2011) and as a method for behavior recognition (Billing et al., 2010). Even though the evaluation overall generated good results, PSL is subject to combinatorial explosion both when the number of sensors and actuators increase, and when the demonstrated behavior requires modeling of long temporal dependencies. PSL can however efficiently model short temporal dependencies in a semi-reactive way and should thus be a good platform for implementing a hierarchical system similar to the HAMMER and MOSAIC architectures.

In this chapter, we present and evaluate a new version of PSL based on Fuzzy Logic. While keeping the core idea of the original PSL algorithm, the new version can handle continuous and multi dimensional data in a better way. To distinguish between the two, the new fuzzy version of the algorithm is denoted FPSL, whereas the previous discrete version is denoted DPSL. A detailed description of FPSL is given in Section 2. An evaluation with comparisons between the two algorithms is presented in Section 3, followed by a discussion and conclusions in section 4.

### **2. Predictive Sequence Learning**

FPSL builds fuzzy rules, referred to as *hypotheses h*, describing temporal dependencies between a sensory-motor event *et*+1 and a sequence of passed events *et*−|*h*|+1,*et*−|*h*|+2,...,*et* , defined up until current time *t*.

$$h: \left(\mathbf{Y}\_{t-|h|+1} \text{ is } E\_{|h|-1}^{\hbar} \wedge \mathbf{Y}\_{t-|h|+2} \text{ is } E\_{|h|-2}^{\hbar} \wedge \dots \wedge \mathbf{Y}\_{t} \text{ is } E\_{0}^{\hbar}\right) \stackrel{\mathbb{C}}{\Leftrightarrow} \mathbf{Y}\_{t+1} \text{ is } \bar{E}^{\hbar} \tag{1}$$

Υ*i* is the event variable and *Eh* (*e*) is a fuzzy membership function returning a membership value for a specific *e*. The right hand side *E*¯*h* is a membership function comprising expected events at time *t* + 1. |*h*| denotes the length of *h*, i.e., the number of left-hand-side conditions of the rule. Both *E* and *E*¯ are implemented as standard cone membership functions with base width *ε* (e.g. Klir & Yuan, 1995).

A set of hypotheses can be used to compute a prediction *e*ˆ*t*+1 given a sequence of passed sensory-motor events *η*, defined up to the current time *t*:

*E*ˆ (*et*+1) :

*E*ˆ is converted to crisp values using a squared *center of sum* defuzzification:

exp � ∑ *h*

PSL is able to produce a prediction is referred to as the *coverage φ* (*η*):

*φ* (*η*) =

is, referred to as the *trust c*ˆ:

**2.2 Generating hypotheses**

Algorithm 0.1.

2011).

*c*ˆ � *M*ˆ � = ⎧ ⎨ ⎩ � *h E*¯

Robot Learning from Demonstration Using Predictive Sequence Learning 241

*e*ˆ = ∑ *e eE*ˆ (*e*) 2

∑ *e E*ˆ (*e*)

The amount of entropy in *M*ˆ also bears information about how reliable a specific prediction

0

The trust is important since it allows a controller to evaluate when to rely on PSL, and when to choose an alternate control method. The proportion of time steps in *η* for which *c*ˆ *>* 0 and

� *M*ˆ (*h*) �

*c*ˆ*i >* 0 *otherwise*

*M*ˆ (*h*)log2

*t* ∑ *i*=1 � 1 0

Hypotheses can be generated from a sequence of sensory-motor events *η*. During training, PSL continuously makes predictions and creates new hypotheses when no matching hypothesis produces the correct prediction *E*¯. The exact training procedure is described in

For example, consider the event sequence *η* = *abccabccabcc*. Let *t* = 1. PSL will search for a hypothesis with a body matching *a*. Initially, the context set *C* is empty and consequently PSL will create a new hypothesis (*a*) ⇒ *b* which is added to *C* with confidence 1, denoted *C* (*a* ⇒ *b*) = 1. The same procedure will be executed at *t* = 2 and *t* = 3 such that *C* ((*b*) ⇒ *c*) = 1 and *C* ((*c*) ⇒ *c*) = 1. At *t* = 4, PSL will find a matching hypothesis (*c*) ⇒ *c* producing the wrong prediction *c*. Consequently, a new hypothesis (*c*) ⇒ *a* is created and confidences are updated such that *C* ((*c*) ⇒ *c*) = 0.5 and *C* ((*c*) ⇒ *a*) = 1. The new hypothesis receives a higher confidence since confidence values are calculated from the creation time of the hypothesis (Equation 4). The predictions at *t* = 5 and *t* = 6 will be correct and no new hypotheses are created. At *t* = 7, both (*c*) ⇒ *a* and (*c*) ⇒ *c* will contribute to the prediction *E*ˆ. Since the confidence of (*c*) ⇒ *a* is higher than that of (*c*) ⇒ *c*, *E*ˆ will defuzzify towards *a*, producing the wrong prediction (Equation 10). As a result, PSL creates a new hypothesis (*b*, *c*) ⇒ *c*. Similarly, (*c*, *c*) ⇒ *a* will be created at *t* = 8. PSL is now able to

predict all elements in the sequence perfectly and no new hypotheses are created.

Source code from the implementation used in the present work is available online (Billing,

*h* (*et*+1) *M*ˆ (*h*) (9)

2 (10)

*t* (12)

*otherwise* (11)

� *M*ˆ = ∅

$$\eta = (e\_1, e\_2, \dots, e\_l) \tag{2}$$

The process of matching hypothesis to data is described in Section 2.1. The PSL learning process, where hypotheses are generated from a sequence of data, is described in Section 2.2. Finally, a discussion about parameters and configuration is found in Section 2.3.

### **2.1 Matching hypotheses**

Given a sequence of sensory-motor events *η* = (*e*1,*e*2,...,*et*), a match *αt* (*h*) of the rule is given by:

$$\mathfrak{a}\_{\mathfrak{h}}\left(\hbar\right) : \bigwedge\_{i=0}^{|h|-1} E\_i^{\mathfrak{h}}\left(e\_{t-i}\right) \tag{3}$$

where ∧ is implemented as a *min*-function.

Hypotheses are grouped in fuzzy sets *C* whose membership value *C* (*h*) describes the confidence of *h* at time *t*:

$$C\left(h\right) = \frac{\sum\_{k=t^h}^t \alpha\_k\left(h\right) \bar{E}^h\left(e\_{k+1}\right)}{\sum\_{k=t^h}^t \alpha\_k\left(h\right)}\tag{4}$$

*t h* is the creation time of *h* or 1 if *h* existed prior to training. Each C represents a *context* and can be used to implement a specific behavior or part of a behavior. The *responsibility signal λt* (*C*) is used to control which behavior that is active at a specific time. The combined confidence value *C*˜ *t* (*h*) is a weighted sum over all *C*:

$$
\tilde{\mathbf{C}}\_{t}\left(h\right) = \frac{\sum \mathbf{C}\left(h\right)\lambda\_{t}\left(\mathbf{C}\right)}{\sum \lambda\_{t}\left(\mathbf{C}\right)}\tag{5}
$$

*C*˜ *t* can be seen as a fuzzy set representing the active context at time *t*. Hypotheses contribute to a prediction in proportion to their membership in *C*˜ and the *match set M*ˆ . *M*ˆ is defined in three steps. First, the best matching hypotheses for each *E*¯ is selected:

$$M = \left\{ h \mid \mathfrak{a}\left( h\right) \ge \mathfrak{a}\left( h'\right) \text{ for all } \left\{ h' \mid \vec{E}^{h'} = \vec{E}^h \right\} \right\} \tag{6}$$

The longest *h* ∈ *M* for each RHS is selected:

$$\tilde{M} = \left\{ h \mid |h| \ge |h'| \text{ for all } \left\{ h' \in M \mid \bar{E}^{h'} = \bar{E}^h \right\} \right\} \tag{7}$$

Finally, the match set *M*ˆ is defined as:

$$\hat{M}\left(h\right) = \begin{cases} \text{a } \left(h\right) \tilde{\mathbb{C}}\left(h\right) & h \in \tilde{M} \\ 0 & \text{otherwise} \end{cases} \tag{8}$$

The aggregated prediction *E*ˆ (*et*+1) is computed using the Larsen method (e.g. Fullér, 1995):

.

.

.

.

.

$$
\triangle \left( e\_{t+1} \right) : \bigvee\_{h} \bar{E}\_h \left( e\_{t+1} \right) \hat{M} \left( h \right) \tag{9}
$$

*E*ˆ is converted to crisp values using a squared *center of sum* defuzzification:

$$\mathcal{E} = \frac{\sum\_{\mathcal{E}} e \hat{\mathcal{E}} \left( \boldsymbol{\varepsilon} \right)^{2}}{\sum\_{\mathcal{E}} \hat{\mathcal{E}} \left( \boldsymbol{\varepsilon} \right)^{2}} \tag{10}$$

The amount of entropy in *M*ˆ also bears information about how reliable a specific prediction is, referred to as the *trust c*ˆ:

$$\hat{\mathcal{E}}\left(\hat{\mathcal{M}}\right) = \begin{cases} 0 & \hat{\mathcal{M}} = \mathcal{D} \\ \exp\left[\sum\_{h} \hat{\mathcal{M}}\left(h\right) \log\_{2}\left(\hat{\mathcal{M}}\left(h\right)\right)\right] & \text{otherwise} \end{cases} \tag{11}$$

The trust is important since it allows a controller to evaluate when to rely on PSL, and when to choose an alternate control method. The proportion of time steps in *η* for which *c*ˆ *>* 0 and PSL is able to produce a prediction is referred to as the *coverage φ* (*η*):

$$\phi\_{\phi}(\eta) = \frac{\sum\_{i=1}^{t} \begin{cases} 1 & \text{ $\mathcal{E}\_{i}$ } > 0 \\ 0 & \text{otherwise} \end{cases}}{t} \tag{12}$$

#### **2.2 Generating hypotheses**

6 Robot Control

The process of matching hypothesis to data is described in Section 2.1. The PSL learning process, where hypotheses are generated from a sequence of data, is described in Section 2.2.

Given a sequence of sensory-motor events *η* = (*e*1,*e*2,...,*et*), a match *αt* (*h*) of the rule is

*i*=0 *Eh*

Hypotheses are grouped in fuzzy sets *C* whose membership value *C* (*h*) describes the

*t* ∑ *k*=*th*

*h* is the creation time of *h* or 1 if *h* existed prior to training. Each C represents a *context* and can be used to implement a specific behavior or part of a behavior. The *responsibility signal λt* (*C*) is used to control which behavior that is active at a specific time. The combined confidence

> ∑ *C*

> > ∑ *C*

*t* can be seen as a fuzzy set representing the active context at time *t*. Hypotheses contribute to a prediction in proportion to their membership in *C*˜ and the *match set M*ˆ . *M*ˆ is defined in

*f or all*

 *h*� | *E*¯ *h*�

*h*� ∈ *M* | *E*¯ *h*�

*h* ∈ *M*˜

 *h*� *αk* (*h*) *E*¯ *h* (*ek*+1)

*αk* (*h*)

*C* (*h*) *λt* (*C*)

*t* ∑ *k*=*th*

Finally, a discussion about parameters and configuration is found in Section 2.3.

*αt* (*h*) :

*C* (*h*) =

*C*˜ *t* (*h*) =

three steps. First, the best matching hypotheses for each *E*¯ is selected:

*h* | *α* (*h*) ≥ *α*

*h* | |*h*| ≥

*M*ˆ (*h*) =

*h*� *f or all*

*α* (*h*) *C*˜ (*h*) 0

The aggregated prediction *E*ˆ (*et*+1) is computed using the Larsen method (e.g. Fullér, 1995):

**2.1 Matching hypotheses**

confidence of *h* at time *t*:

where ∧ is implemented as a *min*-function.

*t* (*h*) is a weighted sum over all *C*:

*M* =

*M*˜ =

The longest *h* ∈ *M* for each RHS is selected:

Finally, the match set *M*ˆ is defined as:

given by:

. *t*

. *C*˜

.

.

.

value *C*˜

*η* = (*e*1,*e*2,...,*et*) (2)

*i* (*et*−*i*) (3)

*λt* (*C*) (5)

= *E*¯ *h*

= *E*¯ *h*

*otherwise* (8)

(4)

(6)

(7)

Hypotheses can be generated from a sequence of sensory-motor events *η*. During training, PSL continuously makes predictions and creates new hypotheses when no matching hypothesis produces the correct prediction *E*¯. The exact training procedure is described in Algorithm 0.1.

For example, consider the event sequence *η* = *abccabccabcc*. Let *t* = 1. PSL will search for a hypothesis with a body matching *a*. Initially, the context set *C* is empty and consequently PSL will create a new hypothesis (*a*) ⇒ *b* which is added to *C* with confidence 1, denoted *C* (*a* ⇒ *b*) = 1. The same procedure will be executed at *t* = 2 and *t* = 3 such that *C* ((*b*) ⇒ *c*) = 1 and *C* ((*c*) ⇒ *c*) = 1. At *t* = 4, PSL will find a matching hypothesis (*c*) ⇒ *c* producing the wrong prediction *c*. Consequently, a new hypothesis (*c*) ⇒ *a* is created and confidences are updated such that *C* ((*c*) ⇒ *c*) = 0.5 and *C* ((*c*) ⇒ *a*) = 1. The new hypothesis receives a higher confidence since confidence values are calculated from the creation time of the hypothesis (Equation 4). The predictions at *t* = 5 and *t* = 6 will be correct and no new hypotheses are created. At *t* = 7, both (*c*) ⇒ *a* and (*c*) ⇒ *c* will contribute to the prediction *E*ˆ. Since the confidence of (*c*) ⇒ *a* is higher than that of (*c*) ⇒ *c*, *E*ˆ will defuzzify towards *a*, producing the wrong prediction (Equation 10). As a result, PSL creates a new hypothesis (*b*, *c*) ⇒ *c*. Similarly, (*c*, *c*) ⇒ *a* will be created at *t* = 8. PSL is now able to predict all elements in the sequence perfectly and no new hypotheses are created.

Source code from the implementation used in the present work is available online (Billing, 2011).

**2.3 Parameters and task specific configuration**

in the interval [0, 1] (Klir & Yuan, 1995).

in more hypotheses being created during learning.

the previous version of PSL.

errors but also a low coverage.

were however fed directly into PSL.

use of FPSL as a method for LFD.

**3.1 Sensor prediction**

the parameters *ε* and *α*ˆ.

**3. Evaluation**

A clear description of parameters is important for any learning algorithm. Parameters always introduce the risk that the learning algorithm is tuned towards the evaluated task, producing better results than it would in the general case. We have therefore strived towards limiting the number of parameters of PSL. The original design of PSL was completely parameter free, with the exception that continuous data was discretized using some discretization method. The version of PSL proposed here can be seen as a generalization of the original algorithm (Billing et al., 2011) where the width *ε* of the membership function *E* determines the discretization resolution. In addition, a second parameter is introduced, referred to as the *precision constant α*ˆ. *α*ˆ is in fuzzy logic terminology an *α-cut*, i.e., thresholds over the fuzzy membership function

Robot Learning from Demonstration Using Predictive Sequence Learning 243

*ε* controls how generously FPSL matches hypotheses. A high *ε* makes the algorithm crisp but typically increases the precision of predictions when a match is found. Contrary, a low *ε* reduces the risk that FPSL reaches unobserved states at the cost of a decreased prediction performance. The high value of *ε* can be compared to a fine resolution data discretization for

*α*ˆ is only used during learning, controlling how exact a specific *E*¯ has to be before a new hypothesis with a different *E*¯ is created. A large *α*ˆ reduces prediction error but typically results

Both *ε* and *α*ˆ controls the tolerance to random variations in the data and can be decided based on how exact we desire that FPSL should model the data. Small *ε* in combination with large *α*ˆ will result in a model that closely fits the training data, typically producing small prediction

Two tests were performed to evaluate the performance of FPSL and compare it to the previous version. A simulated Robosoft Kompai robot (Robosoft, 2011) was used in the Microsoft RDS simulation environment (Microsoft, 2011). The 270 degree laser scanner of the Kompai was used as sensor data and the robot was controlled by setting linear and angular speeds. Demonstrations were performed via tele-operation using a joypad, while sensor and motor data were recorded with a temporal resolution of 20 Hz. The dimensionality of the laser scanner was reduced to 20 dimensions using an average filter. Angular and linear speeds

The first test (Section 3.1) was designed to compare FPSL and DPSL as prediction algorithms, using sensor data from the simulated robot. The second test (Section 3.2) demonstrates the

The two versions of PSL were compared using a series of tests of prediction performance. Even though DPSL and FPSL are similar in many ways, a comparison is not trivial since DPSL works on discrete data whereas FPSL uses continuous data. Prediction performance of DPSL will hence depend on how the data is discretized while the performance of FPSL depends on

To capture the prediction performance of the two algorithms using different configurations, a series of tests were designed. 10 discretization levels were chosen, ranging from a fine resolution where DPSL could only produce predictions on a few samples in the test set, to a low resolution where DPSL rarely met unobserved states. Laser readings were discretized

**Algorithm 0.1** Predictive Sequence Learning (PSL)

**Require:** *ψ* = (*e*1,*e*2,...,*eT*) where *T* denotes the length of the training set **Require:** *α*ˆ as the precision constant, see text

```
1: let t ← 1
 2: let η = (e1,e2,...,et)
 3: let C ← ∅
 4: let Eˆ as Eq. 9
 5: if Eˆ (et+1) < αˆ then
 6: let hnew = CreateHypothesis (η, C) as defined by Algorithm 0.2
 7: C (hnew) ← 1
 8: end if
 9: Update confidences C (h) as defined by Equation 4
10: set t = t + 1
11: if t<T then
12: goto 2
13: end if
```
**Algorithm 0.2** CreateHypothesis

**Require:** *η* = (*e*1,*e*2,...,*et*) **Require:** *C* : *h* → [0, 1] **Require:** *α* as defined by Eq. 3 1: **let** *M*ˆ (*h*) as Eq. 8 2: **let** *M*¯ = *h* | *E*¯ *h* (*et*+1) ≥ *α*ˆ ∧ *M*ˆ (*h*) *>* 0 where *α*ˆ is the precision constant, see Section 2.3 3: **if** *M*¯ = ∅ **then** 4: **let** *E*∗ be a new membership function with center *et* and base *ε* 5: **return** *hnew* : (Υ*t is E*∗) ⇒ Υ*t*+1 *is E*¯ 6: **else** 7: **let** ¯ *h* ∈ *M*¯ 8: **if** *C* ¯ *h* = 1 **then** 9: **return** *null* 10: **else** 11: **let** *E*∗ be a new membership function with center *et*−|¯ *h*| and base *ε* 12: **return** *hnew* : Υ*t*−|¯ *h*| *is E*∗, Υ*t*−|¯ *h*|+1 *is E*¯ *h* |¯ *h*|−1 ,..., Υ*t is E*¯ *h* 0 ⇒ Υ*t*+1 *is E*¯ 13: **end if** 14: **end if**

## **2.3 Parameters and task specific configuration**

A clear description of parameters is important for any learning algorithm. Parameters always introduce the risk that the learning algorithm is tuned towards the evaluated task, producing better results than it would in the general case. We have therefore strived towards limiting the number of parameters of PSL. The original design of PSL was completely parameter free, with the exception that continuous data was discretized using some discretization method. The version of PSL proposed here can be seen as a generalization of the original algorithm (Billing et al., 2011) where the width *ε* of the membership function *E* determines the discretization resolution. In addition, a second parameter is introduced, referred to as the *precision constant α*ˆ. *α*ˆ is in fuzzy logic terminology an *α-cut*, i.e., thresholds over the fuzzy membership function in the interval [0, 1] (Klir & Yuan, 1995).

*ε* controls how generously FPSL matches hypotheses. A high *ε* makes the algorithm crisp but typically increases the precision of predictions when a match is found. Contrary, a low *ε* reduces the risk that FPSL reaches unobserved states at the cost of a decreased prediction performance. The high value of *ε* can be compared to a fine resolution data discretization for the previous version of PSL.

*α*ˆ is only used during learning, controlling how exact a specific *E*¯ has to be before a new hypothesis with a different *E*¯ is created. A large *α*ˆ reduces prediction error but typically results in more hypotheses being created during learning.

Both *ε* and *α*ˆ controls the tolerance to random variations in the data and can be decided based on how exact we desire that FPSL should model the data. Small *ε* in combination with large *α*ˆ will result in a model that closely fits the training data, typically producing small prediction errors but also a low coverage.

## **3. Evaluation**

8 Robot Control

**Require:** *ψ* = (*e*1,*e*2,...,*eT*) where *T* denotes the length of the training set

6: **let** *hnew* = *CreateHypothesis* (*η*, *C*) as defined by Algorithm 0.2

9: Update confidences *C* (*h*) as defined by Equation 4

*h* | *E*¯ *h* (*et*+1) ≥ *α*ˆ ∧ *M*ˆ (*h*) *>* 0

5: **return** *hnew* : (Υ*t is E*∗) ⇒ Υ*t*+1 *is E*¯

 Υ*t*−|¯

= 1 **then**

4: **let** *E*∗ be a new membership function with center *et* and base *ε*

11: **let** *E*∗ be a new membership function with center *et*−|¯

*h*| *is E*∗, Υ*t*−|¯

*h*|+1 *is E*¯ *h* |¯ *h*|−1

where *α*ˆ is the precision constant, see Section 2.3

*h*| and base *ε*

⇒ Υ*t*+1 *is E*¯

*h* 0

,..., Υ*t is E*¯

**Algorithm 0.1** Predictive Sequence Learning (PSL)

**Require:** *α*ˆ as the precision constant, see text

1: **let** *t* ← 1

3: **let** *C* ← ∅ 4: **let** *E*ˆ as Eq. 9 5: **if** *E*ˆ (*et*+1) *< α*ˆ **then**

8: **end if**

10: **set** *t* = *t* + 1 11: **if** t<T **then** 12: **goto** 2 13: **end if**

7: *C* (*hnew*) ← 1

**Algorithm 0.2** CreateHypothesis **Require:** *η* = (*e*1,*e*2,...,*et*) **Require:** *C* : *h* → [0, 1] **Require:** *α* as defined by Eq. 3

1: **let** *M*ˆ (*h*) as Eq. 8

3: **if** *M*¯ = ∅ **then**

*h* ∈ *M*¯

*h*

9: **return** *null*

12: **return** *hnew* :

2: **let** *M*¯ =

6: **else** 7: **let** ¯

8: **if** *C* ¯

10: **else**

13: **end if** 14: **end if**

2: **let** *η* = (*e*1,*e*2,...,*et*)

Two tests were performed to evaluate the performance of FPSL and compare it to the previous version. A simulated Robosoft Kompai robot (Robosoft, 2011) was used in the Microsoft RDS simulation environment (Microsoft, 2011). The 270 degree laser scanner of the Kompai was used as sensor data and the robot was controlled by setting linear and angular speeds.

Demonstrations were performed via tele-operation using a joypad, while sensor and motor data were recorded with a temporal resolution of 20 Hz. The dimensionality of the laser scanner was reduced to 20 dimensions using an average filter. Angular and linear speeds were however fed directly into PSL.

The first test (Section 3.1) was designed to compare FPSL and DPSL as prediction algorithms, using sensor data from the simulated robot. The second test (Section 3.2) demonstrates the use of FPSL as a method for LFD.

## **3.1 Sensor prediction**

The two versions of PSL were compared using a series of tests of prediction performance. Even though DPSL and FPSL are similar in many ways, a comparison is not trivial since DPSL works on discrete data whereas FPSL uses continuous data. Prediction performance of DPSL will hence depend on how the data is discretized while the performance of FPSL depends on the parameters *ε* and *α*ˆ.

To capture the prediction performance of the two algorithms using different configurations, a series of tests were designed. 10 discretization levels were chosen, ranging from a fine resolution where DPSL could only produce predictions on a few samples in the test set, to a low resolution where DPSL rarely met unobserved states. Laser readings were discretized

Fig. 2. Results from the prediction evaluation (Section 3.1). Upper plot shows prediction errors for FPSL (solid line) and DPSL (dashed line). Lower plot shows coverage , i.e. the proportion of samples for which the algorithm generated predictions, see Equation 12.

Robot Learning from Demonstration Using Predictive Sequence Learning 245

This evaluation can be seen as a continuation of previous tests with DPSL using a Khepera robot (Billing et al., 2011). The evaluation is here performed in a more complex environment, using a robot with much larger sensor dimensionality. Initial tests showed that DPSL has sever problems to handle the increased sensor dimensionality when used as a controller. A discretization resolution of about 2 m appeared necessary in order to produce satisfying discrimination ability. Even with this relatively low resolution, the 20 dimensional data produced a very large state space causing DPSL to frequently reach unrecognized states. DPSL could control the robot after intense training in parts of the test environment, but could

Vertical bars represent standard deviation.

**3.2 Target reaching**

Fig. 1. Simulation Environment (Microsoft, 2011) used for evaluations. Blue stars and yellow dots represent starting positions used for demonstrations and test runs, respectively. The green area marked with a *G* represents the target position. The white area under star 10 is the robot.

over 0.8 m for the finest resolution, up to 8 m for the lowest resolution. Motor data was discretized over 0.06m/s for the finest resolution up to 0.6 m/s for the lowest resolution. Similarly, 10 *ε* values were chosen, corresponding to a cone base ranging from 0.8 m to 8 m for laser data, and 0.06 m/s up to 0.6 m/s for motor data. *α*ˆ was given a constant value of 0.9, corresponding to a error tolerance of 10% of *ε*.

10 data files were used, each containing a demonstration where the teacher directed the robot from a position in the apartment to the TV, see Figure 1. A rotating comparison was used, where PSL was tested on one demonstration at a time and the other nine demonstrations were used as training data. Prediction performance was measured in meters on laser range data.

### **3.1.1 Results**

The results from the evaluation are illustrated in Figure 2. While the *ε* value of FPSL cannot directly be compared to the discretization level used for DPSL, the two parameters have similar effect on coverage. Prediction error is only calculated on the proportion of the data for which prediction are produced, and consequently, prediction error increases with coverage.

Fig. 2. Results from the prediction evaluation (Section 3.1). Upper plot shows prediction errors for FPSL (solid line) and DPSL (dashed line). Lower plot shows coverage , i.e. the proportion of samples for which the algorithm generated predictions, see Equation 12. Vertical bars represent standard deviation.

### **3.2 Target reaching**

10 Robot Control

Fig. 1. Simulation Environment (Microsoft, 2011) used for evaluations. Blue stars and yellow dots represent starting positions used for demonstrations and test runs, respectively. The green area marked with a *G* represents the target position. The white area under star 10 is the

over 0.8 m for the finest resolution, up to 8 m for the lowest resolution. Motor data was discretized over 0.06m/s for the finest resolution up to 0.6 m/s for the lowest resolution. Similarly, 10 *ε* values were chosen, corresponding to a cone base ranging from 0.8 m to 8 m for laser data, and 0.06 m/s up to 0.6 m/s for motor data. *α*ˆ was given a constant value of 0.9,

10 data files were used, each containing a demonstration where the teacher directed the robot from a position in the apartment to the TV, see Figure 1. A rotating comparison was used, where PSL was tested on one demonstration at a time and the other nine demonstrations were used as training data. Prediction performance was measured in meters on laser range

The results from the evaluation are illustrated in Figure 2. While the *ε* value of FPSL cannot directly be compared to the discretization level used for DPSL, the two parameters have similar effect on coverage. Prediction error is only calculated on the proportion of the data for which prediction are produced, and consequently, prediction error increases with coverage.

robot.

data.

**3.1.1 Results**

corresponding to a error tolerance of 10% of *ε*.

This evaluation can be seen as a continuation of previous tests with DPSL using a Khepera robot (Billing et al., 2011). The evaluation is here performed in a more complex environment, using a robot with much larger sensor dimensionality. Initial tests showed that DPSL has sever problems to handle the increased sensor dimensionality when used as a controller. A discretization resolution of about 2 m appeared necessary in order to produce satisfying discrimination ability. Even with this relatively low resolution, the 20 dimensional data produced a very large state space causing DPSL to frequently reach unrecognized states. DPSL could control the robot after intense training in parts of the test environment, but could

Fig. 3. Results from test runs in simulated environment (Section 3.2). Each bar corresponds to one starting position, see Figure 1. The green part of the bar represents number of successful runs where the robot reached and stopped at the target position in front of the TV. The yellow part represents runs when the robot successfully reached the target, but did not stop.

Robot Learning from Demonstration Using Predictive Sequence Learning 247

The evaluation shows that FPSL produces significantly smaller prediction errors in relation to the coverage than DPSL (Section 3.1). This was expected since FPSL can be trained to produce small prediction errors by keeping a high precision constant *α*ˆ, while the coverage is still kept high by using a large *ε*. In contrast, when using DPSL, one must choose between a small prediction error with low coverage or a high coverage at the price of an increased prediction error. As can be seen in Figure 2, FPSL is also affected by the precision/coverage tradeoff, but not nearly as much as DPSL. Furthermore, the number of generated hypotheses will increase

While FPSL performs much better than DPSL on large and multidimensional state spaces, it should not be seen as a general solution to the dimensionality problem. The increased number of hypotheses results in increased processing and memory requirements. Furthermore, FPSL is still not able to ignore uncorrelated dimensions in data, making it subject to the curse of dimensionality. One potential solution is to modify the size of membership functions in relation to the information content of the dimension. However, initial tests did not produce satisfying results and further experiments in this direction were postponed to future work.

with *α*ˆ, which also has a positive effect on coverage for multidimensional data.

The test was executed 10 times from each starting position.

not compete with FPSL in a more realistic setting. We therefore chose to not make a direct controller comparison, but rather show the behavior of FPSL in an applied and reproducible setting.

FPSL was trained on 10 demonstrations showing how to get to the TV from various places in the apartment, see Figure 1. The performance of FPSL as a method for LFD was evaluated by testing how often the robot reached the target position in front of the TV starting out from 10 different positions than the ones used during training. FPSL controlled the robot by continuously predicting the next sensory-motor event based on the sequence of passed events. The motor part of the predicted element was sent to the robot controller. A standard reactive obstacle avoidance controller was used as fallback in cases where FPSL did not find any match with observed data. The task was considered successfully executed if the target position was reached without hitting any walls or obstacles. The experiment was repeated ten times, producing a total of 100 test runs.

## **3.2.1 Results**

FPSL successfully reached the target position in front of the TV (the green area in Figure 1) in 79% of the test runs. In 68 runs, it stopped in front of the TV as demonstrated, but in 11 runs it failed to stop even though it reached the target position. The distribution over the 10 starting positions illustrated in Figure 3.

## **4. Discussion**

Applied as a robot controller, PSL is a semi-reactive generative model that produces both actions and expected observations, based on recent sensory-motor events. We believe that this approach to robot learning has great potential since the behavior can be learnt progressively and previous knowledge contributes to the interpretation of new events. It is also general in the sense that very little domain specific knowledge is introduced. Memories are stored as sequences of sensory-motor events that in principle can represent any behavior. While PSL efficiently can represent behaviors with short temporal dependencies, it is subject to combinatorial explosion when the behavior requires representations over longer time spans. We argue that the gradually extending memory of PSL, from being purely reactive to containing representations over longer time when needed, provides a good bias in learning. It will however make learning of behaviors that do require long temporal dependencies slow. The fuzzy version of PSL presented in this work does not directly provide a solution to this problem, but is one step towards integrating PSL in a hierarchical structure as discussed in Section 1.2.

The seeds to FPSL came from the observation that a lot of training was required in order to cover the state space of DPSL with satisfying resolution. A better tradeoff between high precision in prediction and coverage would make PSL a more competitive alternative for real world LFD scenarios. Without sacrificing the strong attributes of the original PSL algorithm, such as the model free design, few parameters and progressively growing representations, FPSL was designed.

Expressing PSL with Fuzzy Logic is in many ways a natural extension and generalization of the discrete algorithm. By using a discrete uniform membership function *E* and a *max* operator for defuzzification, FPSL becomes very similar to DPSL. Even though the processing of continuous values does add significant processing requirements in comparison to DPSL, FPSL can still be efficiently implemented as a fuzzy rule controller.

12 Robot Control

not compete with FPSL in a more realistic setting. We therefore chose to not make a direct controller comparison, but rather show the behavior of FPSL in an applied and reproducible

FPSL was trained on 10 demonstrations showing how to get to the TV from various places in the apartment, see Figure 1. The performance of FPSL as a method for LFD was evaluated by testing how often the robot reached the target position in front of the TV starting out from 10 different positions than the ones used during training. FPSL controlled the robot by continuously predicting the next sensory-motor event based on the sequence of passed events. The motor part of the predicted element was sent to the robot controller. A standard reactive obstacle avoidance controller was used as fallback in cases where FPSL did not find any match with observed data. The task was considered successfully executed if the target position was reached without hitting any walls or obstacles. The experiment was repeated ten

FPSL successfully reached the target position in front of the TV (the green area in Figure 1) in 79% of the test runs. In 68 runs, it stopped in front of the TV as demonstrated, but in 11 runs it failed to stop even though it reached the target position. The distribution over the 10 starting

Applied as a robot controller, PSL is a semi-reactive generative model that produces both actions and expected observations, based on recent sensory-motor events. We believe that this approach to robot learning has great potential since the behavior can be learnt progressively and previous knowledge contributes to the interpretation of new events. It is also general in the sense that very little domain specific knowledge is introduced. Memories are stored as sequences of sensory-motor events that in principle can represent any behavior. While PSL efficiently can represent behaviors with short temporal dependencies, it is subject to combinatorial explosion when the behavior requires representations over longer time spans. We argue that the gradually extending memory of PSL, from being purely reactive to containing representations over longer time when needed, provides a good bias in learning. It will however make learning of behaviors that do require long temporal dependencies slow. The fuzzy version of PSL presented in this work does not directly provide a solution to this problem, but is one step towards integrating PSL in a hierarchical structure as discussed in

The seeds to FPSL came from the observation that a lot of training was required in order to cover the state space of DPSL with satisfying resolution. A better tradeoff between high precision in prediction and coverage would make PSL a more competitive alternative for real world LFD scenarios. Without sacrificing the strong attributes of the original PSL algorithm, such as the model free design, few parameters and progressively growing representations,

Expressing PSL with Fuzzy Logic is in many ways a natural extension and generalization of the discrete algorithm. By using a discrete uniform membership function *E* and a *max* operator for defuzzification, FPSL becomes very similar to DPSL. Even though the processing of continuous values does add significant processing requirements in comparison to DPSL,

FPSL can still be efficiently implemented as a fuzzy rule controller.

setting.

**3.2.1 Results**

**4. Discussion**

Section 1.2.

FPSL was designed.

times, producing a total of 100 test runs.

positions illustrated in Figure 3.

Fig. 3. Results from test runs in simulated environment (Section 3.2). Each bar corresponds to one starting position, see Figure 1. The green part of the bar represents number of successful runs where the robot reached and stopped at the target position in front of the TV. The yellow part represents runs when the robot successfully reached the target, but did not stop. The test was executed 10 times from each starting position.

The evaluation shows that FPSL produces significantly smaller prediction errors in relation to the coverage than DPSL (Section 3.1). This was expected since FPSL can be trained to produce small prediction errors by keeping a high precision constant *α*ˆ, while the coverage is still kept high by using a large *ε*. In contrast, when using DPSL, one must choose between a small prediction error with low coverage or a high coverage at the price of an increased prediction error. As can be seen in Figure 2, FPSL is also affected by the precision/coverage tradeoff, but not nearly as much as DPSL. Furthermore, the number of generated hypotheses will increase with *α*ˆ, which also has a positive effect on coverage for multidimensional data.

While FPSL performs much better than DPSL on large and multidimensional state spaces, it should not be seen as a general solution to the dimensionality problem. The increased number of hypotheses results in increased processing and memory requirements. Furthermore, FPSL is still not able to ignore uncorrelated dimensions in data, making it subject to the curse of dimensionality. One potential solution is to modify the size of membership functions in relation to the information content of the dimension. However, initial tests did not produce satisfying results and further experiments in this direction were postponed to future work.

**5. References**

vol.3.

Alissandrakis, A., Nehaniv, C. L. & Dautenhahn, K. (2002). Imitation With ALICE: Learning

Robot Learning from Demonstration Using Predictive Sequence Learning 249

Argall, B. D., Chernova, S., Veloso, M. & Browning, B. (2009). A survey of robot learning from

Billard, A., Epars, Y., Cheng, G. & Schaal, S. (2003). Discovering imitation strategies through

Billing, E. A. (2009). *Cognition Reversed - Robot Learning from Demonstration*, Lic. thesis, Umeå

Billing, E. A. & Hellström, T. (2010). A Formalism for Learning from Demonstration, *Paladyn:*

Billing, E. A., Hellström, T. & Janlert, L. E. (2010). Behavior Recognition for Learning

Billing, E. A., Hellström, T. & Janlert, L. E. (2011). Predictive learning from demonstration, *in*

Brass, M., Bekkering, H., Wohlschläger, A. & Prinz, W. (2000). Compatibility between

Brooks, R. A. (1986). A Robust Layered Control System For A Mobile Robot, *IEEE Journal of*

Byrne, R. W. & Russon, A. E. (1998). Learning by Imitation: a Hierarchical Approach, *The*

Calinon, S. (2009). *Robot Programming by Demonstration - A Probabilistic Approach*, EFPL Press. Calinon, S., Guenter, F. & Billard, A. (2007). On Learning, Representing and Generalizing a

de Rengervé, A., D'halluin, F., Andry, P., Gaussier, P. & Billard, A. (2010). A study

Demiris, J. & Hayes, G. (1997). Do robots ape?, *Proceedings of the AAAI Fall Symposium on*

Demiris, J. & Hayes, G. R. (2002). *Imitation as a dual-route process featuring predictive and learning components: a biologically plausible computational model*, MIT Press, pp. 327–361. Demiris, Y. (1999). *Movement Imitation Mechanisms in Robots and Humans*, PhD thesis,

demonstration, *Robotics and Autonomous Systems* 57(5): 469–483.

University, Department of Computing Science, Umeå, Sweden.

Arkin, R. C. (1998). *Behaviour-Based Robotics*, MIT Press.

Billing, E. A. (2011). www.cognitionreversed.com.

*Automation*, Anchorage, Alaska.

*Robotics and Automation* 2(1): 14–23.

*Epigenetic Robotics*, Lund, Sweden.

*Socially Intelligent Agents*, pp. 28–31.

University of Edinburgh.

Berlin, pp. 186–200.

*Journal of Behavioral Robotics* 1(1): 1–13.

imitative cues., *Brain and cognition* 44(2): 124–43.

*Journal of Behavioral and Brain Sciences* 16(3).

Brooks, R. A. (1991). New Approaches to Robotics, *Science* 253(13): 1227–1232.

*on Systems, Man and Cybernetics, Part A: Systems and Humans* 32: 482–496. Alissandrakis, A., Nehaniv, C. L., Dautenhahn, K. & Saunders, J. (2005). An Approach

to Imitate Corresponding Actions Across Dissimilar Embodiments, *IEEE Transactions*

for Programming Robots by Demonstration: Generalization Across Different Initial Configurations of Manipulated Objects, *Proceedings of 2005 International Symposium on Computational Intelligence in Robotics and Automation*, Ieee, Espoo, Finland, pp. 61–66.

categorization of multi-dimensional data, *Proceedings of the 2003 IEEE/RSJ Intl. Conference on Intelligent Robots and Systems*, Vol. 3, Las Vegas, Nevada, pp. 2398–2403

from Demonstration, *Proceedings of IEEE International Conference on Robotics and*

J. Filipe, A. Fred & B. Sharp (eds), *Agents and Artificial Intelligence*, Springer Verlag,

observed and executed finger movements: comparing symbolic, spatial, and

Task in a Humanoid Robot, *IEEE Transactions on Systems, Man and Cybernetics, Part B. Special issue on robot learning by observation, demonstration and imitation* 37(2): 286–298.

of two complementary encoding strategies based on learning by demonstration for autonomous navigation task, *Proceedings of the Tenth International Conference on*

We found the results from the controller evaluation very promising (Section 3.2). The application environment has been scaled up significantly in comparison to previous work (Billing et al., 2011) and we are now able to perform learning in a fairly realistic setting. When observing these results one should remember that PSL does not solve a spatial task. There is no position sensor or internal map of the environment and the robot is still able to navigate from almost any position in the environment, to a specific target location. The goal is better described as an attractor in a dynamic system, where the robot in interaction with the environment finally reaches a stable state in front of the TV (Figure 1).

An interesting observation is that it is often difficult to predict how well PSL will be able to handle a specific situation. For example, starting position 6 was not passed during any demonstration, but PSL still managed to control the robot such that it reached the target in 7 out of 10 test runs. On the other hand, position 5 and 10 produced worse results than expected. Even though these starting positions were spatially close to several positions passed during the demonstrations, the directions at which the robot reached these positions were different, producing different laser scans, and PSL could consequently not find a suitable match. In some of the cases, inappropriate matches were found and the robot turned in the wrong direction. In other cases, no match at all was found causing the robot to fall back on the reactive controller for a longer period and usually getting stuck in a corner.

The amount of training data used in this evaluation was fairly small. Only one demonstration from each starting position was performed. One reason why FPSL is able to solve the task despite the small amount of training is that all data potentially contribute to every action selection, independently of where in the demonstration it originally took place. Techniques that represent the whole behavior as a sequence with variations, typically require more training since information from the beginning of the demonstration does not contribute to action selection in other parts of the behavior. PSL does not relies on common features within a set of demonstrations and consequently does not require that demonstrations are compared or temporally aligned, see Section 1. In its current design, PSL is of course unable to perform a program-level imitation since it always rely on sensory-motor events, but it does not suffer from a large diversity in the demonstration set as long as the recent sensory-motor events bear necessary information to select a suitable action.

### **4.1 Conclusions and future work**

In this chapter, we show that PSL can be used as a method for LFD, in a fairly realistic setting. The move from a discrete state space used in previous work to the continuous state space appears to have a positive effect on generalization ability and prediction performance, especially on multi-dimensional data. The next step is to conduct experiments with the physical Kompai robot (Robosoft, 2010) in an attempt to verify the results in the real world. The fuzzy version of PSL proposed here, and specifically the introduction of context sets *C*, should be seen as one step towards integrating PSL in a hierarchical architecture. The higher level controller may be another instance of PSL working on a lower temporal resolution, or a completely different control system interacting with PSL by changing the responsibility *λt* (*C*) for each context (Equation 5). For an actual interaction to take place, PSL also has to feed information upwards, to higher level controllers. In previous work on behavior recognition (Billing et al., 2010), we have shown that PSL can be used to compute a bottom-up signal providing information about how well each context corresponds to present circumstances. While this has not been the focus of this chapter, we intend to evaluate these aspects of PSL in future work.

### **5. References**

14 Robot Control

We found the results from the controller evaluation very promising (Section 3.2). The application environment has been scaled up significantly in comparison to previous work (Billing et al., 2011) and we are now able to perform learning in a fairly realistic setting. When observing these results one should remember that PSL does not solve a spatial task. There is no position sensor or internal map of the environment and the robot is still able to navigate from almost any position in the environment, to a specific target location. The goal is better described as an attractor in a dynamic system, where the robot in interaction with the

An interesting observation is that it is often difficult to predict how well PSL will be able to handle a specific situation. For example, starting position 6 was not passed during any demonstration, but PSL still managed to control the robot such that it reached the target in 7 out of 10 test runs. On the other hand, position 5 and 10 produced worse results than expected. Even though these starting positions were spatially close to several positions passed during the demonstrations, the directions at which the robot reached these positions were different, producing different laser scans, and PSL could consequently not find a suitable match. In some of the cases, inappropriate matches were found and the robot turned in the wrong direction. In other cases, no match at all was found causing the robot to fall back on the

The amount of training data used in this evaluation was fairly small. Only one demonstration from each starting position was performed. One reason why FPSL is able to solve the task despite the small amount of training is that all data potentially contribute to every action selection, independently of where in the demonstration it originally took place. Techniques that represent the whole behavior as a sequence with variations, typically require more training since information from the beginning of the demonstration does not contribute to action selection in other parts of the behavior. PSL does not relies on common features within a set of demonstrations and consequently does not require that demonstrations are compared or temporally aligned, see Section 1. In its current design, PSL is of course unable to perform a program-level imitation since it always rely on sensory-motor events, but it does not suffer from a large diversity in the demonstration set as long as the recent sensory-motor events bear