Scribd
Upload
38 views

Application of Reinforcement Learning To A Two Dof Robot Arm Control

This document summarizes a study that applied reinforcement learning techniques to control a two degree-of-freedom robot arm. The researchers developed several approaches to optimize the computational efforts of reinforcement learning for this application. Specifically, they proposed: 1) Unevenly distributing the state space based on the target position to reduce the number of states. 2) Creating a library of learned policies for different tasks/positions to accelerate learning for new tasks. 3) Integrating a fuzzy logic system into the reward function to simplify its approximation of multiple parameters. The experimental results showed that these new approaches improved the learning performance and solution quality compared to conventional reinforcement learning methods.

Uploaded by

marco
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
38 views

Application of Reinforcement Learning To A Two Dof Robot Arm Control

This document summarizes a study that applied reinforcement learning techniques to control a two degree-of-freedom robot arm. The researchers developed several approaches to optimize the computational efforts of reinforcement learning for this application. Specifically, they proposed: 1) Unevenly distributing the state space based on the target position to reduce the number of states. 2) Creating a library of learned policies for different tasks/positions to accelerate learning for new tasks. 3) Integrating a fuzzy logic system into the reward function to simplify its approximation of multiple parameters. The experimental results showed that these new approaches improved the learning performance and solution quality compared to conventional reinforcement learning methods.

Uploaded by

marco
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

0415

Annals of DAAAM for 2009 & Proceedings of the 20th International DAAAM Symposium, Volume 20, No. 1, ISSN 1726-9679
ISBN 978-3-901509-70-4, Editor B. Katalinic, Published by DAAAM International, Vienna, Austria, EU, 2009
Make Harmony Between Technology and Nature, and Your Mind will Fly Free as a Bird
Annals of DAAAM International

APPLICATION OF REINFORCEMENT LEARNING TO A TWO DOF ROBOT ARM


CONTROL

ALBERS, A[lbert]; YAN, W[enjie] & FRIETSCH, M[arkus]

Abstract: Automatic manipulators control poses a complex because the inertia of the upper arm changes due to a variable
challenge for control systems, which have to deal with various angle (Denzinger & Laureyns, 2008).
dynamic and nonlinear effects. This paper presents a novel
approach for motion control of a two DOF robot arm based on
reinforcement learning. A new method to reduce the high
computational efforts, which come along with this method, is
presented. In order to accelerate the convergence of the
learning process, a fuzzy logic system is integrated in the
reward function. For further optimization of the implemented
algorithm a library of already learned motions is created. The
experimental result shows a significant improvement of
learning efficiency.
Key words: reinforcement learning, robotics, computational
intelligence

1. INTRODUCTION Fig. 1. Robot arm model

One of the biggest challenges in current research in robotics Since the redundancy of the inverse kinematic, the angle set
is, that robots “leave” their well structured environment and are which is closer to the target state will be selected as end
confronted with new tasks in a more complex environment. position. Because our work is focused on how to reach the goal,
Due to this, it can only be successful resp. useful, when it is the motion accuracy of intermediate states is not interested thus
able to adapt itself and learn from experiences. Reinforcement the interval of a state close to the target could be much smaller
Learning (RL), a branch of machine learning (Mitchell & Tom, than of a state far from it. A relative position is adopted to
1997), is one possible solution. RL is a learning process, which normalize the distribution of state space, whose values are
uses reward and punishment from the interaction with always changed depending on the target position. Moreover, a
environment to learn a policy for achieving tasks. Various RL fuzzy-logic system is integrated in the reward function for
methods e.g. Q-learning (Watkins, 1989) have been studied in evaluating the executed actions and a coordinate transformation
the recent decades and it is shown that two problems must be is applied to represent different tasks with one Q-table in
considered. At first, the high computational efforts: RL is library.
disturbed by the “curse of dimensionality” (Bellman, 1957),
which refers to the tendency of a state space to grow 2.2 Unevenly distribution of state space
exponentially in its dimension, that is, in the number of state The state space of robot arm consists of its angles and angle
variables (Sutton & Barto, 1998). Secondly, a Q-table is created velocities. A Q-table in this case is a 6-dimensional hyperspace,
for one specific task. It requires an extreme large space to store here means angle, velocity and action:
policies for all possible tasks, which restricts the practical
(1)
application of this learning method strongly.
In (Martin & De Lope, 2007), the author presents a According to the approach of (Martin & De Lope, 2007),
distributed architecture in RL to solve the first problem, which we can decompose the Q-table in and
uses some small, low dimensional Q-tables instead of a global . With the help of the fixed target position,
high-dimensional Q-table with the evaluations of actions for all each position can be calculated as a constant value plus a
states. In this paper, another ways based on an optimized state difference as shown in the follow equations:
space representation are proposed for improving the learning
ability of RL
(2)
.
2. APPROACH If the state space is not built according to but to , the
2.1 Overview target position can always be located at . In
The schematic of robot arm with two degree of freedom is this case, the state space can be easily divided with different
illustrated in Fig. 1. The system is dynamic and nonlinear intervals.
0416

• State space with unevenly and evenly distribution of angle


position with an increment of (Fig. 4)
(3)
• Learning with and without the use of learned policy (Fig. 5)
A state space for a joint can be created with
elements and for both joints with elements. In
comparison, elements are needed to
generate a conventional state space with fixed increments (we
assume that so as to get the same accuracy).

2.3 Q-table library


As mentioned before, the movement based on the relative
position is independent of the absolute position . For two Fig. 4. Test results with different state space
movements with the same , we can change the coordinate
system so as to make both of them the same. Because the
dynamic model changes with , it is necessary to save the
Q-tables after or . By using ten discrete values for
every dimension, a cubic shaped library with 1000 elements is
created for different . The library is searched
through these three values for a new task and the Q-table is
loaded as initial policy in reinforcement learning.
Fig. 5. Test results with and without policy file
2.4 Reward function with fuzzy logic system
A fuzzy logic (Michels & Kai, 2002) is an extension of 4. CONCLUSION
boolean logic to multi-valued logic, which can be used to
approximate complex function with simple and well understood This paper introduced novel approaches for optimizing the
control conditions. Because a reward function is composed of computational efforts in reinforcement learning by means of an
different parameters and challenging to describe in mathematic, unevenly distribution of state space and building the policy
a fuzzy logic system is integrated in the reward function so as library so as to accelerate the learning process for new tasks.
to reduce the modeling expenses. The angle and velocity of The experimental results showed that the learning performance
each joint are defined as input fuzzy sets and the reward value can be improved significantly with this new approach. At the
as output. The reward values for both joints are calculated same time, the quality of calculated solutions is better than the
before summed to one. The fuzzy rule is depicted in Fig. 2 and solutions with conventional methods. For these reasons, we
the reward function of single joint in Fig. 3.
conclude that our approach is a suitable technique to enhance
the learning ability of reinforcement learning for the given
application. However, this method can be only used for point-
to-point motion control and still difficult for online learning.
The presented methods are currently being implemented on a
higher DOF robot arm.
Fig.2. Control rule for fuzzy logic system
5. REFERENCES

Bellman, R. E. (1957). Dynamic Programming, ISBN: 978-


0691079516, Princeton University Press, Princeton, NJ
Denzinger, J. & Laureyns, I. (2008). A study of reward
functions in reinforcement learning on a dynamic model of
a two-link planar robot, Proceedings of DAAAM 2008,
ISBN: 978-3-901509-68-1, Vienna, Austria, Oct 2008
Martin H, J. A. & De Lope, J. (2007). A Distributed
Reinforcement Learning Architecture for Multi-Link
Robots, Proceedings of ICINCO 2007, Angers France, May
2007
Fig. 3. Reward function for single joint Michels & Kai (2002). Fuzzy-Regelung: Grundlagen, Entwurf,
Analyse, Springer, ISBN: 3540-43548-4, Berlin, Heidelberg
3. EXPERIMENTAL RESULTS Mitchell, Tom M. (1997). Machine learning, McGraw-Hill,
ISBN 0-07-042807-7, New York
The experiments confirm the success of our approaches. Sutton, R. S. & Barto, A. G. (1998). Reinforcement Learning:
The figures show the results of the operating process of An Introduction, MIT Press, ISBN: 978-0-262-19398-6,
performed simulations: Cambridge

You might also like