requirement to the involved parameters are followed and detected relations are described in a qualitative or - if possible - in a quantitative way. For instance, relation between strut length and workspace dimension can be described by kinematic problem, see section 3. Furthermore, it is necessary to determine the influence of the identified reconfiguration parameters to other requirements and detect possible goal-conflicts. It is obvious, that an extension of the strut length will lead to a higher moved mass, which might decrease the performance of the robotic

In Paragraph 2.1 general approaches for reconfigure robotic systems were introduced, in case of parallel robots for industrial applications three reconfiguration approaches are of major relevance. These different concepts are discussed in detailed, following the classification

• *Static reconfiguration (SR)* requires switching off the robot and rearranging machine parts such as drives, struts, or joints as well as complete sub-assemblies manually (see section

• *Dynamic reconfiguration type 1 (DR I)* is carried out during operation of the system. Physical dimensions are not changed, but different configurations of the kinematic structure are used. Here singularities of type 1 have to be passed by the robot (see paragraph *system*

• *Dynamic reconfiguration type 2 (DR II)* is realized during operation of the system. The changing of kinematic properties is effected by the adjustment of geometric parameters such as strut length or modification of the DoF by joint couplings in order to avoid or

Table 1 illustrates the description of *SR*, *DR I* and *DR II* by characterizing the robot status and the configuration chance. The detailed discussion of the concepts is made in the following sections, whereas section 5.4 points out two case studies of reconfigurable parallel robots. **Reconfiguration Approach Robot Status Configuration Change**

**Static (SR)** off manual rearrangement of

**Dynamic Type I (DR I)** on usage of different configuration

**Dymanic Type II (DR II)** on adjustment of geometries by

According to the term of definition of static reconfiguration, the rearrangement of mechanical components is the objective to deal with. Based on a starting concept, the introduced methodology supports identification of reasonable reconfiguration modules (*RMi*) of the parallel robot. Within this modularization it is possible to allocate several requirements to the system components. In this way it can be sorted out, which key property of the system is affected by what reconfiguration strategy. Figure 6 shows the *RM*1...3 and their interrelation

as well as the interfaces between single machine components of parallel robots.

Table 1. Reconfiguration approaches for parallel robotic systems.

components/ subassemblies

by passing through singularities

machine components

replace singularities (see paragraph *auxiliary drives and adaptive machine elements*).

**5. Reconfiguration approaches for parallel robotic systems**

system.

2.2.1).

proposed by Krefft (2006):

*inherent reconfiguration*).

**5.1 Static reconfiguration**

Fig. 6. SysML model for the modularization of the system components of a parallel robot.

The application of different grippers (*RM*1) according to the requirement spread accomplishes the type of requirement, that result from the variation of the manufactured product. In this case, the workpiece size, geometries, surface or material properties are meant. The workspace or the required structural stiffness are only slightly influenced by the gripper. To integrate different kinds of grippers, e.g. mechanical or pneumatic ones, adequate constructive and control interfaces have to be designed. Furthermore, the changing manner of energy supply must be considered e.g. at the mobile platform. The change in mass and moments of inertia influences operation forces, accuracy and performance.

The strut/joint assembly (*RM*2) can be reconfigured through the installation of different strut kits. Here, diverse strut length results in variations of the position and dimension of the workspace. In addition, changes in strut length affect performance and accuracy due to changes in moved masses and transmission ratio of the kinematic chain, thence possible acceleration, maximum speed and eigenfrequencies. In terms of a modular construction kit, the interfaces between modules have to correspond with one another.

In general the drives are the base of the characteristic kinematic chain of a parallel robot and represent the 3rd reconfiguration module (*RM*3) together with the drive/joint fixture. They are assembled at or near the frame platform and are distinguishable between rotational and translational DoF. According to static reconfiguration, the specific properties such as weight or number and accessibility of the application points, must be observed during the design process for reconfigurability. Therefore, the frame can determine the ability for the static reconfiguration of parallel robots. If the frame provides a set of patterned holes to assemble the drives or base assembly points of the struts, the workspace dimension and position as well as stiffness can be influenced according to the identified requirement spread.

In order to show the benefits of static reconfiguration concerning the variation of the workspace position and dimension, a simple 2-DoF parallel mechanism is considered. The kinematic 2-RPR structure consists of passive rotary joints (R) at the base and the end-effector platform and active prismatic joints (P) (see Fig. 7).

The point of origin is fixed at base *A* and the Tool Centre Point (TCP) is defined by the vector *X* = (*xp*, *yp*)*T*. The distance between base *A* and *B* is given by b. The vector

a) Kinematic structure of the RPR mechanism.

*q*1 = *xp yp* = *x*2 *p* + *y*2 *p* (4) of Parallel Robotic Systems 13

Requirement Oriented Reconfiguration of Parallel Robotic Systems 399

(a) (b) (c)

to be varied to fulfil different tasks. Since the main drawback of parallel mechanisms is the small workspace-to-installation-space ratio, dynamic reconfiguration type I facilitates the passing of singularties. Hence, several workspaces are combined during operation and the useable overall workspace is enlarged. The different workspaces are mathematically defined by different solutions of the direct kinematic problem (DKP) (assembly modes) or the indirect kinematic problem (IKP) (working modes). Each mode constitutes its own workspace. The workspaces of different working modes are divided by configuration space singularities, where assembly mode workspaces are distinguished by actuator singularities. In order to determine each of the possible configurations a vector **k** of binary configuration parameters

where *kIKP* expresses the different solutions of the IKP and *kDKP* denotes the different set of drive positions (solutions of the DKP). Each sprocket chain has two solutions of the IKP. Hence, a parallel mechanism with *nSC* sprocket chains and *nDKP* solutions of the DKP can have 2*nSC* · *nDKP* theoretical configurations, whereas not every solution is a real one and therefore relevant for the reconfiguration strategy. In order to use different workspaces or configurations generally (several) change-over configurations has to be captured to

drive 3

The described approach of DR I (passing through singularities to enlarge the workspace) should now be demonstrated using the example of the CRC 562 TRIGLIDE robot. The robot is based on the Linear-Delta structure. It consists of three linear driven kinematic chains, which guide the working platform. Each chain has a parallelogram structure in order to keep the end-effector platform in a constant orientation. This arrangement allows three DoF,

x y z

drive 2

**k** = (*k*1, *k*2, ...)=(**k***IKP*, **k***DKP*), (8)

drive 1

Fig. 8. Influence of the reconfiguration parameter on the position and dimension of the workspace: a) varied strut length *l*1, *l*2; b) varied frame diameter *b* and c) varied angle *α* at

the frame connection level.

reconfigure the mechanism.

(+1 or -1) has been established by Budde (2008) with

Fig. 9. Kinematic structure of the TRIGLIDE robot.

$$q\_2 = \begin{bmatrix} \cos(\alpha) \cdot (b - \chi\_p) \\ \sin(\alpha) \cdot b \cdot \chi\_p \end{bmatrix} = \tag{5}$$

$$=\sqrt{\cos(\alpha)\cdot(b-\mathfrak{x}\_p))^2+(y\_p+b\cdot\sin(\alpha))^2}\tag{6}$$

b) Relations for kinematic analysis.

Fig. 7. Kinematic structure and relation for kinematic analysis of the considered RPR mechanism.

*q* = (*q*1, *q*2)*T* includes the actuated strut variables, whereas each strut can be moved between the intervals [*qmin*, *qmax*]. The parameter *α* describes the angular displacement *B* of base *B* to *A*. For the kinematic analysis the solution of the relation between end-effector coordinates and drive parameter (*f*(*X*, *q*) = 0) is essential. Here, the direct and inverse kinematic problem constitutes the motion behavior constrained by the geometric dimensions. The direct kinematic problem calculates the position and orientation of the TCP out of the given drive displacement. If the TCP coordinates are defined, the inverse kinematic problem assesses the actuator displacement. The relation given in Figure 7 b) is important for kinematic analysis. In paragraph 3 the existence and kinematic influences of singularities of parallel robots were discussed briefly. In case of the shown planar 2-DoF parallel mechanism the configuration space singularities can be associated with the boundary of the respective workspace.

$$\text{The first-order coupling between the two-dimensional } \mathcal{N} \text{-matrices is the only possible } \mathcal{N} \text{-matrices with } \mathcal{N} = \{0, 1, 2, \dots, N\} \text{ and } \mathcal{N} = \{0, 1, 2, \dots, N\}.$$

Referring to the kinematic structure in Figure 7 a) a workspace analysis to point out several reconfiguration concepts was done using MATLAB. First, the variation of the interval [*qmin*, *qmax*] can be associated as a change of strut kits. Here length range can be varied according to the boundary of each requirement in the chosen use case. The second reconfiguration concept is to reposition the assembly points. This can be shown by the variation of the parameter b in a way that the distance between *A* and *B* is in- or decreasable. Furthermore, the change *α* expresses the repositioning of *B* not only in x-, but also in y- direction to *B*. The following Figure 8 demonstrates the possibilities of workspace repositioning by changing the drive assembly points and varying strut length. In field (a) the range *l* with respect to [*qmin*...*qmax*] of the actuated struts is changed. Illustration (b) shows the repositioning of the drive carrying struts in x-direction. In the kinematic model this is equivalent to the adjustment of parameter *b*. Furthermore in (c) the reconfiguration of the base assembly points in y-direction by varying the angel *α* between the drives is shown.

### **5.2 Dynamic reconfiguration type I**

Actuation singularities are existent for the condition

Dynamic reconfiguration of type I focuses on the passing through singularities during operation without adding specific components. Therefore, it also can be constituted as a system inherent ability of the parallel robot to reconfigure itself. Based on the works of *Budde* [Budde (2010), Budde (2007)] this reconfiguration approach is introduced in detail, using the TRIGLIDE structure as an example.

As pointed out before, workspace dimension is an essential system parameter, which has

12 Will-be-set-by-IN-TECH

=

Fig. 7. Kinematic structure and relation for kinematic analysis of the considered RPR

*q* = (*q*1, *q*2)*T* includes the actuated strut variables, whereas each strut can be moved between the intervals [*qmin*, *qmax*]. The parameter *α* describes the angular displacement *B* of base *B* to *A*. For the kinematic analysis the solution of the relation between end-effector coordinates and drive parameter (*f*(*X*, *q*) = 0) is essential. Here, the direct and inverse kinematic problem constitutes the motion behavior constrained by the geometric dimensions. The direct kinematic problem calculates the position and orientation of the TCP out of the given drive displacement. If the TCP coordinates are defined, the inverse kinematic problem assesses the actuator displacement. The relation given in Figure 7 b) is important for kinematic analysis. In paragraph 3 the existence and kinematic influences of singularities of parallel robots were discussed briefly. In case of the shown planar 2-DoF parallel mechanism the configuration space singularities can be associated with the boundary of the respective workspace.

Referring to the kinematic structure in Figure 7 a) a workspace analysis to point out several reconfiguration concepts was done using MATLAB. First, the variation of the interval [*qmin*, *qmax*] can be associated as a change of strut kits. Here length range can be varied according to the boundary of each requirement in the chosen use case. The second reconfiguration concept is to reposition the assembly points. This can be shown by the variation of the parameter b in a way that the distance between *A* and *B* is in- or decreasable. Furthermore, the change *α* expresses the repositioning of *B* not only in x-, but also in y- direction to *B*. The following Figure 8 demonstrates the possibilities of workspace repositioning by changing the drive assembly points and varying strut length. In field (a) the range *l* with respect to [*qmin*...*qmax*] of the actuated struts is changed. Illustration (b) shows the repositioning of the drive carrying struts in x-direction. In the kinematic model this is equivalent to the adjustment of parameter *b*. Furthermore in (c) the reconfiguration of the base assembly points in y-direction by varying the angel *α* between the drives is shown.

Dynamic reconfiguration of type I focuses on the passing through singularities during operation without adding specific components. Therefore, it also can be constituted as a system inherent ability of the parallel robot to reconfigure itself. Based on the works of *Budde* [Budde (2010), Budde (2007)] this reconfiguration approach is introduced in detail, using the

As pointed out before, workspace dimension is an essential system parameter, which has

*q*1 = *xp yp* = *x*2 *p* + *y*2

> *cos*(*α*) · (*b* − *xp*) *sin*(*α*) · *b* · *yp*

b) Relations for kinematic analysis.

*cos*(*α*) · (*b* − *xp*))2 + (*yp* + *b* · *sin*(*α*))2 (6)

*q*2 =  *p* (4)

= (5)

a) Kinematic structure of the RPR mechanism.

Actuation singularities are existent for the condition

**5.2 Dynamic reconfiguration type I**

TRIGLIDE structure as an example.

mechanism.

Fig. 8. Influence of the reconfiguration parameter on the position and dimension of the workspace: a) varied strut length *l*1, *l*2; b) varied frame diameter *b* and c) varied angle *α* at the frame connection level.

to be varied to fulfil different tasks. Since the main drawback of parallel mechanisms is the small workspace-to-installation-space ratio, dynamic reconfiguration type I facilitates the passing of singularties. Hence, several workspaces are combined during operation and the useable overall workspace is enlarged. The different workspaces are mathematically defined by different solutions of the direct kinematic problem (DKP) (assembly modes) or the indirect kinematic problem (IKP) (working modes). Each mode constitutes its own workspace. The workspaces of different working modes are divided by configuration space singularities, where assembly mode workspaces are distinguished by actuator singularities. In order to determine each of the possible configurations a vector **k** of binary configuration parameters (+1 or -1) has been established by Budde (2008) with

$$\mathbf{k} = (k\_1, k\_2, \ldots) = (\mathbf{k}\_{IKP}, \mathbf{k}\_{DKP})\_\prime \tag{8}$$

where *kIKP* expresses the different solutions of the IKP and *kDKP* denotes the different set of drive positions (solutions of the DKP). Each sprocket chain has two solutions of the IKP. Hence, a parallel mechanism with *nSC* sprocket chains and *nDKP* solutions of the DKP can have 2*nSC* · *nDKP* theoretical configurations, whereas not every solution is a real one and therefore relevant for the reconfiguration strategy. In order to use different workspaces or configurations generally (several) change-over configurations has to be captured to reconfigure the mechanism.

Fig. 9. Kinematic structure of the TRIGLIDE robot.

The described approach of DR I (passing through singularities to enlarge the workspace) should now be demonstrated using the example of the CRC 562 TRIGLIDE robot. The robot is based on the Linear-Delta structure. It consists of three linear driven kinematic chains, which guide the working platform. Each chain has a parallelogram structure in order to keep the end-effector platform in a constant orientation. This arrangement allows three DoF,

of Parallel Robotic Systems 15

Requirement Oriented Reconfiguration of Parallel Robotic Systems 401

x y z

x y z

Fig. 11. Main-workspaces in working configuration (top); Selected changeover

(a) (b)

Fig. 12. Effect principle and realized prototype of the adaptive revolute joint used for

B2

Fig. 13. Kinematic scheme and geometries of the RRRRR-mechanism (a) and solutions of the

B1

L11

C

q1

L12

L22

A1 A2 LB

(b)

C´

B2

q2 L21

q2 L21

manipulator with two DoF. The cranks at the base points *A*1 and *A*2 are actuated. The two passive rods *L*12 and *L*22 are coupled to each other by a passive revolute joint in the end-effector point C. The kinematic chain originating at base *A*1 and the corresponding crank is connected by the described adaptive revolute joint, which is marked by ⊗ (see Figure 13).

configurations (middle, buttom) of the TRIGLIDE robot.

Faxial Mfriction Faxial + Factuator Mfriction

C

y

(a)

q1

L12

L22

A1 A2

x LB

clearance

outer ring

dynamic reconfiguration type II.

B1

L11

inner ring

DKP (b).

additionally a fourth rotary axis can be mounted at the end-effector platform. Thus, a hybrid kinematic structure occurs. In Figure 9 the kinematic structure of the TRIGLIDE robot without the rotary axis is shown.

Fig. 10. Prototype of the TRIGLIDE robot (left); Usable working-configurations and resulting overall workspace (right).

In case of the TRIGLIDE robot the change between different configuration workspaces is accomplished by passing through singularities. Implemented strategies to go through the different singularity type are described in Budde (2010). In Figure 10 the prototype of the TRIGLIDE robot is shown as well as the desired main-workspaces and the resulting overall workspace.

As mentioned before the so called change-over configurations have to be captured in order to reach the main-workspaces, which are mostly capable to fulfil requirement spreads regarding to workspace. Several configurations are shown in Figure 11. The presented approach to reconfigure the TRIGLIDE robot leads to challenging tasks in control. The necessary steps and the adapted control strategy is explained in Maass (2006).

To complete the explanation of the named reconfiguration approaches for parallel robots, the next paragraph deals with *DR II* with a focus on adaptive machine elements.

### **5.3 Dynamic reconfiguration type II**

Unlike the aforementioned reconfiguration strategies, the adjustment of geometric parameters is the subject of *DR II*. Here, the variation of strut length or the modification of kinematic DoF by joint couplings or lockings e.g. in order to avoid singularities or dislocate them are reasonable reconfiguration scenarios. Since this reconfiguration strategy demands no shut-down of the system, specific machine elements are required. In Schmitt (2010) an adaptive revolute joint is presented. Besides the basic functions of a passive joint, this revolute joint enables adaption of friction and stiffness. This functionality is based on the so-called quasi-statical clearance adjustment in the bearings using piezoelectric actuators Stechert (2007). This clearance modification between the inner and outer ring leads to an adaption of friction and stiffness (see Fig. 12 a)). Depending on the previous friction moment caused by the preload of the bearings the realized prototype enables continuous locking of rotary DoF by friction magnification of 440 percent [Inkermann (2011)]. In Figure 12 the first prototype of the adaptive revolute joint is illustrated.

In order to demonstrate a reconfiguration strategy of parallel mechanisms in the sense of *DR II* with an adaptive machine element a simple RRRRR-mechanism is further considered, following Schmitt (2010). The RRRRR-mechanism is a fully closed-loop planar parallel 14 Will-be-set-by-IN-TECH

additionally a fourth rotary axis can be mounted at the end-effector platform. Thus, a hybrid kinematic structure occurs. In Figure 9 the kinematic structure of the TRIGLIDE robot without

Fig. 10. Prototype of the TRIGLIDE robot (left); Usable working-configurations and resulting

In case of the TRIGLIDE robot the change between different configuration workspaces is accomplished by passing through singularities. Implemented strategies to go through the different singularity type are described in Budde (2010). In Figure 10 the prototype of the TRIGLIDE robot is shown as well as the desired main-workspaces and the resulting overall

As mentioned before the so called change-over configurations have to be captured in order to reach the main-workspaces, which are mostly capable to fulfil requirement spreads regarding to workspace. Several configurations are shown in Figure 11. The presented approach to reconfigure the TRIGLIDE robot leads to challenging tasks in control. The necessary steps and

To complete the explanation of the named reconfiguration approaches for parallel robots, the

Unlike the aforementioned reconfiguration strategies, the adjustment of geometric parameters is the subject of *DR II*. Here, the variation of strut length or the modification of kinematic DoF by joint couplings or lockings e.g. in order to avoid singularities or dislocate them are reasonable reconfiguration scenarios. Since this reconfiguration strategy demands no shut-down of the system, specific machine elements are required. In Schmitt (2010) an adaptive revolute joint is presented. Besides the basic functions of a passive joint, this revolute joint enables adaption of friction and stiffness. This functionality is based on the so-called quasi-statical clearance adjustment in the bearings using piezoelectric actuators Stechert (2007). This clearance modification between the inner and outer ring leads to an adaption of friction and stiffness (see Fig. 12 a)). Depending on the previous friction moment caused by the preload of the bearings the realized prototype enables continuous locking of rotary DoF by friction magnification of 440 percent [Inkermann (2011)]. In Figure 12 the first

In order to demonstrate a reconfiguration strategy of parallel mechanisms in the sense of *DR II* with an adaptive machine element a simple RRRRR-mechanism is further considered, following Schmitt (2010). The RRRRR-mechanism is a fully closed-loop planar parallel

next paragraph deals with *DR II* with a focus on adaptive machine elements.

the adapted control strategy is explained in Maass (2006).

prototype of the adaptive revolute joint is illustrated.

the rotary axis is shown.

overall workspace (right).

**5.3 Dynamic reconfiguration type II**

workspace.

Fig. 11. Main-workspaces in working configuration (top); Selected changeover configurations (middle, buttom) of the TRIGLIDE robot.

Fig. 12. Effect principle and realized prototype of the adaptive revolute joint used for dynamic reconfiguration type II.

manipulator with two DoF. The cranks at the base points *A*1 and *A*2 are actuated. The two passive rods *L*12 and *L*22 are coupled to each other by a passive revolute joint in the end-effector point C. The kinematic chain originating at base *A*1 and the corresponding crank is connected by the described adaptive revolute joint, which is marked by ⊗ (see Figure 13).

Fig. 13. Kinematic scheme and geometries of the RRRRR-mechanism (a) and solutions of the DKP (b).

of Parallel Robotic Systems 17

Requirement Oriented Reconfiguration of Parallel Robotic Systems 403

A1 A2 LB

case studies are explained and the benefits of reconfigurable robots are shown. The first one deals with the reconfiguration of a pick-and-place robot to an assembly robot, using the approach of static reconfiguration. Another case study is the classification of solar cells under consideration of their efficiency. This case study will illustrate an example of dynamic

**5.4.1 Example I - reconfiguration of a "'pick-and-place"' robot into an assembly robot** In order to highlight the introduced procedure and the concept of static reconfiguration in this section case study is presented. Therefore, a "'pick-and-place"' robot should be reconfigured for an assembly task, while handling object and DoF of the system remain the same. To demonstrate the reconfiguration concept the kinematic structure described in section 5.1 is used. The use case assembly is characterised by a small workspace. However, higher process forces and accuracy are required. At the same time the performance (in this case numbers of assembly cycles per minute) compared with the "'pick-and-place"' task is not such important. The lifetime cost should be minimal. With regard to the introduced reconfiguration modules (see section 5.1), a change of the rack radius (*RM*3) leads to a meaningful reconfiguration

Fig. 15. Relation network for the static reconfiguration of a tow DoF parallel mechanism.

C

LR

q1

L11

**(3)**

Fig. 14. Strategy of passing through singularity by blocking the adaptive joint.

B1

1=40°

A1 A2 LB

q1

LR

C L22

B1

reconfiguration of type I.

concept.

L12

L11 1=40°

**(1)**

L12

L21

B2

B1

L11 1=40°

L22

path through singularity

L21

B2

**(2)**

A1 A2 LB

L21

singular position

B2

q1

L22 L12

C

LR

The kinematic description of the mechanism with its two closed loop chains *i* = 1, 2 can be geometrically done with the cartesian end-effector coordinates **X** = [*xc*, *yc*] *T*, the base coordinates **A** = [*xAi*, *yAi*] *T*, which can be derived by *LB*, the actuator angles **qi** and the given geometric parameters *Lii* with respect to the base frame {0}. In eq. 9 the kinematic describtion of the mechanism is given, considering the parameters and variables shown in Figure 13 (a).

$$\mathbf{F} = \begin{bmatrix} \begin{bmatrix} \mathbf{x}\_c \\ \mathbf{y}\_c \end{bmatrix} - \begin{bmatrix} \mathbf{x}\_{Ai} + \cos(q\_i) \cdot L\_{i1} \\ \mathbf{y}\_{Ai} + \sin(q\_i) \cdot L\_{i1} \end{bmatrix} \end{bmatrix}^2 - L\_{i2}^2 = \mathbf{0}. \tag{9}$$

Expression 9 provides a system of equations which relate the end-effector coordinated **X** to the actuator coordinates **qi**. The equation for the direct kinematic problem (DKP) has two solutions, in case of the RRRRR-mechanism. They constitute to different configurations of the mechanism (see Figure 13 (b)). Changing these configurations an actuator singularity occurs, when moving from position *C* to *C*� . In general actuator singularities can occur in the workspace or separate workspaces into different areas.

In order to reconfigure the mechanism or to use these different workspaces adequate strategies have to be developed, in case of *DR II* using the adaptive revolute joint. Assembling the adaptive revolute joint in point *B*1 (see Fig. 13 (a)) and blocking it entirely, it is possible to treat the mechanism as a RRRR 4-bar structure, with one actuated revolute joint at base point *A*1. According to the theorem of GRASHOF different kinds of 4-bar mechanisms exist, depending on the geometric aspect ratio of the links [Kerle (2007)]:

> *lmin* + *lmax < l* � + *l* �� ability to turn around *lmin* + *lmax* = *l* � + *l* �� ability to snap-through *lmin* + *lmax > l* � + *l* �� no ability to turn around.

*lmin* and *lmax* are the lengths of the shortest and the longest link in the structure and l', l" are the lengths of the remaining links. In case of the shown 5-bar mechanism with the adaptive revolute joint, one link length can be adjusted according to the desired mechanism property. The resulting length *LR* can be calculated by

$$\mathbf{L}\_{R} = \sqrt{\mathbf{L}\_{12}^{2} + \mathbf{L}\_{11}^{2} - 2 \cdot L\_{12} \cdot L\_{11} \cdot \cos(\Theta\_{1})}.\tag{10}$$

Beside the aforementioned possibility to reconfigure the robot, it is feasible to develop strategies to pass through an actuator singularity in an assured manner by blocking and releasing the mechanism at point *B*1 by means of the adaptive revolute joint. This reconfiguration strategy is shown in Fig. 14.

The aspect ratios in the example are chosen by *Li*1/(*Li*2, *LB*) = 3/4. If the mechanism is close to a singular position (Fig. 14.1), the adaptive revolute joint will be locked, (*θ*1 = 40◦). The resulting 4-bar mechanism moves on a path, which is defined by a segment of a circle of radius *LR* with center at *A*1 through the singular position (Figure 14.2). Subsequently, the mechanism is in another configuration, which is not a singular position (Fig. 14.3) and the adaptive joint can be released.

#### **5.4 Case studies**

In the previous paragraphs different approaches to reconfigure parallel robotic systems were explained on a theoretical level. In order to give examples and fields of application two 16 Will-be-set-by-IN-TECH

The kinematic description of the mechanism with its two closed loop chains *i* = 1, 2 can

geometric parameters *Lii* with respect to the base frame {0}. In eq. 9 the kinematic describtion of the mechanism is given, considering the parameters and variables shown in Figure 13 (a).

> *xAi* + *cos*(*qi*) · *Li*1 *yAi* + *sin*(*qi*) · *Li*1

Expression 9 provides a system of equations which relate the end-effector coordinated **X** to the actuator coordinates **qi**. The equation for the direct kinematic problem (DKP) has two solutions, in case of the RRRRR-mechanism. They constitute to different configurations of the mechanism (see Figure 13 (b)). Changing these configurations an actuator singularity

In order to reconfigure the mechanism or to use these different workspaces adequate strategies have to be developed, in case of *DR II* using the adaptive revolute joint. Assembling the adaptive revolute joint in point *B*1 (see Fig. 13 (a)) and blocking it entirely, it is possible to treat the mechanism as a RRRR 4-bar structure, with one actuated revolute joint at base point *A*1. According to the theorem of GRASHOF different kinds of 4-bar mechanisms exist, depending

*lmin* and *lmax* are the lengths of the shortest and the longest link in the structure and l', l" are the lengths of the remaining links. In case of the shown 5-bar mechanism with the adaptive revolute joint, one link length can be adjusted according to the desired mechanism property.

Beside the aforementioned possibility to reconfigure the robot, it is feasible to develop strategies to pass through an actuator singularity in an assured manner by blocking and releasing the mechanism at point *B*1 by means of the adaptive revolute joint. This

The aspect ratios in the example are chosen by *Li*1/(*Li*2, *LB*) = 3/4. If the mechanism is close to a singular position (Fig. 14.1), the adaptive revolute joint will be locked, (*θ*1 = 40◦). The resulting 4-bar mechanism moves on a path, which is defined by a segment of a circle of radius *LR* with center at *A*1 through the singular position (Figure 14.2). Subsequently, the mechanism is in another configuration, which is not a singular position (Fig. 14.3) and the adaptive joint

In the previous paragraphs different approaches to reconfigure parallel robotic systems were explained on a theoretical level. In order to give examples and fields of application two

�� ability to turn around

�� ability to snap-through

�� no ability to turn around.

*T*, which can be derived by *LB*, the actuator angles **qi** and the given

− *Li*2

. In general actuator singularities can occur in the

11 − 2 · *L*12 · *L*11 · *cos*(Θ1). (10)

2

*T*, the base

2 = 0. (9)

be geometrically done with the cartesian end-effector coordinates **X** = [*xc*, *yc*]

coordinates **A** = [*xAi*, *yAi*]

**F** =

occurs, when moving from position *C* to *C*�

The resulting length *LR* can be calculated by

reconfiguration strategy is shown in Fig. 14.

can be released.

**5.4 Case studies**

 *xc yc* −

workspace or separate workspaces into different areas.

on the geometric aspect ratio of the links [Kerle (2007)]:

*lmin* + *lmax < l*

*lmin* + *lmax* = *l*

*lmin* + *lmax > l*

**L***R* =

 *L*2 12 + *L*2

� + *l*

� + *l*

� + *l*

Fig. 14. Strategy of passing through singularity by blocking the adaptive joint.

case studies are explained and the benefits of reconfigurable robots are shown. The first one deals with the reconfiguration of a pick-and-place robot to an assembly robot, using the approach of static reconfiguration. Another case study is the classification of solar cells under consideration of their efficiency. This case study will illustrate an example of dynamic reconfiguration of type I.

### **5.4.1 Example I - reconfiguration of a "'pick-and-place"' robot into an assembly robot**

In order to highlight the introduced procedure and the concept of static reconfiguration in this section case study is presented. Therefore, a "'pick-and-place"' robot should be reconfigured for an assembly task, while handling object and DoF of the system remain the same. To demonstrate the reconfiguration concept the kinematic structure described in section 5.1 is used. The use case assembly is characterised by a small workspace. However, higher process forces and accuracy are required. At the same time the performance (in this case numbers of assembly cycles per minute) compared with the "'pick-and-place"' task is not such important. The lifetime cost should be minimal. With regard to the introduced reconfiguration modules (see section 5.1), a change of the rack radius (*RM*3) leads to a meaningful reconfiguration concept.

Fig. 15. Relation network for the static reconfiguration of a tow DoF parallel mechanism.

of Parallel Robotic Systems 19

Requirement Oriented Reconfiguration of Parallel Robotic Systems 405

The values of the cell efficiency are subject to a distribution e.g. Gaussian, which means that a certain percentage rate (depends on the standard deviation *σ*) of the classes corresponding to the cell efficiency are distributed with a dedicated expectation value of *μ*. The remaining

Under this assumption the concept of DR I, the TRIGLIDE robot makes use of the two workspaces. The high frequented boxes to classify the solar cells are located in the first, the less ones in the second workspace. Due to the required time *TR* to change the configuration, it is not efficient to vary workspace every cycle. Therefore, the proposed box distribution according to the expectation value *μ* is feasible. In Figure 17 (a) the workspaces with the corresponding intervals of the Gaussian distribution and in (b) the concept of the storage boxes are shown. The indices of the boxes *cmno* describe the actual workspace (m), the column

TR

(a)

(b)

Fig. 17. Exemplarily probability distribution of supplied solar cells and used workspaces (a) and top view of a feasible conceptual design of a classification facility with the TRIGLIDE

Since the introduced application examples highlight the advantages of reconfigurable parallel robotic systems to fulfil changing requirement, specific restrictions have to be considred in order to apply reasonable reconfiguration concepts. For this reason in the following section major aspects for the assessment and their interrelations to the presented static and dynamic

*μ* ± *nd* · *σ*, (11)

P (X = x )i

xi

+ n - nd d

6. Drop the cell in the box

classes, which belong to

P (X = x )i

7. Pick up the next cell → return to process step 3

where *nd* constitutes the desired interval of *σ*.

(n) and the row (o), so that the position is dedicated.

xi

robot without periphery (b).

reconfiguration strategies are proposed.

As can be seen in Figure 15 with the new use case the requirements *process forces*, *accuracy* and *workspace dimension* are refined. Since the workspace is directly influenced by the rack radius, the accuracy is influenced via the sensitivity. For increasing rack radius sensitivity in x-direction also increases and leads to higher accuracy in this direction. At the same time accuracy decreases in y-direction. In addition, the accuracy is influenced by the elastic deformation of the structure. The dimension of the deformation correlates with the value of the process forces and the stiffness of the structure. For the same stiffness higher process forces result in higher deformation and, therefore, decreased accuracy. This leads to a goal conflict which can be reduced with higher stiffness of the structure. At the same time the stiffness can be increased due to an adaption of the rack radius. Here, a higher rack radius results in a higher stiffness.

Fig. 16. Exemplary use cases for reconfiguration of a "'pick-and-place"' robot (a) into an assembly robot (b).

Static Reconfiguration from "'pick-and-place"' to assembly system in this case is carried out by increasing the rack radius. Figure 16 highlights the reconfiguration, showing the two use cases. In the first use case push-buttons are delivered via a first conveyer belt and placed into a container on a second belt. For this reason the workspace is elongated. After reconfiguration the same push-buttons are taken from a container and assembled into a casing. As the push-buttons provide a snap-in fastening operational forces in y-direction are needed and forces in x-direction might appear due to alignment deviations. Hence, the second configuration complies with the demanded changes.

### **5.4.2 Use case II - solar cell classification**

The end-of-line process-step within the fabrication of crystalline solar cells is the classification of the cells according to their efficiency. Here, the "'flash-test"' is conducted, where each cell is lighted by a defined flash (time, brightness per area) comparable to the sun irradiation. The result determines the quality class of a cell. In order to sort the cells according to their class after the flash-test, handling devices are necessary. Due to the high number of classes, which are physically represented as square boxes, the handling robot has to provide a relatively huge workspace. Additionally, the requirement of cycle time about 1 sec. per "'pick-and-place"' operation is very challenging. This combination makes the application of parallel robots feasible, if it is possible to overcome the poor workspace-to-installation-space ratio. The process flow of the classification can be summarized as following:

6. Drop the cell in the box

18 Will-be-set-by-IN-TECH

As can be seen in Figure 15 with the new use case the requirements *process forces*, *accuracy* and *workspace dimension* are refined. Since the workspace is directly influenced by the rack radius, the accuracy is influenced via the sensitivity. For increasing rack radius sensitivity in x-direction also increases and leads to higher accuracy in this direction. At the same time accuracy decreases in y-direction. In addition, the accuracy is influenced by the elastic deformation of the structure. The dimension of the deformation correlates with the value of the process forces and the stiffness of the structure. For the same stiffness higher process forces result in higher deformation and, therefore, decreased accuracy. This leads to a goal conflict which can be reduced with higher stiffness of the structure. At the same time the stiffness can be increased due to an adaption of the rack radius. Here, a higher rack radius results in a

Fig. 16. Exemplary use cases for reconfiguration of a "'pick-and-place"' robot (a) into an

Static Reconfiguration from "'pick-and-place"' to assembly system in this case is carried out by increasing the rack radius. Figure 16 highlights the reconfiguration, showing the two use cases. In the first use case push-buttons are delivered via a first conveyer belt and placed into a container on a second belt. For this reason the workspace is elongated. After reconfiguration the same push-buttons are taken from a container and assembled into a casing. As the push-buttons provide a snap-in fastening operational forces in y-direction are needed and forces in x-direction might appear due to alignment deviations. Hence, the second

The end-of-line process-step within the fabrication of crystalline solar cells is the classification of the cells according to their efficiency. Here, the "'flash-test"' is conducted, where each cell is lighted by a defined flash (time, brightness per area) comparable to the sun irradiation. The result determines the quality class of a cell. In order to sort the cells according to their class after the flash-test, handling devices are necessary. Due to the high number of classes, which are physically represented as square boxes, the handling robot has to provide a relatively huge workspace. Additionally, the requirement of cycle time about 1 sec. per "'pick-and-place"' operation is very challenging. This combination makes the application of parallel robots feasible, if it is possible to overcome the poor workspace-to-installation-space ratio. The

higher stiffness.

assembly robot (b).

configuration complies with the demanded changes.

process flow of the classification can be summarized as following: 1. Conveying the cells to the classifying facility via feed band

5. Position the cell at the determined box according to the efficiency value

2. Determination of the efficiency value (flash test)

4. Picking up the cell with the robot

3. Define the right box/position of the robots' end-effector

**5.4.2 Use case II - solar cell classification**

7. Pick up the next cell → return to process step 3

The values of the cell efficiency are subject to a distribution e.g. Gaussian, which means that a certain percentage rate (depends on the standard deviation *σ*) of the classes corresponding to the cell efficiency are distributed with a dedicated expectation value of *μ*. The remaining classes, which belong to

$$
\mu \pm n\_d \cdot \sigma\_\prime \tag{11}
$$

where *nd* constitutes the desired interval of *σ*.

Under this assumption the concept of DR I, the TRIGLIDE robot makes use of the two workspaces. The high frequented boxes to classify the solar cells are located in the first, the less ones in the second workspace. Due to the required time *TR* to change the configuration, it is not efficient to vary workspace every cycle. Therefore, the proposed box distribution according to the expectation value *μ* is feasible. In Figure 17 (a) the workspaces with the corresponding intervals of the Gaussian distribution and in (b) the concept of the storage boxes are shown. The indices of the boxes *cmno* describe the actual workspace (m), the column (n) and the row (o), so that the position is dedicated.

Fig. 17. Exemplarily probability distribution of supplied solar cells and used workspaces (a) and top view of a feasible conceptual design of a classification facility with the TRIGLIDE robot without periphery (b).

Since the introduced application examples highlight the advantages of reconfigurable parallel robotic systems to fulfil changing requirement, specific restrictions have to be considred in order to apply reasonable reconfiguration concepts. For this reason in the following section major aspects for the assessment and their interrelations to the presented static and dynamic reconfiguration strategies are proposed.

of Parallel Robotic Systems 21

Requirement Oriented Reconfiguration of Parallel Robotic Systems 407

Object Handling

The need to increase flexibility of production systems via reconfiguration is mainly driven by the customers of the end product. Rising demands for individual products and shorter periods for product changes force the manufacturers to think about reconfigurable production equipment. In order to present ideas and strategies for reconfigurable robotic systems this contribution firstly shows the general meaning of reconfiguration and gives a brief literature

Such as parallel robots are in focus in this chapter, this special class of robotic systems was explained and the essential benefits as well as disadvantages, in particular the occurrence of singularities within the workspace and at its borders were introduced. Based on high potential of parallel robots to cover a huge range of requirements, an adequate requirements management referring to robotic reconfiguration has been introduced. Thereby, the identification of requirement spreads and the derivation of reconfiguration parameters were considered. The main part of the contribution examines the distinctive static and dynamic (type I and II) reconfiguration approaches of parallel robotic system. Each strategy is first discussed theoretically before two case studies are presented. In order to assess the different reconfiguration approaches, a matrix-based scheme is proposed, which serves the

Potential aims of further research activities should deal with the validation of the adapted development procedure with respect to reconfiguration issues. Further, regarding parallel robots, the proposed reconfiguration strategies should be surveyed to their transferability to other parallel structures. Another aspect, to be investigated, is the control implementation of the *SR* or *DR* concepts, in order to reduce efforts of changes. The treatment of an entirely changed kinematic structure after reassembling its mechanical components (*SR*) or the

review by classifying the most common types into offline and online approaches.

Workspace

Opertaion Force

Degree of Freedom

Accuracy

Concept-Requirement Interactions Concept-Complexity Interactions

Aspects of Complexity

Performance

Duration Time

Mechanical Complexity

Computational Complexity

Parameter Range

Robustness

Reconfiguration type Reconfiguration concept

Fig. 18. Reconfiguration-approach-matrix.

**7. Conclusion and further research**

designer as a decision support.

Reconfiguration-Relevant Requirements

Strut kits; diverse length Varing fixture distances Gripping principle; geometry, DoF

Adaptive revolut joint Air cylinders (Krefft, 2006b)

Singularity passing; inertia of TCP

Static (SR)

Dynamic Type II (DR II) Dynamic Type I (DR I)

Legend

## **6. Assessment of reconfiguration concepts**

As pointed out before the reconfiguration of robotic systems should not be considered as an end itself. It should rather follow an aim and should be subject of a systematic analysis. According to Steiner [(Steiner (1998), Steiner (1999))] changeability (reconfiguration) may not be cost efficient for systems which:

However, different use cases within the life time (e.g. handling muffins, packing croissants) and specific secondary functions (e.g. gripping principle) call for reconfiguration of robotic systems. Demands of high life-cycle times emphasize this need. In order to support the assessment of the before mentioned reconfiguration concepts a set of distinguishing aspects was derived. These aspects point out for which type of requirement spread a static or dynamic reconfiguration should be considered. Furthermore, the complexity of each reconfiguration approach is taken into account considering:

The matrix in Figure 18 indicates which reconfiguration strategy is contributing to what reconfiguration-relevant requirement, see section 4.2. In applying the different approaches it is important to show up more detailed realizations. For this reason different reconfiguration concepts are stated following the introduced concepts. For instance, static reconfiguration can be realized by different strut kits and varying fixture distances. Relevant reconfiguration concepts are given for each approach.

While static reconfiguration concepts do not demand specific computational adaption and have high potencies to change several system properties such as workspace and accuracy, long time is needed to carry out the reconfiguration. The system has to be switched off, in order to rearrange several components. Thus, static reconfigurations approaches offer robust solutions since the configurations are changed manually. Regarding to dynamic reconfiguration concepts duration time is the essential advantage. However, to realize dynamic reconfiguration of a parallel robotic system specific components (DR II) have to developed or challenging control tasks have to be solved (DR I). Besides this in particular DR II features high potential to affect system properties.

This brief assessment of the introduced reconfiguration approaches should not be seen as a strict selecting tool. It rather helps to reason different concepts in early design phases of parallel robotic systems.

20 Will-be-set-by-IN-TECH

As pointed out before the reconfiguration of robotic systems should not be considered as an end itself. It should rather follow an aim and should be subject of a systematic analysis. According to Steiner [(Steiner (1998), Steiner (1999))] changeability (reconfiguration) may not

• are highly precendented systems in slowly changing markets and no customer variety,

• are developed for ultrahigh performance markets with no performance loss allowables. However, different use cases within the life time (e.g. handling muffins, packing croissants) and specific secondary functions (e.g. gripping principle) call for reconfiguration of robotic systems. Demands of high life-cycle times emphasize this need. In order to support the assessment of the before mentioned reconfiguration concepts a set of distinguishing aspects was derived. These aspects point out for which type of requirement spread a static or dynamic reconfiguration should be considered. Furthermore, the complexity of each reconfiguration

• *Duration time* characterizes the period needed to adapt the robotic systems performance,

• *Mechanical complexity* indicates the complexity of mechanical modifications and additional

• *Computational complexity* indicates the complexity of computational modifications and additional algorithms, including costs to realize a reconfigurable robotic system.

• *Parameter range* characterizes the potential of a reconfiguration approach to change several

• *Robustness* indicates the vulnerability of a reconfiguration concept due to unforeseen

The matrix in Figure 18 indicates which reconfiguration strategy is contributing to what reconfiguration-relevant requirement, see section 4.2. In applying the different approaches it is important to show up more detailed realizations. For this reason different reconfiguration concepts are stated following the introduced concepts. For instance, static reconfiguration can be realized by different strut kits and varying fixture distances. Relevant reconfiguration

While static reconfiguration concepts do not demand specific computational adaption and have high potencies to change several system properties such as workspace and accuracy, long time is needed to carry out the reconfiguration. The system has to be switched off, in order to rearrange several components. Thus, static reconfigurations approaches offer robust solutions since the configurations are changed manually. Regarding to dynamic reconfiguration concepts duration time is the essential advantage. However, to realize dynamic reconfiguration of a parallel robotic system specific components (DR II) have to developed or challenging control tasks have to be solved (DR I). Besides this in particular

This brief assessment of the introduced reconfiguration approaches should not be seen as a strict selecting tool. It rather helps to reason different concepts in early design phases of

components, including costs to realize a reconfigurable robotic system.

system properties e.g. workspace dimension in a wide range.

• are highly expedient, short life systems without needed product variety,

**6. Assessment of reconfiguration concepts**

be cost efficient for systems which:

• are insensitive to change over time, and

approach is taken into account considering:

operating conditions e.g. varying payload

DR II features high potential to affect system properties.

concepts are given for each approach.

parallel robotic systems.

both manually and automatically.

## **7. Conclusion and further research**

The need to increase flexibility of production systems via reconfiguration is mainly driven by the customers of the end product. Rising demands for individual products and shorter periods for product changes force the manufacturers to think about reconfigurable production equipment. In order to present ideas and strategies for reconfigurable robotic systems this contribution firstly shows the general meaning of reconfiguration and gives a brief literature review by classifying the most common types into offline and online approaches.

Such as parallel robots are in focus in this chapter, this special class of robotic systems was explained and the essential benefits as well as disadvantages, in particular the occurrence of singularities within the workspace and at its borders were introduced. Based on high potential of parallel robots to cover a huge range of requirements, an adequate requirements management referring to robotic reconfiguration has been introduced. Thereby, the identification of requirement spreads and the derivation of reconfiguration parameters were considered. The main part of the contribution examines the distinctive static and dynamic (type I and II) reconfiguration approaches of parallel robotic system. Each strategy is first discussed theoretically before two case studies are presented. In order to assess the different reconfiguration approaches, a matrix-based scheme is proposed, which serves the designer as a decision support.

Potential aims of further research activities should deal with the validation of the adapted development procedure with respect to reconfiguration issues. Further, regarding parallel robots, the proposed reconfiguration strategies should be surveyed to their transferability to other parallel structures. Another aspect, to be investigated, is the control implementation of the *SR* or *DR* concepts, in order to reduce efforts of changes. The treatment of an entirely changed kinematic structure after reassembling its mechanical components (*SR*) or the

of Parallel Robotic Systems 23

Requirement Oriented Reconfiguration of Parallel Robotic Systems 409

Jantapremjit, P., Austin, D. (2001). Design of a Modular Self-Reconfigurable Robot. In: Proc.

Ji, Z., Song, P. (1998). Design of a Reconfigurable Platform Manipulator. In: Journal of Robotic

Jovane, F. et al. (2002). Design issues for reconfigurable PKMs. In: Proc. of the 4th Chemnitz

Kerle, H., Pittschellis, R., Corves, B. (2007). Einführung in die Getriebelehre. Teubner Verlag,

Koren, Y. et al. (1999). Reconfigurable Manufacturing Systems. Annals of the CRIP, Vol. 48,

Krefft, M., Brüggemann, H., Herrmann, G. & Hesselbach, J. (2006). Reconfigurable Parallel

Lee, G.H. (1997). Reconfigurability Consideration Design of Components and Manufacturing

Maass, J. et al. (2006). Control strategies for enlarging a spatial parallel robot's workspace by

Merlet, J.P. (1997). DEMOCRAT: A DEsign MethOdology for the Conception of Robot with

Merlet, J.-P. (2001). Parallel Robots - Solid Mechanisms and its applications, Vol. 74, ISBN

O'Brien, J.-F.; Wen, J.-T. (2001). Kinematic Control of Parallel Robots in the Presence of

Park, F.C., Jin Wook Kim (1998). Manipulability and singularity analysis of multiple robot

Pritschow, G. (2000). Parallel Kinematic Machines (PKM) - Limitations and New Solutions. CIRP Anals-Manifacturing Technology. Vol. 49, No. 1, 2000, pp. 275-295. Riedel, M., Nefzi, M., and Corves, B. (2010). Grasp Planning for a Reconfigurable Parallel

Riedel, M., Nefzi, M., Hüsing, M., and Corves, B. (2008). An Adjustable Gripper as a

Roozenburg, N.F.M, Dorst, K., (1991). Some Guidelines for the Development of Performance

Schmitt, J., Inkermann, D., Raatz, A., Hesselbach, J., Vietor, T. (2010). Dynamic

Robots and Systems, IROS, Grenoble France, pp. 1630-1636.

1402003854, Kluwer Academic Publishers, Dordrecht.

Robots: Combining High Flexibility and Short Cycle Times. In: *Journal of Production*

Systems. In: The International Journal of Advanced Manufacturing Technology, Vol.

change of configuration. In Proc. of the 5th Chemnitz Parallel Kinematics Seminar,

parallel ArchiTecture. In: Proceedings of the International Conference on Intelligent

Unstable Singularities. In: Proceedings of IEEE International Conference on Robotics

systems: a geometric approach, In: Proceedings of the IEEE International Conference

Robot with an Underactuated Arm Structure. In: Mechanical Sciences, Vol. 1, 2010,

Reconfigurable Robot with a parallel Structure. In: Proc. of the 2nd Int. Workshop on Fundamental Issues and Future Research Directions for Parallel Mechanisms and

Specifications in Product Design. In: Proc. of the 8th International Conference on

Reconfiguration of Parallel Mechanisms. In: *EUCOMES - European Conference on*

2001, pp. 38-43.

No. 2, 1999, pp. 527-540.

13, 1997, pp. 376-386.

http://www.molecubes.org/

pp. 33U42. ˝

Manipulators, 2008.

Systems Vol. 15, No. 6, 1998, pp. 341-346.

*Engineering*, Vol. XIII, No.1., pp. 109-112.

Chemnitz (Germany), 2006, pp. 515-530.

and Automation, Seoul, Korea, pp.354-359.

Engineering Design ICED 91, pp.359-366.

on Robotics and Automation, Vol.2, pp.1032-1037.

Parallel Kinematic Seminar, Chemnitz, 2001, pp. 69-82.

Wiesbaden, Germany, ISBN 978-3-8351-0070-1.

of the 2001 Australien Conference on robotics and Automation. Syndney (Australia),

realization of passing through respectively to avoid singular positions (*DR*) are challenging research fields to deal with.

### **8. Acknowledgments**

The authors gratefully thank the Deutsche Forschungsgemeinschaft (DFG) for supporting the CRC 562 "'Robotic Systems for Handling and Assembly"' and the Project "'Application-Oriented Analysis of High Dynamic Handling of Solar Cells with Parallel Robots"'.

### **9. References**

22 Will-be-set-by-IN-TECH

realization of passing through respectively to avoid singular positions (*DR*) are challenging

The authors gratefully thank the Deutsche Forschungsgemeinschaft (DFG) for supporting the CRC 562 "'Robotic Systems for Handling and Assembly"' and the Project "'Application-Oriented Analysis of High Dynamic Handling of Solar Cells with Parallel

Anggreeni, I., van der Voort, M.C. (2008). Classifying Scenarios in a Product Design Process: a

Boudreau, R. and C.M. Gosselin (2001). La synthèse d'une plate-forme de Gough-Stewart pour un espace atteignable prescrit. Mechanism and Machine Theory 36. pp. 327-342. Brouwer, M., van der Voort, M.C. (2008). Scenarios as a Communication Tool in the Design

Budde, C., Helm, M., Last, P., Raatz, A., Hesselbach, J. (2010). Configuration Switching

Budde, C., Last, P., Hesselbach, J. (2007). Development of a Triglide-Robot with Enlarged

Budde, C. (2008). Wechsel der Konfiguration zur Arbeitsraumvergrösserung bei

Clavel, R. (1991). Conception d'un robot parallel rapide a 4 degres de liberte, Ph.D. Thesis,

Fisher, R., Podhorodeski, R.P., Nokleby, S.B. (2004). Design of a Reconfigurable Planar Parallel Manipulator. In: Journal of Robotic Systems Vol. 21, No. 12, (2004), pp. 665-675. Franke, H.-J., Krusche, T. (1999). Design decisions derived from product requirements. In:

Frindt, M. (2001). Modulbasierte Synthese von Parallelstrukturen für Maschinen

Hesselbach, J., Helm, M., Soetebier, S. (2002). Connecting Assembly Modes for Workspace

Inkermann, D., Stechert, C., Vietor, T. (2011). Einsatz multifunktionaler Werkstoffe für die

Conference - Design Synthesis, Enschede.

Conference - Design Synthesis, Enschede.

Rom, Italy, ISBN 978-1-4244-0602-9.

Design, Enschede, pp.371-382.

978-3-642-16784-3.

978-3-8027-8747-8.

EPFL, Lausanne.

978-3-8027-8659-4.

347-356.

231-244.

study towards semi-automated scenario generation. In: Proc. of the 18th CIRP Design

Process: Examples from a Design Course. In: Proc. of the 18th CIRP Design

for Workspace Enlargement In: Schütz, D., Wahl, F. (Ed.), Robotic Systems for Handling and Assembly, Springer-Verlag, Star Series, Berlin, pp. 175-189, ISBN

Workspace In: Proc. of the International Conference on Robotics and Automation,

Parallelrobotern. Ph.D Thesis, TU Braunschweig, Vulkan, Essen, ISBN

Proc. of the 9th CIRP Design Conference - Integration of Process Knowledge into

in der Produktionstechnik. Ph.D, TU Braunschweig, Vulkan, Essen, ISBN

Enlargement. In: Advances in Robotis, Kluwer Academic Publishers, 2002, pp.

Entwicklung einer adaptiven Gelenkachse. In: Proc. of 8. Paderborner Workshop Entwurf mechatronischer Systeme. HNI-Verlagsschriftenreihe Paderborn, 2011, pp.

research fields to deal with.

**8. Acknowledgments**

Robots"'.

**9. References**

**Part 3**

**Vision and Sensors**

*Mechanism Design*, Cluj-Napoca, Rumänien, Springer Berlin / Heidelberg, 2010, pp. 557-565, ISBN 978-90-481-9688-3.

**Part 3**

**Vision and Sensors**

24 Will-be-set-by-IN-TECH

410 Robotic Systems – Applications, Control and Programming

Schmitt, J., Stechert, C., Raatz, A., Hesselbach, J., Vietor, T., Franke, H.-J. (2009).

Setchi, R-M. & Lagos, N. (2004). Reconfigurability and Reconfigurable Manufacturing

Stechert, C. (2010). Modellierung komplexer Anforderungen. Ph.D Thesis, TU Braunschweig,

Stechert, C., Franke, H.-J. (2009). Requirements Models for Collaborative Product Development. In: Proc. of the 19th CIRP Design Conference, Cranfield, pp.24-31. Stechert, C., Franke, H.-J. (2009). Managing Requirements as the Core of Multi-Disciplinary

Stechert, C., Pavlovic, N., Franke, H.-J. (2007). Parallel Robots with Adaptronic Components

Steiner, R. (1998). Systems Architectures and evolvability - definitions and perspectives. In:

Steiner, R. (1999). Enduring Systems Architectures. In: Proc. of the 10th Int. Symp. INCOSE,

Theingi; Chen, I.M.; Angeles, J.; Chuan, Li. (2007). Management of Parallel-Manipulator

Weilkiens, T. (2008). Systems Engineering with SysML/UML - Modeling, Analysis, Design.

Wenger, Ph., Chalbat, D. (1998). Workspace and Assembly Modes in Fully-Parallel

Woelkl, S., Shea, K. (2009). A Computational Product Model for Conceptual Design Using

Wurst, K.-H. & Peting, U. (2002). PKM Concept for Reconfigurable Machine Tools. Proc. of the 4th Chemnitz Parallel Kinematic Seminar, Chemnitz, 2002, pp. 683-696. Yang, G., Chen, I-M., Lim, W.K., Yeo, S.H. (2001). Kinematic Design of Modular

Yim, M., et al. (2007). Modular Self-Reconfigurable Robot Systems - Challenges and

Yim, M.; Duff, D.G.; Roufas, K.D. (2000). "PolyBot: a modular reconfigurable robot," Robotics

and Automation, 2000. In: Proc of ICRA '00., pp.514-520.

557-565, ISBN 978-90-481-9688-3.

*Industrial Informatics (INDIN)*, Berlin.

Congress, Besancon, France, 2007.

Morgan Kaufmann. ISBN:9780123742742

Verlag Dr. Hut, München, ISBN 978-3-86853-545-7.

Proc. of the 8th Annu Symp. INCOSE, Vancouver, 1998.

2009), Cambridge.

1, Is. 3, pp.153-158.

Brighton, 1999.

2009, San Diego.

2007, pp. 43-52

pp.583-600.

*Mechanism Design*, Cluj-Napoca, Rumänien, Springer Berlin / Heidelberg, 2010, pp.

Reconfigurable Parallel Kinematic Structures for Spreading Requirements, In: Proc. of the 14th IASTED International Conference on Robotics and Applications (RA

Systems - State-of-the-art Review, In: *Proc. 2nd. IEEE International Conference on*

Product Development. Cirp Journal of Manufacturing Science and Technology, Vol.

Singularities Using Joint-Coupling. In: Advanced Robotics, Vol. 21, No.5-6,

Manipulators: A Descriptive Study. In: Proc. of the International Conference on Advances in Robotic Kinematics: Analysis and Control. Salzburg, 1998, pp. 117-126.

SysML In: Proc. of the ASME 2009 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE

Reconfigurable In-Parallel Robots. In: Autonomous Robots No. 10, 2001, pp. 83U89.

Opportunities for the Future -. In: IEEE Robotics and Automation Magazine. March

˝

**20**

*India*

**Real-Time Robotic Hand Control**

Jagdish Lal Raheja, Radhey Shyam, G. Arun Rajsekhar and P. Bhanu Prasad

Interpretation of human gestures by a computer is used for human-machine interaction in the area of computer vision [3][28]. The main purpose of gesture recognition research is to identify a particular human gesture and convey information to the user pertaining to individual gesture. From the corpus of gestures, specific gesture of interest can be identified[30][36], and on the basis of that, specific command for execution of action can be given to robotic system[31]. Overall aim is to make the computer understand human body language [14], thereby bridging the gap between machine and human. Hand gesture recognition can be used to enhance human–computer interaction without depending on

Hand gestures are extensively used for telerobotic control applications [20]. Robotic systems can be controlled naturally and intuitively with such telerobotic communication [38] [41]. A prominent benefit of such a system is that it presents a natural way to send geometrical information to the robot such as: left, right, etc. Robotic hand can be controlled remotely by hand gestures. Research is being carried out in this area for a long time. Several approaches have been developed for sensing hand movements and controlling robotic hand [9][13][22]. Glove based technique is a well-known means of recognizing hand gestures. It utilizes sensor attached mechanical glove devices that directly measure hand and/or arm joint angles and spatial position. Although glove-based gestural interfaces give more precision, it limits freedom as it requires users to wear cumbersome patch of devices. Jae-Ho Shin et al [14] used entropy analysis to extract hand region in complex background for hand gesture recognition system. Robot controlling is done by fusion of hand positioning and arm gestures [2][32] using data gloves. Although it gives more precision, it limits freedom due to necessity of wearing gloves. For capturing hand gestures correctly, proper light and camera angles are required. A technique to manage light source and view angles, in an efficient way for capturing hand gestures, is given in [26]. Reviews have been written on the subsets of hand postures and gesture recognition in [6]and [18]. The problem of visual hand recognition and tracking is quite challenging. Many early approaches used position markers or colored bands to make the problem of hand recognition easier, but due to their inconvenience, they cannot be considered as a natural interface for the robot control [24]. A 3D Matlab Kinematic model of a PUMA [1][4][16] robot, is used for executing actions by hand gesture[39]. It can be extended to

any robotic system with a number of specific commands suitable to that system [17].

traditional input devices such as keyboard and mouse.

**1. Introduction**

*Digital Systems Group, Central Electronics Engineering Research Institute (CEERI)/Council of Scientific & Industrial Research (CSIR), Pilani, Rajasthan*

**Using Hand Gestures**

## **Real-Time Robotic Hand Control Using Hand Gestures**

Jagdish Lal Raheja, Radhey Shyam, G. Arun Rajsekhar and P. Bhanu Prasad *Digital Systems Group, Central Electronics Engineering Research Institute (CEERI)/Council of Scientific & Industrial Research (CSIR), Pilani, Rajasthan India*

## **1. Introduction**

Interpretation of human gestures by a computer is used for human-machine interaction in the area of computer vision [3][28]. The main purpose of gesture recognition research is to identify a particular human gesture and convey information to the user pertaining to individual gesture. From the corpus of gestures, specific gesture of interest can be identified[30][36], and on the basis of that, specific command for execution of action can be given to robotic system[31]. Overall aim is to make the computer understand human body language [14], thereby bridging the gap between machine and human. Hand gesture recognition can be used to enhance human–computer interaction without depending on traditional input devices such as keyboard and mouse.

Hand gestures are extensively used for telerobotic control applications [20]. Robotic systems can be controlled naturally and intuitively with such telerobotic communication [38] [41]. A prominent benefit of such a system is that it presents a natural way to send geometrical information to the robot such as: left, right, etc. Robotic hand can be controlled remotely by hand gestures. Research is being carried out in this area for a long time. Several approaches have been developed for sensing hand movements and controlling robotic hand [9][13][22]. Glove based technique is a well-known means of recognizing hand gestures. It utilizes sensor attached mechanical glove devices that directly measure hand and/or arm joint angles and spatial position. Although glove-based gestural interfaces give more precision, it limits freedom as it requires users to wear cumbersome patch of devices. Jae-Ho Shin et al [14] used entropy analysis to extract hand region in complex background for hand gesture recognition system. Robot controlling is done by fusion of hand positioning and arm gestures [2][32] using data gloves. Although it gives more precision, it limits freedom due to necessity of wearing gloves. For capturing hand gestures correctly, proper light and camera angles are required. A technique to manage light source and view angles, in an efficient way for capturing hand gestures, is given in [26]. Reviews have been written on the subsets of hand postures and gesture recognition in [6]and [18]. The problem of visual hand recognition and tracking is quite challenging. Many early approaches used position markers or colored bands to make the problem of hand recognition easier, but due to their inconvenience, they cannot be considered as a natural interface for the robot control [24]. A 3D Matlab Kinematic model of a PUMA [1][4][16] robot, is used for executing actions by hand gesture[39]. It can be extended to any robotic system with a number of specific commands suitable to that system [17].

Real-Time Robotic Hand Control Using Hand Gestures 415

technique to acquire hand gestures and to control robotic system using hand gestures is

• Generation of instructions corresponding to matched gesture, for specified robotic

First step is to capture the image from video stream. It is assumed that while capturing video, black background with proper light soure is used. To minimize number of light

From video stream one frame is captured every 1/3 of a second. Our aim is to identify the frame that contains hand gesture shown by human. For this we are searching the frame in which there is no or least movement. Required frame is identified by comparing three continuously captured frames. *Motion parameter* is determined for each frame by counting total pixels of mismatch. If *motion parameter* is less than the specified threshold value, it is

Analysis of frames to find the frame of interest is carried out by converting the captured frame into a binary frame[23]. Since binary images have values of one or zero., differences of white pixels between newly captured frame and two previously captured frames are determined and they are added together to find *motion parameter.* XOR function is used to

If *frame1, frame2,* and, *frame3* are three matrices containing three frames captured in three

*fr1 = frame1 XOR frame3* (1)

*fr2 = frame2 XOR frame 3* (2)

considered as a frame having least movement of object i.e. the hand of the person.

sources, arrangement of the light sources and camera is made as shown in figure 2.

divided into four subparts:

action.

• Image acquisition to get hand gestures.

**2.1 Image acquisition to get hand gestures**

**Light Source**

**Light Source**

**Camera**

• Extracting hand gesture area from captured frame.

Fig. 2. Arrangement of light sources for image aquisition

find the differences in white pixels [5][15].

continuous time slots respectively then:

• Determining gesture by pattern matching using PCA algorithm.

In this chapter, we have proposed a fast as well as automatic hand gesture detection and recognition system. Once a hand gesture is recognised, an appropriate command is sent to a robot. Once robot recieves a command, it does a pre-defined work and keeps doing until a new command arrives. A flowchart for overall controlling of a robot is given in figure 1.

Fig. 1. Flow diagram for robot control using hand gestures

## **2. Methodology**

The proposed methodology involves acquisition of live video from a camera for gesture identification. It acquire frames from live video stream in some given time interval[15]. In this case frame capture rate for gesture search is 3 frames per second. The overall proposed technique to acquire hand gestures and to control robotic system using hand gestures is divided into four subparts:

• Image acquisition to get hand gestures.

414 Robotic Systems – Applications, Control and Programming

In this chapter, we have proposed a fast as well as automatic hand gesture detection and recognition system. Once a hand gesture is recognised, an appropriate command is sent to a robot. Once robot recieves a command, it does a pre-defined work and keeps doing until a new command arrives. A flowchart for overall controlling of a robot is given in figure 1.

Video Capture of hand movement

Normalization (Improving image brightness)

Smoothing (Removing Noise)

Frame contain object of

Frame

Yes

No

Crop unused extra part (i.e. removal of wrist, arm etc)

Compare and match the gesture using Principal Component Analysis (PCA)

> Identification of Gesture and Extaction of instruction set corresponding to the matched gesture

Passing the instructions to robot for execution of corresponding

Fig. 1. Flow diagram for robot control using hand gestures

Commands.

The proposed methodology involves acquisition of live video from a camera for gesture identification. It acquire frames from live video stream in some given time interval[15]. In this case frame capture rate for gesture search is 3 frames per second. The overall proposed

**2. Methodology**

contains object of interest?

### **2.1 Image acquisition to get hand gestures**

First step is to capture the image from video stream. It is assumed that while capturing video, black background with proper light soure is used. To minimize number of light sources, arrangement of the light sources and camera is made as shown in figure 2.

Fig. 2. Arrangement of light sources for image aquisition

From video stream one frame is captured every 1/3 of a second. Our aim is to identify the frame that contains hand gesture shown by human. For this we are searching the frame in which there is no or least movement. Required frame is identified by comparing three continuously captured frames. *Motion parameter* is determined for each frame by counting total pixels of mismatch. If *motion parameter* is less than the specified threshold value, it is considered as a frame having least movement of object i.e. the hand of the person.

Analysis of frames to find the frame of interest is carried out by converting the captured frame into a binary frame[23]. Since binary images have values of one or zero., differences of white pixels between newly captured frame and two previously captured frames are determined and they are added together to find *motion parameter.* XOR function is used to find the differences in white pixels [5][15].

If *frame1, frame2,* and, *frame3* are three matrices containing three frames captured in three continuous time slots respectively then:

$$\text{fr1} = \text{frame1 XOR frame3} \tag{1}$$

$$\text{fr2} = \text{frame2 XOR frame 3} \tag{2}$$

Real-Time Robotic Hand Control Using Hand Gestures 417

Fig. 4. Binary Image formation (a) Target image (b) HSV conversion (c) Filtered image

To remove these errors, the biggest BLOB (Binary Linked Object) method was applied to the noisy image [24][25]. The error free image is shown in figure 5. The only limitation of this filter was that the BLOB for hand should be the biggest one. In this, masking background

Second step is to extract object of interest from the frame. In our case, object of interest is the part of human hand showing gesture [29]. For this, extra part, other than the hand, is cropped out so that pattern matching can give more accurate results [12]. For cropping extra parts row and column numbers are determined, from where object of interest appears. This is done by searching from each side of binary image and moving forward until white pixels encountered are more than the offset value [25]. Experimental results show that offset value set to 1% of total width gives better result for noise compensation. If size of selected image is

*Min\_col=* minimum value of column number where total number of white pixels are more

*Max\_col=* maximum value of column number where total number of white pixels are more

 *Hor\_offset= m/100* (7) *Ver\_offset=n/100* (8)

would be illuminated, so false detection is not possible from the background [7][10].

(d) Smoothened image (e) Binary image (f) Biggest BLOB

mXn then:

than *Hor\_offset*.

than *Hor\_offset*.

$$\text{mismatch\\_matrix} \equiv \text{fr1 OR } \text{fr2} \tag{3}$$

$$\text{Sum} = \sum\_{i=1}^{r} \sum\_{j=1}^{c} mismatch\\_matrix} \begin{bmatrix} i \end{bmatrix} \begin{bmatrix} j \end{bmatrix} \tag{4}$$

$$\text{Motion parameter } = \frac{sum}{r \, ^\ast c} \tag{5}$$

Here r and c are the number of rows and columns in an image frame. Threshold value is set as 0.01. i.e. if total pixels of mismatch are less than 1% of total pixels in a frame, then it is considered as frame of interest. Required frame is forwarded for further processing and this module again starts searching for next frame of interest.

### **2.2 Extracting hand gesture area from frame of interest**

It may happen that the frame, as shown in fig.3, with a gesture contains extra part along with required part of hand i.e. background objects, blank spaces etc. For better results in pattern matching, unused part must be removed. Therefore hand gesture is cropped out from the obtained frame. Cropping of hand gesture from the obtained frame contains three steps:

Fig. 3. Grayscale image

First step is to convert selected frame into HSV color space. As shown in fig.4, a HSV based skin filter was applied on the RGB format image to reduce lighting effects [8][1]. The thresholds for skin filter used in the experiment were as below:

$$\begin{array}{c} 0.05 \text{$$

The resultant image was segmented to get a binary image of hand[37]. Binary images are bilevel images where each pixel is stored as a single bit (0 or 1). Smoothening was needed, as the output image had some jagged edges as clearly visible in figure 4(c). There can be some noise in the filtered images due to false detected skin pixels or some skin color objects (like wood) in background, it can generate few unwanted spots in the output image as shown in figure 4(e).

416 Robotic Systems – Applications, Control and Programming

[ ][ ] 1 1 Sum = \_ *r c*

Motion parameter \*

module again starts searching for next frame of interest.

Fig. 3. Grayscale image

**2.2 Extracting hand gesture area from frame of interest**

thresholds for skin filter used in the experiment were as below:

Here r and c are the number of rows and columns in an image frame. Threshold value is set as 0.01. i.e. if total pixels of mismatch are less than 1% of total pixels in a frame, then it is considered as frame of interest. Required frame is forwarded for further processing and this

It may happen that the frame, as shown in fig.3, with a gesture contains extra part along with required part of hand i.e. background objects, blank spaces etc. For better results in pattern matching, unused part must be removed. Therefore hand gesture is cropped out from the obtained frame. Cropping of hand gesture from the obtained frame contains three steps:

First step is to convert selected frame into HSV color space. As shown in fig.4, a HSV based skin filter was applied on the RGB format image to reduce lighting effects [8][1]. The

> < < << → < <

The resultant image was segmented to get a binary image of hand[37]. Binary images are bilevel images where each pixel is stored as a single bit (0 or 1). Smoothening was needed, as the output image had some jagged edges as clearly visible in figure 4(c). There can be some noise in the filtered images due to false detected skin pixels or some skin color objects (like wood) in background, it can generate few unwanted spots in the output image as shown in figure 4(e).

0.05 H 0.17 0.1 S 0.3 0.09 V 0.15

*mismatch\_matrix = fr1 OR fr2* (3)

*i j mismatch matrix i j* = = (4)

*r c* = (5)

(6)

*sum*

Fig. 4. Binary Image formation (a) Target image (b) HSV conversion (c) Filtered image (d) Smoothened image (e) Binary image (f) Biggest BLOB

To remove these errors, the biggest BLOB (Binary Linked Object) method was applied to the noisy image [24][25]. The error free image is shown in figure 5. The only limitation of this filter was that the BLOB for hand should be the biggest one. In this, masking background would be illuminated, so false detection is not possible from the background [7][10].

Second step is to extract object of interest from the frame. In our case, object of interest is the part of human hand showing gesture [29]. For this, extra part, other than the hand, is cropped out so that pattern matching can give more accurate results [12]. For cropping extra parts row and column numbers are determined, from where object of interest appears. This is done by searching from each side of binary image and moving forward until white pixels encountered are more than the offset value [25]. Experimental results show that offset value set to 1% of total width gives better result for noise compensation. If size of selected image is mXn then:

$$\text{Hor\\_offset} \equiv \text{m/100} \tag{7}$$

$$\text{Ver\\_offset=n/100} \tag{8}$$

*Min\_col=* minimum value of column number where total number of white pixels are more than *Hor\_offset*.

*Max\_col=* maximum value of column number where total number of white pixels are more than *Hor\_offset*.

Real-Time Robotic Hand Control Using Hand Gestures 419

shows that whether the object continues from the column of Global maxima to first column

Fig. 7. Histogram showing white pixels in each column, with cropping point for hand gesture

or to last column. i.e. whether extra hand is left side of palm or right side of palm.

Fig. 6. Boundary selection for hand cropping

*Min\_row=* minimum value of row number where total number of white pixels are more than *Ver\_offset*.

Max\_row= maximum value of row number where total number of white pixels are more than *Ver\_offset*.

Fig. 5. Error free image

Figure 6 shows Min\_row, Max\_row, Min\_col, and, Max\_col for the selection of the boundary for cropping of hand.

Third step is to remove parts of hand not used in gesture i.e. removal of wrist, arm etc. As these extra parts are of variable length in image frame and pattern matching with gesture database gives unwanted results. Therefore, parts of hand before the wrist need to be cropped out.

Statistical analysis of hand shape shows that either we pose palm or fist, width is lowest at wrist and highest at the middle of palm. Therefore extra hand part can be cropped out from wrist by determining location where minimum width of vertical histogram is found. Figure 7.c and 7.d show global maxima and cropping points for hand gestures in figure 7.a and 7.b respectively.

Cropping point is calculated as:

*Global Maxima = column number where height of histogram is highest (i.e. column number for global maxima as shown in figure 7).*

*Cropping point = column number where height of histogram is lowest in such a way that cropping point is in between first column and column of Global Maxima*

If gesture is shown from opposite side (i.e. from other hand), then *Cropping point* is searched between column of *Global Maxima* and last column. Direction of the hand is detected using continuity analysis of object during hand gesture area determination. Continuity analysis 418 Robotic Systems – Applications, Control and Programming

*Min\_row=* minimum value of row number where total number of white pixels are more than

Max\_row= maximum value of row number where total number of white pixels are more

Figure 6 shows Min\_row, Max\_row, Min\_col, and, Max\_col for the selection of the boundary

Third step is to remove parts of hand not used in gesture i.e. removal of wrist, arm etc. As these extra parts are of variable length in image frame and pattern matching with gesture database gives unwanted results. Therefore, parts of hand before the wrist need to be

Statistical analysis of hand shape shows that either we pose palm or fist, width is lowest at wrist and highest at the middle of palm. Therefore extra hand part can be cropped out from wrist by determining location where minimum width of vertical histogram is found. Figure 7.c and 7.d show global maxima and cropping points for hand gestures in figure 7.a and 7.b

*Global Maxima = column number where height of histogram is highest (i.e. column number for global*

*Cropping point = column number where height of histogram is lowest in such a way that cropping*

If gesture is shown from opposite side (i.e. from other hand), then *Cropping point* is searched between column of *Global Maxima* and last column. Direction of the hand is detected using continuity analysis of object during hand gesture area determination. Continuity analysis

*Ver\_offset*.

than *Ver\_offset*.

Fig. 5. Error free image

for cropping of hand.

cropped out.

respectively.

Cropping point is calculated as:

*point is in between first column and column of Global Maxima*

*maxima as shown in figure 7).*

shows that whether the object continues from the column of Global maxima to first column or to last column. i.e. whether extra hand is left side of palm or right side of palm.

Fig. 6. Boundary selection for hand cropping

Fig. 7. Histogram showing white pixels in each column, with cropping point for hand gesture

Real-Time Robotic Hand Control Using Hand Gestures 421

Principal Component Analysis (PCA)[11] is used in this application to compare the acquired gesture with the gestures available in the database and recognize the intended gesture.

> RGB images take 3 values for each pixel (RED, GREEN, BLUE) but we want gray scale images.

we are taking image of 100 X 100

A column vector contains data for 1 complete image, so size of each column vector will be 10000 X 1. All images will be stored like this in T Matrix, T grows after each turn. So the final size of T will be 10000 X 10.

Mj = (1/P)\*sum (Tj), where P is number of images in Train Database, and Tj is the jth image, Mj is the mean of jth image.

Ai = Ti- Mi , where A represents new difference matrix of images. Size of A will be 10000 X 10

L = A'\*A, A' is transpose of matrix A Size of L will be 10 X

[V D] = eig (L), Diagonal elements of D are Eigenvalues for both L=A'\*A and C=A\*A'

10

we are considering 10 images in Training database[12][21]

Calculate number of images which are

Convert RGB images into Gray images

Reshaping of an image by converting this

Compute the difference image for each

Calculate surrogate of covariance matrix

Calculate Eigenvalues and Eigenvectors

stored in Training Database

Calculate size of an image

into 1-D column vector

Calculate mean of all images

image and merge all images

C=A\*A'

## **2.3 Determining the gesture by resizing and pattern matching**

Cropping results in a frame of 60x80 pixels, so that the gesture can be compared with the database. This particular size of 60x80 is chosen to preserve the aspect ratio of the original image, so that no distortion takes place.

Figure 8.b shows cropped hand gesture from figure 8.a. and figure 8.c. shows the result of scaling and fitting 8.b within 60X80 pixels. Figure 8.d represents final cropped and resized gesture of desired dimension.

Fig. 8. Scaling cropped hand gesture to fit into desired dimension

Fig. 9. (a) PCA basis (b) PCA reduction to 1D

(a) The concept of PCA. Solid lines: the original basis; dashed lines: the PCA basis. The dots are selected at regularly spaced locations on a straight line rotated at 30degrees, and then perturbed by isotropic 2D Gaussian noise. (b) The projection (1D reconstruction) of the data using only the first principal component.

There are many different algorithms available for Gesture Recognition, such as principal Component Analysis, Linear Discriminate Analysis, Independent Component Analysis, Neural Networks and Genetic Algorithms.

420 Robotic Systems – Applications, Control and Programming

Cropping results in a frame of 60x80 pixels, so that the gesture can be compared with the database. This particular size of 60x80 is chosen to preserve the aspect ratio of the original

Figure 8.b shows cropped hand gesture from figure 8.a. and figure 8.c. shows the result of scaling and fitting 8.b within 60X80 pixels. Figure 8.d represents final cropped and resized

(a) (b)

(a) The concept of PCA. Solid lines: the original basis; dashed lines: the PCA basis. The dots are selected at regularly spaced locations on a straight line rotated at 30degrees, and then perturbed by isotropic 2D Gaussian noise. (b) The projection (1D reconstruction) of the data

There are many different algorithms available for Gesture Recognition, such as principal Component Analysis, Linear Discriminate Analysis, Independent Component Analysis,

**2.3 Determining the gesture by resizing and pattern matching**

Fig. 8. Scaling cropped hand gesture to fit into desired dimension

Fig. 9. (a) PCA basis (b) PCA reduction to 1D

using only the first principal component.

Neural Networks and Genetic Algorithms.

image, so that no distortion takes place.

gesture of desired dimension.

Principal Component Analysis (PCA)[11] is used in this application to compare the acquired gesture with the gestures available in the database and recognize the intended gesture.

Real-Time Robotic Hand Control Using Hand Gestures 423

The concept of PCA is illustrated in Figure 9; The graph shows data that originally has two components along the x1 and x2 directions. This is now resolved along the Φ1 and Φ2 directions. Φ1 corresponds to the direction of maximum variance and is chosen as the first principal component. In the 2D case currently being considered, the second principal component is then determined uniquely by orthogonality constraints; in a higherdimensional space the selection process would continue, guided by the variances of the

A comprehensive explanation of the above scheme for gesture recognition[19] within a database of ten images is given in figure 10. For simplicity of calculations and ease of understanding, each figure is taken to be of size of 100X100 pixels. This algorithms was

Different functions corresponding to each meaningful hand gesture are written and stored in database for controlling the robotic Arm. For implementation of robotic arm control, PUMA robotic model has been chosen. Since this model was only available at the time of interface, hand gestures are used to control the various joints of the arm. However, these gestures can be used to control the robotic hand also. Altogether 10 different hand gestures are related to 10 different movements of robotic arm. Movement commands are written as a function in robot specific language. Whenever a gesture is matched with a meaningful gesture from the database, the instruction set corresponding to that gesture is identified and passed to robot for execution. In this way the robotic system can be controlled by hand gesture using live camera. Figure 11 shows movements of robotic hand corresponding to

Fig. 11. Movements of PUMA 762 corresponding to some specific hand gestures

purpose that have been identified as shown in fig. 12.

However, there are a total of 10 actionable gestures, and one background image for resetting

projections.

tested on a PC platform.

four different specific hand gestures.

**2.4 Generation of control Instructions for a robot**

Fig. 10. PCA Algorithm steps for 10 Images

422 Robotic Systems – Applications, Control and Programming

All eigenvalues of matrix L are sorted and those less than a specified threshold are eliminated. So the number of non-zero eigen vectors may be less than (P-1). Let us consider for discussion's sake that it is P-1 i.e. 9 here.

Eigenhands = A\*L\_eig\_vec. Eignevhands is centered image vector, Size of Eigenhands is

Projection of all centered images

temp= Eigenhands'\*A (: i); Projection of centered images into face space. Temp grows after

10000 X 9

each turn.

9 X 9

be 9 X 1

will be stored in Projected Train Images

Reshape test image, Calculate difference by subtracting mean of Train images from test image for which we are calculating

Calculate Projected Test Image by Eigenhands'\*Difference. Size of Eigenhands is 10000 X 9, size of difference is 10000 X 1 So, size of project test image will

Size of Projected Train Image is

equivalent image.

Fig. 10. PCA Algorithm steps for 10 Images

Extracting the PCA features from test

Projected test image

i

Sorting and eliminating eigen values

Projected Train Image

Calculate Eigenhands

The concept of PCA is illustrated in Figure 9; The graph shows data that originally has two components along the x1 and x2 directions. This is now resolved along the Φ1 and Φ2 directions. Φ1 corresponds to the direction of maximum variance and is chosen as the first principal component. In the 2D case currently being considered, the second principal component is then determined uniquely by orthogonality constraints; in a higherdimensional space the selection process would continue, guided by the variances of the projections.

A comprehensive explanation of the above scheme for gesture recognition[19] within a database of ten images is given in figure 10. For simplicity of calculations and ease of understanding, each figure is taken to be of size of 100X100 pixels. This algorithms was tested on a PC platform.

## **2.4 Generation of control Instructions for a robot**

Different functions corresponding to each meaningful hand gesture are written and stored in database for controlling the robotic Arm. For implementation of robotic arm control, PUMA robotic model has been chosen. Since this model was only available at the time of interface, hand gestures are used to control the various joints of the arm. However, these gestures can be used to control the robotic hand also. Altogether 10 different hand gestures are related to 10 different movements of robotic arm. Movement commands are written as a function in robot specific language. Whenever a gesture is matched with a meaningful gesture from the database, the instruction set corresponding to that gesture is identified and passed to robot for execution. In this way the robotic system can be controlled by hand gesture using live camera. Figure 11 shows movements of robotic hand corresponding to four different specific hand gestures.

Fig. 11. Movements of PUMA 762 corresponding to some specific hand gestures

However, there are a total of 10 actionable gestures, and one background image for resetting purpose that have been identified as shown in fig. 12.

Real-Time Robotic Hand Control Using Hand Gestures 425

laboratory. A gesture database consisting of binary images of size 60 X 80 pixels is prestored, so it takes less time and memory space during pattern recognition. Due to the use of cropped image of gesture, our database becomes more effective as one image is sufficient for one type of gesture presentation. So, neither we need to store more than one image for same gesture at different positions of image, nor have we to worry about positions of hand in front of camera. Experimental results show that the system can detect hand gestures with an accuracy of 95 % when it is kept still in front of the camera for 1

**PCA Eigenvector**

**PCA Eigenvector**

**Stored in RAM**

**RAM**

**Image Acquisition**

**Image Acquisition**

**Gesture Recognition**

This research is being carried out at Central Electronics Engineering Research Institute (CEERI), Pilani, India as part of a project "Supra Institutional Projects on Technology Development for Smart Systems". Authors would like to thank Director, CEERI for his

**Minimum Distance**

second.

Fig. 13. Recognition process

active encouragement and support.

**5. Acknowledgment**

Fig. 12. Set of specified gestures used for robot control

## **3. Implementation on FPGA processor**

We have implemented the algorithm on an FPGA, using a Database of 6 images. Each image size is 60X80 pixels. The reduction in number of images, image size and its resolution are due to FPGA memory constraints. The MATLAB code produces 6x6 projection vectors which are stored in the RAM of the FPGA using a JTAG cable [27]. After formation of the database, the system captures an image and projection vector of 6x1 is produced and that is given to the hardware for further processing. The hardware is responsible for calculating the Euclidian Distance [ED] with a dedicated block which we designate as ED block [34]. Then the minimum distance [MD] is found using a dedicated block named MD block. And the index corresponding to the minimum distance is returned to MATLAB to display the matched image from the Database [33]. The whole process is shown in fig. 13.

The main drawback of this implementation is that only part of the algorithm resides on FPGA and rest of the algorthim still runs on PC. Therefore it is not standalone system and requires interface with the PC[13]. FPGA implementation of PCA algorithm is a great challenge due to the limited resources available in commercial FPGA kits. But it is quite important to have a complete system on a FPGA chip [40]. As far as the technology is concerned, FPGA is most suited for implementation of real time image processing algorithm. It can be readily designed with custom parallel digital circuitry tailored for performing various imaging tasks making them well-suited for high speed real time vision processing applications[35].

## **4. Conclusion**

Controlling a robot arm, in real time, through the hand gestures is a novel approach. The technique proposed here was tested under proper lighting conditions that we created in our 424 Robotic Systems – Applications, Control and Programming

We have implemented the algorithm on an FPGA, using a Database of 6 images. Each image size is 60X80 pixels. The reduction in number of images, image size and its resolution are due to FPGA memory constraints. The MATLAB code produces 6x6 projection vectors which are stored in the RAM of the FPGA using a JTAG cable [27]. After formation of the database, the system captures an image and projection vector of 6x1 is produced and that is given to the hardware for further processing. The hardware is responsible for calculating the Euclidian Distance [ED] with a dedicated block which we designate as ED block [34]. Then the minimum distance [MD] is found using a dedicated block named MD block. And the index corresponding to the minimum distance is returned to MATLAB to display the

The main drawback of this implementation is that only part of the algorithm resides on FPGA and rest of the algorthim still runs on PC. Therefore it is not standalone system and requires interface with the PC[13]. FPGA implementation of PCA algorithm is a great challenge due to the limited resources available in commercial FPGA kits. But it is quite important to have a complete system on a FPGA chip [40]. As far as the technology is concerned, FPGA is most suited for implementation of real time image processing algorithm. It can be readily designed with custom parallel digital circuitry tailored for performing various imaging tasks making them well-suited for high speed real time vision

Controlling a robot arm, in real time, through the hand gestures is a novel approach. The technique proposed here was tested under proper lighting conditions that we created in our

matched image from the Database [33]. The whole process is shown in fig. 13.

Fig. 12. Set of specified gestures used for robot control

**3. Implementation on FPGA processor**

processing applications[35].

**4. Conclusion**

laboratory. A gesture database consisting of binary images of size 60 X 80 pixels is prestored, so it takes less time and memory space during pattern recognition. Due to the use of cropped image of gesture, our database becomes more effective as one image is sufficient for one type of gesture presentation. So, neither we need to store more than one image for same gesture at different positions of image, nor have we to worry about positions of hand in front of camera. Experimental results show that the system can detect hand gestures with an accuracy of 95 % when it is kept still in front of the camera for 1 second.

Fig. 13. Recognition process

## **5. Acknowledgment**

This research is being carried out at Central Electronics Engineering Research Institute (CEERI), Pilani, India as part of a project "Supra Institutional Projects on Technology Development for Smart Systems". Authors would like to thank Director, CEERI for his active encouragement and support.

Real-Time Robotic Hand Control Using Hand Gestures 427

[16] Katupitiya, J., Radajewski, R., Sanderson, J., Tordon, M. Implementation of PC Based

[18] Mohammad, Y.; Nishida, T.; Okada, S.; Unsupervised simultaneous learning of

[19] Moezzi and R. Jain, "ROBOGEST: Telepresence Using Hand Gestures". Technical report

[20] M. Moore, A DSP-based real time image processing system. In the Proceedings of the

[21] Nolker, C.; Ritter, H.; Parametrized SOMs for hand posture reconstruction, Neural

[22] Nolker C., Ritter H., Visual Recognition of Continuous Hand Postures, IEEE Transactions on neural networks, Vol 13, No. 4, July 2002, pp. 983-994. [23] K. Oka, Y. Sato and H. Koike. Real-Time Fingertip Tracking and Gesture Recognition.

[24] Ozer, I. B., Lu, T., Wolf, W. Design of a Real Time Gesture Recognition System: High

[25] V.I. Pavlovic, R. Sharma, T.S. Huang. Visual interpretation of hand gestures for human-

[26] Quek, F. K. H., Mysliwiec, T., Zhao, M. (1995). Finger Mouse: A Free hand Pointing

[27] A.K. Ratha, N.K. Jain, FPGA-based computing in computer vision, Computer

[28] Ramamoorthy, N. Vaswani, S. Chaudhury, S. Banerjee, Recognition of Dynamic hand

[29] Sawah A.E., and et al., a framework for 3D hand tracking and gesture recognition

[30] Sergios Theodoridis, Konstantinos Koutroumbas, Pattern Recognition, Elsevier

[31] Seyed Eghbal Ghobadi, Omar Edmond Loepprich, Farid Ahmadov Jens Bernshausen,

International Journal of Computer Science, 35:4, IJCS\_35\_4\_08, Nov 2008.

and robot vision, Montreal, Canada, 28-30 May, 2007, pp. 495-502.

VCL-94-104, University of California, San Diego, 1994.

Joint Conference on , vol.4, no., pp.139-144 vol.4, 2000

IEEE Computer Graphics and Applications: 64-71, 2002.

and Gesture Recognition, pp.372-377, Zurich, Switzerland.

International Workshop on, pp. 128–137, 20-22 Oct 1997.

gestures, Pattern Recognition 36: 2069-2081, 2003.

Publication, Second Edition, 2003.

Machine Intelligence 19(7): 677-695, 1997.

Zurich: pp. 184-188, 1995.

Boston MA, August 1995.

pp. 57-64, 2005.

no., pp.2537-2544, 10-15 Oct. 2009.

Controller for a PUMA Robot. Proc. 4th IEEE Conf. on Mechatronics and Machine Vision in Practice. Australia, p.14-19. [doi:10.1109/MMVIP.1997.625229], 1997. [17] R. Kjeldsen, J. Kinder. Visual hand gesture recognition for window system control, in

International Workshop on Automatic Face and Gesture Recognition (IWAFGR),

gestures, actions and their associations for Human-Robot Interaction, Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, vol.,

6th International conference on signal processing applications and technology,

Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International

Performance through algorithms and software. IEEE Signal Processing Magazine,

computer interaction, A Review, IEEE Transactions on Pattern Analysis and

Computer Interface. Proceedings of International Workshop on Automatic Face

Architecture for Machine Perception, CAMP '97. Proceedings Fourth IEEE

using elements of genetic programming, 4th Canadian conference on Computer

Real Time Hand Based Robot Control Using Multimodal Images, IAENG

## **6. References**

426 Robotic Systems – Applications, Control and Programming

[1] Becerra V.M., Cage, C.N.J., Harwin, W.S., Sharkey, P.M. Hardware retrofit and

[2] L. Brthes, P. Menezes, F. Lerasle, and J. Hayet. Face tracking and hand gesture

[3] Chan Wah Ng, Surendra Ranganath, Real-time gesture recognition system and

[4] Corke, P.I. Operational Details of the Unimation Puma Servo System. Report, CSIRO

[5] Crow, F. Summed-area tables for texture mapping. SIGGRAPH '84: Proceedings of the

[6] J. Davis, M. Shah. Recognizing hand gestures. In Proceedings of European Conference

[7] Daggu Venkateshwar Rao, Shruti Patil, Naveen Anne Babu and V Muthukumar,

[8] Dastur, J.; Khawaja, A.; Robotic Arm Actuation with 7 DOF Using Haar Classifier

[9] K G Derpains, A review of Vision-based Hand Gestures, Internal Report, Department of

[10] Didier Coquin, Eric Benoit, Hideyuki Sawada, and Bogdan Ionescu, Gestures

[11] Fang Zhong, David W. Capson, Derek C. Schuurman, Parallel Architecture for PCA Image Feature Detection using FPGA 978-1-4244-1643-1/08/2008 IEEE. [12] Freeman W., Computer vision for television and games. In Recognition, Analysis, and

[13] Haessig D.,Hwang J., Gallagher S., Uhm M., Case-Study of a Xilinx System Generator

[15] Jagdish Raheja, Radhey Shyam and Umesh Kumar, Hand Gesture Capture and

and Automation, pp. 1901–1906, 2004. New Orleans, Louisiana.

application. Image and Vision computing (20): 993-1007, 2002.

on Computer Vision, ECCV: 331-340, 1994. Stockholm, Sweden.

and Applications (ICCEA), vol.1, no., pp.27-29, 19-21 March 2010.

Computer Science. York University, February 2004.

Robotics and Mechatronics Vol.18 No.6, 2006. pp. 751-759.

Science and Network Security, VOL.6 No.2A, 2006 pp. 216-222.

September 7 - 9, 2009 Palma de Mallorca, Spain.

Division of Manufacturing Technology. Australia, 1993.

IEEE Control Systems Magazine, 24(5): 78-82. 2004.

computed torque control of Puma 560 robot updating an industrial manipulator.

recognition for human robot interaction. In International Conference on Robotics

11th annual conference on Computer graphics and interactive techniques. pp. 207-

Implementation and Evaluation of Image Processing Algorithms on Reconfigurable Architecture using C-based Hardware Descriptive Languages, International Journal of Theoretical and Applied Computer Sciences, Volume 1 Number 1 pp. 9–34, 2006.

Gesture Recognition, Second International Conference on Computer Engineering

Recognition Based on the Fusion of Hand Positioning and Arm Gestures, Journal of

Tracking of Faces and Gestures in Real-Time Systems, page 118, 1999. Copyright©

Design Flow For Rapid Development of SDR Waveforms, Proceeding of the SDR 05 Technical Conference and Product Exposition. Copyright © 2005 SDR Forum. [14] Jae-Ho Shin, Jong-Shill Lee, Se-Kee Kil, Dong-Fan Shen, Je-Goon Ryu, Eung-Hyuk Lee,

Hong-Ki Min, Seung-Hong Hong, Hand Region Extraction and Gesture Recognition using entropy analysis, IJCSNS International Journal of Computer

Recognition Technique for Real-time Video Stream, In The 13th IASTED International Conference on Artificial Intelligence and Soft Computing (ASC 2009),

**6. References**

212.

MEITCA.

**21**

Ebrahim Mattar

*University of Bahrain Kingdom of Bahrain*

**Robotics Arm Visual Servo:**

**with Epipolar Geometry**

**Estimation of Arm-Space Kinematics Relations**

*Intelligent Control & Robotics, Department of Electrical and Electronics Engineering,*

Numerous advances in robotics have been inspired by reliable concepts of biological systems. Necessity for improvements has been recognized due to lack of sensory capabilities in robotic systems which make them unable to cope with challenges such as anonymous and changing workspace, undefined location, calibration errors, and different alternating concepts. Visual servoing aims to control a robotics system through artificial vision, in a way as to manipulate an environment, in a similar way to humans actions. It has always been found that, it is not a straightforward task to combine "Visual Information" with a "Arm Dynamic" controllers. This is due to different natures of descriptions which defines "Physical Parameters" within an arm controller loop. Studies have also revealed an option of using a trainable system for learning some complicated kinematics relating object features to robotics arm joint space. To achieve visual tracking, visual servoing and control, for accurate manipulation objectives without losing it from a robotics system, it is essential to relate a number of an object's geometrical features (object space) into a robotics system joint space (arm joint space). An object visual data, an play important role in such sense. Most robotics visual servo systems rely on object "features Jacobian", in addition to its inverse Jacobian. Object visual features inverse Jacobian is not easily put together and computed, hence to use such relation in a visual loops. A neural system have been used to approximate such relations, hence avoiding computing object's feature inverse Jacobian, even at singular Jacobian postures. Within this chapter, we shall be discussing and presenting an integration approach that combines "Visual Feedback" sensory data with a "6-DOF robotics Arm Controller". Visual servo is considered as a methodology to control movements of a robotics system using certain visual information to achieve a task. Visionary data is acquired from a camera that is mounted directly on a robot manipulator or on a mobile robot, in which case, motion of the robot induces camera motion. Differently, the camera can be fixed, so that can observe the robot motion. In this sense, visual servo control relies on techniques from image processing, computer vision control theory, kinematics,

**1. Introduction**

**1.1 Visual servoing for robotics applications**

dynamic and real time computing.

hitp://xvww.xilinix.coinjpoblication s/xcellonlince/xcell 47/xc svsen47.htm

## **Robotics Arm Visual Servo: Estimation of Arm-Space Kinematics Relations with Epipolar Geometry**

Ebrahim Mattar

*Intelligent Control & Robotics, Department of Electrical and Electronics Engineering, University of Bahrain Kingdom of Bahrain*

## **1. Introduction**

428 Robotic Systems – Applications, Control and Programming

[32] Shin M. C., Tsap L. V., and Goldgof D. B., Gesture recognition using bezier curves for

[33] N. Shirazi and J. Ballagh, "Put Hardware in the Loop with Xilinx System Generator for

hitp://xvww.xilinix.coinjpoblication s/xcellonlince/xcell 47/xc svsen47.htm [34] Stephen D.Brown, R.J. Francis, J.Rose, Z.G.Vranesic, Field Programmable Gate Arrays,

[35] Starner T., Pentland. A., Real-time American Sign Language recognition from video

[36] Triesch J., Malsburg. C.V.D., Robotic gesture recognition. In Gesture Workshop, pages

[37] Tomita, A., Ishii. R. J. Hand shape extraction from a sequence of digitized grayscale

& Workshops, 12-14 Oct, 2009, pp. 1-6, in St. Petersburg, Russia.

http://www.mathworks.com/matlabcentral/fileexchange/14932

Issue 5, May 2004, pp.1011–1024.

233–244, Nara, Japan, 1997.

[39] Walla Walla University, robotic lab PUMA 762.

[40] Xilinx Inc., System Generator for DSP, Users Guide

1992.

1995.

1999.

DSP", Xcell Journal online, May 2003,

visualization navigation from registered 3-d data, Pattern Recognition, Vol. 37,

using hidden markov models. In SCV95, pages 265–270, Ames Street, Cambridge,

images. Proceedings of tweenth International Conference on Industrial Electronics, Control and Instrumentation, IECON '94, vol.3, pp.1925–1930, Bologna, Italy, 1994. [38] Verma R., Dev A., Vision based Hand Gesture Recognition Using finite State Machines

and Fuzzy Logic, International Conference on Ultra-Modern Telecommunications

http://wsssw.xilinx.con1i/products/softwa-e/svstzen/app docs/userLguide.htm

Notes In Computer Science; Vol. 1739, Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction,

[41] Ying Wu, Thomas S Huang, Vision based Gesture. Recognition: A Review, Lecture

## **1.1 Visual servoing for robotics applications**

Numerous advances in robotics have been inspired by reliable concepts of biological systems. Necessity for improvements has been recognized due to lack of sensory capabilities in robotic systems which make them unable to cope with challenges such as anonymous and changing workspace, undefined location, calibration errors, and different alternating concepts. Visual servoing aims to control a robotics system through artificial vision, in a way as to manipulate an environment, in a similar way to humans actions. It has always been found that, it is not a straightforward task to combine "Visual Information" with a "Arm Dynamic" controllers. This is due to different natures of descriptions which defines "Physical Parameters" within an arm controller loop. Studies have also revealed an option of using a trainable system for learning some complicated kinematics relating object features to robotics arm joint space. To achieve visual tracking, visual servoing and control, for accurate manipulation objectives without losing it from a robotics system, it is essential to relate a number of an object's geometrical features (object space) into a robotics system joint space (arm joint space). An object visual data, an play important role in such sense. Most robotics visual servo systems rely on object "features Jacobian", in addition to its inverse Jacobian. Object visual features inverse Jacobian is not easily put together and computed, hence to use such relation in a visual loops. A neural system have been used to approximate such relations, hence avoiding computing object's feature inverse Jacobian, even at singular Jacobian postures. Within this chapter, we shall be discussing and presenting an integration approach that combines "Visual Feedback" sensory data with a "6-DOF robotics Arm Controller". Visual servo is considered as a methodology to control movements of a robotics system using certain visual information to achieve a task. Visionary data is acquired from a camera that is mounted directly on a robot manipulator or on a mobile robot, in which case, motion of the robot induces camera motion. Differently, the camera can be fixed, so that can observe the robot motion. In this sense, visual servo control relies on techniques from image processing, computer vision control theory, kinematics, dynamic and real time computing.

Robotics Arm Visual Servo:

**1.2 Literature surveys**

Estimation of Arm-Space Kinematics Relations with Epipolar Geometry 431

**EGT**, **E**pipolar **G**eometry **T**oolbox, (Eleonora et al., 2004), was also built to grant MATLAB users with a broaden outline for a creation and visualization of multi-camera scenarios. Additionally, to be used for the manipulation of visual information, and the visual geometry. Functions provided, for both classes of vision sensing, the PINHOLE and PANORAMIC, include camera assignment and visualization, computation, and estimation of epipolar geometry entities. Visual servoing has been classified as using visual data within

**I**mage **B**ased **V**isual **S**ervoing (**IBVS**) Using Takagi-Sugeno Fuzzy Neural Network Controller has been proposed by (Miao et. al, 2007). In their study, a T-S fuzzy neural controller based IBVS method was proposed. Eigenspace based image compression method is firstly explored which is chosen as the global feature transformation method. Inner structure, performance and training method of T-S neural network controller are discussed

An Image Based Visual Servoing using Takagi-Sugeno fuzzy neural network controller has been proposed by Miao, (Miao et. al, 2007). In this paper, a TAKAGI-SUGENO Fuzzy Neural Network Controller (TSFNNC) based Image Based Visual Servoing (IBVS) method was proposed. Firstly, an eigenspace based image compression method is explored, which is chosen as the global feature transformation method. After that, the inner structure, performance and training method of T-S neural network controller are discussed

Panwar and Sukavanam in (Panwar & Sukavanam 2007) have introduced Neural Network Based Controller for Visual Servoing of Robotic Hand Eye System. For Panwar and Sukavanam, in their paper a neural network based controller for robot positioning and tracking using direct monocular visual feedback is proposed. Visual information is provided using a camera mounted on the end-effector of a robotics manipulator. A PI kinematics controller is proposed to achieve motion control objective in an image plane. A Feed forward Neural Network (FFNN) is used to compensate for the robot dynamics. The FFNN computes the required torques to drive the robot manipulator to achieve desired tracking. The stability of combined PI kinematics and FFNN computed torque is proved by Lyapunov theory. Gracia and Perez-Vidal in (Gracia & Perez-Vidal 2009), have presented a research framework through which a new control scheme for visual servoing that takes into account the delay introduced by image acquisition and image processing. The capabilities (steadystate errors, stability margins, step time response, etc.) of the proposed control scheme and of previous ones are analytically analyzed and compared. Several simulations and experimental results were provided to validate the analytical results and to illustrate the benefits of the proposed control scheme. In particular, it was shown that this novel control

Alessandro and researchers, as in (Alessandro et. al. 2007), in their research framework, they proposed an image-based visual servoing framework. Error signals are directly computed from image feature parameters, thus obtaining control schemes which do not need neither a 3-D model of the scene, nor a perfect knowledge of the camera calibration matrix. Value of the depth "Z" for each considered feature must be known. Thus they proposed a method to estimate on-line the value of Z for point features while the camera is moving through the scene, by using tools from nonlinear observer theory. By interpreting "Z" as a continuous unknown state with known dynamics, they build an estimator which asymptotically

a control loop (Cisneros, 2004), thus enabling visual-motor (Hand-Eye) coordination.

respectively. Besides that, the whole architecture of TS-FNNC was investigated.

respectively. Besides, the whole architecture of the TS-FNNC is investigated.

scheme clearly improves the performance of the previous ones.

recovers the actual depth value for the selected feature.

Robotics visual servoing has been recently introduced by robotics, AI and control communities. This is due to the significant number of advantages over blind robotic systems. Researchers have also demonstrated that, VISUAL SERVOING is an effective and a robust framework to control robotics systems while relying on visual information as feedback. An image-based scheme task is said to be completely performed if a desired image is acquired by a robotic system. Numerous advances in robotics have been inspired by the biological systems. In reference to Fig. (1), visual servoing aims to control a robotics system through an artificial vision in a way as to manipulate an environment, comparable to humans actions. Intelligence-based visual control has also been introduced by research community as a way to supply robotics system even with more cognitive capabilities. A number of research on the field of intelligent visual robotics arm control have been introduced. Visual servoing has been classified as using visual data within a control loop, enabling visual-motor (hand-eye) coordination. There have been different structures of visual servo systems. However, the main two classes are; **P**osition-**B**ased **V**isual **S**ervo systems (**PBVS**), and the **I**mage-**B**ased **V**isual **S**ervo systems (**IBVS**). In this chapter, we shall concentrate on the second class, which is the Image-based visual servo systems.

Fig. 1. Overall structure of an ANN based visual servoing

## **1.2 Literature surveys**

430 Robotic Systems – Applications, Control and Programming

Robotics visual servoing has been recently introduced by robotics, AI and control communities. This is due to the significant number of advantages over blind robotic systems. Researchers have also demonstrated that, VISUAL SERVOING is an effective and a robust framework to control robotics systems while relying on visual information as feedback. An image-based scheme task is said to be completely performed if a desired image is acquired by a robotic system. Numerous advances in robotics have been inspired by the biological systems. In reference to Fig. (1), visual servoing aims to control a robotics system through an artificial vision in a way as to manipulate an environment, comparable to humans actions. Intelligence-based visual control has also been introduced by research community as a way to supply robotics system even with more cognitive capabilities. A number of research on the field of intelligent visual robotics arm control have been introduced. Visual servoing has been classified as using visual data within a control loop, enabling visual-motor (hand-eye) coordination. There have been different structures of visual servo systems. However, the main two classes are; **P**osition-**B**ased **V**isual **S**ervo systems (**PBVS**), and the **I**mage-**B**ased **V**isual **S**ervo systems (**IBVS**). In this chapter, we

shall concentrate on the second class, which is the Image-based visual servo systems.

Fig. 1. Overall structure of an ANN based visual servoing

**EGT**, **E**pipolar **G**eometry **T**oolbox, (Eleonora et al., 2004), was also built to grant MATLAB users with a broaden outline for a creation and visualization of multi-camera scenarios. Additionally, to be used for the manipulation of visual information, and the visual geometry. Functions provided, for both classes of vision sensing, the PINHOLE and PANORAMIC, include camera assignment and visualization, computation, and estimation of epipolar geometry entities. Visual servoing has been classified as using visual data within a control loop (Cisneros, 2004), thus enabling visual-motor (Hand-Eye) coordination.

**I**mage **B**ased **V**isual **S**ervoing (**IBVS**) Using Takagi-Sugeno Fuzzy Neural Network Controller has been proposed by (Miao et. al, 2007). In their study, a T-S fuzzy neural controller based IBVS method was proposed. Eigenspace based image compression method is firstly explored which is chosen as the global feature transformation method. Inner structure, performance and training method of T-S neural network controller are discussed respectively. Besides that, the whole architecture of TS-FNNC was investigated.

An Image Based Visual Servoing using Takagi-Sugeno fuzzy neural network controller has been proposed by Miao, (Miao et. al, 2007). In this paper, a TAKAGI-SUGENO Fuzzy Neural Network Controller (TSFNNC) based Image Based Visual Servoing (IBVS) method was proposed. Firstly, an eigenspace based image compression method is explored, which is chosen as the global feature transformation method. After that, the inner structure, performance and training method of T-S neural network controller are discussed respectively. Besides, the whole architecture of the TS-FNNC is investigated.

Panwar and Sukavanam in (Panwar & Sukavanam 2007) have introduced Neural Network Based Controller for Visual Servoing of Robotic Hand Eye System. For Panwar and Sukavanam, in their paper a neural network based controller for robot positioning and tracking using direct monocular visual feedback is proposed. Visual information is provided using a camera mounted on the end-effector of a robotics manipulator. A PI kinematics controller is proposed to achieve motion control objective in an image plane. A Feed forward Neural Network (FFNN) is used to compensate for the robot dynamics. The FFNN computes the required torques to drive the robot manipulator to achieve desired tracking. The stability of combined PI kinematics and FFNN computed torque is proved by Lyapunov theory. Gracia and Perez-Vidal in (Gracia & Perez-Vidal 2009), have presented a research framework through which a new control scheme for visual servoing that takes into account the delay introduced by image acquisition and image processing. The capabilities (steadystate errors, stability margins, step time response, etc.) of the proposed control scheme and of previous ones are analytically analyzed and compared. Several simulations and experimental results were provided to validate the analytical results and to illustrate the benefits of the proposed control scheme. In particular, it was shown that this novel control scheme clearly improves the performance of the previous ones.