Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Contents lists available at ScienceDirect

Robotics and Computer Integrated Manufacturing

journal homepage: www.elsevier.com/locate/rcim

Flexible telemanipulation based handy robot teaching on tape masking with T

complex geometry
⁎,1,a
Qilong Yuan , Ching-Yen Weng1,a, Francisco Suárez-Ruizb, I-Ming Chena
a
Robotics Research Centre, School of Mechanical and Aerospace Engineering, Nanyang Technological University, 50 Nanyang Avenue, 639798 Singapore
b
nuTonomy Asia, 77 Ayer Rajah Crescent, 01-30, 139954, Singapore

A R T I C LE I N FO A B S T R A C T

Keywords: Traditional robot teaching methods are cumbersome, tedious and difficult to scale for high-mix low-volume
Flexible manufacturing applications. The tape masking, a common process for surface protection before plasma spraying, spray painting
Human-Robot collaboration and shot peening, is one of those domains where robotic automation lacks flexibility and reliability due to the
Human-Centered automation complexity in task. Fortunately, it is still within the grasps of human-robot collaborative systems. This work
presents a telemanipulation-based robot teaching framework that is able to let the robot manipulator cope with
the taping tasks with complex workpiece geometries. The proposed framework allows quick calibration, variable
motion mapping, and indexing so that the operators can easily set up and guide the robotic taping system to
cover the tapes onto the layers and grooves of different workpieces. This framework enables the operators to
change the motion mapping scale for both large-scale guidance and fine motion dexterous manipulation.
Meanwhile, an indexing function makes it possible for the operators to re-map their poses from the edges of their
comfortable regions. A portable VR system is applied in the telemanipulation system. With its six DoF motion
precisely measured in real-time, the proposed motion remapping algorithms enable the operators to directly
guide the robot in their selected scales. Experimental results show that the proposed framework facilitates robot
programming on the manipulation of the complex workpieces that have multi-layer surfaces and grooves in
between. It also reduces the teaching time comparing to other methods. This system and method improve
teaching efficiency and convenience, which has potential value to be deployed in manufacturing.

1. Introduction multiple Degree-of-freedoms systems. Classic robot teaching ap-

proaches rely on recording the waypoints of the robot’s end-effector,
Nowadays, industry applications require robotic systems to perform and later interpolating across the recorded waypoints for complete
more complex and challenging tasks in the unstructured environment. trajectory generation. The recording processes are time-consuming and
Especially when robotic systems are coordinately and collaboratively lack the adaptivity to complex paths or geometries [14] since it is not
working with humans, dexterous and flexible robot manipulation are easy to collect many waypoints through pure manual teaching.
excepted. Based on this, the feature of user-friendly and ease-of-use to Robotics motion planning software (such as MoveIt! and OpenRave)
enable complex robot manipulations are thereby proposed by [1–3]. can solve complex path planning problems and generate smooth and
Meanwhile, the design of perception and feedback system to ensure efficient robot trajectories for manipulation, provided that the precise
users’ operating awareness and safety are also very important [4–9]. models are available to the robotic software. Otherwise, the simulation
Traditional solutions, such as offline programming or teach pendant will generate errors that enough to make the task manipulation fail.
based teaching methods, to cope with robotic manipulation tasks tend Such an approach has the drawbacks of tedious modeling and calibra-
to have high demands for operators. Typical examples of such ap- tion, especially when there are frequently changing workpieces in high-
proaches in various tasks are explored in previous articles such as pick- mix low-volume manufacturing scenarios. Hence, a more straightfor-
and-place [10], robotic assembly [11], in-hand manipulation [12], and ward and efficient robot teaching solution will be a better solution for
robotic bin-packing [13]. For complex manipulation tasks, traditional these cases.
solutions are difficult to teach a robot due to the high complexity of the A typical example is to cover object surfaces for protection purposes

⁎
Corresponding author.
E-mail address: [email protected] (Q. Yuan).
1
Both authors contributed equally to this work.

https://doi.org/10.1016/j.rcim.2020.101990
Received 16 July 2019; Received in revised form 6 March 2020; Accepted 3 April 2020
Available online 05 May 2020
0736-5845/ © 2020 Elsevier Ltd. All rights reserved.
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 1. Design of the taping tool. (a) Components in the designed taping tool. (b) Illustration of the tape attaching capability of the design of taping tool on corners
and grooves.

by tape masking, which is commonly used in the industrial processes teaching.

such as plasma spraying, spray painting and shot peening. In such a In the robot taping task, the taping tool is required to attach the tape
taping process, the tape needs to be attached to the workpiece surfaces along the workpiece surface following a geodesic curve. In the case of
with sufficient force along a geodesic curve on the surface, while pre- grooves, the taping tool should avoid collisions with surfaces. In such a
venting the tool from damaging the workpiece. In our previous work task, simple robot telemanipulation with constant motion mapping
[15,16], an automatic taping system is developed. The system is com- method [20–23] have difficulties for the following reasons:
posed of a special taping tool for robot taping. Based on the scanned 3D
model of the workpiece, the taping path can be generated via off-line 1. Due to the resolution of the robot guidance in traditional tele-
motion planning algorithms. Next, the generated path is passed to the operation, it is difficult to perform pose fine-tuning while main-
real robot to perform the taping task. In such a system, the drawbacks of taining the geometric constraints.
the teaching method are as follows, 2. The user’s excessive inputs could cause hard contact of the end-ef-
fector tool and may damage the work cell.
1. The workpiece, the robotic system, and the real environment need 3. Constant motion mapping methods are not feasible for flexible task
to be modeled in advance. Besides, a calibration process is required manipulation that has varied motion scales.
to determine the relative positions between the robot and the
workpiece. To enable effective robot teaching in multi-Degree-of-freedom (DoF)
2. The generated motion of the robot in the off-line environment could manipulation under force and path constraints, robust and intuitive
fail to work in one trial due to the differences between the real and telemanipulation-based methods are needed [17,24].
the simulation environment. Therefore, re-calibration and correc- In this work, we propose a Fast Calibration and Variable Mapping
tion of the model parameters can be troublesome and time-con- (FCVM) framework to cope with the aforementioned issues in tele-
suming. manipulation-based teaching. FCVM provides an efficient method for
3. Due to the modeling error, the algorithm could not plan a valid path operators to calibrate the robot heading, a variable motion mapping
for some surface features, such as grooves and edges. method for the operators to guide the robot’s motion using adjustable
motion scaling factors and a flexible indexing function that supports the
Encountering these difficulties in the off-line programming system, operators to keep their poses within their conformable control range.
we started to work on direct robot teaching solutions for these taping The taping result from direct robot reproduction shows the feasi-
tasks and attempted to figure out a more efficient and user-friendly bility of this framework in telemanipulation-based robot teaching,
solution. providing the operator more flexibility to teach high-accuracy paths
Telemanipulation-based teaching enables operators to guide robotic which is hard to achieve under constant mapping based methods. Such
systems online via interactive devices due to the large flexibility from a solution can be applied to more robot tasks and provides an efficient
the high cognitive and perceptual capabilities of the human-in-the-loop way for robot path teaching.
robot teaching process [17–19]. However, telemanipulation-based
teaching still has some limitations, such as portability, low accuracy, 2. System description
and cumbersome motion mapping methods, requiring the teleoperation
systems to be improved for becoming the ultimate tool for human-robot In this work, the developed system is composed of a robotic taping

2
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

system, which has been developed in our previous work [15,16,25] and The operator can visualize the on-site conditions directly during the
a teleoperation system implemented based on the HTC VIVE System. telemanipulation and control the robot action accordingly. To avoid
occlusion, the operator should stand in a proper place such that the
2.1. Robotic taping system VIVE controller is always visible by both base stations. At the same
time, the software side detects the tracking state and alerts the operator
We design a taping tool for a liner of an aerospace engine which has timely once the tracking failure occurs.
a complex surface. The requirement of the design is to make the best The users’ inputs are generated from the HTC VIVE controller whose
use of the taping tool to not only apply tapes on the layers of the 6 DoF motion (translation and rotation) is tracked in real-time by the
workpiece surface but also reach the grooves between these layers. two fixed base stations. With the correct setup, the system can provide
Telemanipulation with such a system is able to prove the feasibility of millimeter level accuracy in position tracking and orientation accuracy
the FCVM in complex surface masking, with a few degrees(details can be found in [27,28].) Through a quick
In this robotic taping system, the new design of the specific taping calibration process and proper motion mapping methods, the human
tool is made for the effective groove and edge taping which cannot be motion can be re-targeted to the robot motion with this expected
achieved in other design [15,26]. As shown in Fig. 1(a), the taping tool manner. Therefore, the human operator can guide the robot with both
is mounted onto the robot end-effector link through the flange inter- large- and fine-scale motions, and also to freely re-adjust his/her hand
face. A tape holder is used to carry the tape. With a guidance roller, the position within a conformable region without disturbing the robot
tape can be straight guided to the tip of the taping tool. A stopping motion. Considering that the continuous adjustment of the motion
roller can maintain the tape for continuous taping patch by patch. The scales will bring more burden in the operation, we currently apply two
spring mechanism of the taping tool provides passive compliance for motion scale modes for quick switching between large and small ma-
the robot to avoid hard contacts and thereby protect the taping tool and nipulation scales, where the scales in these modes depend on the task
the workpieces. requirements. In this work, the motion scales are selected after the
A complex workpiece surface with multiple layers is very common operator’s testing with the telemanipulation tasks. In this paper, we
in engine liner parts. In such a condition, the tip of the taping tool mainly focus on the presentation of the developed system and methods,
would be easily stuck in the grooves between layers due to the lack of while a comprehensive user study is exclusive from the scope of this
human consciousness in teaching. To solve this problem, we use a High- paper.
density polyethylene (HDPE) material to provide compliance while The operation flow of the teleoperation system under the FCVM is
supporting tape attaching. HDPE is stiff enough to bear pressing force, depicted in Fig. 3. After starting the teleoperation system, an operator
and its flexural modulus is suitable to tolerate the deformation on itself firstly calibrates the robot reference frame in the teleoperation system.
so that the problem of hard contact can be alleviated. With the soft Then, the operator records the initial pose of his/her hand and the robot
covering material attached on the surface of the tip of the taping tool, end-effector for motion mapping at the same time. Next, the operator
the taping tool is able to move into the grooves for taping, providing a can activate the telemanipulation by pressing button B1 for large-scale
higher taping coverage and better taping quality, as illustrated in guidance or B2 for fine-motion tuning. During the telemanipulation,
Fig. 1(a), the new tool with a proper tooltip can attach tapes on edges, once the operator wants to take a rest or encounters the limitation of
corners and into grooves of the workpiece with complex surface, as his/her motion range, he/she can exit the telemanipulation by pressing
shown in Fig. 1(b). button B1 and adjust to a more comfortable pose, then the tele-
manipulation can be continued by pressing button B1 or B2, to trigger
different scale of motions.
2.2. Teleoperation system

The teleoperation system enables a human operator to tele- 3. Methodologies

manipulate the robot for the tape masking skill teaching through a
handheld controller, as shown in Fig. 2. In this introduced system, the 3.1. Kinematics notation and frame definition
human operator uses a handheld device HTC VIVE controller to tele-
manipulate the robot arm and the mounted taping tool, the HTC VIVE Through this paper, εTϑ ∈ SE(3) represents a homogeneous trans-
system is used to capture the operator’s hand motion, which is later formation matrix [29], εRϑ ∈ SO(3) represents a rotation matrix, and
εp ∈ 3 represents a translation vector. R (θ) denotes a rotation matrix
mapped to the robot motion through proper motion remapping rules. ϑ Z
that rotates θ along z-axis. Here ε denotes the reference frame and ϑ
denotes the target frame. The definitions of frames used in this paper
are depicted in Fig. 4. As shown in Fig 4 (a), the reference frame FW is
defined to describe the reference frame of the VIVE base stations. The
human frame FH is defined to describe human hand motion. The robot
frame FB is defined to describe the robot motion. The frame of the
handheld controller is denoted by FC. The frame of tip of the robot
taping tool FE, as shown in Fig. 4.

3.2. Fast calibration

In a general case, the reference frame FW and the robot frame FB is

not identical. A heading calibration is required to tell the displacement
between the teleoperation system and the robot reference frame.
Herein, we select the x-axis of FB as a reference. The transformation
between the robot base and the reference frame, WTB, can be obtained
by aligning the x-axis of the VIVE controller FC with the x-axis of FB. By
calculating the yaw angle Ψ from WTC, the controller pose described in
human frame FH presents the operator’s hand pose, and it can be cal-
Fig. 2. Overview of the overall robotic system. culated by

3
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 3. Operation flow of the teleoperation system.

HT ⎡ RZ (−Ψ) 03 × 1⎤ W
C =⎢ TC .
⎣ 01 × 3 1 ⎥ ⎦ (1)

3.3. Variable motion mapping

The illustration of motion mapping is shown in Fig. 5. Let HTC,t

denote the operator’s hand pose at time t and BTE,t denote the pose of
the robot taping tool-tip at time t. The human frame FH is used to define
the initial pose of the operator’s hand. The corresponding pose of the
robot taping tool is defined by frame FE. The motion mapping problem
is to find the corresponding pose of the robot taping tool BTE,t with the
human current pose HTC,t, an initial human pose HTC,0, and an initial
pose of robot taping tool BTE,0. Hence, a mapping function
fM : SE (3) × SE (3) × SE (3) → SE (3) is needed for establishing the re-
lationship among them:
BT = fM ( H TC , t , H TC ,0, BTE ,0).
Fig. 4. Definitions of frames in the system. E,t (2)

Given an object with frame FO, its initial pose described in the

4
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Then, the rotational mapping scalar kr is applied to transform the

rotation variance from the operator’s hand δHRC,t to the rotation var-
iance of the robot taping tool δBRE,t:
δ BRE , t = e kr ωt . (8)
From Lie Algebra [29], it is known that such calculation can ensure
the two motion variance share the same rotating axis, while the rotating
angular variance satisfies θE = kr θC . With Eqs. (6) and (8), we can get
the orientation of the robot taping tool at time t:
BR = e kr ωt BRE ,0 .
E,t (9)
For the translation mapping, the transnational mapping scalar kp is
used as follows:
δ BpE , t = kp δ H pC , t . (10)
Hence, by Eq. (10), we have the translation vector of the robot end-
effector, defined as follows,
Bp = kp δ H pC, t + BpE ,0 .
E,t (11)
Fig. 5. Illustration of the single arm motion remapping. Finally, given the operator’s hand pose RC,t at time t, the corre-H

sponding pose of the robot taping tool can be calculated as follows,

reference frame FW is defined as WTO,0. When the object moves to a new
kr ωt BR kp δ H pC, t + BpE ,0 ⎤
pose WTO,t at time t, we have a transformation from WTO,0 to WTO,t: BT
E,t = ⎡e E ,0
,
WT = δ W TO, t W TO,0,
⎢ 01 × 3 1 ⎥
O, t (3) ⎣ ⎦ (12)
where δ TO,t ∈ SE(3) presents the motion of the object described in the
W
where kp and kr is adjustable for telemanipulating the robot in different
reference frame FW, here we call it pose variance. This pose variance motion scales. kp = 1 and kr = 1 in the telemanipulation means ”copy”
δWTO,t can be decomposed into a rotation variance matrix and a trans- the translation and rotation displacement from the operator to the
lation variance vector δWpO,t: robot, respectively. The value of kp and kr need to be selected separately
W T W T −1
for different task scenarios due to a different range of motions.
δ W TO, t = O, t O,0 The pose of the robot taping tool BTE,t is then employed in the robot
δ W pO, t = Wp
O, t − Wp
O,0 . (4) inverse kinematics to calculate its corresponding joint angels for gen-
erating the robot motion.
Let δ TC and δ TE denote the pose variance of the human hand and
H B

the robot taping tool, respectively. As mentioned previously, the motion

4. Implementation and results
mapping problem defines the relationship between the pose variance of
human motion δHTC,t and the motion of the robot taping tool δBTE,t at
To validate the feasibility of the functions in the developed system
time t. A straight forward motion mapping is to let their pose variance
and evaluate its performances, a set of workpiece surface taping ex-
to be equivalent at any time t ≥ 0:
periments are conducted via the telemanipulation-based teaching
δ BRE , t = δ H RC, t method. The workpiece is an inner liner part of the turbine engine that
δ BpE , t = δ H pC , t . has multiple layers with narrow grooves on its surfaces. Due to such
(5)
complex surface constraints, the robot needs to be telemanipulated in a
However, for dexterous manipulation tasks, the constant mapping fine-motion tuning mode.
cannot fit in various motion ranges with adjustable resolution. In robot
telemanipulation, it is of importance to support the operator works in 4.1. Vertical taping through FCVM
an ergonomic and comfortable motion range while guiding the robot to
work in its own workspace. To meet the requirements of the complex The experimental protocol of this one-patch vertical taping is as
telemanipulation which needs operators’ different motion resolutions, a follows,
variable mapping method is introduced to enable operators to change
the motion mapping scales in both translation and rotation motion. 1. After the initialization and calibration of the system, the operator
According to Eq. (4), we have the rotation variances of the input and prepares his initial pose via holding the VIVE controller. Then, he/
the robot taping tool δHRC,t and δBRE,t as follows she activates the telemanipulation with a large-scale guidance
H −1
mode.
δ H RC , t = HR
C , t RC ,0
2. The operator can take a rest on the telemanipulation system by
δ BRE , t = BR BR −1 .
E,t E ,0 (6) pressing a button on the handheld controller, meanwhile, apply the
indexing function to re-maps his hand pose for telemanipulation in a
For the orientation mapping, in order to let the operator beware of
comfortable range.
the desired robot motion outputs in the telemanipulation, the rotation
3. After the robot is guided to the neighborhood of the workpiece, the
mapping is constructed such that the rotation axes of the δHRC,t and
operator switches to a fine-motion tuning mode.
δBRE,t are equal and has a corresponding rotating angle scalar kr,
4. The operator observes the on-site condition and guides the robot
θE = kr θC . Here, a skew-symmetric matrix of input rotation variance is
taping tool to follow the workpiece surfaces, attaching the tape onto
calculated [29]:
the surface as well as the edges between layers.
ωt = Skew (δ H RC, t ), (7) 5. When the taping tool tip is approaching the narrow grooves on top
of the workpiece, the operator applies indexing and let the taping
where Skew: SO (3) → so (3) is a function that calculates the corre-
tool tip move into the groove slowly without colliding the workpiece
sponding skew-symmetric matrix of a rotation matrix.

5
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 6. Implementation of FCVM in the complex surface masking task.

to apply the tape attachment. than the previous taping tool which cannot handle edge and groove
6. Finally, the robot is telemanipulated to the upper edge of the taping. In addition, the comparison with method [23] in Fig. 8 shows
workpiece. Manual tape cutting is currently needed. that the developed method enable the robot to successfully attach the
tape onto the workspace with complex geometry.
As mentioned, a continuous adjustment of the mapping scalars will
increase the difficulty in operation and it is also not easy for the user to
adapt to many scales of telemanipulation. Therefore, based on the op- 4.2. Comparison- FCVM vs constant mapping
erator’s testing in this task, kp = kr = 0.8 is selected for a large-scale
motion mapping. kp = kr = 0.2 is selected for fine-motion mode motion In order to show the effectiveness of the FCVM method in improving
mapping in this experiment. the telemanipulation based robot teaching, the user conducted robot
The experimental results are shown in Fig. 6. During the teaching teaching of the one patch vertical taping through both FCVM and a
phase, the system responds in an understandable manner such that the constant motion mapping method where kp = kr = 0.8 are selected in
operator can familiarize the teleoperation skill without much effort. In order to let robot cover sufficient manipulation range, and without
Fig. 6(a), fast calibration for determining the robot pose is conducted. indexing. The operator repeats each type of taping for three trails and
Fig. 6(b) shows the recording of the initial pose of the operator’s hand the taught path is reproduced after each type of robot teaching. Fig. 10
and the robot taping tool-tip for motion mapping. Fig. 6(c), the operator provides the teaching and reproducing result of the one patch taping
telemanipulates the robot to a desired pose in the large-scale guidance based on the constant mapping based system. In this large scale motion
mode. In Fig. 6(e), the operator changes to fine-motion tuning mode for mapping, posture fine-tuning is very difficult. The taught path fairly
taping. In Fig. 6(f–h), the operator repetitively uses the indexing follows the surface and cannot handle the edges between layers and the
function for finishing the taping task in the fine-tuning mode. In narrow grooves. On the other hands, the FCVM based method is able to
Fig. 6(i), the operator finishes the task. Then, in Fig. 6(j–l), the robot is teach the robot to tape with better quality, as shown in Fig. 12. With
doing the taping based on the taught motion. As shown in the supple- FCVM, the operator is able to telemanipulate the robot with fine-motion
mentary video, the experiment takes a skilled operator about 5 min to tuning for edge and groove features. The indexing function solves the
complete the teaching process. motion range re-adjustment issue. Fig. 11 provides a sample of the
After the teaching, the recorded motion is directly reproduced by sequential states of the telemanipulation while teaching the taping task,
the robot for validating the feasibility of manipulation in teaching. The where 0.8 denotes the large scale mapping phase, 0.2 denotes the fine-
recorded robot waypoints are taken as key points in the robot path motion phase and 0 denotes the posture re-adjusting phase applying the
generation. The reproduction results are as shown in Fig. 7. Right now indexing functions.
the spring mechanism and compliance tool tip ensure that the tool will The operating times for the three trails of FCVM based system and
not have hard contact with the workpiece surface. In other words, constant mapping based system are 5:43, 6:22, 4:05, and 2:13, 2:09,
current force compliance is a passive reaction. 2:01 separately. During the teaching time, the robot responds properly
Comparing the taping quality of the teaching performance against in both large scale and fine-motion telemanipulation, making the op-
manual method, as shown in Fig. 9, it is obvious that with the FCVM, eration a smooth and enjoyable experience. Since the operator manip-
the taping quality of the telemanipulation-based human teaching and ulates the robot with indexing breaks, five-minute teaching is satisfied
its corresponding reproduction results are in general equivalent to the to the requirements of the task.
manual taping using the designed taping tool, and they are much better

Fig. 7. Task reproduction from the robot. The robot follows the motion generated in the human demonstration for the complex surface masking task. (a) The robot is
moving to an assigned starting pose. (b) The robot starts taping. (c)–(g) The robot is doing the taping. (h) The robot finishes the taping.

6
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 8. Taping quality comparison between previous method [23] and developed method in coping with complex surface.

Fig. 9. Taping results of human manual taping with the previous taping tool, human manual taping with the designed taping tool, human teleoperation under the
FCVM, and the robot reproduction.

4.3. Taping horizontally through FCVM 4.4. Discussion of performances

In the actual taping process, horizontal tapping is common. An ex- The introduced telemanipulation system for tape masking is a setup
periment is conducted to show the flexibility of the FCVM solution. In for the specific application, and there are few similar systems to the best
order to perform a horizontal taping motion, which is hereafter the of authors’ knowledge. Therefore, there is no direct comparison data
operator completes a vertical taping with the tool, the operator applies against similar taping systems. However, in terms of functionality
the indexing function to break the robot, and adjust to a conformable achieved, the telemanipulation task under the high complexity of
control reference pose. Then, he activates the robot telemanipulation to geometry which needs switching between different scales of motions is
guide it to the horizontal direction, through applying the indexing for a much more challenging as compared with general-purpose tele-
few times. A horizontal taping process is followed afterward. The re- manipulation systems.
sults are shown in Fig. 13, and details can be found in the supple- As a comparison of the system performance with manual teaching,
mentary video. The indexing function supports the operator to change our previous works have shown that the manual teaching of taping one
the motion mapping frequently, and manipulate the robot to work more patch, without taping the edge and grooves, will cost an average of five
flexibly in its workspace. minutes [23]. In such a manual teaching condition, only about 15
waypoints need to be recorded before the robot can generate a taping
path based on a linear interpolation method. However, for taping the
grooves and edges, there are more waypoints needed to avoid the

Fig. 10. Constant mapping based telemanipulation taping results (three trails) and their corresponding reproduction results.

7
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 11. Change of motion mapping scale factor along the whole telemanipulation taping process.

collisions between the workpiece and the tools as well as the tape operator’s motion range.
segment. Therefore, such a proposed solution is more efficient when Experimental results show that the FCVM based system works nicely
comparing to manual teaching and our previous methods. in robot teaching with better taping quality and superior user experi-
In summary, the improvements of such system as compared to tra- ence as compared with constant mapping solutions. The operator is able
ditional constant telemanipulation method, the advantage of this in- to teach the robot with not only large-scale guidance but also fine-
troduced system and methods includes: motion tuning in constrained environments. Under the proposed FCVM
framework, an operator is able to telemanipulate a robot to apply tape
1. Fast system initialization and quick calibration and re-calibration. segments on the edges between layers and narrow grooves. The op-
2. Cover large motion scale range and provide various resolution in erator is also able to teach the robot to tape along different orientations
telemanipulation. via the indexing function under this framework. The demonstrated
3. Being able to hand precise motion fine tuning. taping paths can be directly duplicated by the robot, where the taping
4. Flexible to adjust robot pose to manipulate tasks in different regions quality is almost equivalent to the manual taping with the same tool.
of the workspace. Compared with existing robot programming solutions, such as
Offline Programming (OLP) and teach pendant based kinesthetic pro-
gramming methods, the telemanipulation-based teaching method pro-
5. Conclusions vides a very efficient and intuitive method to program complex ma-
nipulation. In the groove taping example, due to the complexity of the
This paper presents a framework, FCVM, which provides a flexible taping path, at least dozens of waypoints are required if the operator
solution in telemanipulation-based robot teaching for manipulation what to manually teach the robot a trajectory to tape the groove while
under complex geometric constraints. In this work, the developed tel- avoiding collisions. In addition, the taught path may fail to satisfy the
eoperation system consists of a handheld controller set, a robot ma- constraints when it is further interpolated to generate an executable
nipulator with specially designed taping tools as the end-effector. Such path for the robot. The proposed FCVM framework helps this teaching
a human-in-the-loop system realized the human-robot telemanipulation method to narrow the gap between human actions and robot reactions,
through corresponding calibration and motion mapping methods. This making robots acquire human skills much easier (faster trajectory
framework enables an operator to set up and teach a robot for masking teaching with high sampling rates) and releasing human operators from
complex workpiece surfaces more efficiently and flexibly via tele- tedious programming. We believe that the FCVM can be applied to
manipulation. With the proposed fast calibration, an operator can easily telemanipulation-based robot skill teaching and reduce the program-
figure out the relative displacement between the robot reference frame ming time for auto-processing of high-mix low-volume product pro-
and the VIVE system for the initialization of the system. With the de- cessing. Such a framework is not only using for taping applications. To
veloped variable motion mapping method, the operator can flexibly name one example, spray painting with robot arm also involves fre-
telemanipulate the robot and switch between large-scale motion (for quent trajectory planning [30,31], especially when there are different
efficient guidance) and fine-tuning motion (for dexterous manipula- types of workpieces. With the use of this framework, operators can
tion). Meanwhile, the indexing function supports the operator in fre- effectively teach the robot remotely while keeping away from the dust
quently changing the motion mapping between different mapping environment.
scales, providing a convenient manner to cope with the limitation of the

Fig. 12. FCVM-based telemanipulation taping results (three trails) and their corresponding reproduction results.

8
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

Fig. 13. Changing the robot taping direction through adjusting the robot tool pose using the indexing function in FCVM.

In this paper, the telemanipulation method with quick calibration architecture for human-robot assembly tasks, Intelligent Robots and Systems
and efficient motion teaching are realized. To enables the robot ma- (IROS), 2017 IEEE/RSJ International Conference on, IEEE, 2017, pp. 1624–1629.
[6] H. Dong, G. Sun, W. Pang, E. Asadi, D.K. Prasad, I. Chen, Fast ellipse detection via
nipulation for more applications, the further development will focus on gradient information for robotic manipulation of cylindrical objects, IEEE Rob.
the trajectories optimization and the pose/force compensation based on Autom. Lett. 3 (4) (2018) 2754–2761, https://doi.org/10.1109/LRA.2018.
sensor feedback. 2836428.
[7] F. Azimifar, M. Abrishamkar, B. Farzaneh, A. Sarhan, H. Amini, Improving tele-
operation system performance in the presence of estimated external force, Robot.
CRediT authorship contribution statement Comput. Integr. Manuf. 46 (2017) 86–93.
[8] C.M.T. Samuel, K.P. Tee, Unified human-robot shared control with application to
haptic telemanipulation, 2019 IEEE/RSJ International Conference on Intelligent
Qilong Yuan: Conceptualization, Methodology, Data curation, Robots and Systems (IROS), (2019), pp. 2221–2226, https://doi.org/10.1109/
Writing - original draft, Funding acquisition, Writing - review & editing. IROS40897.2019.8967719.
Ching-Yen Weng: Conceptualization, Methodology, Software, Data [9] H. Dong, E. Asadi, G. Sun, D.K. Prasad, I. Chen, Real-time robotic manipulation of
cylindrical objects in dynamic scenarios through elliptic shape primitives, IEEE
curation, Writing - original draft, Writing - review & editing, Software.
Trans. Robot. 35 (1) (2019) 95–113, https://doi.org/10.1109/TRO.2018.2868804.
Francisco Suárez-Ruiz: Data curation, Writing - original draft, [10] C.-Y. Weng, W.C. Tan, Q. Yuan, I.-M. Chen, Quantitative assessment at task-level for
Visualization, Investigation, Writing - review & editing. I-Ming Chen: performance of robotic configurations and task plans, J. Intell. Robot. Syst. 96 (3–4)
Visualization, Investigation, Funding acquisition, Resources. (2019) 439–456.
[11] F. Suárez-Ruiz, X. Zhou, Q.-C. Pham, Can robots assemble an ikea chair? Sci. Robot.
3 (17) (2018), https://doi.org/10.1126/scirobotics.aat6385.
Declaration of Competing Interest [12] M. Liarokapis, A.M. Dollar, Deriving dexterous, in-hand manipulation primitives for
adaptive robot hands, Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ
International Conference on, IEEE, 2017, pp. 1951–1958.
The authors declare that they have no known competing financial [13] C.-Y. Weng, W. Yin, Z.J. Lim, I.-M. Chen, A framework for robotic bin packing with
interests or personal relationships that could have appeared to influ- a dual-arm configuration, IFToMM World Congress on Mechanism and Machine
Science, Springer, 2019, pp. 2799–2808.
ence the work reported in this paper. [14] H. Fukui, S. Yonejima, M. Yamano, M. Dohi, M. Yamada, T. Nishiki, Development of
teaching pendant optimized for robot application, Advanced Robotics and its Social
Acknowledgment Impacts (ARSO), 2009 IEEE Workshop on, IEEE, 2009, pp. 72–77.
[15] Q. Yuan, I.-M. Chen, T.S. Lembono, An agile robot taping system – modeling, tool
design, planning and execution, Ind. Robot (2016), https://doi.org/10.1108/IR-01-
This project is partially supported under National Natural Science 2016-0018.
Foundation (51705076) of China. The authors would like to thank our [16] Q. Yuan, I.-M. Chen, T.S. Lembono, S.N. Landén, V. Malmgren, Automatic robot
taping system with compliant force control, Int. J. Adv. Manuf.Technol. 94 (9–12)
colleagues in helping us with the experiments.
(2018) 4105–4113.
[17] G. Niemeyer, C. Preusche, G. Hirzinger, Telerobotics, Springer handbook of ro-
Supplementary material botics, Springer, 2008, pp. 741–757.
[18] F. Conti, O. Khatib, Spanning large workspaces using small haptic devices, First
Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual
Supplementary material associated with this article can be found, in Environment and Teleoperator Systems. World Haptics Conference, IEEE, 2005, pp.
the online version, at doi:10.1016/j.rcim.2020.101990. 183–188.
[19] D. Rakita, B. Mutlu, M. Gleicher, L.M. Hiatt, Shared control–based bimanual robot
manipulation, Sci. Robot. 4 (30) (2019) eaaw0955.
References [20] J. Kofman, X. Wu, T. Luu, S. Verma, Teleoperation of a robot manipulator using a
vision-based human-robot interface, IEEE Trans. Ind. Electron. (2005), https://doi.
[1] H. Li, K. Kawashima, Bilateral teleoperation with delayed force feedback using time org/10.1109/TIE.2005.855696.
domain passivity controller, Robot. Comput. Integr. Manuf. 37 (2016) 188–196. [21] D. Rakita, B. Mutlu, M. Gleicher, A motion retargeting method for effective mi-
[2] K.P. Tee, Y. Wu, Experimental evaluation of divisible human-robot shared control micry-based teleoperation of robot arms, ICHRI, (2017), https://doi.org/10.1145/
for teleoperation assistance, TENCON 2018 - 2018 IEEE Region 10 Conference, 2909824.3020254.
(2018), pp. 0182–0187, https://doi.org/10.1109/TENCON.2018.8650436. [22] Q. Yuan, T.Y. Seng, Q. Lu, I.M. Chen, Task-orientated robot teleoperation using
[3] P. Zhang, Markerless human-robot interface for dual robot manipulators using ki- wearable IMUs, 2017 IEEE International Conference on Cybernetics and Intelligent
nect sensor, Robot. Comput. Integr. Manuf. 30 (2014) 150–159, https://doi.org/10. Systems, CIS 2017 and IEEE Conference on Robotics, Automation and
1016/j.rcim.2013.09.003. Mechatronics, RAM 2017 - Proceedings, (2018), https://doi.org/10.1109/ICCIS.
[4] A. Gupta, C. Eppner, S. Levine, P. Abbeel, Learning dexterous manipulation for a 2017.8274758.
soft robotic hand from human demonstrations, Intelligent Robots and Systems [23] C. Weng, Q. Yuan, F. Suárez-Ruiz, I. Chen, A telemanipulation-based human-robot
(IROS), 2016 IEEE/RSJ International Conference on, IEEE, 2016, pp. 3786–3793. collaboration method to teach aerospace masking skills, IEEE Trans. Ind. Inf. 16 (5)
[5] I. El Makrini, K. Merckaert, D. Lefeber, B. Vanderborght, Design of a collaborative (2020) 3076–3084, https://doi.org/10.1109/TII.2019.2906063.

9
Q. Yuan, et al. Robotics and Computer Integrated Manufacturing 66 (2020) 101990

[24] D. Rakita, B. Mutlu, M. Gleicher, Remote telemanipulation with adapting view- Perception 8 (3) (2017). 2041669517708205
points in visually complex environments, Proceedings of the Robotics: Science and [28] M. Borges, A. Symington, B. Coltin, T. Smith, R. Ventura, Htc vive: analysis and
Systems, (2019). accuracy improvement, 2018 IEEE/RSJ International Conference on Intelligent
[25] Q. Yuan, I.M. Chen, T.S. Lembono, S.N. Landén, V. Malmgren, Strategy for robot Robots and Systems (IROS), IEEE, 2018, pp. 2610–2615.
motion and path planning in robot taping, Front. Mech. Eng. (2016), https://doi. [29] R.M. Murray, Z. Li, S. Sastry, S.S. Sastry, A Mathematical Introduction to Robotic
org/10.1007/s11465-016-0390-1. Manipulation, Boca Raton,FL, 1994.
[26] T.S. Lembono, Q. Yuan, Y. Zou, I.M. Chen, Automatic robot taping: System in- [30] W. Chen, D. Zhao, Path planning for spray painting robot of workpiece surfaces,
tegration, IEEE/ASME International Conference on Advanced Intelligent Math. Prob. Eng. 2013 (2013).
Mechatronics, AIM, (2015), https://doi.org/10.1109/AIM.2015.7222633. [31] L. Scalera, E. Mazzon, P. Gallina, A. Gasparetto, Airbrush robotic painting system:
[27] D.C. Niehorster, L. Li, M. Lappe, The accuracy and precision of position and or- experimental validation of a colour spray model, International Conference on
ientation tracking in the htc vive virtual reality system for scientific research, i- Robotics in Alpe-Adria Danube Region, Springer, 2017, pp. 549–556.

10