18MHC402T – FUNDAMENTALS OF ROBOTICS

CLAT – 4 ASSIGNMENTS
CASE STUDY

M.MELFIN ELISHA DEVOUT

RA2111018010024
B.TECH – MECHATRONICS ENGINEERING
BATCH - A

1
 VISION BASED CONTROL OF ROBOTS

Definition:
Visual servoing, also known as vision based control of robots, is the method of controlling
a robot’s motion using real-time feedback from vision sensors to execute tasks. Computer
vision methods extract features from the robot’s operational scene and determine how the
robot should move in order to articulate the image features or converge to the task
workpiece.
The techniques are heavily reliant on good camera calibration, an accurate kinematic
model of the robot, good IK and FK solvers, and satisfactory camera performance (good
pixel resolution, FPS, shutter speed, etc.).

Fundamental configurations of the robot end-effector (hand) and the camera:

Depending on the positioning of the camera, visual servoing has two paradigms:
1) Eye-in-hand
2) Eye-to-hand

The camera is mounted on the end-effector of the robot in the eye-in-hand configuration
while it is mounted on a fixed location in the world. Thus all camera observations made in
the eye-in-hand paradigm are with respect to the camera which has a known
transformation with respect to the end-effector of the robot ( which is in motion).
Meanwhile, the eye-to-hand paradigm has fixed a camera frame with known
transformation from the robot base or another origin.

2
Position/pose-based visual servoing
PBVS usually uses depth cameras to obtain the 3D pose — position (x, y, z) and
orientation (Euler or Quaternion) — of the regions/objects of interest and compares the
robot’s end-effector’s pose against it. The error term e thus formulated is the Cartesian
pose difference between the two and the servoing scheme attempts to minimize it by
moving the robot around, ideally towards the final desired pose. For example, based on
the location of the object to be grasped seen in the image, the scheme generates the
ideal grasp pose (which may keep changing due to relative change in camera pose) for
the end-effector and tries to converge the robot to it.
Since PBVS works with real-world poses, it needs at least a 6-DOF robot arm to
successfully implement the solution without getting stuck in local minima. It also uses
robot inverse kinematics to convert Cartesian control instructions into joint angle values
for the robot. Since obtaining the 3D pose involves the conversion of information from the
camera frame to the robot frame, camera calibration plays a significant role in the
process.

Image based visual servoing:

Unlike PBVS operating in the real-world 3D pose, IBVS extracts camera features and then
formulates the error in the image plane itself. On an intuitive level, if the position
coordinates of the desired feature in an image are [0.2, 0.5] and the current position
3
coordinates of the image are [0.5, 0.5], the visual servoing scheme tries to converge the
feature to the desired coordinates and moves the robot accordingly. the first step of IBVS
is the projection of the 3D image on a 2D image plane. The image feature extraction in
IBVS is prone to poor camera performance, synchronization issues, and computational
requirements. Owing to the mathematics, to define the control logic for a 6-DOF robot
arm, at least three features are needed in the image. The feature Jacobian (which
encodes the information about the motion of image pixels) is responsible to derive the
control signal for the robot by relating the motion in feature pixels against the same for the
end-effector.

Problems concerning visual control of robots:

In the field of visual servoing, the major challenges or problems lie in the uncertainties and
the specification of the desired motion in the image space. Environments to work on can
often be complex and unpredictable with variable lighting conditions, occlusions and
diverse objects. So, developing a robust perception algorithm is made difficult. Vision
based control often demands significant computational resources to process images or
video streams in real time, which can be a limiting factor in resource constrained robots.
Moreover, as time takes everything with it, the quality of camera sensors may degrade
affecting the reliability of the system. Vision sensors often need to be integrated with other
sensors such as lidars or inertial sensors, to improve overall perception and control,
synchronizing and fusion of data from different sensors modalities can be of challenge.

Solution for the concerned problems:

To handle the environment efficiently, robust computer vision and machine learning
algorithms that can handle variable lighting conditions , occlusions and diverse objects
can be developed and implemented. For accelerated image processing and machine
learning tasks, special hardware like GPU or TPU can be employed to reduce the
4
computational load and latency. And redundancy in sensors and system components to
improve fault tolerance and reliability can be incorporated. To handle sensor degradation
or failures, sensor health monitoring system can be implemented. Protocols or safety
mechanisms can be developed to ensure safe robot behavior even in case of sensor
failure or misinterpretations. Integration techniques ensuring consistency and accuracy in
the fused data can be developed to help combine data from multiple sensors.

STRENGTHS WEAKNESS
Human-like perception
High information content Data intensive trainging
Versitality Limitied sensing range
Rapid Technological Advancements Environmental variabilty

SWOT ANALYSIS
FOR VISION BASED
CONTROL OF
ROBOTS

OPPURTUNITIES THREATS
AI advancdement Interferance and cyber secutity
Cost reduction risks
Cross-Industry applications Market competitions
Regulatory challenges

References:
1. https://www.researchgate.net/
2. https://robotacademy.net.au/lesson/vision-based-robot-control/
3. https://control.com/technical-articles/an-overview-of-visual-servoing-for-robot-
manipulators/
4. https://link.springer.com/chapter/10.1007/bfb0015083

5
 Rubric for Case Studies
Criteria Distinguished Proficient/Good Basic/Fair Non- Marks Weighted
/Excellent (3-4 (2 marks) Performance/Poor Allocated Marks
(5 marks) marks) (0-1 mark) 25 marks obtained
Identification of the Excellent understanding Good understanding in Some understanding in Unable to identify 5
main issues and/or in the identification of the identification of the identification of issues in the case study
problems. issues in the case study issues in the case study issues in the case study

Literature research Excellent research Good research and Limited research and Incomplete research 5
& review. with clearly documented associations between and associations
documented associations between the problems or between the problems
associations between problems or questions questions and key or questions and key
problems or questions and key course course concepts and course concepts and
and key course concepts and some (2 little (1 or more) use no use of
concepts and good (3 or more) use of of corroborating corroborating
or more) use of corroborating sources. sources. sources.

Analysis of the key

•Excellent
nt Excelle
corroborating sources.
analysis of Good analysis of the Adequate analysis of No analysis 5
Issues using SWOT the issues. Excellent issues. Good the issues. Reasonable of the key issues. Very
Analysis tool. presentation on key understanding on key understanding on key poor understanding on
elements of SWOT. elements of SWOT. elements of SWOT. key elements of SWOT.

Provide suggestions Excellent and very Good and appropriate Adequate suggestion Poor suggestion and 5
on appropriate suggestion suggestion on solutions and appropriate least appropriate
Appropriate solutions. solutions to the issues in to the issues in the case solutions to the issues solutions to the issues in
the case study study in the case study the case study

Writing Exceptional use of Significant use of Adequate use of Inadequate use of proper 5
Skills/Professional proper grammar, proper grammar, proper grammar, grammar, spelling and
Presentation. spelling and writing spelling and writing spelling and writing writing effective
effective sentences that effective sentences that effective sentences that sentences that make
make logical sentences. make logical sentences make logical sentences logical sentences.

Criteria Comments
Total 25
Assignment Identification of the main
(Case Study) issues and/or problems.
Points Literature research &
review.
Case Study
FinalScore: 0 Analysis of the key issues using
SWOT analysis tool
Provide suggestions on
appropriate solutions
Writing Skills