Raspberry Pi Robotic Projects

Richard Grimmett

Chapter No. 4 "Adding Vision to Raspberry Pi"

In this package, you will find:

A Biography of the author of the book A preview chapter from the book, Chapter NO.4 "Adding Vision to Raspberry Pi" A synopsis of the books content Information on where to buy this book

About the Author

Richard Grimmett has been fascinated by computers and electronics from his very first programming project, which used Fortran on punch cards. He has a bachelor's and master's degree in electrical engineering and a PhD in leadership studies. He also has 26 years of experience in the radar and telecommunications industries (he even has one of the original brick phones). He now teaches computer science and electrical engineering at Brigham Young University - Idaho, where his office is filled with his many robotic projects. He recently completed a book on using BeagleBone Black for robotic projects, and is now excited to release this title for those who prefer Raspberry Pi. I would certainly like to thank my wife Jeanne and family for providing me with a wonderful, supportive environment that encourages me to take on projects such as this one. I would also like to thank my students; they showed me that amazing things can be accomplished by those who are unaware of all the barriers.

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Raspberry Pi Robotic Projects

The success of Arduino, Raspberry Pi, and other such processors has fueled a community of open source developers who now provide amazing capabilities at no cost. The continued support of the community now makes producing complex robotic projects something almost anyone with an interest in the area can do. Raspberry Pi Robotic Projects is an attempt to organize that set of information and make it available to a wide audience of possible robotics developers. I can only hope it will help inspire a new generation that will be as comfortable with robots as this generation is with personal computers.

What This Book Covers

Chapter 1, Getting Started with Raspberry Pi, will show you how to power up your Raspberry Pi, connect it to a keyboard, mouse, display, and a remote computer, and how to begin to access potential computing power. Chapter 2, Programming Raspberry Pi, will help you learn the basics of programming Raspberry Pi, both in Python and C programming languages. Chapter 3, Providing Speech Input and Output, will help you teach Raspberry Pi to both speak and listen. Chapter 4, Adding Vision to Raspberry Pi, will teach you to use a standard USB webcam to allow your robotic projects to see. Chapter 5, Creating Mobile Robots on Wheels, will show you how to connect Raspberry Pi to a mobile-wheeled platform and control its motors so that your robots can be mobile. Chapter 6, Making the Unit Very Mobile Controlling the Movement of a Robot with Legs, will show you how to make your robot able to walk. Chapter 7, Avoiding Obstacles Using Sensors, shows you how to sense the world around you. As your robot is now mobile, you'll want to avoid or find objects. Chapter 8, Going Truly Mobile The Remote Control of Your Robot, shows you how to control your robot wirelessly as you'll want your robot to move around untethered by cables.

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 9, Using a GPS Receiver to Locate Your Robot, shows you how to use a GPS receiver so that your robot will know its location since your robot may get lost if it is mobile. Chapter 10, System Dynamics, focuses on how to bring together all the capabilities of the system to make complex robots. Chapter 11, By Land, By Sea, and By Air, teaches you how to add capabilities to robots that sail, fly, and even go under water.

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

In the previous chapter, you learned how to communicate with Raspberry Pi via voice. In this chapter, we are going to add vision with a webcam; you'll use this functionality in lots of different applications. Fortunately, adding hardware and software for vision is both easy and inexpensive. To do this, you'll have to add a USB webcam to your system. Having the standard USB interface on your board opens a wide range of amazing possibilities. Furthermore, there are several amazing open source libraries that offer complex capabilities, which we can use in our projects without spending months coding them. Vision will open a set of possibilities for your project. These can range from simple motion detection to advanced capabilities such as facial recognition, object identication, and even object tracking. The robot can also use vision to detect its surroundings and avoid obstacles. In this chapter, we will cover the following topics: Connecting our USB camera to our Raspberry Pi and viewing the images Downloading and installing OpenCV, a full-featured vision library Using the vision library to detect colored objects

To add vision to our projects, we'll need a Raspberry Pi with a LAN connection and a 5V power supply. We'll also need to add a USB webcam; try to nd a recently manufactured one. You may have an older webcam sitting on your project shelf, but it will probably cause problems as Linux may not have driver support for these devices, and the money you save will not be worth the frustration you might have later. You should stick with webcams from major players, such as Logitech or Creative Labs.

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

In most cases, you won't need to connect this device through a powered USB hub; however, if you encounter problems, for example, if the system does not recognize that your webcam is connected, realize that lack of USB power could be the problem.

Connecting the USB camera to Raspberry Pi and viewing the images

The rst step in enabling computer vision is connecting the USB camera to the USB port. I am using a Logitech C110 camera in my example. To access the USB webcam, I like to use a Linux program called guvcview. Install this by entering sudo apt-get install guvcview. Connect your USB camera and make sure your LAN cable is plugged in. Then, apply power to Raspberry Pi. After the system is booted, you can check to see if Raspberry Pi has found your USB camera. Go to the /dev directory and type ls. You should see something as shown in the following screenshot:

[ 84 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

Look for video0, as this is the entry for your webcam. If you see it, the system knows your camera is there. Now, let's use guvcview to see the output of the camera. Since it will need to output some graphics, you either need to use a monitor connected to the board, as well as a keyboard and mouse, or you can use vncserver as described in Chapter 1, Getting Started with Raspberry Pi. If you are going to use vncserver, make sure you start the server on Raspberry Pi by typing vncserver via SSH. Then, start up VNC Viewer as described in Chapter 1, Getting Started with Raspberry Pi. Open a terminal window and type sudo guvcview. You should see something as shown in the following screenshot:

[ 85 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

The video window displays what the webcam sees, and the GUVCViewer Controls window controls the different characteristics of the camera. The default settings of the Logitech C110 camera work ne. However, if you get a black screen for the camera, you may need to adjust the settings. Select the GUVCViewer Controls window and the Video & Files tab. You will see a window where you can adjust the settings for your camera, as shown in the following screenshot:

[ 86 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

The most important setting is Resolution. If you see a black screen, lower the resolution; this will often resolve the issue. This window will also tell you what resolutions are supported by your camera. Also, you can display the frame rate by checking the box to the right of the Frame Rate setting. Be aware, however, that if you are going through vncviewer, the refresh rate (how quickly the video window will update itself) will be much slower than if you're using Raspberry Pi and a monitor directly. Once you have the camera up and running and the desired resolution set, we can go on to download and install OpenCV.
You can connect more than one webcam to the system. Follow the same steps, but connect to cameras via a USB hub. List the devices in the /dev directory. Use guvcview to see the different images. One challenge, however, is that connecting too many cameras can overwhelm the bandwidth of the USB port.

Downloading and installing OpenCV a fully featured vision library

Now that you have your camera connected, you can begin to access some amazing capabilities that have been provided by the open source community. The most popular of these for computer vision is OpenCV. To do this, you'll need to install OpenCV. There are several possible ways of doing this; I'm going to suggest ones that I follow to install it on my systems. Once you have booted the system and opened a terminal window, type the following commands in the following order:
sudo apt-get update If you haven't done this in a while, it is a good idea

to do this now before you start. You're going to download a number of new software packages, so it is good to make sure everything is up to date. previous chapter. In case you skipped that part, you will have to refer to it now, as you need this package. code and decode audio and video streams.

sudo apt-get install build-essential We have done this in a

sudo apt-get install libavformat-dev This library provides a way to sudo apt-get install ffmpeg This library provides a way to transcode

audio and video streams.

[ 87 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

sudo apt-get install libcv2.3 libcvaux2.3 libhighgui2.3 This command shows the basic OpenCV libraries. Note the number in the command. This will almost certainly change as new versions of OpenCV become available. If 2.3 does not work, either try 2.4 or google for the latest version of OpenCV. sudo apt-get install python-opencv This is the Python development kit needed for OpenCV, as you are going to use Python. sudo apt-get install opencv-doc This command will show the

documentation for OpenCV just in case we need it.

sudo apt-get install libcv-dev This command shows the header le and static libraries to compile OpenCV. sudo apt-get install libcvaux-dev This command shows more development tools for compiling OpenCV. sudo apt-get install libhighgui-dev This is another package that provides header les and static libraries to compile OpenCV.

Make sure you are in your home directory, and then type cp -r /usr/share/doc/ opencv-doc/examples. This will copy all the examples to your home directory. Now you are ready to try out the OpenCV library. I prefer to use Python while programming simple tasks; hence, I'll show the Python examples. If you prefer the C examples, feel free to explore. In order to use the Python examples, you'll need one more library. So type sudo apt-get install python-numpy, as you will need this to manipulate the matrices that OpenCV uses to hold images. Now that you have these, you can try one of the Python examples. Switch to the directory with the Python examples by typing cd /home/pi/examples/python. In this directory, you will nd a number of useful examples; we'll only look at the most basic, which is called camera.py. If camera.py is not created, you can create it by typing in the code shown in the next few pages. You can try running this example; however, to do this you'll either need to have a display connected to Raspberry Pi or you can do this over the vncserver connection. Bring up the LXTerminal window and type python camera.py. You should see something as shown in the following screenshot:

[ 88 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

The camera window is quite large; you can change the resolution of the image to a lower one, which will make the update rate faster and the storage requirement for the image smaller. To do this, edit the camera.py le and add two lines, as shown in the following screenshot:

[ 89 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

Here is an explanation of the Python code:

import cv2.cv as cv This line imports the OpenCV library so you can

access its functionality. time functionality.

import time This line imports the time library so you can access the cv.NamedWindow("camera", 1) This line creates a window that you will

use to display your image.

capture = cvCaptureFromCAM(0) This line creates a structure that knows how to capture images from the connected webcam. cv.SetCaptureProperty(capture, 3, 360) This line sets the image

width to 360 pixels.

cv.SetCaptureProperty(capture, 4, 240) This line sets the image

height to 240 pixels.

while True: Here you are creating a loop that will capture and display

the image over and over until you press the Esc key.

it in the img data structure.

img = cv.QueryFrame(capture) This line captures the image and stores cv.ShowImage("camera", img) This line maps the img variable to the

camera window you created previously.

If cv.WaitKey(10) == 27: This if statement checks if a key has been

pressed, and if the pressed key is the Esc key, it executes the break, which stops the while loop and the program reach its end and stop. You need this statement in your code because it also signals OpenCV to display the image now.

Now run camera.py, and you should see the following screenshot:

[ 90 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

You may want to play with the resolution to nd the optimum settings for your application. Bigger images are greatthey give you a more detailed view on the worldbut they also take up signicantly more processing power. We'll play with this more as we actually ask our system to do some real image processing. Be careful if you are going to use vncserver to understand your system performance, as this will signicantly slow down the update rate. An image that is twice the size (width/ height) will involve four times more processing. Your project can now see! You will use this capability to do a number of impressive tasks that will use this vision capability.

[ 91 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

Using the vision library to detect colored objects

OpenCV and your webcam can track objects. This might be useful if you are building a system that needs to track and follow a colored ball. OpenCV makes this amazingly simple by providing some high-level libraries that can help us with this task. I'm going to do this in Python, as I nd it much easier to work with than C. If you feel more comfortable with C, these instructions should be fairly easy to translate. Also, performance will be better if implemented in C, so you can create the initial capability in Python and then nalize the code in C. If you'd like, you can create a directory to hold your image-based work. To do this, perform the following steps: 1. From your home directory, create a directory named imageplay by typing mkdir imageplay while in your home directory. Then, switch from the home directory to the imageplay directory by typing cd imageplay. 2. Once there, let's bring over your camera.py le as a starting point by typing cp /home/ubuntu/examples/python/camera.py camera.py. Now you are going to edit the le until it looks something as shown in the following screenshot:

[ 92 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

Let's look specically at the following changes you need to make to camera.py:
cv.Smooth(img,img,cv.CV_BLUR,3) You are going to use the OpenCV

library rst to smoothen the image, taking out any large deviations.

creates a default image that can hold the hue image you create in the next statement.

hue_img = cv.CreateImage(cv.GetSize(img), 8, 3) This statement

cv.CvtColor(img,hue_img, cv.CV_BGR2HSV) This line creates a new

image that stores the image as per the values of hue (color), saturation, and value (HSV) instead of the red, green, and blue (RGB) pixel values of the original image. Converting to HSV focuses our processing more on the color as opposed to the amount of light hitting it. are going to create yet another image, this time a black and white image that is black for any pixel that is not between two certain color values.

threshold_img = cv.CreateImage(cv.GetSize(hue_img), 8, 1) You

color range. In this case, I have a green ball and I want to detect the color green. For a good tutorial on using hue to specify color, try http://www. tomjewett.com/colors/hsb.html. Also, http://www.shervinemami. info/colorConversion.html includes a program that you can use to determine your values by selecting a specic color. the original image in it.

cv.InRangeS(hue_img, (38,120, 60), (75, 255, 255), threshold_ img) The (38, 160, 60), (75, 255, 255) parameters determine the

cv.ShowImage("Colour Tracking", img) This shows a window with cv.ShowImage("Threshold", threshold_img) This shows a window

with just the threshold image.

[ 93 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

Now run the program. You'll need to either have a display, keyboard, and mouse connected to the board, or you can run it remotely using vncserver. Run the program by typing sudo python ./camera.py. You should see a single black image, but move this window and you will expose the original image window as well. Now take your target (I used my green ball) and move it into the frame. You should see something as shown in the following screenshot:

Notice the white pixels in our threshold image showing where the ball is located. You can add more OpenCV code that gives the actual location of the ball. In our original image le of the ball's location, you can actually draw a rectangle around the ball as an indicator. Edit the camera.py le to look as follows:

[ 94 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

Start by editing just below the line cv.InRangeS(hue_img, (38,120, 60), (75, 255, 255), threshold_img). The lines used are as follows:
storage = cv.CreateMemStorage(0) This line creates some memory for

you to manipulate images in.

image that are within the threshold. There could be more than one, so you may want to capture them all.

contour = cv.FindContours(threshold_img, storage, cv.CV_RETR_ CCOMP, cv.CV_CHAIN_APPROX_SIMPLE) This nds all the areas on your

[ 95 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

points = [] The creates an array for us to hold all the different possible

color points.

while contour: Now add a while loop that will let you step through all the possible contours. By the way, it is important to note that if there is another larger green blob in the background, you will "nd" that location. Just to keep this simple, we'll assume your green ball is unique.

rectangle for each area of color. The rectangle is dened by the corners of a rectangle around the "blob" of color.
contour = contour.h_next() This will prepare you for the next contour,

rect = cv.BoundingRect(list(contour)) This gets a bounding

if one exists.

for the size in pixels of the rectangle.

size = (rect[2] * rect[3]): This calculates the diagonal length of the rectangle you are evaluating. The data structure rect contains four integers; 0 and 1 for the pixel values of the lower-left corner of the box, and 2 and 3

if size > 100: Here you check to see if the area is big enough to be of concern. 100 tells your program to not worry about any rectangles

that are less than 100 pixels in area. You may want to vary this based on the application.

pt1 = (rect[0], rect[1]) Dene a pt1 variable and set its two values to the x and y coordinates of the left side of the blob's rectangular location.

and set its two values to the x and y coordinates of the right side of the blob's rectangular location.
cv.Rectangle(img, pt1, pt2, (38, 160, 60)) Now you add a rectangle to your original image by identifying where it is located.

pt2 = (rect[0] + rect[2], rect[1] + rect[3]) Dene a pt2 variable

Now that the code is ready, you can run it. You should see something as shown in the following screenshot:

[ 96 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Chapter 4

You can now track your object. Now that you have the code, you can modify the color or add more colors. You also have the location of your object, so later you can attempt to follow the object or manipulate it in some way. OpenCV is an amazing, powerful library of functions. You can do all sorts of incredible things with just a few lines of code. Another common feature you may want to add to your projects is motion detection. If you'd like to try, there are several good tutorials; try looking at the following links:
http://derek.simkowiak.net/motion-tracking-with-python/ http://stackoverflow.com/questions/3374828/how-do-i-trackmotion-using-opencv-in-python https://www.youtube.com/watch?v=8QouvYMfmQo https://github.com/RobinDavid/Motion-detection-OpenCV

[ 97 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Adding Vision to Raspberry Pi

Having a webcam connected to your system provides all kinds of complex vision capabilities. You can get 3D vision with OpenCV using two cameras. There are several good places; for example, the code in the samples/cpp directory that came with OpenCV has a sample stereo_match.cpp. For more information, refer to http://code.google.com/p/opencvstereovision/source/checkout.

Summary
As we learned in this chapter, your projects can now speak and see! You can issue commands, and your projects can respond to changes in the physical environment sensed by the webcam. In the next chapter, you will add mobility using motors, servos, and other methods.

[ 98 ]

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book

Where to buy this book

You can buy Raspberry Pi Robotic Projects from the Packt Publishing website: .
Free shipping to the US, UK, Europe and selected Asian countries. For more information, please read our shipping policy.

Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet book retailers.

www.PacktPub.com

For More Information: www.packtpub.com/raspberry-pi-robotic-projects/book