Our team designed a small autonomous vehicle to compete in two robotics competitions: (1) the Autonomous Vehicle Challenge at the 2023 National Robotics Challenge (NRC) and (2) the Intelligent Vehicle Challenge at the 2023 Arkansas Space Grant Consortium (ASGC) Symposium. To earn points in both competitions, the robot needed to autonomously navigate through an obstacle course by driving around five multi-colored bucket waypoints, driving under a small arch, and driving over a ramp. An image of the course layout provided by 2023 NRC contest manual is shown above.
We designed our autonomous system using vision-based supervised machine learning, inspired by the open-source API Donkey Car, which we had great success training with in a pervious project. To teach our robot to navigate, we first manually drive the vehicle around the obstacle course using a wireless controller. The robot records two sets of data while driving: (1) images captured by a mounted camera and (2) the user's driving commands (these are paired with each image as truth label). After driving the car for 10-15 laps, we optimize a Convolutional Neural Network (CNN) using the recorded data. Through an iterative training process, the CNN identifies patterns between what the robot sees in each image and what driving command the user sends to the vehicle. Then, when the CNN receives a new set of image data, it predicts the driving commands for each image. Once the CNN's predictions reach an acceptable level of accuracy, we test the autopilot algorithm live on the course, allowing the trained network to drive the robot autonomously.
This train_and_deploy folder contains all of the software for our autonomous vehicle, including:
- a config.json file that limits the vehicle's maximum throttle and defines the vehicle's steering trim;
- motor.py and servo.py scripts that contain functions for initializing and deploying the vehicle's motor/motor driver and steering servo/PWM board;
- a collect_data.py script that is used to manually drive the vehicle with a wireless controller while collecting steering, throttle, and camera data;
- a train.py script that trains a CNN using PyTorch and generates a .pth autopilot file containing the trained parameters;
- an autopilot.py script that drives the vehicle autonomously using a .pth autopilot file;
- and a cnn_network.py file that defines the neural network architectures accessed by the train.py and autopilot.py scripts.
For more information about the project, see our final_presentation.pdf. And be sure to check out our wiki pages for information about the competition, using the robot, configuring the raspberry pi, and resolving various issues.
After testing several variations of CNN architectures, we had the most success with Donkey Car's fastai architecture (see the "Linear" class). The figure below shows how we modified the CNN structure to accomodate for our input image size and achieve the best results. Our network architecture has five convolution layers and three fully connected layers. Each image in the network has an input size of 3x120x160 (3 color channels, 120 pixel horizontal width, and 160 pixel vertical height) and an output size of 2 prediction values: steering and throttle. When a dataset is loaded, the recorded images are split into training images (90-95% of the data) and test images (10-5%). During the training process, the network uses the Mean Square Error (MSE) loss function, Adam optimizer, a learning rate of 1E-3, and batch sizes of 125 (train) and 125 (test). The neural network iteratively trains for typically 10-20 epochs. We found the most success when using datasets with a size between 15-20k images.
- Analysis of Autopilot Performance at NRC and ASGC
- Successful Autopilot Performance at ASGC Challenge
We built an autonomous ground vehicle using a modified RC car, a Raspberry Pi, and a vision-based neural network. We can connect a bluetooth joystick with the car to engage the throttle, adjust the steering, and record data for neural network training: camera images and user joystick input. After collecting driving data on a course, we can train the parameters of a convolutional neural network to reduce the loss between predictions and ground truth values. Finally, we can import the trained parameters into a neural network autopilot file that autonomously controls the steering and throttle of the vehicle when live camera images are received. The autopilot performs well when operating indoors or under a shaded area outside. However, when driving outside in the sun, the performance accuracy decreased dramatically.
One area of investigation that still needs work is determining the best method for training the neural network. Our most successful tests have been data sets collected continuously, usually with between 15 thousand and 20 thousand images. We would like to see this project taken to the next level, with a neural network model that is general enough to operate under changing conditions (background, weather, time of day) without additional data collection. We tried combining data collected on the course under these different circumstances and training a model, but they performed very poorly. New sensors could potentially help with this. There have been similar projects that use additional information like stereo depth vision and absolute GPS position data based on the start point to make more general models.
While we found a neural network architecture that produced functional models, it was very picky about the weather conditions. We only had successful models when the robot was out of sunlight, and the surroundings were not too bright. This could have been due to the camera itself creating image artifacting from the light, or it could have been because of the model putting too much weight on the brightness of colors. Either way, we recommend trying other neural network architectures to see if there are any that perform better than Donkey Car's fastai architecture.
- National Robotics Challenge
- Arkansas Space Grant Consortium
- Donkey Car API (inspired this project)
- Donkey Car CNN architecture: fastai
We completed this design project during our senior year at the University of Central Arkansas. In May 2023, we each graduated with a Bachelor's of Science in Engineering Physics.
- Dr. Lin Zhang (Thank you for your constant support and enthusiasm!)