Image Processing for Machine Vision
Teaching Robots to See
Unmanned aerial vehicles, known as drones, now buzz around on missions in fields like security, search and rescue, cargo transportation, and surveying. In 2016, the first delivery drone dropped off a pizza in New Zealand. It had a human team and a pilot controlling it. A year later, a Prime Air drone delivered sunscreen to attendees at an Amazon conference. [1] (And as a pale redhead, I can’t wait to live in a world where sunscreen falls from the sky whenever I need it!)
Unmanned Flight
These drones were flying under human guidance or in controlled settings. Newer programs, like Amazon’s Prime Air and Alphabet’s Wing, use unmanned systems, which as the name suggests, can fly themselves to their programmed destination, which is a much more involved task. [2] Murphy’s law dictates that anything that can go wrong on the flight will, and the UAV will have to avoid disaster on its own.
In that case, why use images to direct flight? Why fly autonomously at all? UAVs’ new applications place more demands on their flight ability. They are being used to fly through and view disaster zones with limited connection to their operator, deliver cargo in crowded residential areas which have more obstacles than a controlled test area, and map large areas cooperatively, which requires taking, storing and transmitting large numbers of images. The UAV can pick its flight direction using GPS locating, but reacting to sudden obstacles requires it to be able to analyze its surroundings on the fly. [3]
The UAV System
A typical UAV system consists of the UAV, any other UAVs which are in its group, and its Ground Control Station, or GCS. Generally, the UAV has cameras on the underside of the main body, which are aimed forwards for navigation purposes or down for target location and surveying purposes. The UAV’s flight path is directed by its operator via the Ground Control Station, or GCS, which it communicates with via WiFi. The onboard computer contains the image processing algorithms, flight control software, cameras, and the communication interface to the ground control station. The ground station has a computer for interacting with the onboard system, communication interface, and data storage for the collected images. The overall goal is to create a system that can react to changing situations on its own, using imaging sensors and onboard processing to allow the robot to analyze its current environment and react appropriately. [4]
Machine Vision
Since the drones are directing themselves, they often have to analyze their surroundings with pictures and change their course of action. A UAV’s images may be low quality, noisy, or oriented differently compared to the other images it’s taken, so each image must be processed prior to being analyzed. These algorithms have to be adjusted to work with the UAV’s limited computing power. UAVs then use the corrected image for navigation, obstacle avoidance, and target tracking, or to add to a survey of the area. [5]
The UAV’s understanding of its environment relies heavily on its vision and using the images for decision-making requires several subfields of machine vision. These are similarly, and confusingly, named, but the diagram below shows how the data moves from one step to the next.
Image Processing, the first step, changes an image by improving its quality, preparing it for further processing, or converting it to another form. In the next step, machine learning takes available data and returns information about the current situation. Our robot would do this using computer vision to extracts data from its images. This data is fed into machine vision algorithms, which makes decisions about the environment. Finally, robot vision, the overall goal, takes images and creates physical actions in response to the robot’s current position. [6]
UAVs can take many hundreds of images on each flight, each of which will have different orientations as the UAV turns, noise due to the compression and transfer to the Ground Control Station, and variances due to the UAV’s motion and vibration. After the images have been processed into a meaningful format, other processes extract the notable features and locations that allow the UAV to locate targets, determine its own position, or control its flight path. Finally, the needed images must be compressed to be sent to the GCS, while maintaining the quality needed for the UAV’s current task. All of this has to be done quickly enough for the UAV to react to changes and fly around obstacles.
What’s next?
Fortunately, there are many algorithms, or processes, designed to process images for use in machine vision. Unfortunately for the UAVs, these algorithms were developed for stationary full size computers, and most are either not fast enough for real-time use or require more processing power than the UAV can carry.
Improving the on-board image processing algorithms will make the UAVs’ flights safer and their imaging more reliable and useful, but that will be another article.
Citations
[1] Intelligence, I. (2022, January 6). Why Amazon, ups and even Domino’s is investing in drone delivery services. Business Insider. Retrieved March 17, 2022, from https://www.businessinsider.com/drone-delivery-services
[2] Rubin, B. F. (2017, March 24). Amazon Prime Air Drone completes its first US public delivery. CNET. Retrieved March 17, 2022, from https://www.cnet.com/tech/computing/amazon-prime-air-first-drone-delivery-us-public-sunscreen-mars/
[3] S. Ehsan and K. McDonald-Maier, “On-Board Vision Processing For Small UAVs: Time to Rethink Strategy,” Conference on Adaptive Hardware and Systems, 2009.
[4] UAV system B. Ludington, E. Johnson, and G. Vachtsevanos, “Augmenting UAV autonomy,” IEEE Robotics & Automation Magazine, 2006.
[5] A. Ferrick, J. Fish, E. Venator, and G. Lee, “UAV obstacle avoidance using image processing techniques,” Technologies for Practical Robot Applications, 2012.
[6] A. Owen-Hill, Robot Vision vs Computer Vision: What’s the Difference? [Online]. Available: https://blog.robotiq.com/robot-vision-vs-computer-vision-whats-the-difference.
