Why Test Robots? – Field Testing Series – Part 1

This post is part of a series on testing robots with a focus on field testing. It will start with why we test robots, work its way through planning for a field test, to performing the test, and finally what to do after the test. We will conclude with a description of many common tests that we subject our robots to and how to perform them.
I hope you enjoy this series and please leave comments below.
Links:
Why Test Robots? – Field Testing Series – Part 1
Designing for Success – Field Testing Series – Part 2
Test Preparation – Field Testing Series – Part 3
Test Execution – Field Testing Series – Part 4
Test Analysis – Field Testing Series – Part 5
Common Robot Tests – Field Testing Series – Part 6
—————————————————–

Why do we test robots? Specifically how do we test field robots? These questions form the basis for developing reliable systems in order to remove robots from the domain of the engineers and into the domain of users. When designing for reliability there are design considerations and then there is the testing to validate the design decisions, prove system functionality , and gain confidence that a system will perform as expected. Testing also allows scientists and engineers to learn the weaknesses, problems, and constraints of the robot and improve upon them.

When designing a motor all you need to test is that the rotor spins (I know I am oversimplifying). However testing robotic systems differs from many other applications. Robots are a complex blend of hardware and software. Robots often have many degrees of motion, moving components, dynamic affects, multiple sensors and need to operate in harsh conditions. Robots are expected to work in extreme and inhospitable environments where they can be exposed to temperature extremes, shocks, radiation, and people. These aspects as well as others makes testing robotic systems very complex.

Field robots are a particularly difficult class of robots to test and requires specific testing that differs from many other applications. Field robots are machines that operate outside of the carefully constructed factories and often work in harsh, unstructured and changing environments. This requires that robots be tested in many different scenarios as the possible robot workspace is very large and can almost be infinite. These systems tend to be complex and offer many opportunities for things to go wrong. Only through careful testing can the reliability of the robot be ascertained and obtained. Another unique aspect of field robotics is the safety concerns. Many field robots can be large and/or moving quickly which can be a lethal combination to any human in the vicinity.

Another common argument against field testing and often a sign of someone who has not worked significantly with real hardware is that simulation can be used. Simulation is great during design and for initial testing however there are many flaws that will not be found in simulation and can only be found through actual testing in applicable conditions. Many theorist prefer simulation due to the ease of implementing a new algorithm and the hardship of testing, however if you ask anyone who has extensive experience building systems they will tell you that things seldom work perfectly the first time in a field test and as a result of the real testing they discovered new details of their robot and areas for improvement. These discoveries can be negative such as things that need to be fixed or changed, or they can be positive such as the robot operated better than expected or by tweaking something an entirely new method can be used to achieve the desired outcome. In the DARPA Grand Challenges for building autonomous cars there were several teams (that I know of and probably more) that had tested their cars and everything worked well. However during the evaluations the cars would start exhibiting weird behaviors after a while. Many of these were traced to things that only become apparent after running for an extended time such as memory leaks. Remember the old adage “test as you fly, fly as you test”.

Brian Wilcox a principal member of NASA JPL (Jet Propulsion Labs) said that there are technical and programmatic reasons for field testing and relying on lab testing and simulation alone. Field testing gets you “truly natural terrain with bigger scale” his example was when you are planning for an asteroid mission the only way to get that type of scale on realistic and natural terrain is from doing field tests. Brian Wilcox also remarked that “simulations are doomed to succeed since they are made on your assumptions”. In a recent talk I attended by Astronaut Dr. Jay Apt he stressed the importance of testing. Dr. Apt brought many examples of why testing is a requirement and in his words “testing must be a requirement” since people often make conceptual errors, and those assumptions must also be tested. Dr. Apt continued that we must test since we can not think of everything, due to “failure of imagination to see complications”. One of the examples we brought up was a Mars Orbiter; in this case mission failure was because of a failure to estimate certain forces, and some of the bolts being installed backwards.So it is not just me calling for increased testing, I also have an astronaut on my side.

This is the first part in my series for testing robots. Stay tuned for more information on designing for testing, running tests, finding sites, analyzing the results and common robot tests.

As always comments are welcome (and encouraged)

Image Source: http://www.nasa.gov/centers/ames/research/technology-onepagers/fieldRobotics.html

Liked it? Take a second to support David Kohanbash on Patreon!