Test Preparation – Field Testing Series – Part 3

This post is part of a series on testing robots with a focus on field testing. It will start with why we test robots, work its way through planning for a field test, to performing the test, and finally what to do after the test. We will conclude with a description of many common tests that we subject our robots to and how to perform them.
I hope you enjoy this series and please leave comments below.
Links:
Why Test Robots? – Field Testing Series – Part 1
Designing for Success – Field Testing Series – Part 2
Test Preparation – Field Testing Series – Part 3
Test Execution – Field Testing Series – Part 4
Test Analysis – Field Testing Series – Part 5
Common Robot Tests – Field Testing Series – Part 6
—————————————————–

Planning for a field test requires a lot of work. In an interview with Brian Wilcox he said “don’t underestimate pre-planning” and “do thought experiments of what to do while looking for Oh-Shits”. Doing this work in advance can mean the difference between a successful test and a failed test. One of the first steps in planning a field test in deciding what are your goals. These goals should be aligned with the system requirements document. By using the systems requirement document you can prioritize the different tests based on time and resources. Once the text objective is determined a test plan can be created that highlights the test and details for performing and analyzing the test. As the test plan is further developed people need to monitor for scope creep (as people decided to add more things to the test) and watch out for the number of tests increasing out of control. For example to test a robot three times for statistical significance in two sites with four different configuration easily yields twenty-four tests. This is assuming that all of the tests are valid and that none need to be repeated. If you decide to do those tests while adjusting another parameter between two values you have now just doubled your tests to forty-eight. It is easy to see how this can quickly get out of control. Test designers should think about statistical significance when designing tests, but also remember the logistics and time that it takes for each test. Many times in robotics people will try to reduce the number of tests by reducing the number of times a specific test is performed. Doing this is sometimes useful for initial qualitative and quantitative testing however as a system matures it needs to have statistical significance in order to prove that it performs as expected. It is not just me making these claims, when I asked Brian about how to improve the probability of success for a field test he said make sure to have clear objectives, success criteria, a scientific methodology, and a test plan.

After determining what tests need to be performed the next step is finding an appropriate field site. Doing a feasibility study can help make sure that the site meets the needs of the test and that the limitations are known in advance. When performing this site survey in addition to if the site will work for the desired tests other factors need to be considered. Some of these items include weather conditions, travel distance and availability of lodging, are food sources close and convenient or do they all close for the night before you finish testing for the day, site access including roads, permits, locals, public access, and security, site power, internet connectivity, and transportation. Dr. Rob Ambrose a Principal Investigator at NASA always asks how close to home depot is the test site, this is an example of an important question when finding a test site, the availability for supplies that you need can be critical and make or break a field test. Many times there are local sites that can meet the testing needs of a robot with a little creativity.

Once the robot is deemed ready for the field test it is important to go through all the test scenarios to make sure that the robot is ready to perform the task and so that people can get experience and work out any kinks in the test procedure. Having this formal review lets the team know what aspects of the robot are not mature and where they can improve the robots performance to increase the likelihood of successful testing. Debugging and fixing a problem is usually easier and more comfortable to do in the lab environments then in the field environment. If something is not working you should not assume that it will magically work in the field. On one field test we brought multiple cameras, lasers, and ultrasonic sensors to the field without having fully tested the software. We spent the entire field trip just trying to make things work properly. Our solution was to operate for over a week on minimal sleep and extend the trip several days. Even with this solution we just got the minimum set of data that we required and were not able to get some of the sensors to work optimally. Be honest with yourself to determine if the robot is ready for a field test. The first time you operate the robot should not be in a remote field site, you should test the robot locally to determine if the desired functionality has been achieved.

At this point a logistics plan needs to be in place. This is both part of the test plan and a sanity check on the test plan. Step through each test plan to make sure that it will work. Pay attention to things such as; Who is going to carry the generator? Will our chase vehicles also be able to follow the rover or does difficult terrain lay in its path? Making sure you have a way to source all the needed items is also important. How will you get all of the items to the field site? How will you pick them up after the field test? Is there clean drinking water? How should everyone be dressed? All these items can be taken for granted but they should not be. If you are operating in a remote site this step becomes even more critical. Make sure to have printed maps and instructions and not rely on GPS or cell phones that do not always have the best service in these remote sites. Before leaving for a field site it is good to be familiar with your surrounding and have directions to common places such as hospitals, repair centers, and general stores.

Being ready for the field test includes being prepared for events that can happen in the field. To this extent it is important to go through the various robotic subsystems (electrical, mechanical, etc..) and bring relevant spare parts as well as the tools required to fix and maintain the robot. I have seen many cases where a component that was operating flawlessly for many months suddenly stops performing in the field. Some of these cases were as simple as not having a spare fuse or not having the appropriate allen wrench while others were more subtle such as having the log generate more directories than allowed by the operating system that was running the robot. Having spares and tools divided into what you regularly need and what you hope-not-to-need can be useful. It is also important to have copies of the datasheets for the various components. Something that is often overlooked is environmental protection (rain, snow, dust, bugs, etc..). Having the appropriate tarps, covers, etc.. readily available is always of importance.

When packing for the trip using an organized fashion can increase productivity in the field. Knowing where things can be found is a powerful tool when operating in the field with limited resources. This can also be a safety concern. For example knowing where the first aid kit is and/or fire extinguisher. One approach is to put a tag on each item (or box) that labels what it is and have a master list of all items; for example a pallet can have a paper with what it contains and each box within the pallet can be labeled and tagged with its contents. Having a single person that overseas packing can also help ensure that things are organized and items are in their proper place. There are many ways to pack successfully, which ever way you choose to do this make sure that you have a complete manifest and an easy way of finding each item.

One of the challenges that arises during and after testing is how to represent and log data. Proper test design requires a plan that is both executable and will allow for direct correlation between the various elements of interest. It is important to establish a log naming convention before the test begins and to make sure that everybody is aware of it. This naming convention should be simple to use, simple to understand, and intuitive. I have seen many cases where log file names seem to make complete sense in the field but upon returning from the field it is hard to identify which log corresponds to which event. Further trying to use those logs several years after the test becomes next to impossible. Many people will have a secondary document that lists what test corresponds to what log number. Those documents can get separated from logs and often lost when trying to examine data in the future, it also relies on a human to enter that information into this secondary document. When naming log files I recommend a convention similar to <Date>_<test ID>_<Rover Name>_<Site Name>_<Test Objective>. For example a log name can be 20130621_001_mySuperRobot_myFavoriteTestSite_slope_slimbing_25degrees_attempt1. The ability to add comments into log files also can prove useful when trying to interpret what happened within a log file. Data needs be synchronized when stored in log files. This can be difficult if things are being logged across multiple systems/computers. In the worst case make sure that the system clocks are at least aligned with each other. In ideal cases there is a hardware method of synchronization employed between all devices such as a pulse per second bus or a common clock. While discussing data collection it is also important to determine how to ground truth the data. In order to ground truth and verify your data you should have an external way of measuring things. Some examples can be GPS or a Leica Total Station.

Safety is paramount. When dealing with robotic systems it is important to think about the safety of operators, bystanders, and the robot. Safety precautions must be developed to prevent injuries and death. In current generation systems it is common to put a blinking light and siren on large and potentially dangerous robots. In new and future systems where robots are more integrated with people these safeguards might disappear. When a robot is being tested it is important to keep unnecessary people away and build appropriate safeguards into the robot. E-stops and m-stops (e-stops and m-stops terminology are often used interchangeably) are also important. They should be engaged whenever a person is working on a robot and the robot should not move. Depending on the type of robot determines how the e-stops are used. On robots that might have human operators inside having an e-stop inside the robot is necessary. On fast robots users might not have time or be able to press an e-stop so other safety systems are needed. Some examples of that include laser curtains that have a control really to cut power or wireless e-stops. It is important to verify that e-stops work as intended and still work when other systems fail. In the case of wireless e-stops it should have an auxiliary power source and functionally be isolated from the rest of the system so it still works if other systems fail.

Basic safety should also not be ignored having a stocked first aid kit and people with knowledge of basic first aid is important. As people empty the first aid kit make sure to refill it. There are also items that are specific to a location. For example you might need to carry emergency blankets, medications, flares, radios, fire extinguishers, gas alarms, laser safety, masks, spare water, spare food, etc… Keeping members of the test team alert and healthy is also important. Testing should be planed so that team members are well rested, have water, and decent food. People should stay in groups so that if someone needs help there is always someone there that can help. Having four people in a group is good whenever possible. With four people in a group if someone gets hurt one person can stay and help while the other two people can go for help.

If multiple groups are involved in the field test it is important to organize among the groups to determine which tests are going to happen where and when. Often various groups will have different expectations. If all of the groups are participating in the test and are testing different systems all of the groups should know how to stop other systems if they are out of control. Further every group should know where the other groups are operating.

There are many examples of missions failing due to poor preparation. The Vanguard Rocket was supposed to the USA’s first launch vehicle. However in order to beat the Soviets the launch was pushed ahead without proper design and the result was that out of 11 launches only 3 succeeded. Another example where a lack of pre-planning caused disaster was with the Corporal rocket/missile. There were many failures that were tracked to poor training, not having telemetry from the rocket, no documentation, no operations training, and no coordinator between the different groups involved. All of those problems could have been resolved with a bit of pre-planning. It is better to spend the time and plan thoroughly then to be forced to explain to a review board some colossal failure, injury, or death. Having people communicate concern and have people listen is critical while preparing for a test. Once you are ready for a field test a pre-mortem should be conducted to discuss what can fail and what the fix is.(This paragraph is based on a talk from Astronaut Dr. Jay Apt.)

The next post is all about executing the test that we just prepared for. Stay tuned.

As always comments are welcome (and encouraged)

Image Source: http://frc.ri.cmu.edu/atacama/imageGalleries/2013/Field/IMG_2742-large.jpg

Liked it? Take a second to support David Kohanbash on Patreon!