Perception in Smoke, Dust or Fog

I recently had the fortune to attend a talk titled Multi-Modal Data for Perception in Smoke-Filled Underground Mines by Joe Bartels. It was an interesting talk about pushing sensor technology into difficult environments that can have smoke, dust or fog . The following are notes that I took during the talk, as always I add a little of my own content to the post. I want to thank Joe for giving the talk and sharing the images that are used below.

LIDAR

Standard LIDAR often fails in smoke, however newer multi echo return LIDAR’s do better.

Standard LIDAR operation showing initial pulse and the returned pulse

LIDAR operation in smoke showing initial pulse and the multiple returned pulse. Center pulse is the highest power return. Last pulse is the final, pulse but often not strong enough to detect.

Point cloud showing LIDAR initial returns and final returns. Note that the initial returns are just a cloud in the center around where the robot drove from the smoke. The final return represents the real structure of the walls, the resolution and quality of the walls is less than if there was no smoke. (Generated with the Hokuyo UTM-30LX-EW)

Sonar & Radar

Sonar and Radar works better penetrating the smoke, but lacks the resolution of LIDAR. This can be a problem for creating detailed maps.
Sonar also suffers from a lot of multipath (making RADAR usually better in fog)

Cameras

This work was focused in mines where external lighting is required for visual cameras, as opposed to LIDAR/Radar/Sonar which do not need illumination to operate.

Cameras generally do very bad since all of the photons are scattering, and the scatter from different lighting produces scatter everywhere. (See Thermal Imagery image below)

Lighting

Typical fog lights are angled downwards at the road to minimize light scattering and minimize having the light reflecting back into the camera. Also the lights should be placed low so that they are far away from the sensor camera.

Road as viewed with different car lighting methods.

The downward light helps illuminate a path on the ground, but is not high enough for seeing the environment and building a map (such as for SLAM or data purposes).

Scattered light is not on the epi-polar lines which interferes with the principles of epi-polar geometry and stereo reconstruction.
You can add a line filter to the camera which allows only a single line of light at a time into the camera sensor. This prevents the scattered light from entering the camera sensor. The downside to this is the reduced data returned from the camera.

Thermal Cameras

• Thermal cameras often work well and see through smoke. They are based on emissivity (energy emitted from an object) as opposed to the light reflecting from the object (such as with a typical camera).
• Emissivity of 0 would be a mirror, while an emissivity of 1 would be a true blackbody.
• A problem is that the differential response is small in non man-made materials.
• 2 Types of sensors: short wave & long wave. The short wave sensors are more expensive, have better resolution, but less visibility through smoke.
• Lacks the sharp features for many mapping applications or scan matching algorithms.
• You need to be careful since lights and people can appear to be similar temperatures
• Wet surfaces can be confusing to interpret

Same image as shown from a visual camera and a thermal camera

Calibrating Thermal Cameras

People often use a standard calibration target however this is far from ideal
The next step that people often try is to artificially heat/cool a target, however it is hard to make the target a constant temperature and the temperature changes are short-lived. When you do this the black will heat up quickly. When viewed on a thermal camera the checkerboard colors get inverted.
A better way to calibrate thermal cameras is to make a target of dissimilar materials (multi-modal) that have different thermal properties.

Standard calibration target viewed from visual and thermal cameras

Standard calibration target that is heated viewed from visual and thermal cameras

Calibration target made from a white surface with black circles attached of a different material

“Smoke” Generation

Different types of smoke/fire produce different perception challenges.
Smoke Candles – White smoke that hangs around. It goes to the ceiling and then slowly drops down.
Wood Fire – Lighter color smoke that hangs near the top of the room. Carbon particles are suspended in the air.
Diesel Fire – Dark smoke that sticks to the ceiling and is very persistent. Carbon particles are suspended in the air.
Dust – Harder than smoke since the particles are bigger, however the particle suspension time in the air is shorter. Radar tends to be a good choice in dusty environments.
Fog – Fog is based on small water droplets suspended in the air. Fog tends to stay lower in the space and not hug the ceiling. Particularly cold fog stays near the ground, as it heats up it rises before dissipating. This can be simulated by putting dry ice into hot water.

Warnings

From personal (observed) experience, I need to warn you that if you choose to generate smoke in an indoor environment you need to be careful about alarming others and potentially setting off smoke alarms. Smoke has a way of seeping through cracks and being observed in places that you did not expect. Also if you think a smoke candle is a dud, do not throw it in the dumpster right away, wait a little in case it starts smoking and somebody calls the fire department.

Also as with everything else we do, be careful, and do not do dumb things.

I hope you found the information above interesting, I know I did.
Have you worked in smoke? Leave it in the comments below.

Liked it? Take a second to support David Kohanbash on Patreon!