LIDAR (Laser Scanner) Review Comparisons

by on May 15, 2014

LIDAR's Used

This is part of a series of posts talking about some of the common LIDAR’s used in robotics. At the end of these sensor profile articles there will be a final post that compares the sensors based on data collected by each of the sensors.

Posts:
Evaluating LIDAR Guide
LIDAR Fundamentals
SICK TiM551 – LIDAR Overview
SICK LMS111 – LIDAR Overview
SICK LMS511 – LIDAR Overview
Hokuyo UTM-30LX – LIDAR Overview
LIDAR Review Comparisons

I hope this series (and particularly the final comparison post) proves useful.
Enjoy!


Recommendations

Here are my high level recommendations. Obviously other things like power, mass, size, and ruggedness need to play into your choice of LIDAR’s.

TiM551-2050001 ($2000) – I would highly recommend this sensor for applications that need obstacle avoidance/detection at distances of less than 8 meters. However due to the 1 degree resolution and ±60mm range accuracy this sensor is not ideal for mapping applications.

LMS111-10100 ($5700) – I would recommend this sensor for applications that need obstacle avoidance/detection or mapping at distances of less than 18 meters.

UTM-30LX ($5590) – I would recommend this sensor for applications that need obstacle avoidance/detection at a range from 18 meters to 30 meters. This sensor is not the best for mapping; while it has a resolution of 0.25 degrees the range tends to be a little noisy and have an accuracy of ±50mm.

LMS511-10100 ($7000) – I would recommend this sensor for applications that need obstacle avoidance/detection at a range from 30 meters to 40 meters (80 if really needed). I would also recommend this sensor for the most detailed mapping work of this group. You might need to do some basic filtering of the data.

Object Detection

This test was designed to show the range response in different sensors when exposed to the same objects. You can clearly see how it is easier to pick out items with some of the sensors over the other ones. I should also point out that these are zoomed in a bit and in practice for obstacle avoidance probably would not matter. For mapping this can have a large impact.

LIDAR test objects

TiM551 Point Cloud
With the SICK TiM551 it was pretty difficult to “see” which points correlated with which item. Of note the glass was “invisible” to the sensor, and it had no problem detecting the mirror.

LMS111 Point Cloud
The SICK LMS111 was pretty clear and I had no problem detecting which points were for which objects. You could make a decent map from this.

Hokuyo Point Cloud
The Hokuyo UTM-30LX had a nice result with its 0.25 degree resolution.

LMS511 Point Cloud
The SICK LMS-511 was also very easy for me to determine what I was seeing in the point cloud. This sensor is good for mapping. If you are doing short-range obstacle avoidance then this sensor is probably overkill.

LIDAR test objects

Environmental Tests

Pointed at the Sun Test

All of the lasers performed well in a sunlit outdoor environment. When pointed directly at the sun that portion of the data was reported as invalid and the rest of the sensor data (for example from the sides) seemed to be good.

Medium Rain Test

Rain Satellite Data
During this test there was a heavy drizzle (or medium rain) as can sort of be seen in the satellite image above.

In this test I had the sensor under my roof and the rain started to fall 1m in front of the sensor. There was a silver car (2009 Honda Accord) parked 5.8m in front of the sensor that acted as my target for this.

All of the sensors performed well. There were two observed phenomena. The first is random points when the laser beam hits a water drop. The second is the surfaces that I detected was not as clean and appeared to have more noise in the rain. Here is my summary of these effects:

Sensor Random Points Noisy Surfaces
SICK TiM551 Few random points. This could be because it only has 1 degree resolution compared to the higher resolution in the other sensors. The surface jumped the least with this sensor. Only around 1cm.
SICK LMS111 Very few random points. Almost as good as the TiM551. There was about 8cm of movement in the points on the surface of the car.
Hokuyo UTM-30LX A lot of spurious points from the rain. About 2cm of movement in the points on the car bumper.
SICK LMS511 This is kind of tricky. The LMS511 was the noisiest of the sensors I evaluated which I think might be due to how I used the multiple returns. With that said I did not notice any more random points then normal when operating in the rain. The points on the car jumped around by about 4cm.

Puddle Testing

For this test I filled a ceramic flower-pot with about 2cm of water to the top of the flower-pot.
I am not sure if it is better to see the top of the water or the surface below the water. I think for a mobile robot I would rather see the bottom of the water so that it does not become an obstacle. In some early LIDAR’s I used the sensors would report incorrectly when exposed to puddles and would make our autonomy software think that the puddle was an obstacle.

Puddle
TiM551 Puddle Point Cloud
LMS111 Puddle Point Cloud
Hokuyo Puddle Point Cloud
LMS511 Puddle Point Cloud

All three of the SICK sensors did a good job detecting the bottom of the container and saw right through the water. The Hokuyo had this weird 20-30cm variation where the 2cm of water was.

Laser Accuracy

sensor evaluation

Summary of various sensor performance from this paper

Spot Size Comparison

This was a bit of a difficult test for me since I do not have a good camera that does not have an IR filter (and I did not want to break one of my cameras). So to do this test I used a Sony Handycam DCR-TRV38 in nightshot mode. This was good as it could see the laser points, however I could not adjust image settings so the image quality is bad in some cases (sorry about that).

When I did this test I setup the laser about 100 inches (254cm) from a white poster. On the white poster I taped a US Quarter (0.955 inch / 2.4cm) to help with size comparisons.

TiM551 IR points

SICK TiM551: The laser points are diagonal from the middle left to the bottom right. It is about half as tall as the quarter.


LMS111 IR points

SICK LMS111: Also really hard to see. The points are a little bigger than a quarter.


LMS111 IR points overlapping

SICK LMS111: While hard to see it is interesting to see the overlap between the points.


Hokuyo IR spot size

Hokuyo UTM-30LX: This one is nice and easy to see. The laser points are nice and small. I was pleased to see this since the URG sensor they make is really narrow but also really tall (I looked at this one in the past for other things).


LMS511 IR points

SICK LMS511: Points are about the same height as a quarter.


I would like to thank Hokuyo/Kirinsonfor lending me the UTM-30LX for this review and SICK for the other LIDARs.

More information about the SICK product line can be found here.

More information about the Hokuyo UTM-30LX can be found here.

Disclaimer: I am only borrowing these sensors and will return it when I am done with this evaluation. I have received no money or goods from the company in exchange for this review.

Liked it? Take a second to support David Kohanbash on Patreon!

Comments

Hi
I am assuming you are above water. I think you will really want to test your sensor with how it handles water/sun. Particularly I would be concerned with the sun reflecting off from the water.

You might need to add some filtering in software to handle those cases.

Leave a Reply