The world's most in-depth and scientific reviews of tech gear

How We Tested Robot Vacuums

Friday November 9, 2018
Some of our fleet  ready to go!
Some of our fleet, ready to go!

To see which robot vacuum is really the best for getting your floors to shine, we conducted extensive research, then bought all of the best 'bots currently on the market today to test head-to-head. We've spent over two years reviewing and testing these products, so have come to love — and hate — all of their various quirks and other odd behaviors that we have seen over the years. We've conducted over 30 different evaluations to score and rank these products, grouping them into six weighted rating metrics. The sections below detail our exact testing and scoring procedures in each test. If you are curious about how specific models performed and which robot vacuum truly reigned supreme, head on over to our comprehensive robot vacuum best-in-class review here.

The Roomba 980 is particularly adept at getting itself out of tight places.
The Roomba 980 is particularly adept at getting itself out of tight places.

Navigation


The most important of our testing metrics, the set of tests that make up Navigation account for 30% of the overall score. While your mind may initially jump to cleaning as the most important aspect of these products, we have found that this is secondary to the robot's navigation and home coverage efficiency. None of these products can match the raw cleaning power of an upright or stick vacuum, but instead excel by providing regular, automated cleans of your home. However, the automated part of that falls apart if your robot can't navigate and constantly requires assistance to be freed.

This robot fearlessly ventures under low furniture to clean up all kinds of messes  even pet hair!
This robot fearlessly ventures under low furniture to clean up all kinds of messes, even pet hair!

First, we looked at how competently each vacuum navigated a room full of furniture. For this test, we made a room full of essentially wireframe furniture that matched the dimensions of standard furniture. This allowed us to more accurately see and photograph the robot's progress as they cleaned.

A selection of our test furniture for the navigation and home coverage tests.
A selection of our test furniture for the navigation and home coverage tests.

We scored each robot on how long it took to navigate the room, whether or not it ever became stuck and required assistance, and if there were any particularly troublesome spots that bogged down the 'bot.

Area rugs can prove difficult for some of the robots.
Area rugs can prove difficult for some of the robots.

Additionally, we also deducted points if we noticed the robot being particularly rough on the furniture.

The Neato Connected backing up into an obstacle. It can be a little rough getting out of confined spaces  and leaves some light scuffs on furniture.
The Neato Connected backing up into an obstacle. It can be a little rough getting out of confined spaces, and leaves some light scuffs on furniture.

Next, we looked at how well each robot docked. We timed each robot from when we pressed the "Home" button to it successfully charging on its base to determine scores, testing each robot when it was in the same room as its dock and when it was in a different room.

The Eufy recharging for its next cleaning session.
The Eufy recharging for its next cleaning session.

For the final two tests of this metric, we evaluated and scored how each robot did at navigating around and over a section of high contrast flooring, as well as how each one did at cleaning under low furniture. We awarded the most points that successfully cleared the transition without triggering any false-positives for edges with its edge detection sensors, fewer points if the 'bots simply avoided the problem area, and the least points if the robot became stuck and required assistance to be freed.



For cleaning under furniture, we looked for robots that could go anywhere they physically fit for awarding the most points.

The D7 can easily clean under lower pieces of furniture  even reaching places that most upright vacuums won't.
The D7 can easily clean under lower pieces of furniture, even reaching places that most upright vacuums won't.

Some robots had exceptionally high profiles or wouldn't venture under furniture that was much higher than they were, causing them to lose points.

The Xiaomi covered an impressive amount of our test room.
The Xiaomi covered an impressive amount of our test room.

Home Coverage


Next, we looked at how efficiently each vacuum can clean your home. This differs from the previous metric, as it judges where each robot actually cleans, compared to its ability to navigate around obstacles and essentially not become stuck. It accounts for 20% of the total score for each robot vacuum.

First — and most important for this metric — we tested how well each vacuum does at cleaning multiple rooms. The majority of the vacuums have mapping capabilities, allowing them to clean multiple rooms at once, returning to their docking station and recharging as necessary.

The systematic robots reliably cleaned much more of the floor than the semi-random models.
The systematic robots reliably cleaned much more of the floor than the semi-random models.

However, there are still a few that bounce around randomly, meaning they can't really reliably clean multiple rooms, causing them to be penalized in this assessment.

Semi-random robots can tend to over-clean areas while missing others.
Semi-random robots can tend to over-clean areas while missing others.

We also checked that each vacuum could clear a standard threshold, which they all could do easily.

Next, we tested how well each vacuum did at cleaning a single room, as well as how effective its spot cleaning abilities are.

The Connected is one of the best at keeping your carpets clean.
The Connected is one of the best at keeping your carpets clean.

Carpet Cleaning


After scoring how well each robot moved throughout your home, we moved on to scoring how each robot did at actually cleaning. We started off by evaluating and scoring how well each robot did at keeping soft floors clean, using a variety of different debris as sample messes, as well as comparing how close each vacuum cleaned to the walls or other edges or a room. This metric is responsible for 15% of the total score.

For all of our debris tests, we spread a consistent amount of the material on the floor, then scored each robot vacuum by comparing the before and after photos. We thoroughly cleaned the carpet in between trials with an upright vacuum, to ensure that there was no residual mess carried over between products.

For the first test, we used flour as our fine debris. We only conducted this test on flat, low-pile carpet, as extracting flour from fluffy carpets is really beyond the current cleaning capabilities of these products.

The POWERbot left a bit more residual flour behind than our top models.
The POWERbot left a bit more residual flour behind than our top models.


After flour, we repeated the test with both rice and oats, for our medium debris, though this time we did the test on both flatter, low-pile carpet and on fluffier, medium-pile carpet.

The Samsung surprised us by the considerable amount of leftover oats that remained after cleaning.
The Samsung surprised us by the considerable amount of leftover oats that remained after cleaning.

Finally, we used the small size of Mini-Wheat for our large debris test, to see if the vacuums had sufficient power to crush them up or had the ability to just suck them up whole. We also did this test on both types of carpet.

For the edge test, we spread coffee grounds along the edges of a robot pen we made, then ran each vacuum.

The 980 left a wider strip of debris around the edges of the pen than the D-shaped robots.
The 980 left a wider strip of debris around the edges of the pen than the D-shaped robots.

We used the coffee grounds for maximum contrast and scored each product on how little residual was left behind.

This robot continued its below average cleaning performance on hard floors.
This robot continued its below average cleaning performance on hard floors.

Hard Surface Cleaning


For our next metric, we repeated the exact same set of tests that we did on carpet, though this time they were all conducted on a section of hardwood laminate flooring. This group of tests is also worth 15% of the total score.

The Samsung R7070 did great at cleaning the surface flour off.
The Samsung R7070 did great at cleaning the surface flour off.

We paid particularly close attention to see if the vacuums flung debris around, rather than cleaning it, as they were much more prone to do this on the smoother surface.

Senior canine tester Chewie seems bemused as the Neato D5 collects the excess fur he shed.
Senior canine tester Chewie seems bemused as the Neato D5 collects the excess fur he shed.

Pet Hair


Next, we scored how well each robot dealt with the extra hair that the furry members of your home leave behind. We visited a local groomer who kindly donated a substantial amount of leftover fur to use as our supply, then spread out a predetermined amount on a section of both fluffy and flat carpet. These tests account for 10% of the total score.

Cleaning the pet hair from the end of the extractor of the Roomba 980.
Cleaning the pet hair from the end of the extractor of the Roomba 980.

We scored each vacuum on how much it picked up, as well as how much actually ended up in the collection bin of each product, rather than tangled up in the main extractor brush or the undercarriage of the robot.

The Neato D7 has some of the most impressive set of smart features that we have seen.
The Neato D7 has some of the most impressive set of smart features that we have seen.

Smart Connect


For the final 10% of the total score, we evaluated how Smart each 'bot is. More and more devices are offering smart connectivity, allowing them to be controlled remotely and interface with systems like Google Home, Amazon's Alexa, or Apple's Siri, and robot vacuums are no different.

We scored each product on its WiFi reliability, the amount of functionality offered through each companion app — if there is one — and how easy it is to integrate each product into various smart home ecosystem. Additionally, we also looked at the ease of setting up a schedule and if there were controls on the robot itself.

Some of our all-time favorites when it comes to robot vacuums.
Some of our all-time favorites when it comes to robot vacuums.

Conclusion


Hopefully, this article has provided you with some more insight into how we tested these products and why they scored what they did in our complete review of robot vacuums. For even more background information on these products, such as why you would even want one and what to look for when shopping for a new one, head on over to our Buying Advice guide for a complete breakdown of the products.