How We Tested Robot Vacuums

David Wise and Austin Palmer

Last Updated:
June 13, 2018


To see which robot vacuum is really the best for getting your floors to shine, we conducted extensive research, then bought all of the best 'bots currently on the market today to test head-to-head. We've spent over two years reviewing and testing these products, so have come to love — and hate — all of their various quirks and other odd behaviors that we have seen over the years. We've conducted over 30 different evaluations to score and rank these products, grouping them into six weighted rating metrics. The sections below detail our exact testing and scoring procedures in each test. If you are curious about how specific models performed and which robot vacuum truly reigned supreme, head on over to our comprehensive robot vacuum best-in-class review here.


The most important of our testing metrics, the set of tests that make up Navigation account for 30% of the overall score. While your mind may initially jump to cleaning as the most important aspect of these products, we have found that this is secondary to the robot's navigation and home coverage efficiency. None of these products can match the raw cleaning power of an upright or stick vacuum, but instead excel by providing regular, automated cleans of your home. However, the automated part of that falls apart if your robot can't navigate and constantly requires assistance to be freed.

First, we looked at how competently each vacuum navigated a room full of furniture. For this test, we made a room full of essentially wireframe furniture that matched the dimensions of standard furniture. This allowed us to more accurately see and photograph the robot's progress as they cleaned.

We scored each robot on how long it took to navigate the room, whether or not it ever became stuck and required assistance, and if there were any particularly troublesome spots that bogged down the 'bot.

Additionally, we also deducted points if we noticed the robot being particularly rough on the furniture.

The Neato Connected backing up into an obstacle. It can be a little rough getting out of confined spaces  and leaves some light scuffs on furniture.
The Neato Connected backing up into an obstacle. It can be a little rough getting out of confined spaces, and leaves some light scuffs on furniture.

Next, we looked at how well each robot docked. We timed each robot from when we pressed the "Home" button to it successfully charging on its base to determine scores, testing each robot when it was in the same room as its dock and when it was in a different room.

For the final two tests of this metric, we evaluated and scored how each robot did at navigating around and over a section of high contrast flooring, as well as how each one did at cleaning under low furniture. We awarded the most points that successfully cleared the transition without triggering any false-positives for edges with its edge detection sensors, fewer points if the 'bots simply avoided the problem area, and the least points if the robot became stuck and required assistance to be freed.

For cleaning under furniture, we looked for robots that could go anywhere they physically fit for awarding the most points.

Some robots had exceptionally high profiles or wouldn't venture under furniture that was much higher than they were, causing them to lose points.

Home Coverage

Next, we looked at how efficiently each vacuum can clean your home. This differs from the previous metric, as it judges where each robot actually cleans, compared to its ability to navigate around obstacles and essentially not become stuck. It accounts for 20% of the total score for each robot vacuum.

First — and most important for this metric — we tested how well each vacuum does at cleaning multiple rooms. The majority of the vacuums have mapping capabilities, allowing them to clean multiple rooms at once, returning to their docking station and recharging as necessary.

However, there are still a few that bounce around randomly, meaning they can't really reliably clean multiple rooms, causing them to be penalized in this assessment.

We also checked that each vacuum could clear a standard threshold, which they all could do easily.

Next, we tested how well each vacuum did at cleaning a single room, as well as how effective its spot cleaning abilities are.

Carpet Cleaning

After scoring how well each robot moved throughout your home, we moved on to scoring how each robot did at actually cleaning. We started off by evaluating and scoring how well each robot did at keeping soft floors clean, using a variety of different debris as sample messes, as well as comparing how close each vacuum cleaned to the walls or other edges or a room. This metric is responsible for 15% of the total score.

For all of our debris tests, we spread a consistent amount of the material on the floor, then scored each robot vacuum by comparing the before and after photos. We thoroughly cleaned the carpet in between trials with an upright vacuum, to ensure that there was no residual mess carried over between products.

For the first test, we used flour as our fine debris. We only conducted this test on flat, low-pile carpet, as extracting flour from fluffy carpets is really beyond the current cleaning capabilities of these products.

After flour, we repeated the test with both rice and oats, for our medium debris, though this time we did the test on both flatter, low-pile carpet and on fluffier, medium-pile carpet.

Finally, we used the small size of Mini-Wheat for our large debris test, to see if the vacuums had sufficient power to crush them up or had the ability to just suck them up whole. We also did this test on both types of carpet.

For the edge test, we spread coffee grounds along the edges of a robot pen we made, then ran each vacuum.

We used the coffee grounds for maximum contrast and scored each product on how little residual was left behind.

Hard Surface Cleaning

For our next metric, we repeated the exact same set of tests that we did on carpet, though this time they were all conducted on a section of hardwood laminate flooring. This group of tests is also worth 15% of the total score.

We paid particularly close attention to see if the vacuums flung debris around, rather than cleaning it, as they were much more prone to do this on the smoother surface.

Senior canine tester Chewie seems bemused as the Neato D5 collects the excess fur he shed.
Senior canine tester Chewie seems bemused as the Neato D5 collects the excess fur he shed.

Pet Hair

Next, we scored how well each robot dealt with the extra hair that the furry members of your home leave behind. We visited a local groomer who kindly donated a substantial amount of leftover fur to use as our supply, then spread out a predetermined amount on a section of both fluffy and flat carpet. These tests account for 10% of the total score.

Cleaning the pet hair from the end of the extractor of the Roomba 980.
Cleaning the pet hair from the end of the extractor of the Roomba 980.

We scored each vacuum on how much it picked up, as well as how much actually ended up in the collection bin of each product, rather than tangled up in the main extractor brush or the undercarriage of the robot.

Smart Connect

For the final 10% of the total score, we evaluated how Smart each 'bot is. More and more devices are offering smart connectivity, allowing them to be controlled remotely and interface with systems like Google Home, Amazon's Alexa, or Apple's Siri, and robot vacuums are no different.

We scored each product on its WiFi reliability, the amount of functionality offered through each companion app — if there is one — and how easy it is to integrate each product into various smart home ecosystem. Additionally, we also looked at the ease of setting up a schedule and if there were controls on the robot itself.


Hopefully, this article has provided you with some more insight into how we tested these products and why they scored what they did in our complete review of robot vacuums. For even more background information on these products, such as why you would even want one and what to look for when shopping for a new one, head on over to our Buying Advice guide for a complete breakdown of the products.

  • Share this article:

Follow Us

Related Review

The Best Robot Vacuums of 2018

by David Wise and Austin Palmer