How We Tested Robot Vacuums

By:
David Wise and Austin Palmer

Last Updated:
Thursday
September 21, 2017

Share:
The finished results of the Neato Connected after the completion of our edge and corner cleaning test.
The finished results of the Neato Connected after the completion of our edge and corner cleaning test.

We spent over 2 months pushing each of these robotic vacuums to their limits, running each of them countless times over a variety of messes. We set up tests to really determine the cleaning ability of each robot, their self-sufficiency at navigating a room, as well as their attention to detail in cleaning corners and edges. We also looked at the ease of use at setup for performing basic functions, like setting up a cleaning schedule, spot cleaning, and changing cleaning settings.

We attempted to make every test as objective as possible, but on some categories, like difficulty of set-up and just overall ease of use, we used the general consensus of our impartial testers. We timed each robot as it cleaned the various scenarios that we set up, and estimated the percentage of the room that was cleaned for our room navigation metric. Our cleaning performance tests were conducted with a specific amount of test debris, and compared before and after measurements, eliminating any category that all robots performed equally well in.

The Neato D75 after performing our Flour Line test. It left behind not trace of flour in the area it cleaned.
The Neato D75 after performing our Flour Line test. It left behind not trace of flour in the area it cleaned.

We repeated this with different debris on different surfaces, as well as comparing before and after photos of each test to evaluate if the robots missed cleaning any spots and confirm our measurements. We kept both the overall amount of debris, as well as its relative concentration consistent across all tests, weighing the quantities of debris with our scale, as well as evenly spreading it out over a predetermined area. We aggregated the results of these tests, and used these scores for our Carpet Cleaning, Hard Surfaces, Corners and Edges and Pet Hair rating metrics.

Evaluating the edge cleaning performance of the Samsung in our test pen.
Evaluating the edge cleaning performance of the Samsung in our test pen.

After assessing the cleaning performance, we looked at the robot's self-reliance, or how well the robots did at navigating commonly encountered obstacles, like shoelaces or USB cords, and how they did at navigating what we felt was an average room full of furniture. We timed each trial of the robot, as well as estimated the percentage of the room that was actually cleaned.

Our final round of testing focused on ease of use. While this was the most subjective round of tests, we felt our assessment was fair, only docking points where we really felt that a product fell short, and was exceptionally difficult for us to use.

  • Share this article:
 

Follow Us


Table of Contents


    Related Review
    The Connected cleaning up flour from a hard surface.

    The Best Robot Vacuums of 2017

    by David Wise and Austin Palmer

    Unbiased.