• Amazon invests billions of dollars in robots to boost e-commerce efficiency and profitability.
• Back in 2015, the Amazon Picking Challenge tried to spur more research into warehouse automation.
• The competition inspired some of the company's most advanced robots, including Sparrow and Robin.

Amazon is investing billions of dollars in robots to make its e-commerce business more efficient and profitable. This huge initiative started out a lot smaller.

A decade ago, the company launched a competition for university engineering teams called the Amazon Picking Challenge. It called on researchers to design robots for a common warehouse task: Grabbing products from a shelf and putting them in a box.

As a tech reporter, this quirky project intrigued me. At the time in early 2015, Google was testing self-driving cars, a technology that emerged from a similar academic competition known as the DARPA Grand Challenge. What if Amazon was trying to replicate this magic, but with robots rather than automobiles?

Then, a funny thing happened. The Amazon Picking Challenge faded away. It was renamed and only lasted a few years. I chalked this up to another bad call and moved on.

I only thought about this challenge again late last year. That's when Amazon unveiled a next-generation warehouse in Louisiana that has 10 times more robots moving products around and, yes, picking them up with dexterity. The facility processes orders 25% faster and 25% more efficiently, and it will likely be the future of the company's e-commerce operation.

Ten years after the Amazon Picking Challenge, the fruits of this nerdy competition have finally emerged. It follows an uncannily similar timeline to the DARPA Grand Challenge, which started in 2004 and resulted in Google's driverless cars hitting the road roughly a decade later.

So, with the help of Business Insider reporter Eugene Kim, I investigated how Amazon's huge new fleet of picking robots came to be, and how this competition laid the foundation for a new wave of automation that's about to crash over the warehouse and logistics industry.

From pallets to picking

It started with an acquisition. In 2012, Amazon paid $775 million for Kiva Systems, which designed flat robots that zip around warehouse floors. 

This helped move pallets of goods around, but humans still needed to pick items. Getting a robot to spot the correct product in a box, then grab it just hard enough to pick it up, but not damage it — that's incredibly difficult.

This is where the Amazon Picking Challenge came in. Instead of hacking away at this problem itself, the company wanted to focus the broader academic community on the task.

The risk was that any valuable inventions would be out in the public sphere, and Amazon might not directly benefit from them. But the potential gains were much bigger, according to executives and roboticists.

"Amazon doesn't compete with robotics companies," said former Amazon Robotics chief Brad Porter, who runs robotics startup Cobot now. "When facing an unsolved research problem in robotics AI like bin picking, Amazon benefits if anyone solves that problem as long as Amazon can get access to the technology to improve their operations."

"The challenge Amazon was trying to solve was how to motivate researchers to focus on this problem," Porter added. "The Picking Challenge very much succeeded in doing that."

Oreos, Sharpies, and dog toys

The first competition took place over two days in late May 2015 in Seattle, with more than 25 teams from colleges including MIT, Duke, Rutgers, and Georgia Tech.

The contestants had to design a robot that could pick products from a typical shelf found on a Kiva Systems warehouse pod, and then put those items into containers. The picker had to be fully autonomous, and each robot was given 20 minutes to pick 12 target items from the shelves. Contestants had to open-source their creations.

Companies, including ABB, Fanuc, and Rethink Robotics, founded by industry pioneer Rodney Brooks, provided hardware for contestants to repurpose and tinker with.

The products were a preselected set of 25 items commonly sold on Amazon.com, including packs of Oreo cookies, boxes of Sharpie pens, and dog toys.

Some were easier to pick. There were simple cuboids, like a box of coffee stirrers or a whiteboard eraser. Others were trickier. For instance, a box of Cheez-Its could not be removed from the bin without first tilting it, adding another complex step for the robots. Smaller items, such as an individual spark plug, were more difficult to detect and properly grasp.

Vacuum arms and 'catastrophic failure'

Among all 26 teams, a total of 36 correct items were picked, versus seven incorrect items. Another four products were dropped by robots in the competition.

About half of the teams scored zero points, and two teams couldn't get their robots working well enough to even attempt the challenge, according to a research paper analyzing the results.

Problems ranged from the highly technical to the mundane. Some of the same items came packed differently, which made them even more difficult to pick. One team's machine had a vacuum hose that got accidentally wound around the robotic arm.

With each system having hundreds of components, the failure of any one of these could lead to "catastrophic failure of the overall system — as witnessed during the competition," the researchers wrote. 

The main finding from this first Amazon Picking Challenge was that human warehouse workers were a lot better than machines at picking products.

"A human is capable of performing a more complex version of the same task at a rate of ∼400 sorts/hour with minimal errors," the researchers wrote. "While the best robot in the APC achieved a rate of ∼30 sorts/hour with a 16% failure rate."

But the conclusion was hopeful, too: The contest showed that robotics could substantially increase warehouse automation and order fulfillment in the near future.

The competition was renamed the following year as the Amazon Robotics Challenge, and the tasks evolved to be more complex.

Suction and other benefits

Tye Brady, chief technologist at Amazon Robotics, was involved in those later Amazon Robotics Challenges. 

In a recent interview with Business Insider, he said research on robotic manipulation exploded from 2016 through 2018, with many institutions publishing their results and insights. This helped spread valuable knowledge across the industry, speeding up progress. 

At least two professors started graduate-level classes related to Amazon's challenge, and these programs are still churning out experts with valuable practical applied knowledge in robotics, Brady explained. 

"When you get a whole bunch of smart people together in a room and think about focused problems, some great things are going to happen, and that's really what happened," he said. "It inspired a lot of the work that we have today that we see in, for example, our Sparrow and Robin manipulation systems that are real-world products delivering packages inside of our fulfillment centers."

In that first competition in 2015, some robotics teams used grippers that mimicked the way a human hand picks things up. Other teams tried suction instead, with some researchers even strapping off-the-shelf vacuum cleaners to their robots.

Gripping proved more problematic because the robots didn't receive enough information to know when to release or add pressure at the right times. This could result in squashed or crushed products or dropped items.

Sucking the items up so they stuck to the end of robot arms was a more successful approach.

"The idea of high flow suction was novel. Bring your favorite vacuum cleaner and start picking up objects. That was kind of clever," Brady said. "This idea, we used suction inside of our Robin and our Sparrow arms. It's very good."

The boss has noticed

Amazon unveiled Robin, its first robotic arm, in 2021. This machine picks up packages from conveyor belts and places them on other mobile robots called Pegasus.

Sparrow followed in 2023. This was Amazon's first robotic arm to handle individual items rather than packages. It uses computer vision and AI to pick more than 200 million different items from containers and place them into totes.

Amazon CEO Andy Jassy has taken notice. At the AWS re:Invent conference in December, he should have been talking about cloud computing. But he took time away from that subject to wax lyrical about Sparrow. 

"It has to discern which item is which. It has to know how to grasp that item, given the size of it and the materials and the flexibility of that material. And then it has to know where in the receiving bin it can put it," Jassy said. "These are all inventions that are critical to us changing the processing time and the cost to serve our customers."

Wall Street has noticed, too. Morgan Stanley recently estimated that Amazon's warehouse robots could save the company as much as $10 billion a year.

"The big story is we're just getting started," said Brady.