The RealReal's reserve system blocked shoppers when an AI agent repeatedly re-added items to a cart after each hold expired. The system was built for human behavior, not agent behavior. I learned this building automated systems: automation exploits edge cases a person would never think of. You need checks for the obvious things, not just the clever ones.
The RealReal, a secondhand luxury retailer, found out this week that its item reserve system worked perfectly until it encountered something it was never designed for: an AI agent with infinite patience. The Verge reported that using Meta's Muse, it was possible to repeatedly re-add items to a shopping cart as soon as a 20-minute hold expired, effectively blocking other shoppers from purchasing those products for hours. The RealReal responded by tightening its bot defenses. But the real problem wasn't the bot. It was that the system assumed human behavior.
Why this happened
I run a distribution business that serves over 500 retailers with more than 650 stock keeping units. When I first automated our daily sales and stock report, I built it to do what people did. But I also learned something building automated systems like BabyJunctions, my content site: an unattended pipeline makes mistakes a person would never make, and exploits edge cases a person would never think of.
With BabyJunctions, I discovered that the system could write about the same topic twice because nothing stopped it from doing so. It could also pair product images with the wrong products. Neither error felt catastrophic on its own, but both were things a person would catch reflexively. A human looks at the page and thinks, "I already wrote this." A system does not. A human glances at the image and product and recognizes the mismatch. The system does not. So I built checks into the pipeline for things that seem obvious.
The RealReal's reserve system was built for human shoppers. A human looking at a designer handbag on hold might be torn between two options, but eventually they either buy or move on. They do not spend six hours adding and re-adding the item to their cart. An AI agent with a simple instruction to secure the item does exactly that, with no hesitation.
What this means if you are automating something
Most of the automation I build for clinics and service businesses is designed around how the people in those businesses work now. An AI voice agent schedules an appointment the way a person would handle the call. A WhatsApp automation sends appointment reminders the way the office used to send them by text. Automation that mirrors human processes usually works. The problem surfaces when the system does what you asked it to do, but in a way you did not anticipate.
- Does your system assume a person will get bored, tired, or move on to something else? A workflow will not.
- Does it rely on someone noticing something wrong on a page or in a record, or catching a duplicate before it causes a problem? An automated system will not notice.
- Does it assume that a user will behave rationally, or lose interest, or give up after a few failed tries? An agent or a workflow will keep trying.
- If you store data from your website or from a process, have you thought about what another system could do with access to it?
The deeper thing
The RealReal's response was to make it harder for all AI agents to log in, add items, and check out. Amazon has blocked agents entirely from shopping on its site. Both companies recognized that if humans stop visiting their websites directly and instead access them through AI agents, the retailer loses its direct relationship with the customer. But that is a separate problem from the one in this story.
The immediate problem is that when you automate a system or a process, you need to think about behavior you did not expect. If you are automating something, ask yourself: What could go wrong if this runs without a person watching it? What assumptions about human behavior am I building into this? What would an agent do with this system if it did not get bored, or feel rushed, or make a judgment call?
Once you answer those questions, you can add checks. That is what I do now.
