Amazon has blocked Meta's AI agent, Muse, from buying anything on Amazon.com. Users trying to shop through it now get an error message telling them an unauthorised AI agent violates Amazon's terms. Most of the coverage I've seen frames this as two tech giants squaring off. I don't think that's the interesting part.
The interesting part, to me, is the reason underneath it. According to the reporting, Muse has one of the lower hallucination rates among AI models, and it's still far from zero. If Muse places a bad order, Amazon is the one who deals with the angry customer and the angry vendor, not Meta. That is a liability question dressed up as a competitive one, and I've built enough unattended systems to know exactly why a company would rather wait than find out the hard way.
An agent that can act is only as good as its ability to stop
I run a publishing pipeline called BabyJunctions where no human writes the articles. It has put out more than 199 of them without anyone drafting a single one by hand. The lesson from building that had nothing to do with how good the writing got. It was that a system running without anyone watching needs to be better at stopping than at starting.
Early on, a bad run would publish a weak article, because nothing in the chain was allowed to say no. There was no step whose job was to check the output and refuse to ship it if something looked wrong. I rebuilt the whole thing around checks that fail closed: if a step can't verify its own output, nothing goes out, and I get told instead. Skipping a day of publishing costs almost nothing. Publishing something wrong under your own name costs a lot, and you usually find out about it from someone else, not from your own system telling you first.
That's the exact shape of Amazon's problem with Muse, just with money and inventory instead of articles. An agent that can complete a purchase needs to be just as good at declining to complete one. I'd bet the gap between Muse's current hallucination rate and zero is exactly the gap Amazon isn't willing to absorb on someone else's behalf.
I've shipped the version of this that goes wrong
One of my own automations ran a scheduled job twice by mistake. Nothing was stopping a second run from starting while the first was still going, so the same messages went out to the same people twice. It wasn't dangerous, but it was embarrassing, and it's exactly the kind of failure nobody thinks about until the day it happens. I added locking so a job can only run once at a time, and if the previous run hasn't finished, the new one simply exits rather than piling on top of it.
That fix took an afternoon. What it taught me took longer to sink in: the failure modes that matter in unattended systems are rarely the ones you designed for. They're the boring, structural ones, like two copies of the same job running at once, or a step that has no way of saying no to bad output. Nobody puts those failure modes in a demo. They only show up once something is running for real, at a scale where a human isn't watching every step.
The messy part isn't the AI, it's the data underneath it
There's another problem that gets even less attention than hallucination rates, and it's the one I'd worry about most if I were Amazon: the state of the data an agent has to act on. I once connected an ERP system to an accounting platform for a client, and the connection itself was the easy part. The hard part was that the same product had different names and codes in each system, so once the connection worked, every report built on top of it was wrong in small ways. Cleaning and matching that data took longer than wiring the systems together in the first place.
Now imagine an AI agent placing purchase orders against a catalogue with that same kind of quiet inconsistency, three listings that are really one product, a size or a spec mismatch nobody caught. A human buyer would notice the price looks off, or that the item photo doesn't match the description, and pause. An agent doesn't pause unless you've explicitly built the thing that tells it to pause. Amazon runs one of the largest, messiest product catalogues on the internet. Letting an outside agent transact against it, before anyone has proven it knows when to stop, is not a decision I'd rush either.
What this means if you're building on top of agents, not just watching them
None of this is really about Meta or Amazon specifically. It's about what happens once you hand a repeatable task to a system that runs without someone watching every output. I've seen the upside of that too. A daily sales report that used to be compiled by hand, and that almost nobody read, now lands as a short summary on the owner's WhatsApp every morning, and it actually gets read. The data didn't change. Where it arrived, and how much attention it demanded, did.
That's the difference between automation that quietly makes something better and automation that quietly makes something worse: whether the person who built it thought about what happens when the system is wrong, not just when it's right. A report that goes unread is a mild failure. A duplicate WhatsApp blast is an embarrassing one. An agent placing a bad order on someone else's marketplace, at whatever scale Meta intends to run Muse, is a different order of failure entirely, and Amazon clearly isn't willing to be the one who finds out what it looks like first.
