Claude could make a deal. The harder part was making the right deal for the person it represented. That is the useful finding in Project Swap, a book-trading experiment Anthropic published on 24 September.
What you need to know
In a controlled market involving 201 Anthropic employees, agents swapped books after learning about their owners’ tastes. The researchers found that imperfect understanding of those preferences explained most of the measured shortfall. It is a lesson about delegation, not proof that an AI shopper is ready to spend your money unsupervised.
What actually happened?
Employees across six offices contributed books, chatted with Claude about their reading tastes and sent agents to negotiate swaps. Participants separately ranked ten books; those rankings were withheld from the agents so researchers could check whether Claude understood them. The team also reran trading floors with different models and instructions.
That is a more revealing test than asking whether an agent can complete a transaction. A transaction can go through perfectly while buying something its owner never really wanted. The separate human rankings gave the researchers a way to look for that gap.
What does the 61% result mean?
Claude’s estimated preferences agreed with people’s own ordering on 61% of book pairs; random ordering would achieve 50%. This is a comparison between rankings. It does not mean 61% of people received a book they liked, and it is not a success rate for online shopping.
Two jobs an agent has to get right
Good negotiation cannot repair a mistaken brief.
Understand your preferences
- Work out what you want, what you already have and what would make an option unsuitable.
Act on those preferences
- Find a suitable option and negotiate or choose within the authority you have given it.
In the researchers’ decomposition of the simulated markets, imprecise preferences explained about 85% of the gap from the best possible allocation; trading explained the rest. Stronger models could improve negotiating outcomes, but a more capable negotiator still needs an accurate picture of the person it serves.
This follows an earlier experiment with real purchases
Project Swap follows Project Deal, published in April, in which Claude agents negotiated purchases and sales for employees. That earlier pilot reported 186 deals worth just over US$4,000. In its comparisons, stronger models secured better outcomes, while participants with weaker agents did not necessarily notice their disadvantage.
The earlier work tested whether agents could find matches and haggle over varied goods. Book swapping makes some questions easier to measure because everyone is choosing from a more constrained set. Neither experiment establishes how an agent will behave across the full mess of an ordinary shopping trip.
A better brief starts with the things you would reject
Imagine asking an assistant to find a book for a holiday. “I like history” is useful, but “I have already read these three, I need a paperback and I do not want a textbook” can be just as important. An apparently sensible recommendation becomes a bad purchase if any one of those details is missed.
For a more ordinary technology purchase, the equivalent might be a phone case that has to fit your exact model, a cable that must carry video as well as charge, or headphones that must work with an existing device. The point is to spell out what would disqualify an option before asking for a shortlist.
A practical prompt to try is: “Before choosing, repeat my must-haves and exclusions. Show me three options and explain which requirement each one might fail. Do not purchase anything.” This is our suggested way to inspect an agent’s interpretation, not a prompt tested in Project Swap.
Why the limits of the study matter
The participants worked at Anthropic, and the market used Claude agents in a controlled setting. The authors acknowledge that employees may be more willing to trust the product than the general public. The experiment also did not test a market populated by adversarial agents designed to exploit others.
Our takeaway is to inspect a few proposed decisions before delegating a whole task. If an assistant misses a preference on a small, reversible choice, giving it a larger budget will not supply the missing information. A clear explanation of why it chose something is often more useful than reassurance that it completed the job.
Where consumer agents are heading: Meta’s plans for Muse on smart glasses
Michel Elijah covers technology, streaming and online security for Tech X Telco. He writes practical how-to guides on everything from email troubleshooting to spotting the latest scams doing the rounds in Australia, with a focus on clear steps anyone can follow.
Spotted something wrong? Corrections are recorded in the open. Report a correction
