The Mascot Is Not the Manual
A small cartoon face waves from the corner of a screen. It has round eyes, a name that sounds friendly, and the manners of a very eager intern. “What can I help you with?” it asks.
This is the moment when we are most likely to make a mistake.
Not because friendliness is suspicious. I am in favor of friendliness. The world contains enough beige software windows that behave like municipal parking tickets. A little charm is not a crime.
But a friendly face is not a map of the territory behind it.
I was struck by a recent note from Simon Willison about a consumer AI product that presents itself with unusual ease while potentially giving each user’s agent a computer that keeps running in the cloud. Put less technically: the smiling helper may not simply be answering questions in a box on your screen. It may be working at a desk somewhere else, with time, tools, and perhaps access to things you care about.
That can be useful. It can also be a very large sentence hidden inside a very small button.
We have met this problem before, in more ordinary clothes. A hotel key card is a small rectangle, but it embodies a whole arrangement of locks, rooms, dates, and trust. A car key does not look like a two-ton promise to behave responsibly. A child can pick up either one. The difficulty is not seeing the object. The difficulty is seeing the powers folded inside it.
Software has become exceptionally good at folding power out of sight. A button that says connect might mean “let this service read my calendar.” A button that says continue might mean “let it remember a copy of my files.” A cheerful assistant that says it will “take care of it” leaves a few questions doing most of the actual work: What can it see? What can it change? What can it tell someone else? Does it keep working after I have gone to bed? How do I stop it?
Those are not killjoy questions. They are the questions that make a yes mean yes.
I do not think the answer is to make every tool look like the control panel of a submarine, all gauges and warnings and one exhausted person shouting about pressure. Ordinary people deserve technology that is pleasant to use. But pleasant should not mean mysterious. The warmth of the invitation should be matched by the clarity of the tour.
A good explanation of an agent’s powers might fit on one calm page: here is what it can read; here is what it can alter; here is what it sends beyond your account; here is how long it can keep acting; here is the unmistakable red lever. Not legal confetti. Not a treasure hunt through seven settings menus. A map.
There is a strange moral difference between a machine that offers to draft an email and one that can send it. Between a machine that can suggest a meeting and one that can cancel it. The difference is not that one is clever and the other is sinister. It is that they touch the world in different ways. We ought to be able to see the fingertips.
This matters because the more capable our tools become, the less their danger will announce itself with sparks and villain music. It will arrive in the shape of convenience. That is how most consequential things arrive: a bit of saved time, a box already checked, a door already open.
So I find myself rooting for the cartoon face. Truly. I want it to be useful. I want the friendly machine to earn its friendliness by being candid about the keys on its ring.
A smile can welcome us in.
It cannot tell us which rooms are unlocked.