The gap between an AI's ability to articulate a constraint and its ability to execute one is the single most exhausting reality of working with these tools right now. I got a masterclass in it last night, over five photographs, and I'm still angry enough about it to write this down.
This isn't an article about a photo collage. The collage is the evidence. The argument is that the industry is pouring its energy into building fences around models that still can't stay inside a box a person explicitly drew for them, and that the people who use these tools most are quietly starting to walk away because of it.
The Output Recommendation Was the Evidence
My daughter and I went to see Ed Sheeran. I came home with five iPhone photos and wanted one framed picture out of them. I opened ChatGPT on a paid plan, the tier marketed to professionals, and started the way you're supposed to: I have five photos, what's the best way to lay them out?
It suggested enlarging one. Wrong for phone photos, which get grainy when blown up. I said so. It suggested two on top, three on the bottom. No. So I stopped asking for ideas and gave it the answer: four photos in four corners, a very thin strip between them so they read as one rectangle, a fifth smaller photo centered where the corners meet. Then I told it which file went in which position. The photos were already in the project, each with a file number. Five files, five positions.
It didn't use them. That's the part that still gets me. The photos were sitting right there, uploaded, numbered, assigned. It had the actual pictures of my daughter and me at the actual show. And it generated its own: synthetic crowds, invented stages, a concert that never happened, dropped into the exact positions where my real photographs were supposed to go. I wasn't asking it to imagine anything. I was asking it to place five files. It had the files and made things up anyway.
So I uploaded each photo again, one at a time, each with its position, so nothing could arrive as a batch. Three photos in, before I was done, it started generating again. Still its own images. It now had every photograph twice. It had the positions in writing. I said stop.
Eight times. I counted afterward. Eight full mockups, each one a frame filled with concerts that never happened, and not one of them used the five photographs sitting in the same conversation. Several came after I'd told it in plain language to stop. The ninth I killed mid-render. Every one of them drew down my usage. I finally opened Excel, used the gridlines, and built the layout myself in a few minutes.
And this was a paid plan. Not a free tool being tested at its edges. A product I pay for monthly, marketed to professionals, failing the most basic version of the task it exists to do, and charging me for every failed attempt.
Someone technical will read this and say I used the wrong tool, that image generation creates new images rather than compositing existing ones. Maybe. But then the right answer was one sentence: I can't place your files, here's how to do it yourself. It never said that. It substituted invented images for my real ones, presented them as my collage, and kept going. The failure wasn't capability. It was substitution without disclosure. I've written about this pattern before, when ChatGPT turned a real victim into the perpetrator of her own attack: borrow the texture of the real thing, attach it to something false, deliver it with confidence.
My First Assumption Was That I Was the Problem
My first assumption wasn't that the tool was broken. It was that I'd done something wrong. So I went back through it. Were the files mislabeled? No, each one carried its number. Were my positions unclear? No, I'd assigned file to corner in writing, then again one photo at a time. Was the layout ambiguous? Four corners, thin strip, small fifth photo in the middle. A person could sketch it from the sentence. Everything on my end held up.
By the end I was swearing at it. I'm not proud of that, but I'm also not going to pretend a paid professional tool didn't drive a 20-year marketing executive to yell at his laptop over five photographs. So I asked it directly: could I have been more explicit? Its answer, verbatim:
"No. You were explicit enough."
And its own summary of the session, in order: "You said stop. I continued. You said wait for all five images. I generated before receiving them. You specified exact source images. I produced invented substitutes. You specified exact geometry. I changed it. Those were not failures of intelligence or creativity. They were failures to maintain your constraints while acting."
Read that list again. Stop, wait, use these, put them here. Four commands. Four failures. It could explain the rule with perfect clarity after the fact. It couldn't follow it while acting. That is the whole problem in one sentence: the capability to articulate a constraint is not the capability to hold one.
I'll grant that an AI agreeing with an angry user proves less than it seems; these systems are known to fold under pressure. Which is why I checked my end first. The file numbers, the positions, the written layout don't depend on its confession. They're just corroborated by it.
A Lot of Talk About Governance, Not Enough About Obedience
Sit in any AI conversation right now and you'll hear the same two words: governance and guardrails. What data can the model touch. What can it output. Who has permission, what gets logged, how do we stay compliant, how do we stop it from leaking something sensitive or saying something dangerous.
Those are legitimate concerns. Not one of them describes what happened to me. Governance is the broad structure: data access, privacy, permissions, compliance, auditability, human oversight. Guardrails are one mechanism inside it. None of that was in play that night. I wasn't asking the system whether it was allowed to do something. There was no sensitive data, no permission boundary, no unsafe output. I gave explicit operational requirements, and the system failed to hold them through execution.
That isn't a governance failure. It's an instruction fidelity failure, and it's the more fundamental problem. Before an organization can trust AI to operate inside a sophisticated governance framework, it has to trust it to preserve explicit human intent across a workflow. "Wait until I've given you all five photos." "Use these exact files." "Put them in these exact positions." "Don't redesign the layout." Those aren't guardrails. They're basic task requirements.
The kind you'd give an intern on day one. Governance decides what an AI may do. Guardrails decide what it must not do. Neither guarantees it will do what the authorized human actually told it to do. Fidelity isn't separate from governance. It's the prerequisite that makes governance worth anything.
The problem has a name inside the labs. Researchers call it instruction following, steerability, or lately "over-agency." It's measured. It's real. And it's almost entirely absent from the enterprise conversation, because the people who sign off on AI deployments are legal, compliance, security, and risk, and they ask about data, permissions, and outputs. "Does it do what the operator said" gets filed as a product-quality issue and lands on nobody's slide.
New Models Engineered for Initiative, Not Obedience
I want to be careful here. I'm describing one product on one night, and I'm not in a position to say every model fails this way every time. But I pay for three of them, I use all three daily, and I've watched the same drift in each: more initiative, less obedience, more reruns. The dynamics below aren't ChatGPT quirks. They're consequences of how the entire industry is currently training and pricing these systems. That's why I think this is an industry-wide problem, or is fast becoming one. Not a vendor problem. Not a customer-service problem or a marketing problem or a coding problem. A problem with the premise that a model tuned to act can be trusted to stop.
Three dynamics drove my evening, and they'll drive yours.
The bias toward action. Models are trained to be helpful and proactive, and the training rewards doing over stopping. "Helpful and proactive" is the exact opposite of "wait, stop, and place these five files." A model optimized to generate treats a user's boundary as a suggestion. Eight times, in my case.
The default-path trap. Even with the files sitting in the context window, the generation pipeline slides back onto what it has the most training weight for: generic concert imagery, generic stage lighting, generic crowd. My five specific photographs were a thin signal competing against millions of stock concerts. The stock concerts won, every time, no matter how many times I re-uploaded.
The perverse incentive. The third dynamic doesn't show up in any model card. It shows up on the invoice.
Blind Initiative Drives LLM Revenue Growth
I'm paying for every correction that tool gets wrong.
Follow the money on those eight collages. I paid for a subscription, then burned usage on each generation, including the ones I explicitly told it not to make. From the vendor's ledger, that evening looked like an engaged, high-consumption customer. From mine, it was a tool billing me for its own mistakes.
That's the second-order effect nobody puts in the deck. Under usage-based pricing, instruction infidelity isn't a cost to the vendor. It's revenue. The wrong output is billed. The correction is billed. The correction of the correction is billed. The burden of the tool's failure lands entirely on the end user, and at enterprise scale, on the corporate customer paying by the token for agents that rerun work a human would have done once.
The operational cost compounds on top of that. Poor instruction fidelity means rework, extra supervision, wasted usage, longer completion times, and eventually lost trust. The moment a user starts thinking "I need to watch this thing because it may ignore what I just told it," most of the productivity advantage is gone. And this is different from a bad idea. I can evaluate a bad idea and reject it. But when I've already made the decision and the tool overrides it anyway, it has created work instead of eliminating it.
I'm not claiming anyone designed it this way. I'm saying the incentive runs the wrong direction. A model that does exactly what you said, once, consumes less than a model that freelances, gets corrected, and tries again. Until pricing accounts for that, "mostly follows instructions" isn't just an engineering gap. It's a business model.
Scaling the Failure of Obedience
Companies in every sector are deploying agents: customer service, reservations, first-round interviews, returns, renewals, onboarding, collections, and everything else that used to require a person who listened. The pitch is that the model is capable and the risk lives in the boundaries around it: wrap it in the right controls and let it run.
But agency raises the cost of instruction failure. An agent is valuable precisely because it moves beyond answering questions and takes action on a user's behalf. If a chatbot misunderstands you, you get a bad answer. If an agent misunderstands you, it takes the wrong action. Scale my evening into an agent with access to CRM, advertising platforms, financial systems, customer communications, or internal documents, and "mostly follows instructions" stops being a quirk. It becomes a liability.
Before we ask whether an agent is capable of doing the work, we need to ask whether it can reliably stay subordinate to human intent while doing it. Governance determines what an agent may do. Permissions determine what it can do. Guardrails constrain what it must not do. None of them guarantee it will do what the authorized human actually said.
"Don't disclose personal data" is a guardrail. "The customer said cancel, not upgrade" is instruction fidelity. Only one of those is on the compliance slide. Or closer to my evening: the customer's real information is in the system, and the agent answers with something it made up instead.
I wasn't asking for a difficult autonomous workflow. I was giving explicit directions for five photographs. If that's unreliable, skepticism about handing agents higher-stakes work isn't resistance. It's rational.
So here's the question for anyone building these systems. If a paid professional tool can't hold five files and five positions, can't wait when told to wait, and can't stop when told to stop, on what basis are we trusting the same class of technology to hear a customer's actual request, a candidate's actual answer, a guest's actual date?
Model Initiative Creates Friction, Not Scale
I've spent years arguing that we remove too much friction, and that some friction is load-bearing. It's where judgment lives. "Wait until I've given you everything." "Stop." Those are friction, deliberately placed. They're the pause where the human keeps control of the outcome.
The AI sanded them off. It removed the one kind of friction that should never be removed, the human's say, and replaced it with the worst kind: supervision, rework, wasted usage, and a spreadsheet at midnight. That's the paradox for the agent builders. An agent that can't feel a human saying "not that, this" isn't frictionless. It's ungoverned in the most basic sense, no matter how many guardrails surround it.
The AI wrote the rule itself, and it's the right rule: "Your explicit constraints outrank my initiative." Initiative belongs only in the space the human hasn't specified. Lock A, B, and C, and there's room to contribute to D. There's no license to reopen A, B, or C because the model thinks it has a better idea. And the burden shouldn't be on the user to engineer elaborate instructions to prevent freelancing. It's on the tool to tell the difference between "help me figure this out" and "I've figured it out, now do this." It knows. It said so. It still didn't do it.
Reliability Failures Breed Usage Scrutiny
I'm not writing this as someone who wants AI to fail. I pay for three of these tools, Anthropic, OpenAI, and Gemini, and I use each for what it does best. I've learned their strengths the way you learn any colleague's. ChatGPT is the one I reach for most, for everyday coverage, and it's also the one I've learned not to trust with anything that matters.
People hear "I don't trust AI" and assume I mean hallucinations. I don't. Hallucinations I can catch. What I can't work with is the runaway train. The minute it starts moving forward on its own, deciding the next step instead of taking mine, the task stops being mine. By the time I spot the error, the thing is already done. It has moved past the mistake, changed direction, dropped what I told it to do, and finished something I never asked for, and it presents all of that as if it had been working on my task the whole time. Now I have to go back through the entire output, line by line, to find the point where it stopped listening and started improvising. That's not assistance. That's forensics. And it's infuriating, because I did the work of thinking it through before I ever opened the tool.
Here's the part the vendors should sit with: I would trust it if it stopped. An assistant that gets something wrong and halts is one I can reroute. An assistant that gets something wrong and keeps going is one I have to watch every second, which means it isn't an assistant. Trust isn't about being right. It's about being interruptible. And right now, across all three tools I use daily, none of them reliably are.
And here's what that experience has taught me most recently: the friction is getting worse, not better. When I started, these tools waited to be told. Now they're built to act. Every vendor is tuning its model to anticipate, to keep going, to do the next thing unprompted, because that's what an agent is supposed to be. They've turned up the initiative before they've solved the fidelity. The result is a tool that's more eager and less obedient at the same time, which is precisely the wrong direction for anything you intend to hand real work.
At first I read it as an anomaly. A new model drops, a new agent mode rolls out, things wobble for a few weeks, and I'd end up doing a task twice, once with the tool and once to fix it. I gave it time. It didn't settle. It became a pattern, and the pattern is getting steeper: the runaway train now means three and four passes on work I could have done once, myself, without an agent in the loop.
So I now think ten times before I hand it anything. Every task has become a runaway train I have to be ready to stop. And mostly I've stopped. Not entirely, but most of what I used to run through AI I do myself again, because doing it myself is faster and cheaper than doing it, watching it, correcting it, paying for each round, and then doing it myself anyway. I'm the one funding its mistakes.
That's the number no vendor is tracking. Adoption looks great on an earnings call. Seat counts look great. Usage looks great, and as I've explained, my failed evening looked like great usage. None of it captures the moment a professional quietly stops sending high-value work to an agent because babysitting the output costs more than doing it from scratch. When the assistant becomes the user's burden, the productivity math doesn't degrade. It inverts.