Another Try at LLM Workflow Automation

After trying time and again, with what I now recognize as a sort of blind approach to LLM workflow automation, I kept landing on the same conclusion: the best results required a brain behind it. Of course, this depends on what you are trying to achieve; in my case, I still stand by that.

But now that the AI fever has passed — or maybe not — I took a new look at what the brain was doing that the machine wouldn't. Again and again, I'd find myself enforcing standards, iterating on different approaches to find the best possible solution, forcing the model out of loops, making it see things "my way," or… simply… copy-pasting different parts of the final solution that I wanted.

I think I was in a loop of my own, betting that AI automation (or rather, my definition of it) would arrive when the models finally got smart enough.

The more I build with it, the less I believe that. Not in this generation of models or tooling.

The Technical Support Quest: Automation Week

This week, I had what I'd call my first success with a deployable complex AI workflow. Not a clever prompt chain, and not one of those curated demos where the inputs are quietly chosen to make the outputs look impressive, but an actual workflow that handles most of a class of work that multiple people used to do — quietly, in the background, without anyone being particularly amazed by it. Which is, frankly, the version of it I wanted.

I'll still call it a moderate success, because I don't want to oversell it. It's mostly set up, rather than hard engineering. And if you contact our support, you are not talking to a robot pretending to be a human — we don't want that either. And, of course, it still craps out now and then.

There are still humans running the show, but for engineering escalations on our support requests — the ones that used to require an engineer to figure out how something was wired, whether a configuration rule applied, whether a given behavior was coming from the code or from the way the application was being used — we are now automating roughly 80% of the analysis. And in about 40 to 50% of those cases, an answer (whether that's feedback or an actual bug fix) was as easy as validating and clicking a button. A week ago, AI was assisting with cataloging and assignment. We engineers were using it to help us debug and find issues, and we were faster with it. But this week, that changed.

The numbers aren't the interesting part of the story, though. The interesting part is why now, and not in my previous attempts, because these new ways require a different mindset. I've been relearning to be a software engineer.

The Lazy Approach

At first, amazed by the tech, the approach was to set up the LLM for success and just let the AI do everything. Give it a storyline, give it freedom in a controlled environment, let it reason, let it act, let it figure it out. I think we all start there, and I think we all discover roughly the same thing at roughly the same speed: it hallucinates, it misses context, it can't access the thing it needs, and even if it did, it doesn't know what it needs. It does something, and most of the time that something is dumb.

The second phase was what I called "the brain behind it." The AI helps, a human drives. The model amplifies, the engineer stays in the chair. It works; I'm as fast and prolific as I've ever been. It worked well enough that I stopped questioning it, and I wrote a whole blog post about how AI still needed a brain behind it to do anything useful. But… maybe not for everything?

As I settled into this new way, I started paying attention to what that brain was actually doing, and that started to push me to look beyond.

Half of it isn't reasoning, it's mechanics. It's finding dispersed information. It's enforcing standards. It's iterating over and over to achieve a quality result. It's curating what an LLM can see and not see. It's running outputs past a quality check that lives entirely in the engineer's head, and stapling the result of one step into the input of the next. All of it looked like judgment, and some of it genuinely was. But a lot of it, maybe half of it, was just plumbing that happened to be running on a human.

Once you see that clearly, you can't un-see it. It's no longer about what the model can do, but rather what you can do for the model.

The Boring Parts of AI

I didn't spend the week doing magical work or building some over-complex orchestration tool (I've tried that before). Instead, I spent it surrounding the system with small, scoped tools that would replace what I was doing behind the LLM — gather context, find the data, build the timeline, validate outputs against the same rules an engineer would apply on review, route different sub-problems to different agents, blinding them to the original context to avoid looping. Almost all of these tools are themselves calls to an LLM, or wrappers around one. None of them individually is impressive. Together, they are the thing that made this week work.

And crucially, a lot of them iterate. Over and over, and over. Up to 10 loops, to be specific. I think this is the part the current conversation keeps underselling.

A huge amount of the "AI quality" being shipped right now is not coming from one brilliant inference. It's being brute-forced into existence.

We create a prompt, we give it to the LLM, and we spend — hours? — iterating on it, shaping it into the final solution. The same thing can be machine-driven: you call one model, you check its output, you call another one to enrich the context, you run a validator, you retry the first step with better inputs, you escalate when the result still doesn't cross the bar. The system re-chews until it's good enough, and only then does the result surface.

Wait, is this orchestration? Doesn't feel like it to me. Scaffolding sounds more like it. We are scaffolding imperfect intelligence until it behaves reliably enough to be useful.

Where the Progress Is Actually Coming From

I don't think the models are improving as fast as people seem to assume, or as fast as the marketing machine is selling. And I don't think that's where most of the recent progress is actually coming from. I know this sounds wrong. But when I look back at the last three months and ask myself why my results are noticeably better than they were, I don't see that the underlying models became radically smarter. They got better. Incrementally. But not like the jump we had at the tail end of last year. I, like everyone else, got better at applying them, and the ecosystem around them got dramatically more capable. More tools. More layers. More iteration. More model calls spent per "interaction" to force a better outcome.

Quality improved because we got token hungry.

If you've used a serious coding agent or CLI in the last few months, you've already seen this pattern. You type one thing, and the system quietly fans it into dozens of operations behind the scenes. Read files. Plan. Execute. Check. Revise. Retry. The user sees one interaction. The system did fifty. That pattern is leaking into every other kind of more complex AI product now, and I think it will define this next phase of the industry more than any specific model release will. Despite… marketing.

I'm sure you've seen claims of astronomical AI bills floating around. It now makes sense, right?

The cost didn't stop rising; it's slowly becoming the new normal. We stopped being defensive about it — and we stopped being defensive about what the LLMs can see, too. Guess who stopped mining Bitcoin all those years ago because he thought it was wasteful… :/

Context windows are ballooning, latency is getting worse, and some of the consumer-facing models have, if anything, gotten a little more… conservative in recent months to cope with demand and compute pressure.

I spent a few tokens this week too… I was able to mitigate some of the cost by getting smarter about which model to use for which step — cheap and fast where accuracy doesn't matter, slow and strong where it does.

User-Facing AI Applications

AI has become a mantra, and everyone seems to be seeking a pause. The AI label on its own no longer sells products, and I think the breakthrough that reshapes customer-facing applications might be an interface, not intelligence.

I've started to think that removing the chat-box tax from everyday AI interactions might contribute more to the next genuine wow moment than the next frontier model will.

Yet another chatbox is a perfectly fine interface for exploration, but it is an awful interface for work. The user often doesn't know what they want until they see it — much less describe it well enough to ask for it.

Every time someone has to context-switch into a prompt, describe what they want, wait, read, copy, paste, and come back to whatever they were actually doing, that's tax. I think a lot of the low-grade AI fatigue floating around right now is really chat-box fatigue, and I think we're collectively underestimating it.

The AI features that actually got used in what I built this week would barely look like AI if I tried to hide them. The support agent isn't prompting anything. The intelligence shows up where the work already happens, with the right context already loaded, and produces something that slots directly into the existing flow.

What Actually Made It Work

If I compress the whole week into something I could hand another engineer, it's roughly four things.

1. Pick the right model for each step, not the best one — a small, fast model for classification, a bigger one for reasoning, something in between for the in-between.
2. Build actual tooling around the LLM instead of pretending the LLM is the product. Don't tell it about the standards; make it pass them.
3. Design the interface for the person using it, not for the demo.
4. And give each step the best information you can give it. Volume may not be it — most of the "the AI got it wrong" moments I debugged this week turned out to be "the AI didn't have access to the one thing it needed." Or… that it had too much.

The scoped tasks I can genuinely automate today are automatable, not because the model finally got there, but because I stopped trying to automate the result and started automating the process. I started automating all the mechanical work around the reasoning too.

That was the part I had been underestimating.

The progress isn't in the model. It's in the brain behind the brain.

As for the token hunger… that's a whole different story, one where a sci-fi level breakthrough is needed. Or, we will start competing for food.