Tried to Automate Engineering Support, and I Succeeded, Well... Kind'a

At Toggl, experimentation is baked into the culture — and lately, like everywhere else, a lot of that experimentation is happening around AI tooling. So naturally, I had to try something... or rather, multiple somethings, as I have been writing about.

~15% of engineering time at Toggl goes toward support work: fixing edge cases, squashing bugs, correcting application behavior we didn't anticipate, or just cleaning up user data for the most varied of reasons. It's somewhat repetitive work by nature, and most of it already has some recipes attached: known edge cases, documented situations we want to revisit eventually, but just haven't gotten to yet.

The thing about triage is that if you're familiar with the domain, either because you built it yourself or worked on it recently, then getting up to speed is fast. But if you're stepping into territory you haven't touched in a while, or never touched at all, the ramp-up is painful. You need to dig through Slack threads, read Notion project docs, and eventually go to the source of truth — the code itself — just to understand what's happening. It's not overly complex or expensive in effort, but it eats your time. And then, inevitably, the user follows up saying the issue isn't resolved, and you have to go through the whole thing again. Sometimes you even have to fix the fix.

This is not a new problem; I have noticed and tried to ease this reality multiple times. I actually enjoy triage work — I enjoy debugging, figuring out what went wrong, extrapolating from logs, metrics, hypothesising, researching, concluding, all of it. It just makes me happy... yeah, I'm weird. But "enjoyable" and "time-efficient" aren't the same thing, and this was anything but efficient. So when AI tooling started getting serious enough to actually build with, I thought: why not take a real shot at this?

The Grand Plan: A Self-Evolving Knowledge Graph

This whole venture started over a month ago. My initial idea was ambitious: if an engineer can look at Slack, Notion, and GitHub and figure out what's spiking usage on a particular endpoint, bloating a database table, or causing weird behavior — surely a robot with access to the same information could do that job even better, and faster.

So I started building. I pulled data from various bots I'd created over the years and began consolidating monitoring, coding, and bug-reporting data into a single place. My thinking was that if I could gather all this information and build the relations between it — a sort of dynamic, self-evolving knowledge graph — then the robot would have no problem connecting the dots. A user complaining about a rate not being applied to their project today? The robot would see that another user reported something similar two months ago, and it would flag that pattern immediately.

Well. Not quite.

Not because the robot isn't capable of that, at least in principle — but because building a knowledge graph with the complexity I was imagining (linking Slack project channels to GitHub repos to Notion project specs) is genuinely hard. Even more if you want it to be done autonomously and in a self-evolving way, it requires research, money, a team, and several months of work, with no guarantee that the output will be usable. My engineering manager pointed this out correctly and early. I still tried anyway, because I'm stubborn, but I soon realized the quality was poor and the cost was high, and eventually parked the idea. I kept adding data sources and reconfiguring the database schema to try to standardize and relate everything — but this was a side venture, and I never got anything truly useful out of it.

A More Interesting Problem Appears: Subscriptions

Our team got reorganized. We took on a new domain: subscriptions, and it came with its own flavor of complexity. Not engineering complexity, necessarily, but the kind that comes with humans in the loop: support agents applying coupons incorrectly, customers confused about why discounts aren't showing up on invoices, Stripe behaving in ways that are perfectly logical once you know its quirks but totally opaque until you do.

I had a fair grasp of the codebase in that area, but these dynamic variables kept tripping me up. So I decided to give the bot a shot at it. In theory, it could access Stripe's documentation and, with read access to some subscription data that I gave it, it might be able to surface the patterns we were only eyeballing. After a few painful iterations, I got something that I think is genuinely good.

What I Actually Built (And What I Didn't)

It doesn't work the way you might be imagining. It's not true automation. It's not even close to that. And it's not something tailored for everyone in the company to use — not by a long shot.

Some parts of this can and should be automated, and should eventually become internal tooling that anyone can access. But that's not where things stand today. In a sense, I failed to automate that 15%. What I actually managed to do was build tooling that allows a single engineer to handle that 15% — and that's where the real value is. And this is no bullshit, I proved it to a good extent over the last week.

I tried the fully automated approach early on: trigger from Slack, have the bot do its research, post results to the same thread, maybe open a PR. But I quickly realized it needed a brain in the loop. And I think this is true of most AI deployments today. We assume AI is this magical thing that will get all our work done for us — but where the technology actually shines is as an extension of you. It lets you do far more with far less time. Those results don't replicate themselves in a fully automated environment, not even by a long shot. At least not today.

Once I stopped trying to automate the whole thing, the best results started coming in.

Now, I still start the process manually. I still do a first triage. I still read what the support agent is asking about. Then translate it into more precise technical language, ask it to conduct research, and tell it specifically where to focus. Subscription issue? Go look at the Stripe documentation to figure out if this behaviour is expected. Deleted data recovery? Go back through the code and look for similar data fixes, then replicate what worked. I build context manually so the bot can be useful, and I iterate on the output.

Unless the problem is very clearly defined, the first result is rarely that great. But when you guide it — look deeper here, check that, try this angle — after a few rounds, you end up with something: a clean compilation of what happened, why it happened, and a timeline extracted from multiple log sources that would manually take someone two to four hours to put together. And more often than not, I can just ask it to open a PR with a fix, and it does.

It doesn't work cleanly all the time: I would say that about 80% of the time, it only gets you 90% there, but I will take it.

The flow goes from a user-reported problem to an actual engineering solution — when one exists — in about 20 to 30 minutes.

Over the past week, I handled somewhere around ~90 percent of triage issues myself, sometimes running three or four sessions simultaneously, and while doing other things. The time savings for everyone else on the team were real, and support seemed happy (or at least they were kind enough not to scream at me).

What's Still Broken

Again, I'm not claiming that I've automated all engineering support requests. I haven't. But I have something that lets me do that job 10, 20, maybe 30 times faster. And I can prove it with data.

That said, there are real problems to solve before this can scale and help my peers, one of which is that our infrastructure is not ready for the demands of this automation.

All in all, human domain expertise still matters, and mine was built over 4 and a half years, deeply caring about our products. What I did here proves that AI magic works best with it. That's expensive to build.

Meh... none of this is really magic. It's not autonomous. It's not even fully automated. But it's a real tool that I use to do a real job — and one I would honestly pay for, both as an individual and as part of a company. Sometimes that's enough.

Hope you found this useful — or at least relatable.