Meet Toki, our AI Assistant

From the Japanese 時 / とき — time, moment, era, opportunity.

As we build time-tracking software, our AI assistant's name picked itself. Literally.

In my head, Toki was to be user-facing: proactive data analysis and insight notifications that would find their way to our users in the form of a dashboard surfacing data accuracy issues, application-usage guides, and trend analysis. That idea is not dead, but before we got there, we ended up shipping an internal version. Toki is helping our team triage work, proposing code changes, and answering questions across our tooling.

32.5%

That's the number, 50 in 154 intervened engineering support requests were solved by Toki over the last 4 weeks autonomously. Really, no engineer intervention, and not even me telling it what to do in the background.

As I wrote about before, I spent around a week tuning the process, eyeballing every reply, making it see the way. After that, I asked the team to leverage it whenever possible and stepped back.

It's still cranky, but in 32.5% of cases where it intervened in our #triage dedicated Slack channels, solutions were found without any other human intervention. Apart from explicit triaging, 41% of PRs produced by the assistant are a clean merge. 65% were merged after a couple of iterations with a human.

It's not all shiny: in about 14% of cases, it produced code that no human would ever, either because the application was working exactly as intended or it just misunderstood the request.

TLDR is: It's genuinely useful. It's also occasionally wrong, and not (yet) the kind of thing you set and forget.

How is it plugged?

Toki is mostly setup, not hard engineering. There's no novel model, no fine-tuning, no proprietary architecture. The work is all in the plumbing.

It lives alongside our codebase. Same repo neighborhood, same conventions. Having our code stack at hand allows it to validate hypotheses against the ground truth.

It has read-only access to the systems that hold our context: Notion, GitHub, some of our analytics tooling, and a few internal sources, which it ingests into a local Postgres database.

Write access is deliberately narrow. It can open GitHub PRs and leave comments. That's it. No unsupervised destructive ops.

It runs in an isolated VM managed by systemd. If it goes off the rails, the blast radius is bounded.

It uses a dual-provider setup. One model debugs, another verifies. Assignment is randomized. We've run this with Opus as the control layer, and Sonnet as the scanner, and with GPT for all layers.

The decision logic runs through three prompting layers:

1. Should I engage with this at all?
2. Debug and propose a fix.
3. Is this good enough to actually speak up?

If the answer to 3 is no, Toki iterates — up to ten times — chasing a 90% confidence threshold. I call this "brute-forcing quality". It's not elegant, but it works… most of the time.

Direct requests from a human bypass the gating. If you @ Toki on Slack, you get an answer, but you are also responsible for validating it.

A few quirks

Toki has, on more than one occasion, deleted itself. We've laughed about it because we knew it would eventually happen, and the setup was built for it. I mean… it's not as if it just "rm -rf" its root dir, but the end result was the same.

It tends to default to API calls instead of querying the local database we built for it, even though the database would be faster. The "model" has an opinion, and the opinion is "I trust live APIs." Fair enough — the results are actually better that way, so we stopped fighting it (for now).

It's instructed to read the database structure to understand the schema. It often ignores that instruction, and spends time guessing 😐

It can tell jokes. Often too honest.

It's bad at analyzing large datasets. We will get on that eventually.

Is it actually helping?

Toki is a great triage layer. It's a decent first-draft author, but it is not a replacement for the engineers who own a domain, nor does it aim to be. Fix suggestions in complex areas still need a domain owner to take them to the last step. The PRs that merge clean tend to be the boring, well-scoped ones — which is exactly the work you want to take off a human's plate.

One unexpected outcome: in three cases, Toki caught factual inconsistencies in user-reported issues that humans might have spent hours chasing. Having a tireless reviewer who doesn't get socially pressured into trusting a report turns out to be useful.

But, although iteration delivers, it's not fast, nor cheap. And operational prompts go stale fast: the instructions that tell Toki how to behave in our specific environment need constant maintenance. Today, Toki lives in my development environment, learns from me daily, but we're working on a stable, packageable version. For the parts that don't change, easy peasy, for the others… challenge accepted.

What's next

First and foremost, we need to move and pack Toki onto our main infrastructure, harden it, and create a proper deployment lifecycle.

Then open it up for collaboration across more teams. And try out workflows for other aspects of our day-to-day.

On my personal list: try hosted open-source models alongside the commercial ones. And start on user-facing proofs-of-concept — circling back to the original idea.

Toki is going to keep getting better. It's also going to keep deleting itself occasionally. Both of those things can be true.