We have around 23,000 messages of our own team chat. For a while now we have been running a small in-house classifier on it. It reads a message and decides what it is: should this be a task, is it a calendar thing, is it an issue, or is it just people talking.

Nothing fancy. Old school ML, trained on our own data.

Why we cared

We were planning AI teammates on top of this. The idea was simple: if a message looks like work, wake up the agent. If it doesn't, leave it alone.

That "leave it alone" part matters more than it sounds. Roomie costs $2 per user with all features included. We can't afford to wake up an agent on every message someone sends. Most chat is just chat. If the agent runs on all of it, the cost is gone before the work even starts.

So the question was never "which model is smartest". It was "what is the cheapest thing that tells us correctly when to wake up the agent".

Trying Jev

As soon as Jev came out we had to try it. I didn't have a Jev account at that time, so I went through OpenRouter to get access and started testing on our messages.

Surprisingly, it seemed fine. It understood the messages reasonably well, and calling it was cheap.

The problem wasn't Jev's own cost. It was how often it said "wake up the agent". Every one of those is a full agent run, and that is where the real money goes. Our own classifier was pickier about when to trigger, so it woke the agent far less for the same chat. Put next to each other, ours came out more cost effective, and at our price that decides whether the feature is possible at all.

Trying Laya

Then we tried Laya, a zero-shot classifier. No training on our data, you just describe the categories.

The result was not good either. On our chat it got about 14% right, where our classifier got about 92%. It was also much slower, around 1.5 seconds per message against a few milliseconds for ours.

To be fair to Laya, our chat is a mix of English and Nepali typed in English letters, with our own shorthand. That is not what it was built for. It just didn't fit our case.

What made our classifier better

We didn't swap models. We gave the small one better data.

We fine-tuned it on labelled messages from our own chat. The biggest change was on tasks: before, it caught 57% of the messages that should have been tasks. After, it caught 91%.

Clean, well-defined data did more than a bigger model would have.

A few things we learned on the way

  • How you ask matters a lot. With zero-shot models, small changes in how the question is written changed the results more than switching models.
  • Examples inside a category description act like magnets. Put "e.g. meeting" in the calendar label and every message with the word meeting becomes a calendar event.
  • Don't offer "nothing" as one of the choices. The model picks it far too often.
  • Watch what gets cut. When the conversation was long, the newest message was the one being dropped, which is exactly the one you wanted to classify.

Where this leaves us

I still believe old school ML with clean, well-defined data beats a lot of real world cases. Intelligence seems cheap right now, and usage is all over the place. But sticking to the basics is the only way we have found to stay efficient and affordable.

We are not saying these models are good or bad. This is just our use case. For now, we are sticking with our plain old classifier.

Next we are trying the same thing with a small Qwen model to see if we get a different result. Hopefully we will have something concrete to share after Diwali.