From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents

In plain English

Most AI runs on distant servers. This talk is about AI that runs on the phone itself, with no internet needed. The speaker works on Google’s tools for this and shows two things. First, a small model can use “skills”, short packaged instructions it loads only when needed, to do useful tasks.

Second, very small models (under a billion parameters, a rough measure of size) are weak out of the box but get far better when trained on one narrow job. In one example, a model that picked the right phone action about 46% of the time went above 90% for most actions.

Three ideas worth keeping

• A small model can handle a handful of skills, but the speaker admits chaining several skills in one answer only works sometimes.
• Tiny models need a narrow, focused task, and training them on made-up example data can lift them a long way.
• Running on the device means the app does not grow much, works offline and keeps data on the phone.

What it means for your business

If you want AI inside a mobile app or a tool used in the field, you may not need a big cloud service. A narrow job, such as turning a spoken request into a calendar entry, is the sweet spot. Ask any supplier what the accuracy is today and how it was measured.

What it means for software teams

Pick the smallest model that fits one clear task, then fine-tune it on synthetic examples before reaching for a bigger one. The speaker says eight of ten functions passed 90% after tuning and two stayed in the 80s, so test each function separately. Skills are loaded on demand, which keeps the prompt short.

What we found, and what we did

We gave a small, compressed model extra training (fine-tuning) for one narrow job: choosing the right tool for a request. Compressing it the simple way after training dropped its accuracy to about 90%, so we trained it with the compression already in place, and it kept 97.8% accuracy at about 88 tokens (word pieces) a second.

We also found its limit: a small model on the device wasn’t reliable enough to act as a hard code-review check when many agents were busy at once. We turned it back into an adviser and later moved that job to a cloud reviewer we had measured.

The full story is in From 17% to 97.8%: Making a Laptop-Sized AI Actually Reliable, and we have not tested the Google tools shown here.

Worth watching

• 07:46: how a “skill” is loaded on demand instead of stuffing everything into the prompt.
• 14:19: the 46% starting point and the jump after fine-tuning on synthetic data.
• 17:52: the honest answer on how many skills a small model can juggle.