When people think of AI, they picture the huge models behind the famous chatbots. Those are impressive, and they're also expensive to run, and for plenty of everyday jobs they're more than you need. That's why small language models have become a serious option.
What "small" means
A language model's size is measured in parameters, roughly the number of adjustable settings it learned during training. The largest models have hundreds of billions or more. Small ones have from a few hundred million to a few billion, small enough to run on a single decent computer, and some on a phone.
There's no official line between small and large, and the definitions keep moving as hardware improves. What matters is the practical difference: a small model fits on modest hardware and answers quickly.
What they're good at
Narrow, well-defined jobs. Sorting incoming emails into categories. Pulling names, dates and amounts out of invoices. Tagging support tickets. Rewriting text in a set style. Short summaries. Simple question answering over your own documents.
For this kind of work, a small model that's been fitted to your task, often with a modest amount of your own examples, can match or even beat a general large model on that specific job. It's like hiring a specialist rather than a brilliant generalist.
Why you might want one
Cost. Running a small model takes far less computing power, which matters when you process thousands of documents a day. The savings can be large at volume.
Speed. Smaller models respond faster, which matters for anything interactive like autocomplete, voice assistants or live translation.
Privacy. Because they can run on your own server, or even on the user's device, sensitive data doesn't have to go to an outside provider. For healthcare, legal and finance work, that can simplify compliance considerably.
Offline use. A model on the device works without internet. That's useful for field work and for apps used in areas with poor connectivity.
Control. You choose when the model changes. A hosted service can update or retire a model under you.
Where they fall short
They're weaker at open-ended reasoning, long and complicated instructions, writing that needs real nuance, and broad general knowledge. Ask a small model something far outside its comfort zone and it will stumble more often, and sound just as confident while doing it.
They also still make things up. Smaller doesn't mean more accurate. Grounding the model in your own documents, as we discussed in our piece on RAG, matters just as much.
Running your own model brings its own work too: setting up hardware, keeping it updated, monitoring quality. A hosted large model is simpler if your volume is low.
A sensible approach
Many teams end up using both. A small model handles the high-volume, predictable tasks. A large one steps in for the hard, rare ones. Sometimes a simple router decides which gets each request.
To find out what works for you, take a real sample of your task, perhaps a few hundred examples with correct answers, and test a large model and a couple of small ones on it. Compare quality, speed and cost on your actual data, not on a leaderboard. The results are often a surprise.
Our take
Start with whatever gets you a working prototype fastest, which is usually a large hosted model. Once you know exactly what the task is and the volume justifies it, test whether a smaller model can do the same job for less. Treat model size as a cost decision, not a status symbol.
