You’ve probably heard of large language models, the AI systems that generate human-like text. But what about small language models? Researchers are pushing for smaller, more compact AI systems to solve a range of challenges posed by the growing hunger that AI systems have for data. And it’s not just the generative AI models in the news today that need to become smaller, but the AI systems that could operate industrial facilities, smart cities or autonomous vehicles.
The Challenge of Big AI Models
When you use artificial intelligence, whether on your phone or laptop, most of the actual computing occurs in a data center. That’s because the most popular AI models are computationally expensive — your laptop probably doesn’t have enough computing power to run the query. The AI systems also use significant amounts of energy. It’s said that a single query in a generative AI model — asking a question like “How does generative AI work?” — consumes as much electricity as a lightbulb illuminated for one hour.
That presents two challenges for the use of AI. First, it raises concerns about the sustainability of artificial intelligence because the electricity that powers AI also increases greenhouse gas emissions.
In “The Impact of Technology in 2025 and Beyond: an IEEE Global Study,” a recent survey of global technology leaders, 35% said the usefulness of AI greatly outweighs its energy consumption, while 34% said that energy consumption and usefulness of AI are in good balance. About one-fifth (21%) perceived the benefits of AI to be significant, but the high amount of energy used is still a concern, while 8% perceived the massive amount of energy used outweighs the benefits of AI.
Second, it means that anything that relies on artificial intelligence either needs more power to operate or connectivity to a data center.
Cutting-edge techniques to shrink AI are having some success.
“They (techniques) use substantially less power, often operating in the range of watts rather than the kilowatts or megawatts consumed by large data center systems,” said IEEE Member Jay Shah.
Who Needs Compact AI?
Smaller, more energy efficient AI systems could be used in a diverse range of applications, like autonomous vehicles.
“Next-generation, low-power AI accelerators are crucial for the future of autonomous vehicles in terms of long-term reliability and reduced power consumption,” Shah said. “They could enable real-time decision-making and more compact designs.”
They’d be a boon to robotics systems because they would lower power requirements for robots.
IEEE Senior Member Cristiane Agra Pimentel said compact AI systems would also be useful in industrial settings where smaller controls could automate plant processes.
“The use of compact AI in the industrial area will be increasingly applicable to machine operation controls, product traceability controls and supply chain systems management,” Pimentel said.
Small AI Has Tradeoffs
Large language models are often suited for diverse purposes. They can assist in writing college essays and help you build a website. Compact systems could be optimized for specific systems. They could be designed to work as chatbots for a company or autocomplete computer code.
Additionally, compact AI systems are, for now, less accurate because they often use less data.
“These trade-offs are often acceptable,” Shah said, “given the benefits of lower power consumption, faster inference times and the ability to run AI on edge devices. Researchers and developers continue to work on improving the accuracy of compact AI systems while maintaining their efficiency advantages.”