Analogue chips can slash the energy used to run AI models

An analogue computer chip can run an artificial intelligence (AI) speech recognition model 14 times more efficiently than traditional chips, potentially offering a solution to the vast and growing energy use of AI research and to the worldwide shortage of the digital chips usually used.

The device was developed by IBM Research, which declined New Scientist’s request for an interview and didn’t provide any comment. But in a paper outlining the work, researchers claim that the analogue chip can reduce bottlenecks in AI development.

There is a global rush for GPU chips, the graphic processors that were originally designed to run video games and have also traditionally been used to train and run AI models, with demand outstripping supply. Studies have also shown that the energy use of AI is rapidly growing, rising 100-fold from 2012 to 2021, with most of that energy derived from fossil fuels. These issues have led to suggestions that the constantly increasing scale of AI models will soon reach an impasse.

Another problem with current AI hardware is that it must shuttle data back and forth from memory to processors in operations that cause significant bottlenecks. One solution to this is the analogue compute-in-memory (CiM) chip that performs calculations directly within its own memory, which IBM has now demonstrated at scale.

IBM’s device contains 35 million so-called phase-change memory cells – a form of CiM – that can be set to one of two states, like transistors in computer chips, but also to varying degrees between them.

This last trait is crucial because these varied states can be used to represent the synaptic weights between artificial neurons in a neural network, a type of AI that models the way that links between neurons in human brains vary in strength when learning new information or skills, something that is traditionally stored as a digital value in computer memory. This allows the new chip to store and process these weights without making millions of operations to recall or store data in distant memory chips.

In tests on speech recognition tasks, the chip showed an efficiency of 12.4 trillion operations per second per watt. This is up to 14 times more efficient than conventional processors.

Hechen Wang at tech firm Intel says the chip is “far from a mature product”, but experiments have shown it can work effectively on today’s commonly used forms of AI neural network – two of the best-known examples are called CNN and RNN – and has the potential to support popular applications such as ChatGPT.

“Highly customised chips can provide unparalleled efficiency. However, this has the consequence of sacrificing feasibility,” says Wang. “Just as a GPU cannot cover all the tasks a CPU [a standard computer processor] can perform, similarly, an analogue-AI chip, or analogue compute-in-memory chip, has its limitations. But if the trend of AI can continue and follow the current trend, highly customised chips can definitely become more common.”

Wang says that although the chip is specialised, it could have uses outside the speech recognition task used by IBM in its experiments. “As long as people are still using a CNN or RNN, it won’t be completely useless or e-waste,” he says. “And, as demonstrated, analogue-AI, or analogue compute-in-memory, has a higher power and silicon usage efficiency, which can potentially lower the cost compared to CPUs or GPUs.”