Battery woes are largely solved by pushing much of the inference off to the NPU rather than relying on the GPU. According Todd Lewellen, the Vice President of Intel’s Client Computing Group ,Microsoft insists Copilot runs on the NPU due to the impact on battery life.

I can install a large language model like Meta’s Llama 2 on my M2 MacBook Air today, but I certainly notice when its thinking or responding as my entire machine freezes until its done. That won’t be acceptable for a mainstream application like Copilot — or a future version of Siri.

What are TOPS and why do they matter?

Swipe to scroll horizontally

TOPS are the number of trillions of operations per second that chip can handle. Specifically it is a measure of arithmetic operations the processor can perform in a second and it particularly useful as a measure of performance in AI and machine learning tasks.

Often a TOPS figure will be given for the chip as a whole, CPU and NPU combined but according to Intel, having a higher NPU value is better for overall performance on LLM tasks.

Lewellen told Tom’s Hardware that reaching the magical 40 TOPS for an NPU would “enable us to run more things locally.” 

Although don’t expect to run all of Copilot on your machine just most of the key capabilities. Extra features like some image generation and editing will likely require an internet connection and access to the cloud services for some time.

“The general trend is to offload as much of it (the AI processes) to endpoint (on-device) as possible,” David Feng, VP of Client Computing Group at Intel told Tom’s Guide at MWC.

When can I run AI on my laptop?

There are already a number of AI apps that run services locally, including image and video editors but they’re mainly using the GPU. Intel is actively working with developers to encourage better use of the NPU, but that might not happen until we get the next gen chips.

In the meantime, Qualcomm is running away with the Windows AI space. Its often delayed Snapdragon X Elite chips measure 45 TOPS for the onboard NPU and run Windows.

Qualcomm CFO Dom McGuire predicted the low TOPS count in the Intel chips would be an issue when they were first unveiled. Speaking at MWC in Barcelona in February he told Tom’s Guide there isn’t much you can do with just a handful of TOPS.

“We can support 70+ TOPs of AI performance across all three of our different processor units,” he explained. 

He added that the Intel AI PCs could only handle about 10 TOPS. “There’s not much you can do from an AI user experience with 10 TOPS.”

Even Apple only tops out at 34 TOPS on its highest end A17 Pro mobile chips and the M2 Ultra desktop chip, but with its tighter integration between operating system and hardware, it is able to make better use of the available processing capability and spread loads across the entire system.

The answer is you can run AI on your laptop today. Most of the time it will use the GPU and you’ll see an impact on battery life or overall system performance. However, in the next year or two as second generation AI PCs and new software comes out — local AI will become the norm.

What is for certain is that you’re going to hear a lot more about TOPS in the coming few years as companies seek to capitalise on the capabilities of the NPU for running complex calculations and AI processes without a major battery life impact.

More from Tom’s Guide

Xem chi tiết và đăng kýXem chi tiết và đăng ký