During the last ai-ml SIG meeting, we started a conversation around the idea of adding NPU support to Fedora for F43. However, there seems to be some confusion around exactly what “NPU support in Fedora” would look like and what would need to be done in order to get there.
My primary questions are:
Which NPUs are we talking about? It sounds like we’re talking about AMD and Intel right now
What would we be adding NPU support to? pytorch? ollama? blender?
Are we waiting for any code to be accepted in an upstream (e.g kernel)? If so, will this process be done in time for F43?
What new packages would be needed? Is anyone working on them?
Which existing packages would require modification? are the existing maintainers for those packages onboard with the changes required for NPU support?
If you have answers to any of these questions, have more questions or have other comments, please discuss here. The stated plan was to have “NPU Support” as a feature for F43 so there is some time left but not a ton.
Hi folks, while I am not affiliated with Intel, I have been working on packaging the Linux software stack for their “Intel AI Boost” NPU.
For the Intel NPU hardware:
The “Intel AI Boost” NPU is Intel’s current neural inference acceleration hardware. It is integrated into their current generation “Core Ultra” products. Two generations of the hardware are officially supported on Linux: The NPU 3720 in Meteor Lake and Arrow Lake, and NPU 4000 in Lunar Lake.
At the highest level, Intel has AI Plugins that add various features to The GIMP and Audacity. At a lower level, Intel provides an ONNX Runtime EP for OpenVINO, allowing onnx workloads to run on top of OpenVINO as well as Python libraries that enable NPU acceleration of PyTorch workloads though OpenVINO. At a minimum, even if we ignore the other packages, if we ship OpenVINO and the driver stack, users will be able to build software utilizing the NPU via Intel’s supported APIs.
With the Intel NPU, everything should be available in a stable release to package or is already packaged in Fedora. (The one exception is the GIMP plugin, which targets pre-release version 3.0.) The kernel module has gained new features that require newer kernels, but has basic functionality has been supported since Linux 6.6 LTS.
For the Intel NPU, we need new packages for:
a. the driver (which breaks out into the firmware and the Level Zero API) - I have a draft package ready. Firmware license might need a legal review.
b. the compiler-in-driver (which we can build independently of the driver) - I am working on packaging this ball of code; it all gets squished into a single shared object file.
c. Depending on what high-level software we can package, we would also need new packages for the Audacity or GIMP AI plugins, or the ONNX EP, or the Python libraries, and all of their dependencies. - I have poked at packaging these, they seem mostly straightforward if we can get the lower-level bits working.
For existing packages, OpenVINO has already been updated to build the NPU plugin for F42+, thanks to its maintainer being on top of the discussion. We might consider enabling OpenVINO support in other existing packages to take advantage of it. For example, whipser-cpp can be built with an OpenVINO backend, both useful by itself and a dependency of the Audacity AI Plugin. I imagine for those we will need to get maintainer buy-in to add the OpenVINO dependency to the various high-level packages.
How this npu can work in linux system what i know this npu is really basic and can’t even used for running basic 1.5b models locally using npu only so how this npu can help in the current systems I don’t really know any application that have the capability to use the ondevice ai to do anything as such.