Figuring out NPU support in Fedora

Project Discussion
2

Hi folks, while I am not affiliated with Intel, I have been working on packaging the Linux software stack for their “Intel AI Boost” NPU.

For the Intel NPU hardware:

  1. The “Intel AI Boost” NPU is Intel’s current neural inference acceleration hardware. It is integrated into their current generation “Core Ultra” products. Two generations of the hardware are officially supported on Linux: The NPU 3720 in Meteor Lake and Arrow Lake, and NPU 4000 in Lunar Lake.
  2. At the highest level, Intel has AI Plugins that add various features to The GIMP and Audacity. At a lower level, Intel provides an ONNX Runtime EP for OpenVINO, allowing onnx workloads to run on top of OpenVINO as well as Python libraries that enable NPU acceleration of PyTorch workloads though OpenVINO. At a minimum, even if we ignore the other packages, if we ship OpenVINO and the driver stack, users will be able to build software utilizing the NPU via Intel’s supported APIs.
  3. With the Intel NPU, everything should be available in a stable release to package or is already packaged in Fedora. (The one exception is the GIMP plugin, which targets pre-release version 3.0.) The kernel module has gained new features that require newer kernels, but has basic functionality has been supported since Linux 6.6 LTS.
  4. For the Intel NPU, we need new packages for:
    a. the driver (which breaks out into the firmware and the Level Zero API) - I have a draft package ready. Firmware license might need a legal review.
    b. the compiler-in-driver (which we can build independently of the driver) - I am working on packaging this ball of code; it all gets squished into a single shared object file.
    c. Depending on what high-level software we can package, we would also need new packages for the Audacity or GIMP AI plugins, or the ONNX EP, or the Python libraries, and all of their dependencies. - I have poked at packaging these, they seem mostly straightforward if we can get the lower-level bits working.
  5. For existing packages, OpenVINO has already been updated to build the NPU plugin for F42+, thanks to its maintainer being on top of the discussion. We might consider enabling OpenVINO support in other existing packages to take advantage of it. For example, whipser-cpp can be built with an OpenVINO backend, both useful by itself and a dependency of the Audacity AI Plugin. I imagine for those we will need to get maintainer buy-in to add the OpenVINO dependency to the various high-level packages.