F41 Change Proposal: IPU6 camera support (self-contained)

Project Discussion Change Proposals
,
1

IPU6 camera support

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Integrate support into Fedora for Intel IPU6 attached MIPI cameras using the IPU6 CSI-receiver (isys) driver which has landed in kernel 6.10 together with libcamwera’s 0.3 software ISP support and Firefox’ recent support for using cameras through pipewire.

:link: Owner

:link: Detailed Description

Many new laptops models have a camera-sensor which is directly attached to the laptops CPU/SoC over a MIPI CSI2 databus instead of using a USB webcam module talking the standard USB UVC protocol.

These cameras require a lot of work on the software side to go from the raw Bayer data received over the CSI2 bus to an usable image. This includes both controlling things like exposure and gain settings on the sensor as well as a lot of processing of the raw data, such as debayering and whitebalancing. Adding support for these complex cameras is tricky because:

  • Applications can no longer directly use /dev/video now
  • Supporting ISPs (if supported in the kernel) requires ISP model specific knowledge in userspace
  • Vendor’s 3A algorithms for auto-exposure/gain, auto-whitebalance and auto-focus are secret and need to have open-source counterparts written and tuned
  • Instead of having a single UVC driver this requires CSI-receiver + ISP + sensor drivers in the kernel
  • Different IPU6 laptop models use different sensors, hw-enablement needs to be done on a laptop by laptop basis
  • Good image quality requires per sensor/laptop model tuning

Parts of these challenges are solved by libcamera. For now the aim is a simple stack with good enough image quality for video-conferencing. The plan is to have a stack consisting of:

  1. Mainline kernel sensor driver (currently supported: ov2740, ov01a10, hi556)
  2. Mainline kernel IPU6 CSI receiver driver
  3. libcamera simplepipeline-handler using software ISP for debayering + 3A
  4. pipewire with pipewire libcamera plugin
  5. pipewire support in Firefox (see Jan Grulich’s blog)

:link: Feedback

No feedback yet.

:link: Benefit to Fedora

Currently IPU6 cameras do not work OOTB in Fedora, with the new libcamera software ISP stack these should work OOTB on laptops with supported camera sensors.

:link: Scope

  • Proposal owners:
    • The IPU6 CSI receiver has landed in 6.10 and some bugfixes coming to 6.11 have been added as downstream patches
    • libcamera needs a couple of small downstream-patches to enable the simple pipeline for the IPU6
    • pipewire libcamera plugin needs to be made part of the default Workstation patch-set
  • Other developers: N/A (not a System Wide Change)
  • Release engineering: N/A (not a System Wide Change)
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with the Fedora Strategy: Yes (better hw-support should help getting more users)

:link: Upgrade/compatibility impact

The pipewire-plugin-libcamera needs to be automatically added on Fedora workstation updates to ensure things work. Otherwise there is no upgrade impact.

:link: How To Test

Test plan will be filled in as soon as all necessary bits have landed in rawhide.

:link: User Experience

IPU6 cameras on laptops with supported camera sensors should work OOTB after this change, with the caveat that the image quality may be less then ideal. The hope is that image quality will improve over time as the software ISP and its 3A algorithms get improved. With that said it is unrealistic to expect the image quality to become as good as the proprietary stack which has extensive image quality tuning done on a per laptop basis.

:link: Dependencies

IPU6 support not only depends on kernel and libcamera support taken care of by the proposal owner, but also on pipewire camera support and on Firefox pipewire camera support.

:link: Contingency Plan

ATM IPU6 cameras do not work at all. So unless the new kernel driver actually causes regressions outside of the camera functionality no contingency plan is necessary.

:link: Documentation

N/A (not a System Wide Change)

:link: Release Notes

Fedora 41 has added support for IPU6 cameras on laptops using ov2740, ov01a10 and hi556 sensors. This support requires using applications which support accessing cameras through pipewire such as Firefox.

Last edited by @amoloney 2024-07-01T21:02:08Z

3 Likes
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply giving that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

This change proposal has now been submitted to FESCo with ticket #3243 for voting.

To find out more, please visit our Changes Policy documentation.

4

This change has been accepted by FESCo for Fedora Linux 41. A full list of approved changes to date can be found on the Change Set Page.

To find out more about how our changes policy works, please visit our docs site.

5

If I understand correctly, this would mean Fedora users would be affected by this issue, right? If so, that should probably be considered a blocker for this.
I’d rather not have my laptop battery lifetime halved, after all… especially considering I would not currently benefit from this change.

6

If I understand correctly, this would mean Fedora users would be affected by this issue https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2669, right? If so, that should probably be considered a blocker for this.
I’d rather not have my laptop battery lifetime halved, after all… especially considering I would not currently benefit from this change.

Correct, I have a small list of pipewire camera support issues which I need and plan to resolve before F41 ships and that bug us on that list.

Note that by default when using wireplumber as session manager it will use the always installed v4l2 pipewire plugin for UVC cameras though and AFAICT this is only an issue when using the libcamera plugin. So I think this bug may already be resolved by wireplumber preferring the v4l2 plugin for UVC cameras, but I need to investigate this further.

2 Likes