Lenovo-Fedora world domination plan

The X1 Carbon does NOT support Fedora or Linux of any kind since the Gen 9 fiasco with obscene thermal throttling when Lenovo rewrote the BIOS firmware. Look at other ThinkPads is my advice and DO NOT buy the ones with 11th or 12th gen Intel processors on the X1 Carbon for this reason. Thermal throttling down to 400MHz every 30 seconds under minor load, rendering the computer unusable. The not so amusing part is that this happened on Windows on release of the X1 Carbon with 11th gen processors, and Lenovo supposedly fixed it on Windows after literally months of ignoring their customer base (some users report otherwise, so take that with a grain of salt) but the dptf tables are malformed or something and the fix Lenovo issued doesn’t work on Linux for some reason. This is extensively documented in several Lenovo threads with irate customers wondering why their $2000+ computer is essentially unusable under normal CPU load, particularly on Linux. The X1 Carbon’s firmware is not suitable for running Fedora or any other Linux distro in its current state on 11th and 12th gen processors due to egregious thermal throttling behavior. Don’t say I didn’t warn you if you try to buy one of these.

Hi Joseph,

The US site was bad for getting Linux systems online last year (either Fedora or Ubuntu)
They prefer to have a separate order page for Linux rather than adding it to the general build-your-own page for Windows. This means it is extra work and for various reasons (some reasonable, some not so much) they just didn’t get many Linux platforms online.

Europe is a bit different. There they have taken the approach of just having Linux available as an option of OS alongside Windows. It’s supposed to work so as soon as we enable the Ubuntu/Fedora config the website will make it available - at least that’s my understanding.

There are some downsides - if you choose an invalid config (e.g including MIPI camera) and then change the OS to Linux you just get an error message which isn’t the most user friendly. I’ve asked how last year went for them and if they got complaints.

The other downside is discoverability is pretty poor. I’m asked previously for Linux to be added to the OS selector in the search (just putting Fedora in the search box doesn’t work well) - I’ll send in another nag for that.

Fedora is available on: X1 Carbon, Z13, Z16, P14s AMD G3, P16 AMD G1, P1 G5 and P16

Example on the French site for the Z13: Configure Your PC | Lenovo France

Similar should work for the other platforms and other countries; though looking at the mobile workstations (P-series) those don’t seem to be showing the Linux offerings and I don’t know why. I’ll go and look into that.
I’ve found how the web offerings go up to be full of mystery and there are lots of non-Linux related reasons why systems go up and down and it’s impossible to keep track :frowning:

Mark

Hi Nicholas,

This is going to end up being a long post…and I really don’t want this particular thread to become a thermal thread because people get really hot (ha!) under the collar about this topic…but I’ll put some notes and happy to have the discussion elsewhere (or via email if you have specifics)

You’re correct in that thermal throttling is a challenge in Linux - particularly on the Intel platforms.

What you’re describing is when the system goes into emergency level throttling - the system got too hot, wasn’t cooling fast enough and so the FW heavily throttles the CPU to bring the sytem back down to safe temperatures.

Completely agree it was a challenge on the X1 Carbon 9 - it was also quite problematic on the Carbon 10. I haven’t seen the issues reported on the Carbon 11 and I did some testing myself on that - can you send me the links or details (mpearson-lenovo at squebb dot ca).

Not sure if you’re interested, but I can give some perspective on the challenges.

As a starting note, the Carbon is not a workstation platform - those are bigger and have beefier cooling. If performance is a critical factor then the Carbon might not be the right platform. That being said they are powerful systems and the thermal throttling you’re referencing on the 9 was a valid issue so let’s dig into some of the reasons behind that.

One of the particular challenges for the Linux program is that Intel did not open-source their DPTF thermal framework. It’s why from the Carbon 7 onwards (I think - maybe the 6?) thermal management on Linux has been hard.

I don’t want to throw Intel under the bus for this because their Linux power team are fantastic - but this was an Intel corporate decision and despite significant escalations from us (and I’m guessing other OEMs too) they chose to keep it closed source (this is not the Intel Linux team’s decision).

Our FW team did an implementation that was purely in FW, not requiring the closed source controls that Windows. This was done just for our Linux users. The aim of it was to get the system to behave as closely as possible to Windows.
They did a good job, but there are limitations - it doesn’t have the same fine grained control; it behaves more as a step function with settings - high power/low power/throttle.
There are a lot of variables to control and it for sure took a while to get it right. The thermal team test many particular scenarios but I know we found that some of the manual stress testing Linux users often do highlighted problem areas.
I don’t love the ‘stress all cores for an extended period’ stress tests as they aren’t super realistic of real usage - but we did find video calls in particular, to be particularly problematic scenarios and settings needed adjusting.

Fortunately the DPTF solution has largely (but not entirely) been reverse engineered (kudos to Matthew Garrett for that). You can use thermald in adaptive mode now and I’ve done a bunch of testing with it and it gives a chunk of extra performance. On early platforms it regularly caused thermal shutdown - but I’ve not seen that in a while. It worked well for me on the X1C11.

It is not however officially supported by Intel and that puts us in a bind - the FW team have not been prepared to switch to using it as if there are issues we don’t have any levers with the vendor to get fixes done. We’re investigating this going forward - it’s a really interesting area and I think we’ll see improvements.

Mark

2 Likes

In the case of the French website, that’s good to see. It seems like this maybe the first challenge we address. If there’s not at least a consistent way to get to the Fedora preloaded options, that’s going to be hard to promote. Even if it was something we had to write a guide for, that would still be something we can link to so that people can consistently get to the product. It may have to be done by region it seems.

An option we can try in the meantime is to at last document the exact computers that should support Fedora, even if there’s not a clear way to buy it. This would put Lenovo in the same category as Framework, but at least we would be setting an expectation that certain Lenovo products should work with Fedora if that’s the distro someone is looking for support for.

Here’s the list I currently see from this thread:

  • X1 Carbon, Z13, Z16, P14s AMD G3, P16 AMD G1, P1 G5 and P16
  • Most Thinkpads, all ThinkStation, a bunch of ThinkCenter and all ThinkEdge

If we pulled together a table of all the computers that fall somewhere in these buckets, could you tell us which are good and which are not?

Yeah - this is messy. The dynamic landing page (Buy Linux Laptops Desktops | Ubuntu Laptop | Linux Computer | Lenovo US) was supposed to help with this - but it’s US only (and frankly after last year I find that page depressing - we certify almost 40 different platforms and that’s all that shows up, so I’m … frustrated).

We do have www.lenovo.com/linux which lists all the certified platforms and what they were certified with which is helpful if you want to know which ones you can expect to work well and get support for - but of course doesn’t tell you where you can buy them.

My best case scenario would be a dynamic landing page for each country…but I suspect I won’t be able to get that as it was quite a lot of work the first time (and it’s not perfect…it often pulls in things that aren’t really Linux)
Second best would be so you can at least search by distro (they do already have both Windows variants on there) on the website - and that should be a reasonable ask. I’m seeing if we can get that.

I’m using this discussion for conversation with the Europe web team and it’s been helpful to trigger some things internally…please bear with me a bit while I figure out some pieces.

Mark

2 Likes

This is why I requested early on in the website redesign process to change the way the Workstation site presents getting Fedora Workstation to create a first-class split choice between downloading Fedora Workstation and offering a carousel of computers that have it preloaded to buy. That hasn’t happened yet, though. :slightly_frowning_face:

We desperately need this not just for Lenovo, but for everyone else who preloads Fedora on machines. And when Fedora KDE gets to the point that we do this too, I want this capability too.

1 Like

I came here to suggest this too. Seems like featuring hardware vendors that preload Fedora would catch more attention than simply doing one off articles or podcasts

Wow, that’s an awesome list to see, but how many of those systems were ever offered for sale with Linux after being certified? For example, I’m typing this using a Lenovo T495, which was supposedly certified back in 2019 – Yet at the time I purchased it (Just before Christmas 2019 in the US) I only had a choice of two different Windows flavors. The very first thing I did when unboxing it was to wipe Windows and install Fedora. It would be really nice to be able to tell Lenovo this.

(Meanwhile, every single Linux pre-load Lenovo offers in the US is an Intel platform, representing only a small subset of the still-current models listed on their certified platforms listing. C’mon…)

(Incidentally, a few BIOS updates ago a regression was introduced with respect to USB ports not working across a suspend cycle when a dock is involved. After this happens, you have to completely power-cycle the system to get any USB port to work, even the internal webcam! Additionally, when the battery is nearly depleted, plugging in power won’t charge the battery – you have to unpug and replug it before the battery will start to charge. HTF do I actually report these issues?)

The history of getting systems online is long, messy and confusing :slight_smile:
We had lots of issues with getting the systems online that were mostly process related - those are all now pretty much resolved (it took a long time and we’ve had a couple of snafu’s in manufacturing since, but those are more teething troubles).

Right now the problem is the web teams - and I’m working on that but a lot of it is down to their perception of how much Linux will sell and how much work it is for them. I don’t have a hammer I can use - I have to persuade, convince and (sometimes) nag.

The Europe web team should have a nice approach where if Linux is available it will show up. It doesn’t seem perfect but many systems (including AMD ones) are there. I’m working on them on how we can make Linux easier to find and fix the places where platforms don’t show up.

Yep - I agree. I asked them to put the platforms up - they just didn’t. Sorry :frowning:
The lack of AMD was 100% nothing to do with the Linux team - it blew my mind that the Z-series wasn’t a priority last year for them. I don’t understand it.

This is on the T495? I spent a while investigating an issue with the FW and the USCI driver and I believe that is fixed but let me know if it’s still a problem.

For reporting - general recommendations are:

  • If you have premiere support, use that. They create tickets with the Linux team and we go from there. They sometimes need a bit of hand holding because they don’t deal with much Linux - but there are some awesome support people in that team.

  • Our regular support will usually not help if you didn’t buy the Linux preload - it’s out of their scope. If you did buy the preload then you can use them but generally I’ve been recommending our Linux forums (English Community-Lenovo Community) as you get to short cut to the engineering team.
    One downside there is you may get not useful noise as it’s a community with many opinions - but the Linux section of the forums is one of the most active and there are some great contributions from people outside Lenovo which is pretty cool. I try and monitor every post on there but I’m a bit behind these days - but in most cases someone on the team will pick it up if it’s something we can help with.
    The other downside is it means support gets bypassed so they don’t see as much Linux…and even if it’s a problem exposing other parts of Lenovo to Linux is good. Otherwise we remain under the radar. Shrug.

  • I get tagged on forums/bugzilla/issues/etc all over the place. Including places like here :slight_smile: If it’s something we need to look at I’ll create an internal ticket and then we go from there. It’s not very ‘standard’ but I’m a big believer in the Linux community and engaging directly, so will aim to help wherever I can.

Hope that helps a bit
Mark

3 Likes

This is on the T495? I spent a while investigating an issue with the
FW and the USCI driver and I believe that is fixed but let me know if
it’s still a problem.

Yeah, both of those issues are with the T495 and fully updated Fedora
(currently on F38 with kernel 6.3.x) More specifically:

(1) I don’t recall exactly when the USB regression was introduced, but I
want to say v1.20 or v1.25 of the BIOS. If you’re plugged into the
Thinkpad Ultra Docking station and you disconnect the dock with the
system turned on, the external USB ports stop working. Including the
dock’s ports if you try to re-connect the dock. Internally connected
stuff seems to keep working. FWIW it looks like the external ports are
powered but nothing is able to enumerate.

(2) I don’t recall if the charging bug was always there or not. To
trigger it, the battery has to be very nearly depleted (ie GNOME
presents the “battery critically low, the system will hibernate shortly”
message). If you plug in power, the power LED will blink as usual, but
the charging light next to the USB port doesn’t illuminate. You have to
unplug power and re-plug it (blinking the power LED again) for the
charging light to turn amber and the OS to indicate the system is
plugged in.

Oh, and now that I think of it, (3) – The onboard ethernet doesn’t
detect a link if a cable is not plugged in when the system is powered on
or comes out of suspend. This has always been an issue. The dock port
works fine though.

  • If you have premiere support, use that. They create tickets with the Linux team and we go from there. They sometimes need a bit of hand holding because they don’t deal with much Linux - but there are some awesome support people in that team.

My warranty on this unit expired this past december. But I never had
premier support, and I have to imagine this will go for most folks.

  • Our regular support will usually not help if you didn’t buy the
    Linux preload - it’s out of their scope.

This has been my experience too, but since there’s no way for many/most
folks to actually purchase most Linux-certified Lenovo devices
preloaded with Linux, this becomes a self-defeating situation.

If you did buy the preload then you can use them but generally I’ve
been recommending our Linux forums (English Community-Lenovo Community)
as you get to short cut to the engineering team.

A quick couple of searches shows that none of the issues I’ve seen have
been reported there (indeed, there’s hardly any mention of T495 and
Linux in the same post…)

I’ll see if I can get anyone’s attention there…

  • I get tagged on forums/bugzilla/issues/etc all over the place.
    Including places like here :slight_smile: If it’s something we need to look at
    I’ll create an internal ticket and then we go from there. It’s not
    very ‘standard’ but I’m a big believer in the Linux community and
    engaging directly, so will aim to help wherever I can.

I appreciate you engaging on this stuff. FWIW, It probably does more
for the public perception of non-server Lenovo+Linux than everything
else put together.

  • Solomon

For the sake of keeping things clear, the initiative for getting some kind of representation of supported hardware on the Fedora website is being discussed. Because it includes Lenovo, Slimbook, Framework, and hopefully other future partners, that’s getting peeled off into its own conversation here.

At this point there has been enough interest and confidence from the community to make this Fedora supported web page happen, so we’ll run that as a separate initiative within the Marketing Team in as far as we’re owning it. Based on the conversation in the ticket above, that may change and get organized differently. If you’re interested in helping with that, please track the discussion!

Regarding Lenovo-specific marketing efforts, I would like to have at least a supported list of hardware before we try posting something. There’s a formatting that would be helpful that I can take a crack at building and filling in. That way when we publish marketing efforts like social media posts, articles, or podcast episodes, we have clear documentation on what can be bought with the expectation that it runs Fedora well.

It would be nice to have clearer links on Lenovo’s end or have our own Fedora site to point people to, but those are medium-to-long-term goals right now. If we can take like a month to make this initial list, we have our resource to point people to and then that list can evolve into links to Lenovo or a Fedora landing page. Basically, I would like to get these ducks in a row first, but I don’t want to hold up Lenovo marketing passed the bare minimum we need to confidently share information.

1 Like

Hi @joseph

Supported platforms are all listed on www.lenovo.com/linux - anything on there has completed our Linux process (been thru the QA team, energy certified, preload available, etc).

I think the aim in the other discussion was to provide better ways of finding buyable systems - which is definitely still a problem.

Mark

1 Like

May I suggest auto-expanding those accordions! I didn’t notice them until the 3rd time I visited just now - I thought it was a pretty empty page with no model numbers listed. If it happened to me, it probably happened to others!

Seconding @nikodunk suggestion to make that Lenovo page more readable.

An alternate, concrete, quick (< 5 min, I’d hope) solution to expanding by default is to set the rows with collapse-title class to also have bg-info (or whatever the internet team decides is theme-conforming)


I’m trying to get back to Fedora on the ThinkPad Z13 after a year-long switch to Windows. Hopefully I can submit my own usability misfortunes so they help others.

I already had beef with BitLocker and BIOS. Evidently, updating to the latest BIOS/EC version, resetting everything to default, and clearing all secure boot and encryption keys is not enough to get the Fedora USB to boot—you also need to reset to Setup Mode. It took me a half hour to figure that toggle out.

So far, Fedora 39 freshly installed on the Z13 is quite polished. I don’t have nearly as many hardware complaints as last year… yet. Something related to Displays crashes all the time using an external monitor, and the generated bug reports get auto-rejected for submission because they are “too low value” (something along those lines). I’ll put a proper bug report onto the GNOME or Fedora tracker as well as the Lenovo forums once I get home and recreate the issue.

Mostly, I wish there was a guide for former Marines (“Barney style”, as it’s usually called) that breaks down every. single. tiring. step. “Set line R in BIOS menu 33 to ‘Allow Microsoft CA’”. “Now clear the disk encryption key”. “Reboot”. “Press F12”. and so on. Then again, when I have to make such guides professionally, I get the sense there’s only one customer benefiting, and they aren’t directly paying as much for our product as I get paid while writing the guide. Plus, I understand each customer’s situation is unique, so you can’t blindly say “Enable all peripherals”, “Yes, Control does belong in the corner instead of Fn, you heathens!”, “Set this to maximize stability and battery life. Proper sleep is always more important than max performance!”

I’d also hope one day to discover whichever kernel parameters @mpearson is using. Occasionally, I see Forum posts where someone has entered amdgpu.dcdebugmask=0x10 or acpi_mask_gpe=0x0e gpiolib_acpi.ignore_interrupt=AMDI0030:00@18 and while the regular kernel should eventually have any and all workarounds built-in, there’s a good chance someone at AMD, Lenovo, ELAN, Microsoft, Qualcomm, Realtek, Quectel, or Wacom has already fixed a known problem and it’s just waiting to be merged and tested and put on LVFS or the package manager.


As far as web ordering: Getting a Z13 Fedora preload last year on the German website was super straightforward. The silly “we offer you a flash discount for the next 40 minutes” is a bit predictable and you can get under the Edu pricing if you keep showing up, adding laptops to the cart, and leaving. (In fact, this targeted price discrimination aspect of the website—while understandable and probably praised at business schools worldwide—irritates the hell out of me and almost pushed me off the purchase. I guess its customer behavior metrics worked, because I still landed on the “buy” side of the fence.) Plus—the deciding factor for me—Lenovo has different keyboard layouts for DACH customers, as I’ve never been able to adapt to QWERTZ. A few competing manufacturers will not do a U.S. Int’l keyboard without a phone order (at full retail price, of course) or won’t do it at all.

It does help that the preload is nearly identical to the official recovery ISO (I think 3 documents are added in the home folder) so I don’t have to go through the support website to fresh install Fedora.

1 Like

Hello.
I hope this isn’t my answer, way off topic.
The idea of Fedora coming pre-installed on Lenovo machines is EXCELLENT, and I hope Lenovo generalizes this to all countries… Since here in Brazil, Lenovo installs a distribution that has even been discontinued, called Satux .

1 Like

I’ll leave the details on this one to @mpearson if he wants to explain this one publicly, but we’ve talked to the brazilian Lenovo team, and although the steps they are doing aren’t a lot better, as they’ve just moved from Satux to another internally developed distro called Lux, it’s going to at least be better than the terrible situation it was while they shipped Satux.

1 Like

Is there any >10th gen Lenovo convertible with oled supported by Fedora at the moment? Or any thin & light <1.3kg 13-14" with oled?

I run Silverblue on a HP Spectre x360 13.5" with Intel 10th gen and it works just fine out of the box. Only the keyboard doesn’t work first 10sec after boot or wakeup and the fingerprint is some proprietary thing but it don’t need it.
Compared to other distros, in Fedora 38 2 webcams are listed instead of 1 (I assume it’s the IR sensors for Windows Hello? And by default, any app that uses the webcam selects the incorrect one by default. But that’s a minor thing, there is always a drop down that allows me to select the right one.

I’d love to upgrade to a similarly nice convertible as the Spectre x360 13/14" but then at least with oled screen and preferably a bit lighter. Lenovo is praised for Linux but the focus seems to be on the more traditional looking laptops that they have (reminding me of IBM laptops).

1 Like

Lenovo’s program right now is focused on their business laptops with the ThinkPad line. Here is a list of what Lenovo supports with Fedora and Linux, and you can also check out lenovo.com/linux for the source of that list.

Ultimately it’s Lenovo who decides which programs will get Fedora support and which don’t. The best way to convince them to expand their Linux support is by buying Linux preloads, but of course don’t buy a machine that doesn’t work for you. lol

2 Likes

Sorry for slow reply - catching up on everything after a week off.

@pennrobotics & @nikodunk - thanks for the notes. I have opened a discussion with the web team if we can do something ‘better’ for that page. I agree that people missing the details, which is the primary purpose of that page after all, is not good :slight_smile:

For the Z13 - I am running vanilla Fedora on it with no kernel options. I’ll caveat that with the fact that it’s not my main work machine (couch surfing mostly) but it has been very reliable. It doesn’t see an external monitor often (other than when I’ve been testing) which could be why. I’ve been following the upstream bugs and working with AMD on the GPU driver issues and I’ve seen the amdgpu issues a few times on my Z16, but nowhere near the same level as some people are reporting (I’d go weeks without a problem). I think AMD have made progress on fixing that one so crossing fingers for the 6.6 kernel.

@luca - Agreed on Satux - what a horrible mess. Brazil are now offering ‘Lux’ but I have to highlight that my team have nothing to do with it. It’s a Brazil only project and there are some good pieces and bad pieces about it. They have promised me it won’t be the same as Satux was - but I have concerns about it and I’ve specifically asked them to change the web pages from saying “Linux” to “Lux Linux” (or similar) as I’m really worried about the backlash on the Linux program we have.

On the plus side they are targeting platforms that I don’t get to play with (Ideapad & Legion). I’m optimistically hoping they show those product teams that consumer Linux demand is real and they should invest in supporting Linux fully/correctly. They also have some excellent Linux engineers in Brazil (I’ve been working with some on improving our docking support).

The bits that worry me is that this does not have full FW team support (no LVFS etc), and there is no promise that fixes will go upstream. For me that means it is not a sustainable and correctly delivered Linux project and I, personally, therefore cannot actively support it or recommend.

I hope they succeed because anything that opens Linux doors here is good - but I wish we’d been able to do full and proper Linux support on it as I worry about the experience for users that want to run their choice of distro…which for me is one of the key things about choosing Linux.

@zilexa - yes, for sure there are. For our program for the certified platforms (www.lenovo.com/linux) we aim for full config support so you can choose the model you want. You might find https://psref.lenovo.com/ useful for checking what is supported and where?
As a warning - be very careful as on the X1 Carbon/Yoga the OLED is combined with the MIPI camera (aka ‘computer vision’) and that doesn’t work well with Linux. We’re making progress on that with Intel but it’s still not recommended I’m afraid

Hope I caught everything…
Mark

5 Likes