Lenovo-Fedora world domination plan

The X1 Carbon does NOT support Fedora or Linux of any kind since the Gen 9 fiasco with obscene thermal throttling when Lenovo rewrote the BIOS firmware. Look at other ThinkPads is my advice and DO NOT buy the ones with 11th or 12th gen Intel processors on the X1 Carbon for this reason. Thermal throttling down to 400MHz every 30 seconds under minor load, rendering the computer unusable. The not so amusing part is that this happened on Windows on release of the X1 Carbon with 11th gen processors, and Lenovo supposedly fixed it on Windows after literally months of ignoring their customer base (some users report otherwise, so take that with a grain of salt) but the dptf tables are malformed or something and the fix Lenovo issued doesn’t work on Linux for some reason. This is extensively documented in several Lenovo threads with irate customers wondering why their $2000+ computer is essentially unusable under normal CPU load, particularly on Linux. The X1 Carbon’s firmware is not suitable for running Fedora or any other Linux distro in its current state on 11th and 12th gen processors due to egregious thermal throttling behavior. Don’t say I didn’t warn you if you try to buy one of these.

Hi Joseph,

The US site was bad for getting Linux systems online last year (either Fedora or Ubuntu)
They prefer to have a separate order page for Linux rather than adding it to the general build-your-own page for Windows. This means it is extra work and for various reasons (some reasonable, some not so much) they just didn’t get many Linux platforms online.

Europe is a bit different. There they have taken the approach of just having Linux available as an option of OS alongside Windows. It’s supposed to work so as soon as we enable the Ubuntu/Fedora config the website will make it available - at least that’s my understanding.

There are some downsides - if you choose an invalid config (e.g including MIPI camera) and then change the OS to Linux you just get an error message which isn’t the most user friendly. I’ve asked how last year went for them and if they got complaints.

The other downside is discoverability is pretty poor. I’m asked previously for Linux to be added to the OS selector in the search (just putting Fedora in the search box doesn’t work well) - I’ll send in another nag for that.

Fedora is available on: X1 Carbon, Z13, Z16, P14s AMD G3, P16 AMD G1, P1 G5 and P16

Example on the French site for the Z13: Configure Your PC | Lenovo France

Similar should work for the other platforms and other countries; though looking at the mobile workstations (P-series) those don’t seem to be showing the Linux offerings and I don’t know why. I’ll go and look into that.
I’ve found how the web offerings go up to be full of mystery and there are lots of non-Linux related reasons why systems go up and down and it’s impossible to keep track :frowning:

Mark

Hi Nicholas,

This is going to end up being a long post…and I really don’t want this particular thread to become a thermal thread because people get really hot (ha!) under the collar about this topic…but I’ll put some notes and happy to have the discussion elsewhere (or via email if you have specifics)

You’re correct in that thermal throttling is a challenge in Linux - particularly on the Intel platforms.

What you’re describing is when the system goes into emergency level throttling - the system got too hot, wasn’t cooling fast enough and so the FW heavily throttles the CPU to bring the sytem back down to safe temperatures.

Completely agree it was a challenge on the X1 Carbon 9 - it was also quite problematic on the Carbon 10. I haven’t seen the issues reported on the Carbon 11 and I did some testing myself on that - can you send me the links or details (mpearson-lenovo at squebb dot ca).

Not sure if you’re interested, but I can give some perspective on the challenges.

As a starting note, the Carbon is not a workstation platform - those are bigger and have beefier cooling. If performance is a critical factor then the Carbon might not be the right platform. That being said they are powerful systems and the thermal throttling you’re referencing on the 9 was a valid issue so let’s dig into some of the reasons behind that.

One of the particular challenges for the Linux program is that Intel did not open-source their DPTF thermal framework. It’s why from the Carbon 7 onwards (I think - maybe the 6?) thermal management on Linux has been hard.

I don’t want to throw Intel under the bus for this because their Linux power team are fantastic - but this was an Intel corporate decision and despite significant escalations from us (and I’m guessing other OEMs too) they chose to keep it closed source (this is not the Intel Linux team’s decision).

Our FW team did an implementation that was purely in FW, not requiring the closed source controls that Windows. This was done just for our Linux users. The aim of it was to get the system to behave as closely as possible to Windows.
They did a good job, but there are limitations - it doesn’t have the same fine grained control; it behaves more as a step function with settings - high power/low power/throttle.
There are a lot of variables to control and it for sure took a while to get it right. The thermal team test many particular scenarios but I know we found that some of the manual stress testing Linux users often do highlighted problem areas.
I don’t love the ‘stress all cores for an extended period’ stress tests as they aren’t super realistic of real usage - but we did find video calls in particular, to be particularly problematic scenarios and settings needed adjusting.

Fortunately the DPTF solution has largely (but not entirely) been reverse engineered (kudos to Matthew Garrett for that). You can use thermald in adaptive mode now and I’ve done a bunch of testing with it and it gives a chunk of extra performance. On early platforms it regularly caused thermal shutdown - but I’ve not seen that in a while. It worked well for me on the X1C11.

It is not however officially supported by Intel and that puts us in a bind - the FW team have not been prepared to switch to using it as if there are issues we don’t have any levers with the vendor to get fixes done. We’re investigating this going forward - it’s a really interesting area and I think we’ll see improvements.

Mark

2 Likes

In the case of the French website, that’s good to see. It seems like this maybe the first challenge we address. If there’s not at least a consistent way to get to the Fedora preloaded options, that’s going to be hard to promote. Even if it was something we had to write a guide for, that would still be something we can link to so that people can consistently get to the product. It may have to be done by region it seems.

An option we can try in the meantime is to at last document the exact computers that should support Fedora, even if there’s not a clear way to buy it. This would put Lenovo in the same category as Framework, but at least we would be setting an expectation that certain Lenovo products should work with Fedora if that’s the distro someone is looking for support for.

Here’s the list I currently see from this thread:

  • X1 Carbon, Z13, Z16, P14s AMD G3, P16 AMD G1, P1 G5 and P16
  • Most Thinkpads, all ThinkStation, a bunch of ThinkCenter and all ThinkEdge

If we pulled together a table of all the computers that fall somewhere in these buckets, could you tell us which are good and which are not?

Yeah - this is messy. The dynamic landing page (Buy Linux Laptops Desktops | Ubuntu Laptop | Linux Computer | Lenovo US) was supposed to help with this - but it’s US only (and frankly after last year I find that page depressing - we certify almost 40 different platforms and that’s all that shows up, so I’m … frustrated).

We do have www.lenovo.com/linux which lists all the certified platforms and what they were certified with which is helpful if you want to know which ones you can expect to work well and get support for - but of course doesn’t tell you where you can buy them.

My best case scenario would be a dynamic landing page for each country…but I suspect I won’t be able to get that as it was quite a lot of work the first time (and it’s not perfect…it often pulls in things that aren’t really Linux)
Second best would be so you can at least search by distro (they do already have both Windows variants on there) on the website - and that should be a reasonable ask. I’m seeing if we can get that.

I’m using this discussion for conversation with the Europe web team and it’s been helpful to trigger some things internally…please bear with me a bit while I figure out some pieces.

Mark

2 Likes

This is why I requested early on in the website redesign process to change the way the Workstation site presents getting Fedora Workstation to create a first-class split choice between downloading Fedora Workstation and offering a carousel of computers that have it preloaded to buy. That hasn’t happened yet, though. :slightly_frowning_face:

We desperately need this not just for Lenovo, but for everyone else who preloads Fedora on machines. And when Fedora KDE gets to the point that we do this too, I want this capability too.

1 Like

I came here to suggest this too. Seems like featuring hardware vendors that preload Fedora would catch more attention than simply doing one off articles or podcasts

Wow, that’s an awesome list to see, but how many of those systems were ever offered for sale with Linux after being certified? For example, I’m typing this using a Lenovo T495, which was supposedly certified back in 2019 – Yet at the time I purchased it (Just before Christmas 2019 in the US) I only had a choice of two different Windows flavors. The very first thing I did when unboxing it was to wipe Windows and install Fedora. It would be really nice to be able to tell Lenovo this.

(Meanwhile, every single Linux pre-load Lenovo offers in the US is an Intel platform, representing only a small subset of the still-current models listed on their certified platforms listing. C’mon…)

(Incidentally, a few BIOS updates ago a regression was introduced with respect to USB ports not working across a suspend cycle when a dock is involved. After this happens, you have to completely power-cycle the system to get any USB port to work, even the internal webcam! Additionally, when the battery is nearly depleted, plugging in power won’t charge the battery – you have to unpug and replug it before the battery will start to charge. HTF do I actually report these issues?)

The history of getting systems online is long, messy and confusing :slight_smile:
We had lots of issues with getting the systems online that were mostly process related - those are all now pretty much resolved (it took a long time and we’ve had a couple of snafu’s in manufacturing since, but those are more teething troubles).

Right now the problem is the web teams - and I’m working on that but a lot of it is down to their perception of how much Linux will sell and how much work it is for them. I don’t have a hammer I can use - I have to persuade, convince and (sometimes) nag.

The Europe web team should have a nice approach where if Linux is available it will show up. It doesn’t seem perfect but many systems (including AMD ones) are there. I’m working on them on how we can make Linux easier to find and fix the places where platforms don’t show up.

Yep - I agree. I asked them to put the platforms up - they just didn’t. Sorry :frowning:
The lack of AMD was 100% nothing to do with the Linux team - it blew my mind that the Z-series wasn’t a priority last year for them. I don’t understand it.

This is on the T495? I spent a while investigating an issue with the FW and the USCI driver and I believe that is fixed but let me know if it’s still a problem.

For reporting - general recommendations are:

  • If you have premiere support, use that. They create tickets with the Linux team and we go from there. They sometimes need a bit of hand holding because they don’t deal with much Linux - but there are some awesome support people in that team.

  • Our regular support will usually not help if you didn’t buy the Linux preload - it’s out of their scope. If you did buy the preload then you can use them but generally I’ve been recommending our Linux forums (English Community-Lenovo Community) as you get to short cut to the engineering team.
    One downside there is you may get not useful noise as it’s a community with many opinions - but the Linux section of the forums is one of the most active and there are some great contributions from people outside Lenovo which is pretty cool. I try and monitor every post on there but I’m a bit behind these days - but in most cases someone on the team will pick it up if it’s something we can help with.
    The other downside is it means support gets bypassed so they don’t see as much Linux…and even if it’s a problem exposing other parts of Lenovo to Linux is good. Otherwise we remain under the radar. Shrug.

  • I get tagged on forums/bugzilla/issues/etc all over the place. Including places like here :slight_smile: If it’s something we need to look at I’ll create an internal ticket and then we go from there. It’s not very ‘standard’ but I’m a big believer in the Linux community and engaging directly, so will aim to help wherever I can.

Hope that helps a bit
Mark

3 Likes

This is on the T495? I spent a while investigating an issue with the
FW and the USCI driver and I believe that is fixed but let me know if
it’s still a problem.

Yeah, both of those issues are with the T495 and fully updated Fedora
(currently on F38 with kernel 6.3.x) More specifically:

(1) I don’t recall exactly when the USB regression was introduced, but I
want to say v1.20 or v1.25 of the BIOS. If you’re plugged into the
Thinkpad Ultra Docking station and you disconnect the dock with the
system turned on, the external USB ports stop working. Including the
dock’s ports if you try to re-connect the dock. Internally connected
stuff seems to keep working. FWIW it looks like the external ports are
powered but nothing is able to enumerate.

(2) I don’t recall if the charging bug was always there or not. To
trigger it, the battery has to be very nearly depleted (ie GNOME
presents the “battery critically low, the system will hibernate shortly”
message). If you plug in power, the power LED will blink as usual, but
the charging light next to the USB port doesn’t illuminate. You have to
unplug power and re-plug it (blinking the power LED again) for the
charging light to turn amber and the OS to indicate the system is
plugged in.

Oh, and now that I think of it, (3) – The onboard ethernet doesn’t
detect a link if a cable is not plugged in when the system is powered on
or comes out of suspend. This has always been an issue. The dock port
works fine though.

  • If you have premiere support, use that. They create tickets with the Linux team and we go from there. They sometimes need a bit of hand holding because they don’t deal with much Linux - but there are some awesome support people in that team.

My warranty on this unit expired this past december. But I never had
premier support, and I have to imagine this will go for most folks.

  • Our regular support will usually not help if you didn’t buy the
    Linux preload - it’s out of their scope.

This has been my experience too, but since there’s no way for many/most
folks to actually purchase most Linux-certified Lenovo devices
preloaded with Linux, this becomes a self-defeating situation.

If you did buy the preload then you can use them but generally I’ve
been recommending our Linux forums (English Community-Lenovo Community)
as you get to short cut to the engineering team.

A quick couple of searches shows that none of the issues I’ve seen have
been reported there (indeed, there’s hardly any mention of T495 and
Linux in the same post…)

I’ll see if I can get anyone’s attention there…

  • I get tagged on forums/bugzilla/issues/etc all over the place.
    Including places like here :slight_smile: If it’s something we need to look at
    I’ll create an internal ticket and then we go from there. It’s not
    very ‘standard’ but I’m a big believer in the Linux community and
    engaging directly, so will aim to help wherever I can.

I appreciate you engaging on this stuff. FWIW, It probably does more
for the public perception of non-server Lenovo+Linux than everything
else put together.

  • Solomon

For the sake of keeping things clear, the initiative for getting some kind of representation of supported hardware on the Fedora website is being discussed. Because it includes Lenovo, Slimbook, Framework, and hopefully other future partners, that’s getting peeled off into its own conversation here.

At this point there has been enough interest and confidence from the community to make this Fedora supported web page happen, so we’ll run that as a separate initiative within the Marketing Team in as far as we’re owning it. Based on the conversation in the ticket above, that may change and get organized differently. If you’re interested in helping with that, please track the discussion!

Regarding Lenovo-specific marketing efforts, I would like to have at least a supported list of hardware before we try posting something. There’s a formatting that would be helpful that I can take a crack at building and filling in. That way when we publish marketing efforts like social media posts, articles, or podcast episodes, we have clear documentation on what can be bought with the expectation that it runs Fedora well.

It would be nice to have clearer links on Lenovo’s end or have our own Fedora site to point people to, but those are medium-to-long-term goals right now. If we can take like a month to make this initial list, we have our resource to point people to and then that list can evolve into links to Lenovo or a Fedora landing page. Basically, I would like to get these ducks in a row first, but I don’t want to hold up Lenovo marketing passed the bare minimum we need to confidently share information.

1 Like

Hi @joseph

Supported platforms are all listed on www.lenovo.com/linux - anything on there has completed our Linux process (been thru the QA team, energy certified, preload available, etc).

I think the aim in the other discussion was to provide better ways of finding buyable systems - which is definitely still a problem.

Mark

1 Like

May I suggest auto-expanding those accordions! I didn’t notice them until the 3rd time I visited just now - I thought it was a pretty empty page with no model numbers listed. If it happened to me, it probably happened to others!