"xz" lessons learned: if/how to involve Fedora Magazine in CVE handling?

Project Discussion
, , , , , ,
1

@jflory7 Maybe it makes sense to adjust the Fedora Magazine article once more: the article and the linkedin post already spread the information that testing is NOT enabled by default on F40. And this information already spread to other channels.

I assume that users who already have the false information in mind do not feel corrected if they now read “Fedora Linux 40 Beta users only using stable repositories are NOT impacted”. They might simply come to the conclusion that they are not affected since they never enabled testing manually.

I think you should make clear in the beginning that testing is enabled by default, and unless they changed it themselves, it has to be assumed to be enabled.

With the false information spread already through many channels, I assume some people stop reading after the first section.

@glb I am not sure if you can take care about this? I am not sure if Justin is available at the moment. There is sufficient information about that issue here (above) and in the devel mailing list from @adamwill , @catanzaro and @kevin

Supplement to avoid “double work”: I also send the request to the devel mailing list thread in case some Fedora Magazine people are there

2 Likes
2

Save for trivial things (grammar and spelling mistakes or SEO optimizations), it is highly unusual for a technical correction to be made to an article (something related to the “meaning” of the text or how the “code” operates) without the prior approval of the author whose name is on the article.

It looks like he has been responding to comments. I’m sure he is aware of the issue by this point. Justin can make direct changes to the text of his article. Sorry that I’ve been out and about this weekend and haven’t been keeping a close eye on this issue myself.

It is good that Adam posted a comment to the article pointing out the issue right away. I imagine many people have seen that comment and double checked their systems. The article has mentioned the exact package versions of xz that are impacted from the beginning. I’m hoping/expecting that people are checking those versions to be sure that they have not received an infected copy of xz.

I’m not really comfortable making a significant technical change to such an important article under another user’s name when I don’t have confident and direct knowledge of the details of the problem and, in theory, the author does. Just going by which voices are the “loudest” doesn’t seem right either. I hope you understand. :slightly_smiling_face:

1 Like
3

Of course I understand ! No worries (and no need to be sorry) :slight_smile:

Given the points in the devel mailing list, and the many issues in the communications in between development and the magazine (it seems they mentioned their points already before publishing), I think we need to reconsider if we keep using the magazine for such things in future, or if it maybe makes sense to at least decrease the depth of content and just link to sources that are intended for regular updates. As you said:

Our knowledge in such a situation is evolving slowly and in ways that can be predicted just partly, and distributing information needs to be aligned more closely and in a more “continuous” manner with development/Infra than it is intended for magazine articles.

I think this also can damage the reputation of the magazine, but after the recent posts/information from @catanzaro and @kevin in the mailing list (and the article’s update itself), I saw my update above as the “least invasive” measure, although I didn’t like doing that without consensus with other mods/TL3/4.

I am not so confident that all users who read the first section also read the remaining or even the comments. But I guess we have some stuff to discuss later and to see how we can improve in future.

Given how unexpected this was, the community response was a quite good one, especially at this time of the year.

1 Like
4

This sounds like a very good idea to me. The magazine isn’t designed for a situation where frequent revisions by multiple authors need to be made and it doesn’t track exactly who made such revisions and when. A “wiki” style post on this forum might work better for that sort of thing in the future.

2 Likes
5

I believe the article is amended and addresses the concerns raised by others. If that is not true, let me know.

I understand the concern about whether the Fedora Magazine is the right place for this. Perhaps there could be another place where we maintain the very latest news and updates, although TBH I feel like that should just be the Devel@ mailing list for anyone who absolutely need to follow this play by play, moment by moment.

The Fedora Magazine has one of the greatest reaches of any of our platforms, and it also has the knock-on effect that things we publish on the Magazine are picked up by other global and regional Fedora social media accounts, as well as other Linux/tech news outlets entirely. For this reason, combined with the timely urgency of this notice, the Magazine seemed like the best fit. And we have also done other project-wide announcements through the Magazine as well. I don’t think we should discourage that either?

6

I suggest to review the comments from @adamwill , @catanzaro and @kevin (here and in the devel mailing list) who summed the points up most comprehensibly, and all three are at the source and involved in the issue handling and in the related areas. At first glance, based upon the first section of the article:

→ beta is affected, not just pre-beta (the opposite was added to the article in the update)
→ testing is enabled by default on all including beta (the opposite was spread by the first article version, and it was not devalued by the article’s update: with that in mind, people are likely to read the article update in a way that they are not affected even if they are - in the end, the article and now other channels told them that testing is disabled by default - so, “stable is NOT impacted” sounds good)

I would like to ask you to get in touch with them soon. QA, Infra, workstation developer, I tend to assume they are reliable sources and involved in the handling… Please identify with them how to fix the article if the existing information/posts are not sufficiently reliable in your opinion.

It was already raised if there is the possibility to set the magazine offline to avoid further false information to be spread, and I have to admit that I can understand the underlying frustration of the people who try to handle this issue on a Sunday.

7

I am not sure if that challenges the idea of focusing the content on things that do not change, and only link rather than state information that might change (at least in future we might emphasize such an approach given the development this time).

But as you say, it’s a magazine with impact, and in the mailing list a good point was made about that

That’s the concern many (including me) have atm. You raise the point to make use of the magazine because of its impact, but you don’t question if/how (it is possible) to ensure that the content is corresponding to the situation. The latter seems subordinated to the first. But it should be superordinated. Our handling of CVEs may also contain some marketing thoughts, but it is more than a marketing case.

This is a completely different situation. Please do not compare handling a critical CVE to a marketing case.

Maybe we should open a new topic about that and get a wider discussion about how to handle issues like that in future. Consolidating thoughts, perspectives, alternatives, etc.

8

I take exception to the thinking that the Magazine is a MARKETING tool for the community. It is more than that, and as far as I can see, it may provide a platform for marketing Fedora, but that is but one of it’s functions. Much like the Comm Blog is not marketing either.

1 Like
9

Marketing is to carefully select and tailor information to a target audience. It has the goal to reach as much of the target audience as possible and aims to provide them with the selected information that shall be put in an intended context / interpreted in an intended way. Marketing does not immediately connect with people, which is why its actions are carefully prepared: it is passive means. Once provided, only the outcomes are measured or estimated by some means - adjustments are not intended except if the measurements indicate it to be immediately necessary, and then the adjustments have to be tailored and carefully prepared to avoid further adjustments. This corresponds the means of the Magazine. We might call it today more often “communications” since this term has a less negative reputation, but it actually is the means of marketing. And if I say marketing, I do not mean the term in a negative way - but marketing means are not appropriate for such a situation:

CVE issue handling is as quickly as possible provide necessary, evolving information and keep updating the information in the evolvement of the situation: on one hand, we need to immediately inform people so that they can mitigate the issue based upon best available knowledge (also because potential attackers also get comparable information). On the other hand, the situation is hard to predict and new developments often change past information and thus can change the best possible mitigation. It is more important to be quick (in providing and updating) rather than delimit the target audience, if there is even a target audience since information from us is also relevant for others. We have to ensure that we and the users are always at the peak of the competition of information superiority.

CVE issue handling ends up in many redundancies, and often not everyone is at the same stage, but massive exchange and review make it possible to sufficiently quick respond to information asymmetries. CVE handling means continuous exchange and reciprocal updating.

Fedora Magazine was sufficiently reviewed to find the issue, and we identified the issues in the article early, in both cases, but nevertheless it was not possible to tackle the issue, and as Gregory said, the organization of the magazine does not intend such changes after an article has been published. The Fedora magazine is a great tool (!!!), but as every tool, it is not suitable for everything, and it is not suitable for that.

This is why my suggestion remains to consider its use (in CVE handling it indeed can be a potential tool), but only to make generally aware that there is an issue in a generic way (only add what is for sure) and then do not state content but only link to pages that we can keep updated and that are in the involved in the continuous exchange.

Announcements, such as new Fedora releases, is completely different than CVE handling. Also, it should be verified what went wrong internally, since afaik the information that was partly provided has never been the state of the situation. Communication didn’t work even at the times information was exchanged.

10

Since this has evolved into a separated discussion, I moved the posts to a separated topic to keep both topics focused.

1 Like
11

Added quality-team, rel-eng

12

Added rel-eng

13

Added infrastructure-team

14

A slight addition: you put my comment out of context. the “different situation” referred not to the magazine but to announcements in the magazine.

15

The bug was fixed before Fedora 40 Beta was released, so that’s reasonable.

16

There was a lot of confusion between pre-beta, beta, beta release and final release in the recent days throughout Fedora channels. So I am not sure myself if we are all currently on the same page (and if mine is the correct one : ) But allow me to be a little simplistic (and assume that users are even more likely to end up in confusion than we are):

There are F40 betas out there since 26th March, and the malicious code was reverted later, whereas “pre-beta” indicates that only F40 of the time before any beta received malicious code.

What I read yesterday evening in the mailing list is that it seems the article shall be updated once again since the formulation of the way it currently is was based on the confusion between “beta release” and “final release”. So it ain’t correct, right?

A different formulation is found on the RH article:

… complemented immediately by

So if updated, it got the package, it is still likely to be not vulnerable given Richards actions, but it is still suggested to update to be safe. I tend to assume that this is what we want to tell our users, too?

I assume your argument meant that the malicious code in beta was broken and thus fixed by Richard’s actions?

Anyway, I think at this time it is more about if (or more importantly: how) the magazine is eligible in future for this type of time-critical communication, publishing, continuous exchange and update: so the “lessons learned” from the past days.

17

The malicious code depends on ifuncs, which were not enabled in F40 beta or post-beta. But they were enabled prior to the beta.

18

The Magazine is the correct place to get this info out and also the comm blog, because it is where the community looks for it. There should normally be a link to the actual announcement/bug and associated mitigation/remedy/suggested actions, as part of that link, which would be updated as the project recovered and more is discovered. At least that would leave the article free from post publishing updates on a continuous basis. To be clear as well, updating an article for this type of need would/should be considered acceptable (IMO) since it is a rarity.
I think even novice users understand that newly discovered vulnerabilities take time to resolve and understanding of their ramifications will require testing by experts. The logical next thought is updates will follow.

19

This is why I suggested to provide an article but only provide generic information that is proven to be fixed, and then link to other pages when it comes to details that might need updates.

I have no problem with this, if it works.

But we have seen that this didn’t work, neither the communication about the information that is published, nor the communication that updates are necessary (and what updates).

As glb already said, the magazine’s organization is not intended for that type of publishing and tech changes after publishing.

So if we do that in future, we would need to put efforts into this and evaluate what went wrong, how to solve it, avoid it in future and thus adjust the organization of the magazine / among the magazine and devel/qa/eng/infra.

20

If I might make a suggestion. If readers are unlikely to return to an initial article for the updates, which I suspect might be the case, a follow-up article might be appropriate with links added to the original pointing to the follow-up. It’s a bit cumbersome but more likely to be seen since it is a “new” article.

3 Likes