Rpm-ostree update failing due to error pulling fstatat

Ask Fedora Ask in English
, ,
1

HI, I’ve been having this issue for a week or 2 where I can not update silverblue. rpm-ostree can do some functions, but actual upgrade does not work.

bj@bj-linux ~ [1]> rpm-ostree update
Scanning metadata: 159... done
error: While pulling fedora/36/x86_64/silverblue: fstatat(cd/87ad2b428629f090689019d392f4a93606c85ccbacd9c2c6a0e7a84978def2.dirtree): Input/out

I thought maybe fstatat was a package that my layered packages might rely on, but apparently its a system call in glibc I think? Not sure how to fix it as I can’t rebase or install an old version via rpm-ostree. Somehow my primary and rollback are both affected (which doesn’t make sense as I shouldn’t have been able to upgrade to this last version if rollback version was also affected.

2

I was trying to deploy a random date deployment I found on here just to see if a known good deployment would work and got a similar error.

bj@bj-linux ~ [1]> rpm-ostree deploy 36.20220810.0
Resolving version '36.20220810.0'
1 metadata, 0 content objects fetched; 592 B transferred in 1 seconds; 0 bytes content written
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree 0adbf32... done
error: openat(cd/87ad2b428629f090689019d392f4a93606c85ccbacd9c2c6a0e7a84978def2.dirtree): Input/output error

To me, openat, cd, and fstatat are all things that look for a path, so Im assuming rpm-ostree is looking for a path or file that doesn’t exist on my system for some reason. Not sure how that would happen on immutable system… But definite possibility I screwed something up a long time ago trying to understand Silverblue…

3

Indeed, it looks like it’s having trouble finding something it’s expecting should be there, but the filesystem path doesn’t make much sense to me. I’m curious if you have any errors related to I/O in dmesg?

4

I do see this is dmesg, which seems bad and would explain why random files are missing…

45.117646] BTRFS warning (device sda3): checksum verify failed on 1517420544 wanted 0xfc2ac6fd found 0x032b3332 level 0
[   45.880298] BTRFS warning (device sda3): checksum verify failed on 1517420544 wanted 0xfc2ac6fd found 0x032b3332 level 0
[   45.906865] BTRFS warning (device sda3): checksum verify failed on 1517420544 wanted 0xfc2ac6fd found 0x032b3332 level 0
1 Like
5

If you reboot, it might trigger a fsck on startup that could possibly correct this.

6

I’ve rebooted a few times already and no auto fsck has ran I dont think. I’ll be researching btrfs after work today and seeing the best way to remedy this issue. I do recall a power outage around the time this issue presented itself, so I’m gonna say thats the root cause.

7

If it’s a spinning disk, you might have had a head crash. You might run some tests on that drive with smartctl to verify the sanity of the disk.

8

Start scrub to check: sudo btrfs scrub start -B /

1 Like
9 
bj@bj-linux ~ [1]> sudo btrfs scrub start -B /
[sudo] password for bj: 
scrub done for ec114e5d-ab79-437d-ba96-5a0a55c4452e
Scrub started:    Mon Aug 29 18:02:30 2022
Status:           finished
Duration:         0:11:29
Total to scrub:   353.02GiB
Rate:             511.84MiB/s
Error summary:    csum=4
  Corrected:      0
  Uncorrectable:  4
  Unverified:     0

I had 6 errors before, restored or deleted files as needed, but still have these 4. I think these are probably in the system part of the partition which is why rpm-ostree wont work. Thinking I might just reinstall silverblue and call it a day…

10

Yikes. I recommend running smartctl on that drive since you might have a hardware problem.

11

Haha well smartctl is not installed currently so I don’t think I can layer it in due to rpm-ostree not working. But it is an SSD and all the smart tests look OK. I really think it was the power outage from a few weeks back.

12

You should be about to do it with a toolbox environment.