My daily runner, a desktop machine running Silverblue 40, has accumulated some - I presume - harmless btrfs errors and warnings - that do not affect my daily usage, but I’d like to understand them better and ideally resolve them to avoid eventual breakage.
The system is running kernel 6.11.0 and has two partitions of interest sda1 and sdc3.
1). sdc3 reports:
$ sudo btrfsck --readonly --force /dev/sdc3
[...]
[4/7] checking fs roots
root 257 inode 80864507 errors 1, no inode item
unresolved ref dir 16257806 index 1406 namelen 36 name d5977089-b9dc-4c60-8b35-05cbd7ae7811 filetype 2 errors 5, no dir item, no inode ref
unresolved ref dir 16257806 index 1406 namelen 36 name b63712bd-95de-41fc-b1ad-619dbc843548 filetype 2 errors 2, no dir index
root 257 inode 89736594 errors 1, no inode item
unresolved ref dir 16257806 index 1510 namelen 36 name de482c0c-a592-4dbf-8fd3-09b8a2d0367f filetype 2 errors 5, no dir item, no inode ref
unresolved ref dir 16257806 index 1510 namelen 36 name 4855e2ba-2d7f-4558-af9f-d57d916020d5 filetype 2 errors 2, no dir index
2). sda1 reports:
$ sudo btrfsck --readonly --force /dev/sda1
[...]
[3/7] checking free space tree
free space info recorded 2 extents, counted 1
wanted bytes 3133440, found 3108864 for off 326444789760
cache appears valid but isn't 325374181376
free space info recorded 3 extents, counted 2
wanted bytes 120512512, found 120500224 for off 333829898240
cache appears valid but isn't 332890374144
free space info recorded 4 extents, counted 3
wanted bytes 1642496, found 1626112 for off 338245427200
cache appears valid but isn't 337185341440
wanted bytes 4288512, found 4280320 for off 341476020224
cache appears valid but isn't 340406566912
free space info recorded 6 extents, counted 5
wanted bytes 153001984, found 11952128 for off 372465819648
cache appears valid but isn't 371545079808
[more of the same]
3). the kernel reports on startup (the same is NOT reported for sda):
BTRFS warning (device sdc3): space cache v1 is being deprecated and will be removed in a future release, please use -o space_cache=v2
For the warnings of issue 1) and 2) there are mentions of using btrfs-check --repair but the documentation also mentions that there is potential harm in this command and a expert/maintainer should look over the errors first.
sda1 had two directories written to using the WinBtrfs driver on a Windows 10 machine. That’s when the “off by one” extent errors appeared. I’ve deleted the directories since then but the extent errors remain.
I’ve disabled the monthly scrub of btrfsmaintainence but did a single time scrub a few months back in hopes it would solve some of the warnings.
I’ve not tried a btrfs-check --repair
How old is that drive, how much data (in %) is on it?
The drives are respectively 3 years (sdc) and 1 year (sda) old. They’ve not shown any other issues that could hint at a hardware failure. They are at 70% capacity most of the time.
This is not going to produce reliable results on an active root file system. You can reboot with boot parameter rd.break and run it from a dracut shell. Or you can boot from a current Fedora Live USB and run it on the unmounted root file system.
Good point. I re-did a btrfsck from a Fedora 40 Live USB with sda1 and sdc3 staying unmounted. Full output below.
$ sudo btrfsck /dev/sda1
Opening filesystem to check...
Checking filesystem on /dev/sda1
UUID: 165a6aaf-9b2b-4258-9136-852a0483c107
[1/7] checking root items
[2/7] checking extents
super bytes used 2455000494080 mismatches actual used 2450980777984
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
free space info recorded 2 extents, counted 1
wanted bytes 3133440, found 3108864 for off 326444789760
cache appears valid but isn't 325374181376
free space info recorded 3 extents, counted 2
wanted bytes 120512512, found 120500224 for off 333829898240
cache appears valid but isn't 332890374144
free space info recorded 4 extents, counted 3
wanted bytes 1642496, found 1626112 for off 338245427200
cache appears valid but isn't 337185341440
wanted bytes 4288512, found 4280320 for off 341476020224
cache appears valid but isn't 340406566912
free space info recorded 6 extents, counted 5
wanted bytes 153001984, found 11952128 for off 372465819648
cache appears valid but isn't 371545079808
free space info recorded 3 extents, counted 2
wanted bytes 347402240, found 248090624 for off 404483674112
cache appears valid but isn't 403757334528
free space info recorded 5 extents, counted 4
there is no free space entry for 447780630528-447780749312
cache appears valid but isn't 446707007488
free space info recorded 18 extents, counted 17
there is no free space entry for 719437352960-719437430784
cache appears valid but isn't 718363688960
free space info recorded 14 extents, counted 13
wanted bytes 343552000, found 343478272 for off 768486002688
cache appears valid but isn't 767755812864
free space info recorded 302 extents, counted 301
wanted bytes 147456, found 24576 for off 777902735360
cache appears valid but isn't 777419489280
free space info recorded 5 extents, counted 4
there is no free space entry for 783861850112-783861940224
cache appears valid but isn't 782788198400
free space info recorded 34 extents, counted 33
there is no free space entry for 1073772122112-1073772232704
cache appears valid but isn't 1072698490880
free space info recorded 3 extents, counted 2
there is no free space entry for 1083416346624-1083416371200
cache appears valid but isn't 1082362167296
free space info recorded 3 extents, counted 2
wanted bytes 1093632, found 1048576 for off 1089877266432
cache appears valid but isn't 1088804618240
free space info recorded 3 extents, counted 2
wanted bytes 7991296, found 7925760 for off 1097364758528
cache appears valid but isn't 1096320811008
free space info recorded 3 extents, counted 2
wanted bytes 1777664, found 1769472 for off 1115646386176
cache appears valid but isn't 1114574422016
free space info recorded 3 extents, counted 2
wanted bytes 22716416, found 22601728 for off 1118846672896
cache appears valid but isn't 1117795647488
free space info recorded 7 extents, counted 6
wanted bytes 2793472, found 2703360 for off 1121014079488
cache appears valid but isn't 1119943131136
free space info recorded 4 extents, counted 3
wanted bytes 92401664, found 92319744 for off 1524651397120
cache appears valid but isn't 1523670056960
free space info recorded 3 extents, counted 2
wanted bytes 8331264, found 8318976 for off 1566611398656
cache appears valid but isn't 1565545988096
free space info recorded 4 extents, counted 3
wanted bytes 240275456, found 240263168 for off 1637246414848
cache appears valid but isn't 1636412948480
free space info recorded 6 extents, counted 5
wanted bytes 56123392, found 55844864 for off 1638827782144
cache appears valid but isn't 1638560432128
free space info recorded 7 extents, counted 6
wanted bytes 9465856, found 9281536 for off 1716531466240
cache appears valid but isn't 1715869843456
free space info recorded 3 extents, counted 2
wanted bytes 30949376, found 30384128 for off 1773820952576
cache appears valid but isn't 1772778160128
free space info recorded 3 extents, counted 2
wanted bytes 156418048, found 21913600 for off 1793022836736
cache appears valid but isn't 1792105512960
free space info recorded 7 extents, counted 6
there is no free space entry for 1837202628608-1837202669568
cache appears valid but isn't 1836128927744
free space info recorded 20 extents, counted 19
there is no free space entry for 1842571300864-1842571378688
cache appears valid but isn't 1841497636864
free space info recorded 88 extents, counted 87
there is no free space entry for 1846866321408-1846866345984
cache appears valid but isn't 1845792604160
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 2450980777984 bytes used, error(s) found
total csum bytes: 2389607276
total tree bytes: 4022927360
total fs tree bytes: 1246511104
total extent tree bytes: 274087936
btree space waste bytes: 288642516
file data blocks allocated: 10820840423424
referenced 2604946694144
$ sudo btrfsck /dev/sdc3
Opening filesystem to check...
Checking filesystem on /dev/sdc3
UUID: 03ed6e54-7fed-49c7-b838-03fdabea71df
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 80864507 errors 1, no inode item
unresolved ref dir 16257806 index 1406 namelen 36 name d5977089-b9dc-4c60-8b35-05cbd7ae7811 filetype 2 errors 5, no dir item, no inode ref
unresolved ref dir 16257806 index 1406 namelen 36 name b63712bd-95de-41fc-b1ad-619dbc843548 filetype 2 errors 2, no dir index
root 257 inode 89736594 errors 1, no inode item
unresolved ref dir 16257806 index 1510 namelen 36 name de482c0c-a592-4dbf-8fd3-09b8a2d0367f filetype 2 errors 5, no dir item, no inode ref
unresolved ref dir 16257806 index 1510 namelen 36 name 4855e2ba-2d7f-4558-af9f-d57d916020d5 filetype 2 errors 2, no dir index
ERROR: errors found in fs roots
found 824102596608 bytes used, error(s) found
total csum bytes: 801360456
total tree bytes: 3179642880
total fs tree bytes: 1921187840
total extent tree bytes: 263077888
btree space waste bytes: 741305402
file data blocks allocated: 1997861789696
referenced 893898280960
I’m not sure what happened here, but it’s not small, about 3.7GiB, if my math is right. We should ask upstream.
cache appears valid but isn’t 325374181376
At first glance it seems the thing to do is remove space cache v1 using the btrfs check option for clearing it. And then mount one time using the space cache v2 option, which will set a persistent feature flag, meaning you do not need to make add this mount option anywhere. You’ll just get space cache v2. Except for the small problem that the free space btree is the opposite of the extent btree. Free vs used. Now, if there is a disagreement between the super and extent tree on space, we should probably figure out the proper order of fixing these things.
Still another option is just back up everything and clean install to a new file system. Like, that’s probably faster than pinging upstream and asking them what’s going on, and how to fix it. But maybe there’s a bug needing fixed so if it were me, I’d report it and wait until I get a response. But if you’re OK with a backup, reinstall, restore from backup routine, then you can do that.
/dev/sdc3
I’m not sure what caused this but both copies (DUP metadata) have the same problem. I’m think btrfs check now reliability fixes both error 2 and 5. But I still recommend a backup just in case things go sideways. Also, I don’t think these are urgent problems, therefore you can delay the repair, i.e. they aren’t “contagious” errors that will spread.
Thanks for the analysis Chris. As I do not see performance implications to these warnings and some of these warnings only being caused by using WinBtrfs in the past I’d not bring it to the mailing list to avoid potential waste. I’ll recreate the filesystems on next occassion (either distro hop or new drives).