Installing/updating flatpak apps, home and root mounts become read-only

When Installing/updating flatpak apps, home and root mounts immediately become read-only. This happens with any app install or update.

for example, running:

flatpak update com.slack.Slack

my home directory becomes read-only and the system quickly becomes unusable. If I save the output of mount before and running flatpak as above, I receive a diff similar to this:

< /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on / type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=256,subvol=/root)
---
> /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on / type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=256,subvol=/root)
22c22
< /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on /home type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=257,subvol=/home)
---
> /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on /home type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=257,subvol=/home)
51c51
< /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on /run/timeshift/backup type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=5,subvol=/)
---
> /dev/mapper/luks-c7792035-1f6a-4a92-8bcf-4cccf9fd8755 on /run/timeshift/backup type btrfs (ro,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=5,subvol=/)

Notice the mount point modes changing from rw to ro in the above diff after running the flatpak command.

The output of running flatpak update -vvv com.slack.Slack can be found below. I can’t see anything that alludes to a possible next place to look at why this might be happening. The only errors I see here are relating to it encountering read-only locations.

I’m running Fedora 36 (Linux 5.19.16-200.fc36.x86_64)

I’ve tried running sudo restorecon -vR / in order to see if correctly labelling everything for selinux helps but that doesn’t stop this observed behaviour.

So at this point I’m looking for suggestions of next steps in trying to resolve this issue.


[~] flatpak update -vvv com.slack.Slack                                                                       9:16:41 
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/will/.local/share/flatpak
Looking for updates…
F: Fetching summary index file for remote ‘flathub’
F: Loading https://dl.flathub.org/repo/summary.idx using libsoup
F: Received 2737 bytes
F: Loaded indexed summary file 6a3e8c02f7f0f88ffd9e085e26ed9ae1dc44b84c1d16ec9593fe7f6c8f3bc60c from cache for remote ‘flathub’
F: Transaction: update flathub:app/com.slack.Slack/x86_64/stable[*]
F: Looking for remote metadata updates for flathub
F: marking op update:app/com.slack.Slack/x86_64/stable resolved to cb44e08354a5dcb8b5799a0d2c05ca2470d1e2998e45e816afd21a2165ecc187
F: Calling system helper: GenerateOciSummary
Required runtime for com.slack.Slack/x86_64/stable (runtime/org.freedesktop.Platform/x86_64/22.08) found in remote flathub
Do you want to install it? [Y/n]: 
F: Transaction: install/update flathub:runtime/org.freedesktop.Platform/x86_64/22.08[$old]
F: marking op install/update:runtime/org.freedesktop.Platform/x86_64/22.08 resolved to 253961492abde2982b4c8b9b1e7d45acd72e08e8c84ee7bdc45b491e966072ca
F: Transaction: install/update flathub:runtime/org.freedesktop.Platform.GL.default/x86_64/22.08[$old]
F: Transaction: install/update flathub:runtime/org.freedesktop.Platform.Locale/x86_64/22.08[/en]
F: Transaction: install/update flathub:runtime/org.freedesktop.Platform.VAAPI.Intel/x86_64/22.08[$old]
F: Transaction: install/update flathub:runtime/org.freedesktop.Platform.openh264/x86_64/2.2.0[$old]
F: marking op install/update:runtime/org.freedesktop.Platform.openh264/x86_64/2.2.0 resolved to 435b1135f46af01c742f7680e92a2ca2b63ef7f5cb6b75030caff3cdd80481a9
F: marking op install/update:runtime/org.freedesktop.Platform.VAAPI.Intel/x86_64/22.08 resolved to cb45563d49f0dacf53d4512e60350a286ff20838abc96150b99c425cfd037040
F: marking op install/update:runtime/org.freedesktop.Platform.Locale/x86_64/22.08 resolved to 6db488be45f93406a265d5d8cb294b5ae492c8041f964e8f545702a11e230f46
F: marking op install/update:runtime/org.freedesktop.Platform.GL.default/x86_64/22.08 resolved to 6062874bd1737b867b0c9a538f5f77b454ba1d05365192165136322e8a20959b


 1.	  	org.freedesktop.Platform.GL.default	22.08	i	flathub	< 131.0 MB
 2.	  	org.freedesktop.Platform.Locale	22.08	i	flathub	< 332.9 MB (partial)
 3.	  	org.freedesktop.Platform.VAAPI.Intel	22.08	i	flathub	< 10.8 MB
 4.	  	org.freedesktop.Platform.openh264	2.2.0	i	flathub	< 944.3 kB
 5.	  	org.freedesktop.Platform	22.08	i	flathub	< 214.3 MB
 6.	  	com.slack.Slack	stable	u	flathub	< 72.7 MB

Proceed with these changes to the system installation? [Y/n]: 
Installing 1/6…
F: Calling system helper: GetRevokefsFd
F: Calling system helper: GetRevokefsFd
Warning: Cannot create repo on revokefs mountpoint /var/tmp/flatpak-cache-FIVCU1/org.freedesktop.Platform.GL.default-RFVCU1: Creating temp file: Permission denied
F: Calling system helper: CancelPull
F: Calling system helper: CancelPull
F: flatpak_dir_pull: Using commit 6062874bd1737b867b0c9a538f5f77b454ba1d05365192165136322e8a20959b for pull of ref runtime/org.freedesktop.Platform.GL.default/x86_64/22.08 from remote flathub
Installing 1/6…                        0%  0 bytes/s
Installing 1/6… ████████████████████ 100%
F: Calling system helper: Deploy
Warning: Error deploying: Read-only file system
Installing 2/6…
F: Calling system helper: GetRevokefsFd
F: Calling system helper: GetRevokefsFd
Warning: Failed to get revokefs-fuse socket from system-helper: Failed to create new cache-dir at /var/lib/flatpak/repo/tmp: Read-only file system
Warning: Read-only file system
Installing 3/6…
F: Calling system helper: GetRevokefsFd
F: Calling system helper: GetRevokefsFd
Warning: Failed to get revokefs-fuse socket from system-helper: Failed to create new cache-dir at /var/lib/flatpak/repo/tmp: Read-only file system
Warning: Read-only file system
Installing 4/6…
F: Calling system helper: GetRevokefsFd
F: Calling system helper: GetRevokefsFd
Warning: Failed to get revokefs-fuse socket from system-helper: Failed to create new cache-dir at /var/lib/flatpak/repo/tmp: Read-only file system
Warning: Read-only file system
Installing 5/6…
F: Calling system helper: GetRevokefsFd
F: Calling system helper: GetRevokefsFd
Warning: Failed to get revokefs-fuse socket from system-helper: Failed to create new cache-dir at /var/lib/flatpak/repo/tmp: Read-only file system
Error: Read-only file system
Info: com.slack.Slack was skipped
Changes complete.
error: There were one or more errors
thread 'main' panicked at 'McFly error: DELETE from selected_commands to work (attempt to write a readonly database)', src/history/history.rs:158:17            
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[~] 

Have you tried running the updated with selinux set to permissive just to rule that out?
sudo setenforce 0
Note, this will not persist through reboots.

can you also post the output of:
df -h
systemctl --user status flat\*
After recreating the problem:
journalctl -S today -g flatpak

Thanks

Hi Joe,

RE running sudo setenforce 0: I hadn’t tried this before. I have now, then ran the failing flatpak command and saw the exact same issue occur.

running df -h before the issue:

[~] df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.7G  264K  7.7G   1% /dev/shm
tmpfs           3.1G  2.4M  3.1G   1% /run
/dev/dm-0       1.8T  1.2T  562G  69% /
tmpfs           7.7G   92K  7.7G   1% /tmp
/dev/dm-0       1.8T  1.2T  562G  69% /home
/dev/loop0      128K  128K     0 100% /var/lib/snapd/snap/bare/5
/dev/loop2      374M  374M     0 100% /var/lib/snapd/snap/anbox/186
/dev/loop1      115M  115M     0 100% /var/lib/snapd/snap/core/13886
/dev/loop4       56M   56M     0 100% /var/lib/snapd/snap/core18/2560
/dev/loop5       62M   62M     0 100% /var/lib/snapd/snap/core20/1611
/dev/loop3       56M   56M     0 100% /var/lib/snapd/snap/core18/2566
/dev/loop7       64M   64M     0 100% /var/lib/snapd/snap/core20/1623
/dev/loop6      179M  179M     0 100% /var/lib/snapd/snap/docfetcher/4
/dev/loop8      165M  165M     0 100% /var/lib/snapd/snap/gnome-3-28-1804/161
/dev/loop10     347M  347M     0 100% /var/lib/snapd/snap/gnome-3-38-2004/119
/dev/loop9      347M  347M     0 100% /var/lib/snapd/snap/gnome-3-38-2004/115
/dev/loop11      82M   82M     0 100% /var/lib/snapd/snap/gtk-common-themes/1534
/dev/loop12      92M   92M     0 100% /var/lib/snapd/snap/gtk-common-themes/1535
/dev/loop13     211M  211M     0 100% /var/lib/snapd/snap/inkdrop/48
/dev/loop15      48M   48M     0 100% /var/lib/snapd/snap/snapd/17029
/dev/loop16      48M   48M     0 100% /var/lib/snapd/snap/snapd/17336
/dev/loop14     209M  209M     0 100% /var/lib/snapd/snap/inkdrop/49
/dev/loop17      75M   75M     0 100% /var/lib/snapd/snap/upnote/34
/dev/loop18      80M   80M     0 100% /var/lib/snapd/snap/upnote/36
/dev/nvme0n1p5  974M  263M  644M  29% /boot
/dev/nvme0n1p1   96M   66M   31M  69% /boot/efi
tmpfs           1.6G  232K  1.6G   1% /run/user/1000

systemctl --user status flat\*:

systemctl --user status flat\*
● flatpak-session-helper.service - flatpak session helper
     Loaded: loaded (/usr/lib/systemd/user/flatpak-session-helper.service; static)
     Active: active (running) since Wed 2022-10-26 12:35:46 BST; 2min 7s ago
   Main PID: 3672 (flatpak-session)
      Tasks: 4 (limit: 18765)
     Memory: 1.5M
        CPU: 25ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/flatpak-session-helper.service
             ├─ 3672 /usr/libexec/flatpak-session-helper
             └─ 3678 server --sh -n /run/user/1000/.flatpak-helper/pkcs11-flatpak-3672 --provider p11-kit-trust.so "pkcs11:model=p11-kit-trust?write-protected=>

Oct 26 12:35:46 fedora systemd[2492]: Starting flatpak-session-helper.service - flatpak session helper...
Oct 26 12:35:46 fedora systemd[2492]: Started flatpak-session-helper.service - flatpak session helper.

Recreating the issue by updating Zoom, the output of journalctl -S today -g flatpak (discarding previous boots) is:

-- Boot 4f180310b41f4dc8b5c9609ea35697c2 --
Oct 26 12:35:26 fedora systemd[1]: flatpak-add-fedora-repos.service - Add Fedora flatpak repositories was skipped because of a failed condition check (Conditio>
Oct 26 12:35:46 fedora systemd[2492]: Starting flatpak-session-helper.service - flatpak session helper...
Oct 26 12:35:46 fedora systemd[2492]: Started flatpak-session-helper.service - flatpak session helper.
Oct 26 12:35:46 fedora systemd[2492]: Starting xdg-document-portal.service - flatpak document portal service...
Oct 26 12:35:46 fedora systemd[2492]: Started xdg-document-portal.service - flatpak document portal service.
Oct 26 12:35:46 fedora systemd[2492]: Started app-flatpak-it.mijorus.smile-3384.scope.
Oct 26 12:35:53 fedora gnome-software[3418]: enabled plugins: appstream, fwupd, os-release, repos, fedora-langpacks, fedora-pkgdb-collections, flatpak, hardcod>
Oct 26 12:35:59 fedora systemd[1]: Starting flatpak-system-helper.service - flatpak system helper...
Oct 26 12:35:59 fedora systemd[1]: Started flatpak-system-helper.service - flatpak system helper.
Oct 26 12:35:59 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=flatpak-system-helper comm>
Oct 26 12:38:25 fedora polkitd[1314]: Registered Authentication Agent for unix-process:7132:20061 (system bus name :1.180 [flatpak update us.zoom.Zoom], object>
Oct 26 12:38:28 fedora flatpak[7132]: /var/tmp/flatpak-cache-VTRAU1/org.freedesktop.Platform.GL.default-XVRAU1/repo-affeLI: Pulled runtime/org.freedesktop.Plat>
Oct 26 12:38:28 fedora systemd[1]: var-tmp-flatpak\x2dcache\x2dVTRAU1-org.freedesktop.Platform.GL.default\x2dXVRAU1.mount: Deactivated successfully.
Oct 26 12:38:28 fedora flatpak-system-helper[4509]: system: Pulled runtime/org.freedesktop.Platform.GL.default/x86_64/21.08 from /var/lib/flatpak/repo/tmp/flat>
Oct 26 12:38:28 fedora flatpak-system-[4509]: Unable to remove old checkout: Removing /var/lib/flatpak/.removed/org.gnome.Platform-fd056de9303be38f78472f766dc6>
Oct 26 12:38:28 fedora flatpak-system-[4509]: Unable to remove ongoing pull's src dir at /var/lib/flatpak/repo/tmp/flatpak-cache-AC2BU1: Removing /var/lib/flat>
Oct 26 12:38:28 fedora flatpak-system-[4509]: Unable to remove ongoing pull's src dir at /var/lib/flatpak/repo/tmp/flatpak-cache-AC2BU1: Removing /var/lib/flat>
Oct 26 12:38:29 fedora flatpak-system-[4509]: Unable to remove ongoing pull's src dir at /var/lib/flatpak/repo/tmp/flatpak-cache-AC2BU1: Removing /var/lib/flat>
Oct 26 12:38:29 fedora flatpak-system-[4509]: Unable to remove ongoing pull's src dir at /var/lib/flatpak/repo/tmp/flatpak-cache-AC2BU1: Removing /var/lib/flat>
Oct 26 12:38:29 fedora flatpak-system-[4509]: Unable to remove ongoing pull's src dir at /var/lib/flatpak/repo/tmp/flatpak-cache-AC2BU1: Removing /var/lib/flat>

Thanks

can you check, systemctl --user status flatpak-portal.service

or see why it isn’t running?

Thanks

The return value from:

is similar to:

● flatpak-portal.service - flatpak portal
     Loaded: loaded (/usr/lib/systemd/user/flatpak-portal.service; static)
     Active: active (running) since Sat 2022-10-29 21:31:10 BST; 10h ago
   Main PID: 7247 (flatpak-portal)
      Tasks: 3 (limit: 18765)
     Memory: 149.5M
        CPU: 242ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/flatpak-portal.service
             └─ 7247 /usr/libexec/flatpak-portal

It wasn’t straight after startup but after a little time it seemed to come up. Initially at startup it was returning:

○ flatpak-portal.service - flatpak portal
     Loaded: loaded (/usr/lib/systemd/user/flatpak-portal.service; static)
     Active: inactive (dead)

The observed problem still occurs, and flatpak-portal.service is active before and after the issue.

have you already run?

sudo flatpak repair
flatpak --user repair
flatpak remove --unused
flatpak update --appstream
flatpak update

Thanks

Thanks joe,

I tried all of these commands, rebooting in between, and all bar flatpak update --appstream failed and then caused the issue to reoccur.

By fail I mean they’ll start the task and then once they need to write anything to disk fail with an I/O error, assume due to the incorrect mount state. for example:

[~] sudo flatpak repair                                                                   7:05:18 
Place your finger on the fingerprint reader
[35/160] Verifying flathub:runtime/org.kde.WaylandDecoration.QGnomePlatform-decoration/x86_64/5.15-[49/160] Verifying flathub:runtime/org.freedesktop.Platform.openh264/x86_64/2.0…
[85/160] Verifying flathub:app/com.spotify.Client/x86_64/stable…
[129/160] Verifying flathub:app/app.drey.Dialect/x86_64/stable…
[158/160] Verifying flathub:runtime/org.kde.Platform.Locale/x86_64/5.15-21.08…
Checking remotes...
Pruning objects
Erasing .removed
error: Removing /var/lib/flatpak/.removed: unlinkat(ca_AD.dic): Input/output error

I have exactly the same problem. Whenever I try to upgrade/remove/install something in flatpak, the filesystem gets forced into read-only mode. It started happening a couple of weeks ago.

Dmesg outputs this:

[  863.643353] BTRFS: error (device dm-0: state A) in __btrfs_free_extent:3079: errno=-2 No such entry
[  863.643356] BTRFS info (device dm-0: state EA): forced readonly
[  863.643357] BTRFS: error (device dm-0: state EA) in btrfs_run_delayed_refs:2157: errno=-2 No such entry

Btrfs check shows backref errors:

[root@fedora ~]# btrfs check --force /dev/mapper/luks-b563c27d-374a-4321-ae90-75912ab9ec37
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/mapper/luks-b563c27d-374a-4321-ae90-75912ab9ec37
UUID: c2cd9ebf-5a6f-4e48-a609-bc6c6fb10736
[1/7] checking root items
[2/7] checking extents
data backref 6993911808 root 257 owner 565678 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 6993911808 root 257 owner 565678 offset 0 found 1 wanted 0 back 0x55eb01346bf0
incorrect local backref count on 6993911808 root 257 owner 565670 offset 0 found 0 wanted 1 back 0x55eb029a2350
backref disk bytenr does not match extent record, bytenr=6993911808, ref bytenr=0
backpointer mismatch on [6993911808 8192]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 126735335424 bytes used, error(s) found
total csum bytes: 76479036
total tree bytes: 1763246080
total fs tree bytes: 1564573696
total extent tree bytes: 97239040
btree space waste bytes: 353971334
file data blocks allocated: 2799363825664
 referenced 156993171456

I will try a btrfs repair later. It’d be great to know what caused this state.

Full dmesg log: https://pastebin.com/PiFc4XEd

1 Like

I tried btrfs check --repair and it fixed the backref error, now flatpak works again. If you have the same issue, here is what I did:

  • Boot up a live fedora image.

  • Find your LUKS device and unlock it (if you don’t use LUKS you can skip this):

    [liveuser@localhost-live ~]$ sudo -i
    [root@localhost-live ~]# blkid | grep crypto_LUKS
    /dev/nvme0n1p6: UUID="b563c27d-374a-4321-ae90-75912ab9ec37" TYPE="crypto_LUKS" PARTUUID="21d1dfc3-ecf9-4d69-bbdc-73f38c986961"
    [root@localhost-live ~]# cryptsetup open /dev/nvme0n1p6 foo
    Enter passphrase for /dev/nvme0n1p6:
    
  • Run btrfs check with --repair argument (I suggest to make a backup first):

    [root@localhost-live ~]# btrfs check --repair /dev/mapper/foo
    enabling repair mode
    WARNING:
    
    Do not use --repair unless you are advised to do so by a developer
    or an experienced user, and then only after having accepted that no
    fsck can successfully repair all types of filesystem corruption. Eg.
    some software or hardware bugs can fatally damage a volume.
    The operation will start in 10 seconds.
    Use Ctrl-C to stop it.
    10 9 8 7 6 5 4 3 2 1
    Starting repair.
    Opening filesystem to check...
    Checking filesystem on /dev/mapper/foo
    UUID: c2cd9ebf-5a6f-4e48-a609-bc6c6fb10736
    [1/7] checking root items
    Fixed 0 roots.
    [2/7] checking extents
    data backref 6993911808 root 257 owner 565678 offset 0 num_refs 0 not found in extent tree
    incorrect local backref count on 6993911808 root 257 owner 565678 offset 0 found 1 wanted 0 back 0x55cdc1bd17d0
    incorrect local backref count on 6993911808 root 257 owner 565670 offset 0 found 0 wanted 1 back 0x55cdc2f34190
    backref disk bytenr does not match extent record, bytenr=6993911808, ref bytenr=0
    backpointer mismatch on [6993911808 8192]
    repair deleting extent record: key [6993911808,168,8192]
    adding new data backref on 6993911808 root 257 owner 565678 offset 0 found 1
    Repaired extent references for 6993911808
    No device size related problem found
    [3/7] checking free space cache
    cache and super generation don't match, space cache will be invalidated
    [4/7] checking fs roots
    [5/7] checking only csums items (without verifying data)
    [6/7] checking root refs
    [7/7] checking quota groups skipped (not enabled on this FS)
    found 128671645696 bytes used, no error found
    total csum bytes: 78435552
    total tree bytes: 1765064704
    total fs tree bytes: 1564524544
    total extent tree bytes: 97222656
    btree space waste bytes: 353720545
    file data blocks allocated: 2799964717056
     referenced 158965547008
    
  • Reboot.

I also run the flatpak repair commands advised by @grumpey after this just to be sure. Hope it helps.

3 Likes

@korbel Fantastic! Thank you, this fixed the issue for me. I followed your suggestion and then after rebooting back to Fedora, flatpak update worked perfectly.

:tada:

3 Likes