I am having a strange bug in Fedora IoT.
I am running Fedora IoT on a raspberry pi 4 from an ssd connected to a USB port (rpi4 bootloader allows this). Inside the PI I am running 2 linuxserver io containers from podman, the swag (let’sencrypt reverse proxy container) and the transmission container.
The key job of the swag container is to act as a reverse proxy for the raspberrypi3 that runs nextcoud (from nextcloudpi).
The problem is that infrequently (2 times in the last 2 months) the nextcloud instance becomes unavailable with a 502 Bad Gateway error.
I am using podman’s functionality to have the containers run as user services. Restarting the swag container does not fix the issue.
But the weird thing happens when I tell the fedora iot to reboot (Turning the computer on and off is a surprisingly powerfull repair action.). The fedora-iot rpi4 reboots but becomes unreachable! I can neither ssh in nor ping it successfully! After I reboot it a second time (by taking the power cable off, the fedora iot rpi4 comes up normally, is perfectly accessible and the swag container comes up and works fine!
Any ideas on how I would debug this? I m not sure where to start.
Moreover the fedora-iot image has a software called greenboot that allows for health checks. Any ideas on how I can add a script that checks whether the fedora-iot-rpi4 is unreachable from the outside world and reboots itself in that case? This is addressing the symptom, not the cause and even though I 'd like to get to the bottom of this, it’s all I can think of at the moment.
I just upgraded fedora iot, below is the system information with the image version that had this bug happen:
* fedora-iot:fedora/stable/aarch64/iot Version: 34.20210916.0 (2021-09-16T21:17:51Z) BaseCommit: e40efc690d37503cd9bfbd749bc0f7ac8afd56a6728c9ddbfae4b5342753cfbd |- repo-0 (2021-04-23T10:47:46Z) |- repo-1 (2021-09-16T21:04:49Z) |- repo-2 (2021-07-20T09:24:09Z) `- repo-3 (2021-09-16T18:52:23Z) Commit: aa2f02ae07e41bdc82f0bc181b85d28d24c423eac09ec56aa34f5a993edb7505 |- fedora (2021-04-23T10:47:46Z) |- updates (2021-09-24T20:25:30Z) |- copr:copr.fedorainfracloud.org:iolaum:podcust (2021-05-18T06:15:00Z) `- fedora-cisco-openh264 (2021-02-23T00:49:00Z) StateRoot: fedora-iot GPGSignature: 1 signature Signature made Fri Sep 17 00:17:57 2021 using RSA key ID 1161AE6945719A39 Good signature from "Fedora <email@example.com>" LayeredPackages: cockpit nano cockpit-podman cockpit-ostree