Strange Fedora IoT container and ssh problem

Ask Fedora
,
1

I am having a strange bug in Fedora IoT.

I am running Fedora IoT on a raspberry pi 4 from an ssd connected to a USB port (rpi4 bootloader allows this). Inside the PI I am running 2 linuxserver io containers from podman, the swag (let’sencrypt reverse proxy container) and the transmission container.

The key job of the swag container is to act as a reverse proxy for the raspberrypi3 that runs nextcoud (from nextcloudpi).

The problem is that infrequently (2 times in the last 2 months) the nextcloud instance becomes unavailable with a 502 Bad Gateway error.

I am using podman’s functionality to have the containers run as user services. Restarting the swag container does not fix the issue.

But the weird thing happens when I tell the fedora iot to reboot (Turning the computer on and off is a surprisingly powerfull repair action.). The fedora-iot rpi4 reboots but becomes unreachable! I can neither ssh in nor ping it successfully! After I reboot it a second time (by taking the power cable off, the fedora iot rpi4 comes up normally, is perfectly accessible and the swag container comes up and works fine!

Any ideas on how I would debug this? I m not sure where to start.

Moreover the fedora-iot image has a software called greenboot that allows for health checks. Any ideas on how I can add a script that checks whether the fedora-iot-rpi4 is unreachable from the outside world and reboots itself in that case? This is addressing the symptom, not the cause and even though I 'd like to get to the bottom of this, it’s all I can think of at the moment.

I just upgraded fedora iot, below is the system information with the image version that had this bug happen:

* fedora-iot:fedora/stable/aarch64/iot
                   Version: 34.20210916.0 (2021-09-16T21:17:51Z)
                BaseCommit: e40efc690d37503cd9bfbd749bc0f7ac8afd56a6728c9ddbfae4b5342753cfbd
                            |- repo-0 (2021-04-23T10:47:46Z)
                            |- repo-1 (2021-09-16T21:04:49Z)
                            |- repo-2 (2021-07-20T09:24:09Z)
                            `- repo-3 (2021-09-16T18:52:23Z)
                    Commit: aa2f02ae07e41bdc82f0bc181b85d28d24c423eac09ec56aa34f5a993edb7505
                            |- fedora (2021-04-23T10:47:46Z)
                            |- updates (2021-09-24T20:25:30Z)
                            |- copr:copr.fedorainfracloud.org:iolaum:podcust (2021-05-18T06:15:00Z)
                            `- fedora-cisco-openh264 (2021-02-23T00:49:00Z)
                 StateRoot: fedora-iot
              GPGSignature: 1 signature
                            Signature made Fri Sep 17 00:17:57 2021 using RSA key ID 1161AE6945719A39
                            Good signature from "Fedora <fedora-34-primary@fedoraproject.org>"
           LayeredPackages: cockpit nano cockpit-podman cockpit-ostree
2

Yes, greenboot may be a good solution to this…however before exploring that…are you sure the device is actually rebooting? Have you connected a monitor / serial cable and checked that its up and available?

For greenboot, its quite a simple system. Place a .sh file inside /etc/greenboot/check/required.d

If that script “FAILS”, then it will subsequently run the scripts located in /etc/greenbood/red.d

So for example you could do:

/etc/greenboot/check/required.d/01_check_network.sh

#!/bin/sh
ping -c 1 192.168.1.1 #(or whatever)

/etc/greenboot/red.d/99_reboot.sh

#!/bin/sh
systemctl reboot

Obviously this is a super simple example and you need to be aware that you might get caught in a reboot loop if things go bad. More checks and more graceful handling of failures is probably needed.

1 Like
3

Thank you, I will experiment with the scripts you provided!

Meanwhile while I was looking at the problem I saw that I could see logs from cockpit and there there were some strange entries:

hwmon hwmon0: Undervoltage detected!

I have changed the USBc cable from the power supply to the RPi4 and waiting to see if the problem comes again. I wasn’t able to reproduce it when I forced restart the rpi.

4

The Pi4 has a minimum current requirement and is usually sold with an adequate power supply. Using anything with lesser amperage will give the under voltage problem. Here you can see it is a minimum 15W power supply and canakit provides a 3.5 amp power supply which is more than the minimum.