Fedora IoT greenboot triggeres reboot when system should be healthy

Ask Fedora
, , , , ,
1

My Fedora IoT installation keeps on rebooting after a couple of seconds when coming to the login screen. Even after reinstalling from scratch (so no “try reinstalling and begin from scratch” is possible in this case :/) This seems to be due to some greenboot issue but I do not really know how to combat the issue seen in the log below. It is an excerpt from journalctl from the last boot reduced to the exact time when the system triggers a reboot.

It seems that the “01_repository_dns_check.sh” script failed and hence the greenboot service failed due to unmet dependencies prompting greenboot to reboot due to an “unhealthy” system state… Is this a common issue?

Jul 14 21:23:23 server systemd[4625]: Queued start job for default target default.target.
Jul 14 21:23:23 server systemd[4625]: Created slice app.slice - User Application Slice.
Jul 14 21:23:23 server systemd[4625]: grub-boot-success.timer - Mark boot as successful after the user session has run 2 minutes was skipped because of an unmet condition check (ConditionUser=!@system).
Jul 14 21:23:23 server systemd[4625]: Started systemd-tmpfiles-clean.timer - Daily Cleanup of User's Temporary Directories.
Jul 14 21:23:23 server systemd[4625]: Reached target paths.target - Paths.
Jul 14 21:23:23 server systemd[4625]: Reached target timers.target - Timers.
Jul 14 21:23:23 server systemd[4625]: Starting dbus.socket - D-Bus User Message Bus Socket...
Jul 14 21:23:23 server systemd[4625]: pipewire-pulse.socket - PipeWire PulseAudio was skipped because of an unmet condition check (ConditionUser=!root).
Jul 14 21:23:23 server systemd[4625]: Listening on pipewire.socket - PipeWire Multimedia System Socket.
Jul 14 21:23:23 server systemd[4625]: Starting systemd-tmpfiles-setup.service - Create User's Volatile Files and Directories...
Jul 14 21:23:23 server systemd[4625]: Listening on dbus.socket - D-Bus User Message Bus Socket.
Jul 14 21:23:23 server systemd[4625]: Reached target sockets.target - Sockets.
Jul 14 21:23:23 server systemd[4625]: Finished systemd-tmpfiles-setup.service - Create User's Volatile Files and Directories.
Jul 14 21:23:23 server systemd[4625]: Reached target basic.target - Basic System.
Jul 14 21:23:23 server systemd[4625]: Reached target default.target - Main User Target.
Jul 14 21:23:23 server systemd[4625]: Startup finished in 119ms.
Jul 14 21:23:23 server systemd[1]: Started user@0.service - User Manager for UID 0.
Jul 14 21:23:23 server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user@0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:23 server systemd[1]: Started session-1.scope - Session 1 of User root.
Jul 14 21:23:23 server login[4440]: pam_unix(login:session): session opened for user root(uid=0) by LOGIN(uid=0)
Jul 14 21:23:23 server audit[4440]: USER_START pid=4440 uid=0 auid=0 ses=1 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_umask,pam_lastlog acct="root" exe="/usr/bin/login" hostname=server addr=? terminal=/dev/tty1 res=success'
Jul 14 21:23:23 server audit[4440]: CRED_REFR pid=4440 uid=0 auid=0 ses=1 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/login" hostname=server addr=? terminal=/dev/tty1 res=success'
Jul 14 21:23:23 server audit[4440]: USER_LOGIN pid=4440 uid=0 auid=0 ses=1 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/bin/login" hostname=server addr=? terminal=tty1 res=success'
Jul 14 21:23:23 server login[4440]: ROOT LOGIN ON tty1
Jul 14 21:23:24 server systemd[1]: Starting zezere_ignition.service - Run Ignition for Zezere...
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : Ignition 2.15.0
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : Stage: fetch
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jul 14 21:23:24 server zezere-ignition[4666]: DEBUG    : parsed url from cmdline: ""
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : no config URL provided
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : reading system config file "/usr/lib/ignition/user.ign"
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : no config at "/usr/lib/ignition/user.ign"
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : using config file at "/tmp/zezere-ignition-config-v8houat8.ign"
Jul 14 21:23:24 server zezere-ignition[4666]: DEBUG    : parsing config with SHA512: 84663fcd50e87d5e1c1f83e9cf8cad38387e99389ca88d99e6b84b05a73b62833034ffe796eecf4230abd6f38a46b32786bdf52c53407246a747fe243b3168bf
Jul 14 21:23:24 server zezere-ignition[4666]: INFO     : GET https://provision.fedoraproject.org/netboot/x86_64/ignition/xx:xx:xx:xx:xx:xx: attempt #1
Jul 14 21:23:24 server systemd-resolved[4269]: Using degraded feature set TCP instead of UDP for DNS server xxx.xxx.xxx.xxx.
Jul 14 21:23:24 server 01_repository_dns_check.sh[4610]: The following repository domains haven't responded properly to DNS queries:
Jul 14 21:23:24 server 01_repository_dns_check.sh[4610]: ostree.fedoraproject.org
Jul 14 21:23:24 server greenboot[4593]: Script '01_repository_dns_check.sh' FAILURE (exit code '1'). Continuing...
Jul 14 21:23:24 server 02_watchdog.sh[4681]: Device:        /dev/watchdog0
Jul 14 21:23:24 server 02_watchdog.sh[4681]: Identity:      SP5100 TCO timer [version 0]
Jul 14 21:23:24 server 02_watchdog.sh[4681]: Timeout:       60 seconds
Jul 14 21:23:24 server 02_watchdog.sh[4681]: Timeleft:      60 seconds
Jul 14 21:23:24 server 02_watchdog.sh[4681]: Pre-timeout:    0 seconds
Jul 14 21:23:24 server 02_watchdog.sh[4681]: FLAG           DESCRIPTION               STATUS BOOT-STATUS
Jul 14 21:23:24 server 02_watchdog.sh[4681]: KEEPALIVEPING  Keep alive ping reply          1           0
Jul 14 21:23:24 server 02_watchdog.sh[4681]: MAGICCLOSE     Supports magic close char      0           0
Jul 14 21:23:24 server 02_watchdog.sh[4681]: SETTIMEOUT     Set timeout (in seconds)       0           0
Jul 14 21:23:25 server systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
Jul 14 21:23:25 server rpm-ostree[4688]: Reading config file '/etc/rpm-ostreed.conf'
Jul 14 21:23:25 server systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
Jul 14 21:23:25 server rpm-ostree[4688]: In idle state; will auto-exit in 63 seconds
Jul 14 21:23:25 server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpm-ostreed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server kernel: kauditd_printk_skb: 84 callbacks suppressed
Jul 14 21:23:25 server kernel: audit: type=1130 audit(1689362605.434:163): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpm-ostreed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server rpm-ostree[4688]: client(id:cli dbus:1.21 unit:greenboot-healthcheck.service uid:0) added; new total=1
Jul 14 21:23:25 server rpm-ostree[4688]: client(id:cli dbus:1.21 unit:greenboot-healthcheck.service uid:0) vanished; remaining=0
Jul 14 21:23:25 server rpm-ostree[4688]: In idle state; will auto-exit in 60 seconds
Jul 14 21:23:25 server 02_watchdog.sh[5052]: Device:        /dev/watchdog0
Jul 14 21:23:25 server 02_watchdog.sh[5052]: Identity:      SP5100 TCO timer [version 0]
Jul 14 21:23:25 server 02_watchdog.sh[5052]: Timeout:       60 seconds
Jul 14 21:23:25 server 02_watchdog.sh[5052]: Timeleft:      60 seconds
Jul 14 21:23:25 server 02_watchdog.sh[5052]: Pre-timeout:    0 seconds
Jul 14 21:23:25 server 02_watchdog.sh[5052]: FLAG           DESCRIPTION               STATUS BOOT-STATUS
Jul 14 21:23:25 server 02_watchdog.sh[5052]: KEEPALIVEPING  Keep alive ping reply          1           0
Jul 14 21:23:25 server 02_watchdog.sh[5052]: MAGICCLOSE     Supports magic close char      0           0
Jul 14 21:23:25 server 02_watchdog.sh[5052]: SETTIMEOUT     Set timeout (in seconds)       0           0
Jul 14 21:23:25 server greenboot[4593]: Script '02_watchdog.sh' SUCCESS
Jul 14 21:23:25 server systemd[1]: greenboot-healthcheck.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 21:23:25 server systemd[1]: greenboot-healthcheck.service: Failed with result 'exit-code'.
Jul 14 21:23:25 server systemd[1]: Failed to start greenboot-healthcheck.service - greenboot Health Checks Runner.
Jul 14 21:23:25 server systemd[1]: Dependency failed for boot-complete.target - Boot Completion Check.
Jul 14 21:23:25 server systemd[1]: Dependency failed for greenboot-task-runner.service - greenboot Success Scripts Runner.
Jul 14 21:23:25 server systemd[1]: greenboot-task-runner.service: Job greenboot-task-runner.service/start failed with result 'dependency'.
Jul 14 21:23:25 server systemd[1]: boot-complete.target: Job boot-complete.target/start failed with result 'dependency'.
Jul 14 21:23:25 server systemd[1]: greenboot-healthcheck.service: Triggering OnFailure= dependencies.
Jul 14 21:23:25 server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=greenboot-healthcheck comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 14 21:23:25 server kernel: audit: type=1130 audit(1689362605.465:164): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=greenboot-healthcheck comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 14 21:23:25 server systemd[1]: Starting redboot-task-runner.service - greenboot Failure Scripts Runner...
Jul 14 21:23:25 server greenboot[5057]: Boot Status is RED - Health Check FAILURE!
Jul 14 21:23:25 server greenboot[5057]: Running Red Scripts...
Jul 14 21:23:25 server systemd[1]: Finished redboot-task-runner.service - greenboot Failure Scripts Runner.
Jul 14 21:23:25 server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-task-runner comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server systemd[1]: Reached target redboot.target - Generic red boot target.
Jul 14 21:23:25 server kernel: audit: type=1130 audit(1689362605.491:165): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-task-runner comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server systemd[1]: Starting greenboot-status.service - greenboot MotD Generator...
Jul 14 21:23:25 server systemd[1]: Starting redboot-auto-reboot.service - Reboot on red boot status...
Jul 14 21:23:25 server redboot-auto-reboot[5062]: SYSTEM is UNHEALTHY. Rebooting...
Jul 14 21:23:25 server systemd-logind[4295]: The system will reboot now!
Jul 14 21:23:25 server systemd-logind[4295]: System is rebooting.
Jul 14 21:23:25 server systemd[1]: redboot-auto-reboot.service: Deactivated successfully.
Jul 14 21:23:25 server systemd[1]: Finished redboot-auto-reboot.service - Reboot on red boot status.
Jul 14 21:23:25 server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-auto-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=redboot-auto-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 14 21:23:25 server systemd[1]: Stopping session-1.scope - Session 1 of User root...
2

In what you posted I see no configuration of the network, and the network (or at least a functional dns) is required to access the repo as shown

I would focus on learning why the network is not available first.

3

You are correct. Thanks for pointing that out.
I do not configure the network until my virtual router vm is loaded and I learned that the greenboot application requires dns to work properly meaning if no internet is available at the time of booting then you get stuck in a bootloop. Guess I’ll have to open an issue on GitHub asking if the dns check can be changed to non-critical or similar for people who boot offline machines.