Kexec into an earlier deployment

Project Discussion
1

Hi there,

I have 3 deployments:

$ find /boot/loader/
/boot/loader/entries/ostree-2-fedora.conf
/boot/loader/entries/ostree-3-fedora.conf
/boot/loader/entries/ostree-1-fedora.conf

I’m currently booted into ostree-2, but I want to kexec into ostree-1. It’s a challenge.

Knowing this information and trying to plunk it into a kexec line isn’t working out.
$ cat /boot/loader/entries/ostree-1-fedora.conf
title Fedora Linux 36.20220917.0 (Silverblue) (ostree:2)
version 1
options rhgb quiet root=UUID=506c0afb-14fa-4343-ac68-448dd136cb94 rootflags=subvol=root ostree=/ostree/boot.1/fedora/40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/0 modprobe.blacklist=nouveau nvidia-drm.modeset=1 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1
linux /ostree/fedora-40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/vmlinuz-5.19.8-200.fc36.x86_64
initrd /ostree/fedora-40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/initramfs-5.19.8-200.fc36.x86_64.img

After creating the following line:
sudo kexec -l /ostree/fedora-40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/vmlinuz-5.19.8-200.fc36.x86_64 --initrd=/ostree/fedora-40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/initramfs-5.19.8-200.fc36.x86_64.img --command-line=“BOOT_IMAGE=(hd2,gpt2)/ostree/boot.1/fedora/40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/vmlinuz-5.19.8-200.fc36.x86_64 rhgb quiet root=UUID=506c0afb-14fa-4343-ac68-448dd136cb94 rootflags=subvol=root ostree=/ostree/boot.1/fedora/40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/0 modprobe.blacklist=nouveau nvidia-drm.modeset=1 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1”

I’m getting the following error:
Cannot open `/ostree/fedora-40c69a3fac97cc940679672e24a7f8da885ff0965da1fc7992cb8d4f702bcdec/vmlinuz-5.19.8-200.fc36.x86_64’: No such file or directory

My last boot with the most recent deployment looks like this:
$ cat /proc/cmdline
BOOT_IMAGE=(hd2,gpt2)/ostree/fedora-f64cbc4d92fa112e54d04fcd4debefee14c416a1321b97666205d68ef610bcfb/vmlinuz-6.0.5-200.fc36.x86_64 rhgb quiet root=UUID=506c0afb-14fa-4343-ac68-448dd136cb94 rootflags=subvol=root ostree=/ostree/boot.1/fedora/f64cbc4d92fa112e54d04fcd4debefee14c416a1321b97666205d68ef610bcfb/0 modprobe.blacklist=nouveau nvidia-drm.modeset=1 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1

Could anybody nudge me into the right direction please? Thank you.

This is all because I’m trying to upgrade after getting an error against nvidia-470xx stuff again so I needed to remove the packages, upgrade the os, then try to reinstall the packages again. It’s painful.
In that previous deployment, that one has the nvidia drivers intact and running. After I boot up the old deployment, I’m going to undeploy the newer ones and just stay put I guess.

Thank you in advance. Cheers

2

I’m curious why you might want to kexec instead of reboot and picking the earlier one from grub?

1 Like
3

Thank you for asking, for the curiosity, and for appreciating the context of it all.
I am physically not in front of my computer. I am working remotely. So that ability to choose from the grub menu is not there.

There’s another ability that isn’t there since I am remoting that has become a painpoint as well.

Booting from an external usb drive with different flavor or version of fedora there as well. I can’t get there because I’m not physically in front of the computer to get into the boot selection menu and select a different drive to boot from. I wish this was also possible from Kexec but that’s for another discussion.

Cheers :slight_smile:

3 Likes
5

It is possible to have grub (or other bootloaders) provide their menu over a serial connection and that, in turn, can be routed to a remote connection (e.g. via a connection to another computer that is online or via IPMI’s SoL protocol if it is a server (I have several servers configured this way for remote admin)). But all that requires additional hardware.

6

It is possible over a remote KVM (keyboard-video-mouse, not the hypervisor), but that depends on such things being available in an environment and that’s a big “if”. IPMI sol may or may not work. Usually I see the kernel messages after boot since it’s generally what you would see from the serial connection in qemu, etc. You can reboot it with ipmi, but accessing BIOS or GRUB might require a kvm.

7

You don’t need specialized hardware other than a null modem cable and at least two computers/hypervisors. Even if the computers or servers don’t have IPMI, grub/syslinux/sd-boot can be configured to output their boot menu over the serial terminal. It can then be accessed from the other computer via a command such as screen /dev/ttyS0 115200. In grub you have to add something like:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

With syslinux it is just (must be first line in syslinux.cfg):

serial 0 115200

And I think it is enabled by default with sd-boot (but you might need to increase the menu timeout so it shows).

In all cases, you would also want to add console=ttyS0,115200n8 to your kernel command line so you could also get the default console after the kernel finishes loading.

I’ve found this very useful for remotely administering several servers over the years. I’ve even added rd.break=pre-mount to get it to drop into the dracut shell and done very elaborate recovery operations from there (e.g., I once used this method to rollback the entire server OS to an earlier snapshot when an update went wrong – completely remotely).

1 Like
8

Your answers are valid, thank you for that.

In my current work environment, I may not ipmi nor minicom with a null modem serial cable.

I was indirectly asking for a change request making it possible to kexec to another deployment more easily without resorting to ipmi/minicom.

Thank you for listening.

| Gregory Lee Bartholomew glb Extra! Extra!
November 4 |

  • | - |

You don’t need specialized hardware other than a null modem cable and at least two computers/hypervisors. Even if the computers or servers don’t have IPMI, grub/syslinux/sd-boot can be configured to output their boot menu over the serial terminal. It can then be accessed from the other computer via a command such as screen /dev/ttyS0 115200. In grub you have to add something link:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

With syslinux it is just (must be first line in syslinux.cfg):

serial 0 115200

And I think it is enabled by default with sd-boot (but you might need to increase the menu timeout so it shows).

In all cases, you would also want to add console=ttyS0,115200 to your kernel command line so you could also get the default console after the terminal finishes loading.

I’ve found this very useful for remotely administering several servers over the years. I’ve even added rd.break=pre-mount to get it to drop into the dracut shell and done very elaborate recovery operations from there (e.g., I once used this method to rollback the entire server OS to an earlier snapshot when an update went wrong – completely remotely).

9

It’s funny because all this happened because I was struggling to get nvidia legacy drivers updated.

$ cat myCurrentStruggleForSilverblue36Nvidia470xxDriverInstall1032.txt
sudo rpm-ostree upgrade
sudo rpm-ostree upgrade --check
sudo rpm-ostree upgrade
sudo rpm-ostree uninstall screen
sudo systemctl reboot
sudo rpm-ostree upgrade

I removed some redundant kargs that were repeating to blacklist nouveau video driver:

sudo rpm-ostree kargs --editor
sudo systemctl reboot

I installed the nvidia 3rd party repos:
sudo rpm-ostree install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sync
sudo systemctl reboot

I attempted to install the nvidia legacy 470xx driver stuff:

sudo rpm-ostree install akmod-nvidia-470xx nvidia-settings-470xx xorg-x11-drv-nvidia-470xx-cuda xorg-x11-drv-nvidia-470xx-power

Here’s the output for that:

davidm@io02Silverblue 2022-11-04_10:29:24_EDT : ~
$ sudo rpm-ostree install akmod-nvidia-470xx nvidia-settings-470xx xorg-x11-drv-nvidia-470xx-cuda xorg-x11-drv-nvidia-470xx-power
[sudo] password for davidm:
Checking out tree 8dfda11… done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rpmfusion-free-updates rpmfusion-free rpmfusion-nonfree-updates rpmfusion-nonfree phracek-PyCharm rpmfusion-nonfree-steam google-chrome updates-archive
Updating metadata for ‘rpmfusion-free-updates’… done
Updating metadata for ‘rpmfusion-free’… done
Updating metadata for ‘rpmfusion-nonfree-updates’… done
error: Updating rpm-md repo ‘rpmfusion-nonfree-updates’: cannot update repo ‘rpmfusion-nonfree-updates’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Downloading successful, but checksum doesn’t match. Calculated: 7d5b1ea33c40dac9a4c2685e03b73def1def7221d746acbdc96ef3eaccc4184a56d4eaf5025d5232cd9f015d1652cc3641c7957e2dec6071b275832dc93873eb(sha512) Expected: a83b218875595d16869842612a6eb7c552649719a7fd34b85f5b117f1c25b7035241375eb13ff1dd1cbd9d6c320ae0359da424c0f46ac45d29da7eb179344c70(sha512)
davidm@io02Silverblue 2022-11-04_10:31:01_EDT : ~

Here are all my deployments:
sudo rpm-ostree status

State: idle
Deployments:
● fedora:fedora/36/x86_64/silverblue
Version: 36.20221104.0 (2022-11-04T10:23:21Z)
BaseCommit: 8dfda118f6b68c27a414d835933185c43647282395670ad9ffe894030b251a3e
GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
LayeredPackages: ‘c+±gtk-utils-gtk4-devel’ ‘gcc-c++’ ‘ImageMagick-c+±devel’
‘libstdc+±devel’ ‘qt6-*’ adwaita-icon-theme-devel
alsa-lib-devel autoconf automake avahi-gobject-devel blender
cairo-devel cairo-gobject-devel chromium clang clang-tools-extra
clapper cmake dbus-devel emacs gcc gdk-pixbuf2-devel gh gimp
gitk glib-devel glib2-devel graphene-devel gst123
gstreamer1-devel gstreamer1-plugins-bad-free-devel
gstreamer1-plugins-base-devel gstreamer1-rtsp-server-devel
gstreamermm-devel gtk4-devel gtkmm4.0-devel gtkmm4.0-doc
ImageMagick ImageMagick-devel ImageMagick-doc kexec-tools
libgudev-devel libshaderc-devel libsoup-devel libsoup3-devel
libsqlite3x-devel libtoml-devel libva-utils libva-vdpau-driver
libxcb-devel lld meld meson openfortivpn openssl-devel
pango-devel python3-jinja2 python3-pygments python3-toml-adapt
python3-typogrify qt6-qt3d-examples qt6-qtbase-examples
qt6-qtcharts-examples qt6-qtconnectivity-examples
qt6-qtdeclarative-examples qt6-qtmultimedia-examples
qt6-qtpositioning-examples qt6-qtquick3d-examples
qt6-qtwayland-examples remmina svt-av1 vdpauinfo vkmark
vulkan-tools xrdp youtube-dl
LocalPackages: rpmfusion-free-release-36-1.noarch
rpmfusion-nonfree-release-36-1.noarch

fedora:fedora/36/x86_64/silverblue
Version: 36.20221104.0 (2022-11-04T10:23:21Z)
BaseCommit: 8dfda118f6b68c27a414d835933185c43647282395670ad9ffe894030b251a3e
GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
LayeredPackages: ‘c+±gtk-utils-gtk4-devel’ ‘gcc-c++’ ‘ImageMagick-c+±devel’
‘libstdc+±devel’ ‘qt6-*’ adwaita-icon-theme-devel
alsa-lib-devel autoconf automake avahi-gobject-devel blender
cairo-devel cairo-gobject-devel chromium clang clang-tools-extra
clapper cmake dbus-devel emacs gcc gdk-pixbuf2-devel gh gimp
gitk glib-devel glib2-devel graphene-devel gst123
gstreamer1-devel gstreamer1-plugins-bad-free-devel
gstreamer1-plugins-base-devel gstreamer1-rtsp-server-devel
gstreamermm-devel gtk4-devel gtkmm4.0-devel gtkmm4.0-doc
ImageMagick ImageMagick-devel ImageMagick-doc kexec-tools
libgudev-devel libshaderc-devel libsoup-devel libsoup3-devel
libsqlite3x-devel libtoml-devel libva-utils libva-vdpau-driver
libxcb-devel lld meld meson openfortivpn openssl-devel
pango-devel python3-jinja2 python3-pygments python3-toml-adapt
python3-typogrify qt6-qt3d-examples qt6-qtbase-examples
qt6-qtcharts-examples qt6-qtconnectivity-examples
qt6-qtdeclarative-examples qt6-qtmultimedia-examples
qt6-qtpositioning-examples qt6-qtquick3d-examples
qt6-qtwayland-examples remmina svt-av1 vdpauinfo vkmark
vulkan-tools xrdp youtube-dl

fedora:fedora/36/x86_64/silverblue
Version: 36.20220917.0 (2022-09-17T00:57:59Z)
BaseCommit: f8fce7b43cec814eb03e84a6aa1cbc3c6d5a5baafd2a9184a4b70d79a640ceb0
GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
LayeredPackages: ‘c+±gtk-utils-gtk4-devel’ ‘gcc-c++’ ‘ImageMagick-c+±devel’
‘libstdc+±devel’ ‘qt6-*’ adwaita-icon-theme-devel
akmod-nvidia-470xx alsa-lib-devel autoconf automake
avahi-gobject-devel blender cairo-devel cairo-gobject-devel
chromium clang clang-tools-extra clapper cmake dbus-devel emacs
gcc gdk-pixbuf2-devel gh gimp glib-devel glib2-devel
graphene-devel gst123 gstreamer1-devel
gstreamer1-plugins-bad-free-devel gstreamer1-plugins-base-devel
gstreamer1-rtsp-server-devel gstreamermm-devel gtk4-devel
gtkmm4.0-devel gtkmm4.0-doc ImageMagick ImageMagick-devel
ImageMagick-doc libgudev-devel libshaderc-devel libsoup-devel
libsoup3-devel libsqlite3x-devel libtoml-devel libva-utils
libva-vdpau-driver libxcb-devel lld meld meson
nvidia-settings-470xx openfortivpn openssl-devel pango-devel
python3-jinja2 python3-pygments python3-toml-adapt
python3-typogrify qt6-qt3d-examples qt6-qtbase-examples
qt6-qtcharts-examples qt6-qtconnectivity-examples
qt6-qtdeclarative-examples qt6-qtmultimedia-examples
qt6-qtpositioning-examples qt6-qtquick3d-examples
qt6-qtwayland-examples remmina screen svt-av1 vdpauinfo vkmark
vulkan-tools xorg-x11-drv-nvidia-470xx-cuda
xorg-x11-drv-nvidia-470xx-power xrdp youtube-dl
Pinned: yes
davidm@io02Silverblue 2022-11-04_10:36:30_EDT : ~

After doing all this, I can’t upgrade packages anymore so this is an ongoing painpoint for me.

I also noticed openssl is sitting at 3.0.5-2 when I believe the cve asks to upgrade to openssl 3.0.7 now.

Your answers are valid, thank you for that.

In my current work environment, I may not ipmi nor minicom with a null modem serial cable.

I was indirectly asking for a change request making it possible to kexec to another deployment more easily without resorting to ipmi/minicom.

Thank you for listening.

| Gregory Lee Bartholomew glb Extra! Extra!
November 4 |

  • | - |

You don’t need specialized hardware other than a null modem cable and at least two computers/hypervisors. Even if the computers or servers don’t have IPMI, grub/syslinux/sd-boot can be configured to output their boot menu over the serial terminal. It can then be accessed from the other computer via a command such as screen /dev/ttyS0 115200. In grub you have to add something link:

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

With syslinux it is just (must be first line in syslinux.cfg):

serial 0 115200

And I think it is enabled by default with sd-boot (but you might need to increase the menu timeout so it shows).

In all cases, you would also want to add console=ttyS0,115200 to your kernel command line so you could also get the default console after the terminal finishes loading.

I’ve found this very useful for remotely administering several servers over the years. I’ve even added rd.break=pre-mount to get it to drop into the dracut shell and done very elaborate recovery operations from there (e.g., I once used this method to rollback the entire server OS to an earlier snapshot when an update went wrong – completely remotely).

10

The patches were backported to 3.0.5-2.

1 Like