First boot error "Failed to set up standard input: Inappropriate ioctl for device”

So I try to provision a bare metal device (x86_64), but the first boot always fails.

  • I download the image:
    coreos-installer download -s stable -p metal -f raw.xz -C ./downloads.
  • In my case I directly put that onto an bootabler medium/disk/drive I put in later:
    sudo coreos-installer install /dev/sdd -i ignition/[…].ign --image-file downloads/fedora-coreos-33.20210328.3.0-metal.x86_64.raw.xz
    (though I had some strange errors when retrying)
  • And finally just boot.

What happens

I can see it shows me a message that it setups file partitions for quite some time, which is understandable and correct.
However, I have LUKS encryption with a tang server enabled for the root medium, and I can see the tang server is not actually contacted. That’s somewhat strange, as I know the root partition is correctly encrypted when I look at the disk later.

Instead, it says some startup finished, but then gets into an emergency shell, which it (I guess?) cannot start though, because it yields some strange error related to shell vs script checking, as far as I could find out.

Additional information

I may have a wrong fingerprint for the Tang key used in my CoreOS config, as I’m still trying to figure out how to generate it, respectively which key to use which lay on the server, but IMHO this should show me a very different error, so I guess that is no problem here.

The TL;DR of the error is:

Startup finished[…]
[…]
systemd[…]: emergency.service: Failed to set up standard input: Inappropriate ioctl for device

That last error is repeated multiple times.

Here is a full image:

I can unfortunately not scroll further up and also not access any (log) file, because well… it’s encrypted.

CoreOS config

Here is a redacted version of my quite basic CoreOS config:

variant: fcos
version: 1.3.0
passwd:
  users:
    - name: core
    - # ...
storage:
  files:
    # hostname
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: #******
    # updates
    - path: /etc/zincati/config.d/55-updates-strategy.toml
      mode: 0644
      contents:
        inline: |
          [updates]
          strategy = "periodic"
          [[updates.periodic.window]]
          days = [ "Mon", "Tue", "Wed", "Thd", "Fri", "Sat", "Sun" ]
          start_time = "01:00"
          length_minutes = 60
    # enable ZRAM
    # https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-configure-swaponzram/
    - path: /etc/systemd/zram-generator.conf
      mode: 0644
      contents:
        inline: |
          # This config file enables a /dev/zram0 device with the default settings
          [zram0]
    # internal
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # to hide audit messages from the interactive console
          #kernel.printk=4
    # scripts
    - path: /usr/local/bin/userinit
      overwrite: true
      mode: 0755
      contents:
        local: ./scripts/userinit.sh # some "#!/bin/sh" script is saved and included here locally
  luks:
    # encrypted root
    - name: root
      label: luks-root
      device: /dev/disk/by-partlabel/root
      clevis:
        tpm2: false
        threshold: 1
        tang:
          - url: # *****
            thumbprint: # "*******"
      wipe_volume: true
  # resize root volume to 100% of disk
  # https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_sizing_the_root_partition
  disks:
    - device: /dev/vda
      partitions:
        - label: root
          number: 4
          # 0 means to use all available space
          size_mib: 0
          resize: true
  filesystems:
    - device: /dev/mapper/root
      format: btrfs
      wipe_filesystem: true
      label: root
systemd:
  units:
    # switch to modern cgroups v2: https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/
    - name: cgroups-v2-karg.service
      enabled: true
      contents: |
        [Unit]
        Description=Switch To cgroups v2
        # We run after `systemd-machine-id-commit.service` to ensure that
        # `ConditionFirstBoot=true` services won't rerun on the next boot.
        After=systemd-machine-id-commit.service
        ConditionKernelCommandLine=systemd.unified_cgroup_hierarchy
        ConditionPathExists=!/var/lib/cgroups-v2-karg.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/bin/rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy
        ExecStart=/bin/touch /var/lib/cgroups-v2-karg.stamp
        ExecStart=/bin/systemctl --no-block reboot

        [Install]
        WantedBy=multi-user.target

Questions

What’s the problem here and what does the error message mean?
And how I can further best debug this?

The emergency shell, and some of the log output, only go to the primary console, which is the first serial port by default. If you don’t have a first serial port, that could explain the ENOTTY error. You can switch the primary console to graphical by removing console=ttyS0,115200 from the kernel command line.

Uhm and then?
I see all log output on the screen?

And if so, would that help given it may be too much then and just “scroll-by” as I am not able to read it then?
That’s not really how it was intended I guess…

Ignition failures result in an emergency shell. You should see a summary of the failure, and then you can use journalctl -t ignition to look at the Ignition logs.

Thanks a lot! I actually noticed that is also documented in the doc including how you can actually apply/remove the kernel parameter. (Just this strange error message is mentioned nowhere.)

And indeed I could find the culprit, I used the wrong disk /dev/vda instead of /dev/vda for a bare-metal installation for resizing the root partition. So I now look into extracting that into a variable.

Edit: Uhm actually I have an NVMe disk, which of course uses /var/nvme0n1 or so.