Ignition file not detected from OpenStack user data

macchaffee · August 7, 2020, 1:03am

I’m trying to deploy fedora coreos in OpenStack (I think v3.7, not sure) and I keep getting “Ignition: no config provided by user”

Here is my ignition file:
{"ignition":{"version":"3.1.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEJSIvZLaGSTX6KF+quBc6LYFIOzd5AT/33ztBBMR8wV core"]}]}}

I’ve tried providing this ignition file through ansible and through the UI, with and without a trailing newline, but each time I get “Ignition: no config provided by user” and I can’t SSH to the instance.

Attached is the full console logs, are there any clues in there?

gist.github.com

https://gist.github.com/mac-chaffee/4ba101a8198c318a6e7da3f4c9b6f82d

console.txt

82] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.049285] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.049286] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.049287] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.049293] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.049294] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.049299] Using ACPI (MADT) for SMP configuration information
[    0.049301] TSC deadline timer available
[    0.049303] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.049338] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]

This file has been truncated. show original

console2.txt

 for Ignition (disks) (44s / no limit)
M[K[[0;1;31m*[0m[0;31m*    [0m] A start job is running for Ignition (disks) (44s / no limit)
M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] A start job is running for Ignition (disks) (45s / no limit)
M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] A start job is running for Ignition (disks) (45s / no limit)
M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Ignition (disks) (46s / no limit)
M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (disks) (46s / no limit)
M[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (disks) (47s / no limit)
M[K[     [0;31m*[0m] A start job is running for Ignition (disks) (47s / no limit)
M[K[    [0;31m*[0;1;31m*[0m] A start job is running for Ignition (disks) (48s / no limit)
M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Ignition (disks) (48s / no limit)

This file has been truncated. show original

console3.32.20200629.3.0.txt

[H[J[1;1H[?25l[m[H[J[1;1H[20;7H[mUse the ^ and v keys to change the selection.                       
      Press 'e' to edit the selected item, or 'c' for a command prompt.   [4;80H [7m[4;1H      Fedora CoreOS 32.20200629.3.0 (ostree:0)                                 [m[4;79H[m[m[5;1H                                                                               [m[5;79H[m[m[6;1H                                                                               [m[6;79H[m[m[7;1H                                                                               [m[7;79H[m[m[8;1H                                                                               [m[8;79H[m[m[9;1H                                                                               [m[9;79H[m[m[10;1H                                                                               [m[10;79H[m[m[11;1H                                                                               [m[11;79H[m[m[12;1H                                                                               [m[12;79H[m[m[13;1H                                                                               [m[13;79H[m[m[14;1H                                                                               [m[14;79H[m[m[15;1H                                                                               [m[15;79H[m[m[16;1H                                                                               [m[16;79H[m[m[17;1H                                                                               [m[17;79H[m[m[18;1H                                                                               [m[18;79H[m[18;80H [4;79H[22;1H   The selected entry will be started automatically in 1s.                     [4;79H[22;1H   The selected entry will be started automatically in 0s.                     [4;79H[?25h[H[J[1;1H[H[J[1;1H[    0.000000] Linux version 5.6.19-300.fc32.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc version 10.1.1 20200507 (Red Hat 10.1.1-1) (GCC)) #1 SMP Wed Jun 17 16:10:48 UTC 2020
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-5745fc400d160a75dcc276a1177e34fbf2a198402ed8809c30e4336428c3706e/vmlinuz-5.6.19-300.fc32.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.firstboot rd.neednet=1 ostree=/ostree/boot.1/fedora-coreos/5745fc400d160a75dcc276a1177e34fbf2a198402ed8809c30e4336428c3706e/0 ignition.platform.id=openstack
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

This file has been truncated. show original

Sorry I don’t have much more information. If it sounds like this could be an issue with my openstack provider, I can file a ticket with them to get more information.

siosm · August 7, 2020, 7:55am

You are most likely hitting https://github.com/coreos/ignition/issues/1056

macchaffee · August 7, 2020, 3:14pm

I downgraded to the previous stable version (fedora-coreos-32.20200629.3.0-openstack.x86_64.qcow2) as mentioned in that link, but I’m still getting “Ignition: no config provided by user”. The boot logs look a little better (it at least attempts to contact the metadata server at 169.254.169.254), but still no dice unfortunately: https://gist.github.com/mac-chaffee/4ba101a8198c318a6e7da3f4c9b6f82d#file-console2-txt

siosm · August 7, 2020, 5:15pm

Consider opening an issue in the Fedora CoreOS tracker if you can not find an answer.

jlebon · August 7, 2020, 5:52pm

Hmm, those logs look truncated. Can you paste the full logs when using the previous version? That way we can verify that networking did come up.

Also, how are you passing the userdata? Are you using e.g. openstack server create --user-data config.ign ...?

macchaffee · August 7, 2020, 6:10pm

Ah sorry, I guess only the last ~1000 lines or so are saved in openstack.

I’m passing the user data through the UI like this:

Here are the untruncated logs after pressing “Launch Instance” right after that screenshot was taken:
https://gist.githubusercontent.com/mac-chaffee/4ba101a8198c318a6e7da3f4c9b6f82d/raw/a27373c7297bdda8118bfd870f6aeb62ceeae813/console3.32.20200629.3.0.txt

macchaffee · August 7, 2020, 7:38pm

Oh I’ve realized my mistake. I was booting this instance with a security group that only had “ALLOW IPv4 22/tcp from 0.0.0.0/0” (no outgoing connections allowed). As a result, the instance could not contact the metadata server to download the userdata.

The fix was to expand the security group to allow outgoing TCP traffic, at least to 169.254.169.254.
Here’s an example from the UI to allow all outgoing traffic:

Thanks for your help!