Help with systemd units for containers in ignition file

Project Discussion
1

Hi all,
I am trying to setup a Fedora CoreOS VM on my FreeNAS host, mainly as a testing environment to get to know Docker, but also to potentially run some services in “home production use” in the future.

My plan was to create an ignition file that sets up an NFS-Client to provide storage to all containers and a portainer container.

For all future configuration and messing around with containers I intended to use portainer.

I set up the two mentioned services as systemd units.
Unfortunately the two units both have some issues:

The NFS service can’t reach the host:

[core@coreos ~]$ sudo systemctl status docker.nfs.service 
● docker.nfs.service - NFS Client Container for Shared Volume
   Loaded: loaded (/etc/systemd/system/docker.nfs.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-06-14 14:34:49 UTC; 1h 59min left
  Process: 1548 ExecStartPre=/usr/bin/docker stop docker.nfs.service (code=exited, status=1/FAILURE)
  Process: 1649 ExecStartPre=/usr/bin/docker rm docker.nfs.service (code=exited, status=1/FAILURE)
  Process: 1670 ExecStartPre=/usr/bin/docker pull d3fk/nfs-client (code=exited, status=0/SUCCESS)
 Main PID: 1996 (docker)
    Tasks: 11 (limit: 2288)
   Memory: 75.2M
   CGroup: /system.slice/docker.nfs.service
           └─1996 /usr/bin/docker run --rm --privileged=true --name docker.nfs.service --net=host -v /mnt/shared_nfs:/mnt/nfs-1:shared -e SERVER=192.168.1.2 -e SHARE=/mnt/ssd_data/CoreOSdata d3fk/nfs-client

Jun 14 14:34:47 coreos.fritz.box docker[1548]: Error response from daemon: No such container: docker.nfs.service
Jun 14 14:34:48 coreos.fritz.box docker[1649]: Error: No such container: docker.nfs.service
Jun 14 14:34:48 coreos.fritz.box docker[1670]: Using default tag: latest
Jun 14 14:34:49 coreos.fritz.box docker[1670]: latest: Pulling from d3fk/nfs-client
Jun 14 14:34:49 coreos.fritz.box docker[1670]: Digest: sha256:8ecd17f30c0f0f8e30ebc8c0c605de4e1d1b53e9b103a98b173483c2f7f84b5f
Jun 14 14:34:49 coreos.fritz.box docker[1670]: Status: Image is up to date for d3fk/nfs-client:latest
Jun 14 14:34:49 coreos.fritz.box systemd[1]: Started NFS Client Container for Shared Volume.
Jun 14 14:34:50 coreos.fritz.box docker[1996]: /usr/local/bin/entry.sh: line 5: [192.168.1.2: not found
Jun 14 14:34:50 coreos.fritz.box docker[1996]: /dev/vda4 on /mnt/nfs-1 type xfs (rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
Jun 14 14:34:50 coreos.fritz.box docker[1996]: 192.168.1.2:/mnt/ssd_data/CoreOSdata on /mnt/nfs-1 type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.2,mountvers=3,mountproto=tcp,local_lock>

Portainer is inactive (dead) right after startup:

[core@coreos ~]$ systemctl status docker.portainer.service
● docker.portainer.service - Portainer Admin Container
   Loaded: loaded (/etc/systemd/system/docker.portainer.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2020-06-14 12:34:53 UTC; 46s ago
  Process: 1997 ExecStartPre=/usr/bin/docker stop docker.portainer.service (code=exited, status=1/FAILURE)
  Process: 2012 ExecStartPre=/usr/bin/docker rm docker.portainer.service (code=exited, status=1/FAILURE)
  Process: 2019 ExecStartPre=/usr/bin/docker pull portainer/portainer (code=exited, status=0/SUCCESS)
  Process: 2097 ExecStart=/usr/bin/docker run -d -p 9000:9000 -p 8000:8000 --name docker.portainer.service --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /mnt/shared_nfs:/data portainer/portainer --no-analytics (code=exited, status=0/SUCCESS)
 Main PID: 2097 (code=exited, status=0/SUCCESS)

Jun 14 14:34:49 coreos.fritz.box systemd[1]: Starting Portainer Admin Container...
Jun 14 14:34:49 coreos.fritz.box docker[1997]: Error response from daemon: No such container: docker.portainer.service
Jun 14 14:34:49 coreos.fritz.box docker[2012]: Error: No such container: docker.portainer.service
Jun 14 14:34:49 coreos.fritz.box docker[2019]: Using default tag: latest
Jun 14 12:34:52 coreos.fritz.box docker[2019]: latest: Pulling from portainer/portainer
Jun 14 12:34:53 coreos.fritz.box docker[2019]: Digest: sha256:55c7614b1ad61eabc27214299c42d41bb49e5ef78238c0e5783031f041499284
Jun 14 12:34:53 coreos.fritz.box docker[2019]: Status: Image is up to date for portainer/portainer:latest
Jun 14 12:34:53 coreos.fritz.box systemd[1]: Started Portainer Admin Container.
Jun 14 12:34:53 coreos.fritz.box docker[2097]: ec941c771288e7aa4e692e5adf3591ff1d24a00b6fb9cb2add0becb4e17f2942
Jun 14 12:34:53 coreos.fritz.box systemd[1]: docker.portainer.service: Succeeded.

However when I stop the two services and manually start them by running the same lines as in the Unit’s ExecStart command, the NFS share is mounted as expected and portainer starts up, reachable via web-UI and populates the shared folder.

Could anyone hint me at what I am doing wrong with my systemd setup?

Also if my general approach is bad practice, I’d be eager to learn how this is done right!

Here is my fcc file for my ignition file:

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ecdsa-sha2-nistp521 AAAA...
systemd:
  units:
  - name: docker.nfs.service
    enabled: true
    contents: |-
      [Unit]
      Description=NFS Client Container for Shared Volume
      After=docker.service
      Requires=docker.service

      [Service]
      TimeoutStartSec=0
      Restart=always
      ExecStartPre=-/usr/bin/docker stop docker.nfs.service
      ExecStartPre=-/usr/bin/docker rm docker.nfs.service
      ExecStartPre=/usr/bin/docker pull d3fk/nfs-client
      ExecStart=/usr/bin/docker run --rm --privileged=true --name docker.nfs.service --net=host -v /mnt/shared_nfs:/mnt/nfs-1:shared -e SERVER=192.168.1.2 -e SHARE=/mnt/ssd_data/CoreOSdata d3fk/nfs-client

      [Install]
      WantedBy=multi-user.target
  - name: docker.portainer.service
    enabled: true
    contents: |-
      [Unit]
      Description=Portainer Admin Container
      After=docker.service docker.nfs.service
      Requires=docker.service docker.nfs.service

      [Service]
      TimeoutStartSec=0
      Restart=always
      ExecStartPre=-/usr/bin/docker stop docker.portainer.service
      ExecStartPre=-/usr/bin/docker rm docker.portainer.service
      ExecStartPre=/usr/bin/docker pull portainer/portainer
      ExecStart=/usr/bin/docker run -d -p 9000:9000 -p 8000:8000 --name docker.portainer.service --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /mnt/shared_nfs:/data portainer/portainer --no-analytics

      [Install]
      WantedBy=multi-user.target

Thanks in advance
Leo

2

From the logs, your container may be having some issues on startup, please double-check with whoever is the author:

Additionally, systemd and docker do not really play well together. If you run a container with docker run -d then you are effectively just tracking the client with systemd. The container itself is daemonized and supervisioned by the docker daemon, while the docker client quickly returns. This is what you are seeing in your systemctl status.

3

Hi lucab,
thanks for looking into my issue and for pointing out the problematic “-d” flag. I did find some more reading on docker and systemd.

I came up with this fcc file, which creates a setup that mounts my nfs and install portainer to avoid the commandline after the initial setup:

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ecdsa-sha2-nistp521 AAAA...
systemd:
  units:
  - name: docker.nfs.service
    enabled: true
    contents: |-
      [Unit]
      Description=NFS Client Container for Shared Volume
      After=docker.service
      Requires=docker.service network.target network-online.target

      [Service]
      TimeoutStartSec=0
      Restart=always
      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull d3fk/nfs-client
      # Privileged mode is required due to SELINUX (https://github.com/portainer/portainer/issues/849)
      ExecStart=/usr/bin/docker run --rm --privileged=true --name %n --net=host -v /mnt/shared_nfs:/mnt/nfs-1:shared -e SERVER=192.168.1.2 -e SHARE=/mnt/ssd_data/CoreOSdata d3fk/nfs-client
      ExecStop=/usr/bin/docker stop -t 15 %n

      [Install]
      WantedBy=multi-user.target
  - name: docker.portainer.service
    enabled: true
    contents: |-
      [Unit]
      Description=Portainer Admin Container
      After=docker.service docker.nfs.service
      Requires=docker.service docker.nfs.service

      [Service]
      Type=oneshot
      RemainAfterExit=yes
      TimeoutStartSec=0
      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull portainer/portainer
      ExecStart=-/usr/bin/mkdir -p /mnt/shared_nfs/portainer_data
      # Privileged mode is required for binding to local socket to work due to SELINUX (https://github.com/portainer/portainer/issues/849)
      ExecStart=/usr/bin/docker run --rm --privileged=true -d -p 9000:9000 --name %n --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /mnt/shared_nfs/portainer_data:/data portainer/portainer --no-analytics --host=unix:///var/run/docker.sock
      ExecStop=/usr/bin/docker stop -t 15 %n

      [Install]
      WantedBy=multi-user.target

The “not found” message is indeed an issue within the nfs-client image which however doesn’t seem to affect its function.