NTP broken on Silverblue 29?

Project Discussion
1

I’m enjoying the Silverblue 29 beta a lot, but it seems like I’ve been unable to turn on NTP on any systems (reproducible on multiple hosts).

Attempting to turn on “Automatic Date & Time” in the GNOME Settings just results in the toggle switching itself back off.

Similarly, trying to start the systemd-timesyncd service raises the following error:

Oct 12 02:11:09 X240 systemd[11091]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /usr/lib/systemd/systemd-timesyncd: Permission denied
-- Subject: Process /usr/lib/systemd/systemd-timesyncd could not be executed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The process /usr/lib/systemd/systemd-timesyncd could not be executed and failed.
-- 
-- The error number returned by this process is 13.
Oct 12 02:11:09 X240 systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=238/STATE_DIRECTORY
Oct 12 02:11:09 X240 systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Oct 12 02:11:09 X240 systemd[1]: Failed to start Network Time Synchronization.

There seems to be some kind of permissions error, but I don’t think it’s an SELinux issue as the audit log shows:

type=SERVICE_START msg=audit(1539306880.140:439): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-timedated comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"

which clearly states res=success (if I’m reading it right).

2

This looks like 1559286 – systemd-timesyncd fails to start: Failed to set up special execution directory in /var/lib: File exists

I dug around the GitHub issue linked from there and ended up taking a look at the spec file for systemd. It looks like there are a number of directories/files that should be created by the install of systemd-timesyncd that don’t exist.

https://src.fedoraproject.org/rpms/systemd/blob/f28/f/systemd.spec#_398-405

I tried creating those manually, but it didn’t seem to help anything. :frowning:

# mkdir -p /var/lib/private/systemd/timesync                                                               
# ln -s /var/lib/private/systemd/timesync/ /var/lib/systemd/timesync/                                                                                                                          
# touch /var/lib/private/systemd/timesync/clock  

# systemctl start systemd-timesyncd                              
Job for systemd-timesyncd.service failed because the control process exited with error code.                                                                                                                      
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.                                                                                                                                
[root@mastershake ~]# systemctl status systemd-timesyncd.service             
● systemd-timesyncd.service - Network Time Synchronization                      
   Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: disabled)                                                                                                           
   Active: failed (Result: exit-code) since Fri 2018-10-12 14:19:14 EDT; 7s ago                                                                                                                                   
     Docs: man:systemd-timesyncd.service(8)                                                                                                                                                                       
  Process: 472 ExecStart=/usr/lib/systemd/systemd-timesyncd (code=exited, status=238/STATE_DIRECTORY)
 Main PID: 472 (code=exited, status=238/STATE_DIRECTORY)                             
                                                                                                                                                                                                                  
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart.                                                                                              
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 5.
Oct 12 14:19:14 mastershake systemd[1]: Stopped Network Time Synchronization.                         
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Oct 12 14:19:14 mastershake systemd[1]: Failed to start Network Time Synchronization.
[root@mastershake ~]# journalctl --since 14:19:14 -u systemd-timesyncd
-- Logs begin at Mon 2018-04-30 16:12:58 EDT, end at Fri 2018-10-12 14:19:14 EDT. --
Oct 12 14:19:14 mastershake systemd[1]: Starting Network Time Synchronization...
Oct 12 14:19:14 mastershake systemd[454]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: File exists
Oct 12 14:19:14 mastershake systemd[454]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /usr/lib/systemd/systemd-timesyncd: File exists
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=238/STATE_DIRECTORY
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Oct 12 14:19:14 mastershake systemd[1]: Failed to start Network Time Synchronization.
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart.
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 1.
Oct 12 14:19:14 mastershake systemd[1]: Stopped Network Time Synchronization.
Oct 12 14:19:14 mastershake systemd[1]: Starting Network Time Synchronization...
Oct 12 14:19:14 mastershake systemd[458]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: File exists
Oct 12 14:19:14 mastershake systemd[458]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /usr/lib/systemd/systemd-timesyncd: File exists
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=238/STATE_DIRECTORY
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Oct 12 14:19:14 mastershake systemd[1]: Failed to start Network Time Synchronization.
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Service has no hold-off time, scheduling restart.
Oct 12 14:19:14 mastershake systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 2.
Oct 12 14:19:14 mastershake systemd[1]: Stopped Network Time Synchronization.
...

I’ve tried removing all the directories under /var/lib/private and various permutations, but it just brings us back to the original problem of Failed to set up special execution directory in /var/lib: File exists

It might be worth a shot to file another BZ about this specific problem on Silverblue or maybe a GitHub issue.

1 Like
3

Thanks for attempting to investigate! Good to see the issue is reproducible for more than just myself, even if there’s no simple fix.

It might be worth a shot to file another BZ about this specific problem on Silverblue or maybe a GitHub issue.

Is Red Hat’s Bugzilla the right place for Silverblue issues, or should I be using the Pagure tracker? I’m not familiar with Pagure but it looks easy enough that I could file a bug tomorrow for this issue if I have the time.

4

Start with the Silverblue Pagure tracker and we can go from there

5

Is this fixed? I just did a side-by-side F29 -Workstation → Silverblue and NTP doesn’t appear to work. Is it a missing package or just missing configuration files?

6

Still looks broken on my system, although now with different errors:

$ systemctl status systemd-timesyncd 
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-02-26 14:11:56 EST; 1 day 19h ago
     Docs: man:systemd-timesyncd.service(8)
  Process: 969 ExecStart=/usr/lib/systemd/systemd-timesyncd (code=exited, status=226/NAMESPACE)
 Main PID: 969 (code=exited, status=226/NAMESPACE)

Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 5.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: Stopped Network Time Synchronization.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: Failed to start Network Time Synchronization.

$ journalctl -b -u systemd-timesyncd.service
-- Logs begin at Mon 2018-04-30 16:12:58 EDT, end at Thu 2019-02-28 09:27:21 EST. --
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: Starting Network Time Synchronization...
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[933]: systemd-timesyncd.service: Failed to set up mount namespacing: No such file or directory
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[933]: systemd-timesyncd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-timesyncd: No such file or directory
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: Failed to start Network Time Synchronization.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 1.
Feb 26 14:11:56 mastershake.usersys.redhat.com systemd[1]: Stopped Network Time Synchronization.
...
7

Some googling seems to imply that this happens if ProtectHome=yes but $HOME is a symlink, if so this should probably be filed as a systemd bug. Can you try using systemctl edit to set ProtectHome=no and then start it again?

IIRC I also had to do some edits to the service to make it run, but I can’t recall them off the top of my head.

8

Filed the following 1684143 – systemd-timesyncd fails to start on Silverblue 29

I tried the systemctl edit route, but no luck:

$ systemctl cat systemd-timesyncd.service
# /usr/lib/systemd/system/systemd-timesyncd.service
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Network Time Synchronization
Documentation=man:systemd-timesyncd.service(8)
ConditionCapability=CAP_SYS_TIME
ConditionVirtualization=!container
DefaultDependencies=no
After=systemd-remount-fs.service systemd-sysusers.service
Before=time-sync.target sysinit.target shutdown.target
Conflicts=shutdown.target
Wants=time-sync.target

[Service]
Type=notify
Restart=always
RestartSec=0
ExecStart=!!/usr/lib/systemd/systemd-timesyncd
WatchdogSec=3min
User=systemd-timesync
CapabilityBoundingSet=CAP_SYS_TIME
AmbientCapabilities=CAP_SYS_TIME
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictNamespaces=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RuntimeDirectory=systemd/timesync
SystemCallFilter=@system-service @clock
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
LockPersonality=yes
StateDirectory=systemd/timesync

[Install]
WantedBy=sysinit.target
Alias=dbus-org.freedesktop.timesync1.service

# /etc/systemd/system/systemd-timesyncd.service.d/override.conf
ProtectHome=no

$ sudo systemctl daemon-reload                                                                                                                                                                                   

$ sudo systemctl start systemd-timesyncd                                                                                                                                                                         
Job for systemd-timesyncd.service failed because the control process exited with error code.       
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.                                                                                                                                                                                                       

$ sudo systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization                                                           
   Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: disabled)                  
  Drop-In: /etc/systemd/system/systemd-timesyncd.service.d                                      
           └─override.conf                                                                                                           
   Active: failed (Result: exit-code) since Thu 2019-02-28 09:44:47 EST; 6s ago                                                                                                                                                                                                          
     Docs: man:systemd-timesyncd.service(8)                                                             
  Process: 16682 ExecStart=/usr/lib/systemd/systemd-timesyncd (code=exited, status=226/NAMESPACE)                    
 Main PID: 16682 (code=exited, status=226/NAMESPACE)                                                                                        
                                                                                                                                                                                                                                                                                         
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.                                                                                                                                   
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 5.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Stopped Network Time Synchronization.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.            
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.                                                                                                                                                                    
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Failed to start Network Time Synchronization.

$ sudo journalctl -r -b -u systemd-timesyncd 
-- Logs begin at Mon 2018-04-30 16:12:58 EDT, end at Thu 2019-02-28 09:45:02 EST. --                                                        
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Failed to start Network Time Synchronization.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Stopped Network Time Synchronization.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 5.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Failed to start Network Time Synchronization.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[16682]: systemd-timesyncd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-timesyncd: No such file or directory
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[16682]: systemd-timesyncd.service: Failed to set up mount namespacing: No such file or directory
Feb 28 09:44:47 mastershake.usersys.redhat.com systemd[1]: Starting Network Time Synchronization...
9

So I just checked on my own system, and I have DynamicUsers=no…but if remove it, it still runs perfectly. However, I also have SELinux set to permissive (just because I use systemd-machined, for which the SELinux policies have been incredibly outdated and broken for quite some time). Maybe try a sudo setenforce 0, if not just for testing purposes?

10

In the BZ I filed, I tried enabling the service on another Silverblue host I had and got an SELinux denial. I’ll continue to debug the problem there.

Though worth noting, chronyd works fine on both hosts. So if folks need an NTP client, I would suggest using chronyd in the meantime.

1 Like
11

Yeah, it’s annoying to have to subtract 8 from the hours on my Silverblue desktop but not on the F29 one. :wink: