Failed To Start Rotate Log Files on Boot Fedora 35

[abdalrahman-xyz (Abdalrahman Azab)]

Hey all
this happened to me in the last 2 times i booted my pc

sudo systemctl status logrotate.service output is:

logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
     Active: failed (Result: exit-code) since Tue 2022-04-05 02:48:04 EET; 4min>
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 774 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited>
   Main PID: 774 (code=exited, status=1/FAILURE)
        CPU: 43ms

Apr 05 02:48:02 fedora systemd[1]: Starting Rotate log files...
Apr 05 02:48:04 fedora logrotate[774]: error: failed to rename /opt/netdata/var>
Apr 05 02:48:04 fedora systemd[1]: logrotate.service: Main process exited, code>
Apr 05 02:48:04 fedora systemd[1]: logrotate.service: Failed with result 'exit->
Apr 05 02:48:04 fedora systemd[1]: Failed to start Rotate log files.

i dont really know why this happened
btw i updated my karnel to 5.16.18 and i think this started to happen after that update -maybe its just by convention i dont really know-

so anyone can help to solve this?

fedora 35
karnel 5.16.18
gnome 41.5

[mhdave (Dave Buck)]

Your systemctl status logrotate output seems to be truncated on the right. Regardless, may I suggest running logrotate -v to follow along with its activities and see more about the error? By the way, have you added or modified logrotate configurations recently?
And probably the most help to you, did you try an internet search on logrotate /opt/netdata to see if you could answer your question quickly? I found this: [Bug]: fail on logrotate, permission denied · Issue #12520 · netdata/netdata · GitHub, and it includes a reference to a potential solution.

[abdalrahman-xyz (Abdalrahman Azab)]

may I suggest running logrotate -v to follow along with its activities

u mean by logrotate -v /etc/logrotate.conf ? sry bc im kinda noob btw :\
output is :

[root@fedora ~]# logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file bootlog
reading config file btmp
reading config file chrony
reading config file dnf
reading config file firewalld
reading config file httpd
reading config file iscsiuiolog
reading config file netdata
reading config file ppp
reading config file psacct
reading config file samba
olddir is now /var/log/samba/old
reading config file sssd
reading config file wpa_supplicant
reading config file wtmp
Reading state from file: /var/lib/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 14 logs

rotating pattern: /var/log/boot.log
 after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/boot.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-05 02:48
  log does not need rotating (log has been rotated at 2022-04-05 02:48, which is less than a day ago)

rotating pattern: /var/log/btmp  monthly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/btmp
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-01 00:48
  log does not need rotating (log has been rotated at 2022-04-01 00:48, which is less than a month ago)

rotating pattern: /var/log/chrony/*.log  weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/chrony/*.log
  log /var/log/chrony/*.log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/hawkey.log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/hawkey.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-03 04:23
  log does not need rotating (log has been rotated at 2022-04-03 04:23, which is less than a week ago)

rotating pattern: /var/log/firewalld  weekly (4 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/firewalld
  Now: 2022-04-05 08:07
  Last rotated at 2022-02-08 01:00
  log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)

rotating pattern: /var/log/httpd/*log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/httpd/*log
  log /var/log/httpd/*log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/iscsiuio.log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/iscsiuio.log
  log /var/log/iscsiuio.log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /opt/netdata/var/log/netdata/*.log  after 1 days (14 rotations)
empty log files are not rotated, old logs are removed
considering log /opt/netdata/var/log/netdata/access.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-05 02:48
  log does not need rotating (log has been rotated at 2022-04-05 02:48, which is less than a day ago)
considering log /opt/netdata/var/log/netdata/debug.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-02-14 00:00
  log does not need rotating (log is empty)
considering log /opt/netdata/var/log/netdata/error.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-05 02:48
  log does not need rotating (log has been rotated at 2022-04-05 02:48, which is less than a day ago)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/ppp/connect-errors  after 1 days (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/ppp/connect-errors
  log /var/log/ppp/connect-errors does not exist -- skipping

rotating pattern: /var/account/pacct  after 1 days (31 rotations)
empty log files are not rotated, old logs are removed
considering log /var/account/pacct
  Now: 2022-04-05 08:07
  Last rotated at 2022-02-08 01:00
  log does not need rotating (log is empty)

rotating pattern: /var/log/samba/log.*  weekly (99 rotations)
olddir is /var/log/samba/old, empty log files are not rotated, old logs are removed
considering log /var/log/samba/log.*
  log /var/log/samba/log.* does not exist -- skipping

rotating pattern: /var/log/sssd/*.log  weekly (2 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/sssd/sssd_kcm.log
  Now: 2022-04-05 08:07
  Last rotated at 2022-04-03 04:23
  log does not need rotating (log has been rotated at 2022-04-03 04:23, which is less than a week ago)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/wpa_supplicant.log  30720 bytes (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/wpa_supplicant.log
  log /var/log/wpa_supplicant.log does not exist -- skipping

rotating pattern: /var/log/wtmp  monthly (1 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/wtmp
  Now: 2022-04-05 08:07
  Last rotated at 2022-02-08 01:00
  log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)
set default create context to system_u:object_r:logrotate_var_lib_t:s0

did you try an internet search on logrotate /opt/netdata

ya i looked at that
also i found this which had the same problem but my logrotate.conf Permissions is root:root already

have you added or modified logrotate configurations recently?

nono Ive never needed to change its default configurations before since im still learning bc im a big noob :\

[mhdave (Dave Buck)]

Being a noob isn’t a bad thing, of course, it shows an interest in learning, which is good for all ages. I was a Unix noob back about 45 years ago.
Logrotate keeps track of when it last did a rotation or rotation check, so when you ran it with -v, it had little interest in doing rotations again when it just recently did them (“log does not need rotating”). So unless we take additional actions, nothing interesting will happen until the next rotation interval for these files. The --force option forces it to ignore the recency and do it anyway. But hang on.
Let’s look at the configuration for /opt/netdata/var/log/netdata/*.log. You should find the configuration either in /etc/logrotate.conf or in one of the files in /etc/logrotate.d. Copy and paste that here using the </> markup as you did with your previous copy/paste, that makes it more readable. Also, do an ls -la /opt/netdata/var/log/netdata as well as ls -la /etc/logrotate.d and similarly paste that here so that we can look at permissions.
Now the thing I expect is that there’s a permissions or SELinux protection issue regarding that directory. The posting at Permission issue with logrotate · Issue #2794 · netdata/netdata · GitHub suggests modifying the logrotate configuration for this directory, adding a line to tell logrotate to assume the user identity of netdata when doing this directory. If you want to try that out, make the change recommended in that posting and then run logrotate -v --force /etc/logrotate.d/netdata.conf (presuming that’s where the netdata configuration is on your system).

[computersavvy (Jeff V)]

And about the mention of selinux protection issues.
Most of those can be fixed with the command sudo restorecon -vR / then wait for that to finish. It may report updating many settings but when done the data/files should have the proper context for where they are located.

[abdalrahman-xyz (Abdalrahman Azab)]

as for /etc/logrotate.conf its configuration is :

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may also be configured here.

as for /etc/logrotate.d/netdata :

/opt/netdata/var/log/netdata/*.log {
	daily
	missingok
	rotate 14
	compress
	delaycompress
	notifempty
	sharedscripts
	postrotate
		/bin/kill -HUP `cat /opt/netdata/var/run/netdata/netdata.pid 2>/dev/null` 2>/dev/null || true
	endscript
}

ls -la /opt/netdata/var/log/netdata output is:

total 75556
drwxr-xr-x. 1 netdata netdata       56 Feb 13 20:25 .
drwxrwxr-x. 1 netdata netdata       14 Feb 13 02:26 ..
-rw-r--r--. 1 netdata netdata      276 Feb 13 20:25 access.log
-rw-r--r--. 1 netdata netdata        0 Feb 13 20:25 debug.log
-rw-r--r--. 1 netdata netdata 77364988 Apr  5 19:42 error.log

ls -la /etc/logrotate.d output is:

 total 56
 drwxr-xr-x. 1 root root  176 Apr  3 16:12 .
 drwxr-xr-x. 1 root root 4908 Apr  5 07:57 ..
 -rw-r--r--. 1 root root   91 Mar 31  2021 bootlog
 -rw-r--r--. 1 root root  130 Oct 14  2019 btmp
 -rw-r--r--. 1 root root  160 Dec 16 14:17 chrony
 -rw-r--r--. 1 root root   88 Mar 14 12:56 dnf
 -rw-r--r--. 1 root root   93 Mar 25 22:31 firewalld
 -rw-r--r--. 1 root root  331 Mar 17 18:26 httpd
 -rw-r--r--. 1 root root  172 Mar 11  2021 iscsiuiolog
 -rw-r--r--. 1 root root  239 Feb 13 20:25 netdata
 -rw-r--r--. 1 root root  136 Jul 23  2021 ppp
 -rw-r--r--. 1 root root  408 Jul 23  2021 psacct
 -rw-r--r--. 1 root root  155 Mar 16 18:49 samba
 -rw-r--r--. 1 root root  237 Jan 25 14:03 sssd
 -rw-r--r--. 1 root root  100 Jan 26 19:06 wpa_supplicant
 -rw-r--r--. 1 root root  145 Oct 14  2019 wtmp

also edited netdata.cof :

[abdalrahmanazab@fedora ~]$ sudo su -
[sudo] password for abdalrahmanazab: 
[root@fedora ~]# sudo gedit /etc/logrotate.d/netdata

#added su netdata netdata to it

[root@fedora ~]# semanage fcontext -a -t var_log_t "/opt/netdata/var/log/netdata(/.*)?"
[root@fedora ~]# restorecon -Rv /opt/netdata/var/log/netdata 
Relabeled /opt/netdata/var/log/netdata from unconfined_u:object_r:usr_t:s0 to unconfined_u:object_r:var_log_t:s0
Relabeled /opt/netdata/var/log/netdata/debug.log from system_u:object_r:usr_t:s0 to system_u:object_r:var_log_t:s0
Relabeled /opt/netdata/var/log/netdata/error.log from system_u:object_r:usr_t:s0 to system_u:object_r:var_log_t:s0
Relabeled /opt/netdata/var/log/netdata/access.log from system_u:object_r:usr_t:s0 to system_u:object_r:var_log_t:s0
[root@fedora ~]# logrotate -v --force /etc/logrotate.d/netdata.conf
error: cannot stat /etc/logrotate.d/netdata.conf: No such file or directory
Reading state from file: /var/lib/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 0 logs
set default create context to system_u:object_r:logrotate_var_lib_t:s0
[root@fedora ~]# 

then ls -la /opt/netdata/var/log/netdata again:

[root@fedora ~]# ls -la /opt/netdata/var/log/netdata
total 75572
drwxr-xr-x. 1 netdata netdata       56 Feb 13 20:25 .
drwxrwxr-x. 1 netdata netdata       14 Feb 13 02:26 ..
-rw-r--r--. 1 netdata netdata      276 Feb 13 20:25 access.log
-rw-r--r--. 1 netdata netdata        0 Feb 13 20:25 debug.log
-rw-r--r--. 1 netdata netdata 77378119 Apr  5 20:21 error.log

and ls -la /etc/logrotate.d again :

[root@fedora ~]# ls -la /etc/logrotate.d
total 56
drwxr-xr-x. 1 root root  176 Apr  5 20:20 .
drwxr-xr-x. 1 root root 4908 Apr  5 07:57 ..
-rw-r--r--. 1 root root   91 Mar 31  2021 bootlog
-rw-r--r--. 1 root root  130 Oct 14  2019 btmp
-rw-r--r--. 1 root root  160 Dec 16 14:17 chrony
-rw-r--r--. 1 root root   88 Mar 14 12:56 dnf
-rw-r--r--. 1 root root   93 Mar 25 22:31 firewalld
-rw-r--r--. 1 root root  331 Mar 17 18:26 httpd
-rw-r--r--. 1 root root  172 Mar 11  2021 iscsiuiolog
-rw-r--r--. 1 root root  259 Apr  5 20:20 netdata
-rw-r--r--. 1 root root  136 Jul 23  2021 ppp
-rw-r--r--. 1 root root  408 Jul 23  2021 psacct
-rw-r--r--. 1 root root  155 Mar 16 18:49 samba
-rw-r--r--. 1 root root  237 Jan 25 14:03 sssd
-rw-r--r--. 1 root root  100 Jan 26 19:06 wpa_supplicant
-rw-r--r--. 1 root root  145 Oct 14  2019 wtmp

i think that nothing changed

also should my logrotate.config to be containing su root adm or not
by default it isnt but i read some people r saying that so idk

maybe i have to wait a couple of days / until the next rotation date to see if the same error appeared again or it has been solved ?

Thank you for your encouragement/help btw C:

[mhdave (Dave Buck)]

Unrelated: I see you ran sudo su - then sudo gedit .... The first puts you into a superuser shell, where you could run gedit or other commands, so the second line didn’t need sudo. But, do it as one step: sudo gedit /etc/logrotate.d/netdata – easier.

Back to business … instead of waiting for the next rotation date, you can force logrotate to ignore its previously saved state. sudo logrotate -v --force /etc/logrotate.d/netdata should go through all the steps, and you’ll see whether the change you made to that file makes a difference. If successful, you’ll see that /opt/netdata/var/log/netdata has empty files for access.log, debug.log, error.log, and additional files access.log.(date) and error.log.(date), the latter files not yet compressed as that will happen the next cycle (delaycompress). As logrotate is supposed to do this daily activity, it may have already happened before you read this.
Good luck