Auditbeat 7.7.x with the System Module Socket Dataset enabled, will randomly sta…rt using 100%+ CPU on some servers. This was not an issue prior to 7.7.x.
Restarting the Auditbeat services causes CPU usage to go back to normal for a bit, but it will eventually start having issues again.
This issue doesn't seem to happen on every server, running Auditbeat on ~100 servers with the same config (below), the issue appears to occur on 10-15% of the servers. I see the issue on both OpenSUSE and CentOS servers, on multiple different kernels, and running different apps.
Screenshot showing issue (Percentages on the graph are of total CPU, not of individual cores, this example server has 4 cores, meaning Auditbeat is using one of them completely for itself):
Version Output:
```
auditbeat version
auditbeat version 7.7.1 (amd64), libbeat 7.7.1 [932b273e8940575e15f10390882be205bad29e1f built 2020-05-28 15:20:33 +0000 UTC]
```
System versions:
```
# uname -a
Linux server 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
```
Configuration:
```
###################### Auditbeat Configuration #########################
#========================== Modules configuration =============================
auditbeat.modules:
- module: auditd
resolve_ids: true
failure_mode: silent
backlog_limit: 8192
rate_limit: 0
include_raw_message: false
include_warnings: false
backpressure_stratgey: auto
# Load audit rules from separate files. Same format as audit.rules(7).
audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
audit_rules: |
## Define audit rules here.
## Create file watches (-w) or syscall audits (-a or -A). Uncomment these
## examples or add your own rules.
## If you are on a 64 bit platform, everything should be running
## in 64 bit mode. This rule will detect any use of the 32 bit syscalls
## because this might be a sign of someone exploiting a hole in the 32
## bit API.
-a always,exit -F arch=b32 -S all -F key=32bit-abi
## Executions.
-a always,exit -F arch=b64 -S execve,execveat -k exec
## External access (warning: these can be expensive to audit).
-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access
## Identity changes.
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
## Unauthorized access attempts.
-a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
-a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
- /root
- /usr/local/bin
- /home
exclude_files:
- '(?i)\.sw[nop]$'
- '~$'
- '/\.git($|/)'
- '\.rrd$'
include_files: []
scan_at_start: true
scan_rate_per_sec: 50 MiB
max_file_size: 100 MiB
hash_types: [md5,sha256]
recursive: true
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
- package # Installed, updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
- user # User information
# How often datasets send state updates with the
# current state of the system (e.g. all currently
# running processes, all open sockets).
state.period: 12h
# Enabled by default. Auditbeat will read password fields in
# /etc/passwd and /etc/shadow and store a hash locally to
# detect any changes.
user.detect_password_changes: true
# File patterns of the login record files.
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["<snipped>"]
loadbalance: true
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_tags:
tags: [auditbeat]
- dns:
type: reverse
fields:
server.ip: server.hostname
client.ip: client.hostname
source.ip: source.hostname
destination.ip: destination.hostname
nameservers: ['<snipped>']
tag_on_failure: [_dns_reverse_lookup_failed]
#================================ Logging =====================================
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/auditbeat
name: auditbeat
keepfiles: 2
permissions: 0600
rotateeverybytes: 5242880
#============================== X-Pack Monitoring ===============================
monitoring.enabled: true
monitoring.elasticsearch:
hosts: ["<snipped>"]
protocol: "https"
username: "<snipped>"
password: "<snipped>"
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["<snipped>"]
monitoring.cluster_uuid: "<snipped>"
```
For confirmed bugs, please report:
- Version: 7.7.0 + 7.7.1
- Operating System: OpenSUSE 15.0, OpenSUSE 15.1, CentOS 7, CentOS 8
- Discuss Forum URL: https://discuss.elastic.co/t/auditbeat-120-cpu/234909
- Steps to Reproduce:
1. Install Auditbeat 7.7.x (I used the RPM file and installed via RPM)
2. Configure Auditbeat to use System module with Socket portion
3. Start Auditbeat
4. Wait for Auditbeat to start consuming more CPU than it should
5. Stop Auditbeat
6. Remove Socket portion of System module
7. Start Auditbeat
8. Auditbeat will no longer use more CPU than it should