Deploying Zabbix monitoring stack to Fedora Infra (eventually replace nagios)

Project Discussion
1

Hi all,

Over the last few months on and off I’ve been working to deploy Zabbix to the fedora infra staging. I’ve received a lot of help from others along the way, and we now finally have a working instance ready to be tested [1].

It’s hooked into the Fedora FAS system for authentication, however there is one caveat. This version of Zabbix requires that a user be created on the Zabbix side. So until that is completed you won’t actually be able to login.

I’m considering creating a sysadmin-zabbix group membership as to who has accounts created/read only access, while edit/admin access will be reserved for sysadmin-main users.

There are 2 ansible roles for deployment see [2], zabbix_agent (should auto enrol hosts with server) and zabbix_server (covers deployment of zabbix server itself). The server is running RHEL9 and its running Zabbix LTS 6.x. I went with postgres db running locally on the same machine.

I’d like to invite interested folks that want to come take a look, poke the tyres make themselves known, I’ll create an account on the zabbix staging server which you can then login via your FAS account at [3]. If you have feedback/suggestions regarding the deployment etc nows the time to make them while I have cycles to action it!

Once everyones happy, I’ll go ahead with a production deployment, and begin rolling the zabbix agent out to all hosts. Lot of work to be done to start replacing the nagios monitoring with Zabbix equivalents, so potentially plenty of small pieces of work people interested in contributing might be willing to take on!

  1. https://zabbix.stg.fedoraproject.org ↩︎

  2. https://pagure.io/fedora-infra/ansible/blob/main/f/roles/zabbix ↩︎

  3. https://accounts.stg.fedoraproject.org ↩︎

1 Like
2

Hi all,

Over the last few months on and off I’ve been working to deploy Zabbix to the fedora infra staging. I’ve received a lot of help from others along the way, and we now finally have a working instance ready to be tested [1].

Awesome work. Thanks for doing this!

It’s hooked into the Fedora FAS system for authentication, however there is one caveat. This version of Zabbix requires that a user be created on the Zabbix side. So until that is completed you won’t actually be able to login.

I’m considering creating a sysadmin-zabbix group membership as to who has accounts created/read only access, while edit/admin access will be reserved for sysadmin-main users.

Does zabbix have anything like a ‘guest’ view? With nagios currently we
let anyone look / read-only and thats sometimes helpful in see when
things are broken, etc.

Actually, looking at it we allowed any valid user to login, but just
didn’t give them privs to ack things/supress alerts/do anything.

Perhaps we could just reuse the sysadmin-noc group here? (for the lower
access that can ack/etc) and main for the admin access?

I think I saw that the next major zabbix version should allow creating
accounts automatically, so perhaps we just don’t worry too much for now
and move to that version when it’s out/we can?

There are 2 ansible roles for deployment see [2], zabbix_agent (should auto enrol hosts with server) and zabbix_server (covers deployment of zabbix server itself). The server is running RHEL9 and its running Zabbix LTS 6.x. I went with postgres db running locally on the same machine.

Sounds good. We do have a generic db server, but perhaps in this case it
might be better to be self contained to avoid the dependency on that
server being up/working. :wink:

I’d like to invite interested folks that want to come take a look, poke the tyres make themselves known, I’ll create an account on the zabbix staging server which you can then login via your FAS account at [3]. If you have feedback/suggestions regarding the deployment etc nows the time to make them while I have cycles to action it!

Awesome. I poked around a bit, but not too much yet, will try more next
week.

I wonder if we couldn’t roll the agent out to the rest of .stg. soon?
Might be nice to get more hosts in there and confirm that the agent
works.

Once everyones happy, I’ll go ahead with a production deployment, and begin rolling the zabbix agent out to all hosts. Lot of work to be done to start replacing the nagios monitoring with Zabbix equivalents, so potentially plenty of small pieces of work people interested in contributing might be willing to take on!

There is a zabbix ansible collection. We may want to look at pacakaging
that up and using it to configure things? I think the base agent stuff
is probibly good for much of what nagios does, but the ‘special’ checks
may have to be implemented in ansible, etc. I’m not sure if the
collection does this kind of thing or it’s more base config, but we
should look into it. :wink:

We should perhaps look at rolling to prod after f39 beta freeze, or
after f39 is out. Then we can run both zabbix and nagios for a while and
migrate things over. I’m sure there’s stuff we have added to nagios over
the years we need to reimplement in zabbix.

One of those that leaps to mind would be the ability to send alerts to
our matrix channel (or some subset of them). I’m sure we can figure
something out.

Anyhow, thanks again for moving this forward!

2 Likes
3

Hi Kevin,

Does zabbix have anything like a ‘guest’ view? …

Ok had a quick look, it does have a guest view as it happens, you can login and see the dashboards but not change anything. I’ve enabled it there. On the login page there is a new option to login as guest. It doesn’t require any credentials.

Perhaps we could just reuse the sysadmin-noc group here? (for the lower
access that can ack/etc) and main for the admin access?

Sounds good yeah, I’ll look into a set of permissions/role that can ack notifications, and should be able to have sysadmin-noc accounts be given this role.

I’d prefer to try stick to the LTS versions just to limit maintenance pain, some periodic sync to create users eg by running a playbook, might be less painful than going to 6.4 (which has that new features regarding SAML logins and bein able to auto create accounts).

I wonder if we couldn’t roll the agent out to the rest of .stg. soon?

I’ll start working on this now ;D

There is a zabbix ansible collection.

Yup its pretty powerful, I was playing around with it, can administer a lot of config. We can create custom host templates for targetting systems with a particular role or OS. I’ll take a look at packaging that up so.

4

Hi Kevin,

Does zabbix have anything like a ‘guest’ view? …

Ok had a quick look, it does have a guest view as it happens, you can login and see the dashboards but not change anything. I’ve enabled it there. On the login page there is a new option to login as guest. It doesn’t require any credentials.

Awesome. Thats a great find!

Perhaps we could just reuse the sysadmin-noc group here? (for the lower
access that can ack/etc) and main for the admin access?

Sounds good yeah, I’ll look into a set of permissions/role that can ack notifications, and should be able to have sysadmin-noc accounts be given this role.

+1

I’d prefer to try stick to the LTS versions just to limit maintenance pain, some periodic sync to create users eg by running a playbook, might be less painful than going to 6.4 (which has that new features regarding SAML logins and bein able to auto create accounts).

Sure… agreed. When is the next LTS one coming out?
I mean we can just setup some kind of stopgap now, and once that one
comes out move to it.

I wonder if we couldn’t roll the agent out to the rest of .stg. soon?

I’ll start working on this now ;D

Sounds great. Will be nice to see more data in there…also will tell us
if things scale ok.

There is a zabbix ansible collection.

Yup its pretty powerful, I was playing around with it, can administer a lot of config. We can create custom host templates for targetting systems with a particular role or OS. I’ll take a look at packaging that up so.

Excellent!

This is all moving along nicely…thanks again for driving it forward.

1 Like