Setup Zabbix monitoring with Oracle back end repository



We currently have a number of monitoring tools in place in the organisation (about 5 at last count - and there may be more....). None of them seems to do exactly what the organisation wants as different areas require different functionality, we also have licence and access restrictions in place for some of those tools.

Anyway we are now evaluating Zabbix to see how that can help us and how it compares with other tools already in place.

For those of you not in the know (whom i was one of until very recently), Zabbix is an open source monitoring tool - you can read all the spiel about it here: http://www.zabbix.com/

So lets get down to it then - i want to install this and as i don't have root access and getting anything done by the central admin function is pretty painful - so i don't want to use any prebuilt packages as these can only be loaded by the root user.

Instead i'm doing this the hardcore way and downloading all the sources and i'm going to compile and build all of this myself - and if that's not enough i'm going to use Oracle as the back end repository database where normally this would be mySQL (I don't want to have to learn mySQL - at least not right now anyway....)

My environment in SLES11sp3 and the repo database I'm planning to use is Oracle 11.2.0.4.3

So first things first - and this is the only point i have to involve the unix team - i need a new group creating called zabbix and a new user created called zabbix (with zabbix as it's primary group)

so just

#groupadd zabbix
#useradd -g zabbix zabbix

Now that's out of the way i can get down to the serious task of installing the software - now as i went through this i discovered that more and more sources were needed to be able to set all this up (particularly in the apache/php area) - I'm listing them all here for completeness - i would advise you to read all of this post before you start doing anything as i didn't do things in the most logical order and there were problems along the way.

So here is the complete list of sources

zabbix-2.4.4rc1.tar.gz
httpd-2.2.29.tar.gz
php-5.6.7.tar.gz
libxml2-2.9.2.tar.gz
freetype-2.5.5.tar.gz
libpng-1.6.16.tar.gz
zlib-1.2.8.tar
jpegsrc.v6b.tar.gz

Depending on what your current system has you may need more or less than this - but this did it for me.

First up was the zabbix source itself - i needed to unzip and extract that to get to the files i needed to create the database schema inside oracle - to get to these i did the following

gunzip zabbix-2.4.4rc1.tar.gz
tar -xvf zabbix-2.4.4rc1.tar
cd ./zabbix-2.4.4rc1/database/oracle

In this directory are 3 files

 ls
data.sql  images.sql  schema.sql

The install docs tell you to create a schema to hold the objects but they give no clue as to what privileges it should have - so i took a look at the files and ran the following steps

create tablespace zabbix;
create user zabbix identified by zabbix default tablespace zabbix;
alter user zabbix quota unlimited on zabbix;
grant resource,create session to zabbix;
grant create any directory to zabbix;

So now we have somewhere to create the objects lets login and run the supplied script

sqlplus zabbix/zabbix
SQL>@schema

Now first run through this spewed out loads of errors basically because the column length is too long for oracle (which makes me wonder if this was tested...)

Anyway i did a global replace of

nvarchar2(2048)

with

nvarchar2(2000)

in the schema.sql file and then dropped and recreated the user - i then re-ran it and all was fine.

Next stage is to load the images into the database

To enable this the images.sql file must be edited to change the path to where the images are located - in my case this involved changing this line

CREATE OR REPLACE DIRECTORY image_dir AS '/home/zabbix/zabbix/create/output_png'
/

to this

CREATE OR REPLACE DIRECTORY image_dir AS '/export/home/zabbix/zabbix-2.4.4rc1/misc/images'
/

Then the script runs ok - it just throws a single error at the end which we can ignore as its just a tidy up

DROP DIRECTORY image_dir
*
ERROR at line 1:
ORA-01031: insufficient privileges


So that's images done - now onto some basic config data that has to be loaded

This was the easiest of the three and ran without error - it just needed a commit at the end.

OK - now the database is ready to go - now onto the Zabbix software (and the supporting cast.....)

So first up i got to the directory where i untar'd zabbix and i run the configure command specifying explicit install paths, the fact i want server and agent and the fact i want to use oracle - this seems very happy and gives me a nice output at the end

cd /export/home/zabbix/zabbix-2.4.4rc1
./configure --prefix=/export/home/zabbix --enable-server --enable-agent --with-oracle=/oracle/11.2.0.4.3.DB


Configuration:

  Detected OS:           linux-gnu
  Install path:          /export/home/zabbix
  Compilation arch:      linux

  Compiler:              gcc
  Compiler flags:        -g -O2   -I/oracle/11.2.0.4.3.DB/rdbms/public -I/oracle/11.2.0.4.3.DB/rdbms/demo

  Enable server:         yes
  Server details:
    With database:         Oracle
    WEB Monitoring:        no
    Native Jabber:         no
    SNMP:                  no
    IPMI:                  no
    SSH:                   no
    ODBC:                  no
    Linker flags:          -rdynamic      -L/oracle/11.2.0.4.3.DB/lib
    Libraries:             -lm -ldl -lrt  -lresolv     -lclntsh -lnnz11

  Enable proxy:          no

  Enable agent:          yes
  Agent details:
    Linker flags:          -rdynamic
    Libraries:             -lm -ldl -lrt  -lresolv

  Enable Java gateway:   no

  LDAP support:          no
  IPv6 support:          no

***********************************************************
*            Now run 'make install'                       *
*                                                         *
*            Thank you for using Zabbix!                  *
*              <http://www.zabbix.com>                    *
***********************************************************

I then do

make install

and it's all gone very smoothly....

I now update
serveractive = server shortname
server = server shortname

 in the

/export/home/zabbix/etc/zabbix_agentd.conf

and in

 zabbix_server.conf

i set

dbname=TNSNAMES entry of repo database
dbuser=zabbix
dbpassword=zabbix

And that is zabbix configured (at least from a backend point of view) - now it got more tricky - the front end needed a load of other components to make it work. I went through a load of trial and error to get this to work - some of which is detailed below.....

For the front end then first of all we need apache - lets get that done

Don't fall into the trap of just trying to build apache directly - there are some other bits to do first from the downloaded file - first up is apr

cd /export/home/zabbix/httpd-2.2.29/srclib/apr
./configure --prefix /export/home/zabbix/apr
make install

then apr-util (which is dependant on apr)

cd ../apr-util
./configure --prefix=/export/home/zabbix/apr-util --with-apr=/export/home/zabbix/apr
make
make install

then apache itself (dependent on the 2 we already built)

cd ../..
 ./configure --prefix /export/home/zabbix/apache --with-apr=/export/home/zabbix/apr  --with-apr-util=/export/home/zabbix/apr-util
make
make install

In the apache config file (/export/home/zabbix/apache/conf/httpd.conf ) i switch the port to 3333 (rather than the root only port 80 and to keep it away from other used ports)

So i change

Listen 3333

I then copy the Zabbix front end files under the apache tree

cd /export/home/zabbix/apache/htdocs
mkdir zabbix
cd /export/home/zabbix/zabbix-2.4.4rc1/frontends/php
cp -a * /export/home/zabbix/apache/htdocs/zabbix

Now we can start apache

/export/home/zabbix/apache/bin> ./apachectl start

The basic test page works fine

However as soon as i try the zabbix pages i just get a directory listing and they are not processed correcly. After some basic trial and error it emerges that you have to install php (I had foolishly thought that this somehow came with apache....)

So lets get php - for brevity I'm just going to show the working steps - but this is where i spent most of my effort......

So i get all of these.....

php-5.6.7.tar.gz
libxml2-2.9.2.tar.gz
freetype-2.5.5.tar.gz
libpng-1.6.16.tar.gz
zlib-1.2.8.tar
jpegsrc.v6b.tar.gz

and do the following

gunzip libxml2-2.9.2.tar.gz
tar -xvf libxml2-2.9.2.tar
./configure --prefix=/export/home/zabbix/libxml2 --without-python
make
make install

gunzip freetype-2.5.5.tar.gz
tar -xvf freetype-2.5.5.tar
./configure --prefix=/export/home/zabbix/freetype
make
make install

gunzip zlib-1.2.8.tar.gz
 tar -xvf zlib-1.2.8.tar
 cd zlib-1.2.8
 ./configure --prefix=/export/home/zabbix/zlib
 make
 make install

export LDFLAGS="-L/export/home/zabbix/zlib/lib"
export CPPFLAGS="-I/export/home/zabbix/zlib/include"

gunzip libpng-1.6.16.tar.gz
 tar -xvf libpng-1.6.16.tar
./configure --prefix=/export/home/zabbix/libpng

gunzip jpegsrc.v6b.tar.gz
tar -xvf jpegsrc.v6b.tar
cd jpeg-6b
   mkdir /export/home/zabbix/libjpeg/lib
mkdir  /export/home/zabbix/libjpeg/
mkdir /export/home/zabbix/libjpeg/bin
 mkdir -p /export/home/zabbix/libjpeg/man/man1
./configure --prefix=/export/home/zabbix/libjpeg --enable-shared

Edit the makefile for this one to change ./libtool to just libtool

make 
make install

 gunzip php-5.6.7.tar.gz
tar -xvf php-5.6.7.tar
cd php-5.6.7
./configure --prefix=/export/home/zabbix/php --with-apxs2=/export/home/zabbix/apache/bin/apxs --with-libxml-dir=/export/home/zabbix/libxml2 --enable-bcmath --enable-mbstring --enable-sockets --with-gd --with-gettext --with-oci8=/oracle/11.2.0.4.3.DB --with-png-dir=/export/home/zabbix/libpng --with-freetype-dir=/export/home/zabbix/freetype --with-zlib-dir=/export/home/zabbix/zlib --with-jpeg-dir=/export/home/zabbix/libjpeg
make
make install

Now after all that messing about i now have a working apache/php setup (however there is still a little bit of config i need to set for php itself). Now i wasn't sure where this config file should be located - but fortunately it's easy enough to find out

you just create a simple file info.php in htdocs/zabbix with this content

<?php phpinfo(); ?>

When you call this page from a browser it tells you where the config file should be

In this file (/export/home/zabbix/php/lib/php.ini ) i paste the following content (which is case sensitive i found out....)

max_execution_time = 600
expose_php = off
date.timezone = CET
post_max_size = 32M
upload_max_filesize = 16M
max_input_time = 600
memory_limit = 256M
always_populate_raw_post_data = -1

Now all that is done i just need to check/amend some entries in the httpd.conf for apache so that the php stuff is correctly working

LoadModule php5_module        modules/libphp5.so

<FilesMatch "\.ph(p[2-6]?|tml)$">
    SetHandler application/x-httpd-php
</FilesMatch>

<IfModule mime_module>
    AddType application/x-httpd-php .php
</IfModule>

And in mime.types (not sure if this is actually needed)

application/x-httpd-php                         php

Right now all that lot is in place i can start up everything (note the following 2 env variables must be defined)

 export ORACLE_HOME=/oracle/11.2.0.4.3.DB
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

(oh and make sure your tnsnames entry is set up correctly for your repo DB....)

we start apache
/export/home/zabbix/apache/bin/apachectl start

zabbix server
/export/home/zabbix/sbin/zabbix_server

zabbix agent
/export/home/zabbix/sbin/zabbix_agentd

And everything starts without error......... (no really it does - i've spared you some of the pain)


OK - thats all the back end stuff done - now to go through the web gui to set that lot up - this is just a 'wizard' so is pretty painless once everything is set up OK

So navigating to the web gui is just http://zabbix-server:3333/zabbix/index.php - skipping through the welcome screen - the main screen to get past is this one (here is one of my earlier attempts before everything was set up) - you can't move on until everything is green


Next up is db connection



Now server details - just specify hostname again here


Then we get a summary


And away we go


Now we can navigate to the login screen


And there we go it's working - we just need to update the agent config for the zabbix server host to be the hostname and not localhost and then tick the enable box - see the 3 fields highlighted below



And there we have a fully working system just monitoring a single host - and we can see some graphs and things already - like this one


So now its a case of add more agents and configure monitoring as appropriate.

Note again - no root rights are needed to run or configure this (other than the user creation at the start)

Lets see how the battle of the monitoring tools is won (or lost....)

7 comments:

  1. Hi Rich,
    thank you for what you've done so far and for documenting it!

    ReplyDelete
  2. Hello,

    Are you using a SID or a SERVICE_NAME entry in your tnsnames.ora ?

    Gautier

    ReplyDelete
    Replies
    1. Hi Gautier,
      SERVICE_NAME - though SID should work just as well. SID is kind of the old way of doing things in oracle though - everything should have switched to SERVICE_NAME's long ago... :-)

      Just depends on how your DBA configured the listener.

      Cheers,
      Rich

      Delete
  3. Hi Rich.

    In our case, we have to setup Zabbix to connect into an Oracle database which has an active dataguard. So for failover purposes we have to configure a different connect string in tnsnames.ora file pointing to both PRIMARY and STANDBY DBs, using service_name to point to correct node. No problem to configure zabbix server part, since it uses a tnsnames.ora file to point to correct database, however how to do it for Apache part ? I saw that there is a zabbix.conf.php file but it points to only a single dbhost. Is it possible to perform this kind of setup ?

    Thx

    Fabio

    ReplyDelete
    Replies
    1. Hi Fabio,
      I'm not sure the web gui supports anything other than the basic style of connection string with host/port/database - which then means it cant handle dataguard......

      If you had RAC you'd be OK with the SCAN VIP I guess....

      All I can think is that you have dns alias for zabbix backend database and you can somehow trigger a dns update when the service switches to the other node - all a bit messy though.

      Maybe there is some way of switching the web gui to use oci but I've not looked into that and for now we have a single server with MySQL as the backed for zabbix so I can't even test anything for you...

      Cheers,
      Rich

      Delete
  4. Thanks for your response, Rich.

    Actually we are thinking about all technical possibilities. To create a DNS over Scan Ips or even change zabbix configuration files after a database switchover are being considered.

    Thanks a lot :)

    Fabio

    ReplyDelete
    Replies
    1. Hi Pabio,
      Just be chance I'm looking into some other stuff for MySQL but i think i can see another solution to this - the php file ( /usr/share/zabbix/include/db.inc.php) used to connect to oracle has a construct like this


      case ZBX_DB_ORACLE:
      $connect = '';
      if (!empty($DB['SERVER'])) {
      $connect = '//'.$DB['SERVER'];

      if ($DB['PORT'] != '0') {
      $connect .= ':'.$DB['PORT'];
      }
      if ($DB['DATABASE']) {
      $connect .= '/'.$DB['DATABASE'];
      }
      }

      $DB['DB'] = @oci_connect($DB['USER'], $DB['PASSWORD'], $connect);
      The oci_connect line can i think be changed to just include a full tns entry - or if you get your env right tnsnames i guess - might be worth looking in to that approach - so the connect line could be something like this?


      oci_connect($username, $password, '(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.14)(PORT = 1521)) (CONNECT_DATA = (SERVICE_NAME = ORCL) (SID = ORCL))with extra dataguard host here etc)');

      Delete