View source | Discuss this page | Page history | Printable version   

Release Management/System Monitoring with munin/monit

Bulbgraph.png   This is an old document about monitoring, please go here for latest.




Release Management/Internal

Index


Contents

Introduction

This document gives a brief on why and how to use Monitoring tools to monitor a system/server.

Why

Tools

Munin: It is the server node, which collects data from various munin-nodes or muninlite working on different servers and uses that data to generate the graphs. By default it runs after every 5 minutes (cron).

Munin-node: It is a client node which collects the data from the system (proc and other log files) as specified by its plugins when ever munin (server-node) requests for it. By default this runs on port 4949.

Muninlite: It is a lighter version of munin-node, this is lighter as this is built only using bash script. But it has a draw back too, it has less plugins and it does not works properly on ubuntu systems.

Monit: This tools has an upper hand as it deals with the services (apache/ssh/mysql etc) running on the system and is capable of restarting/stopping/starting/monitoring from the web interface. By default this runs on port 2812.

M/monit: This is a fancy dashboard which in terms of working is same as monit's web interface but is more presentable. By default it runs on port 8080.

Installation & Configuration

Munin and Munin-node

Gentoo

Install And Configure munin

emerge -av munin
emerge --config net-analyzer/munin

Configuring Plugins

sudo -u munin munin-node-configure --shell

Best way to add plugins is to creating a symlink

ln -s /usr/libexec/munin/plugins/<plugin name> /etc/munin/plugins/
rc-update add munin-node default
/etc/init.d/munin-node start

Debian

Install And Configure munin

apt-get install munin munin-node 

Edit /etc/munin/munin.conf

dbdir   /var/lib/munin
htmldir /var/www/munin/
logdir  /var/log/munin
rundir  /var/run/munin

tmpldir /etc/munin/templates

[server1.example.com]
    address 127.0.0.1
    use_node_name yes
# Note:/var/www/munin/ should be your apache root directory or somewhere inside that.
mkdir -p /var/www/munin
chown munin:munin /var/www/munin
/etc/init.d/munin-node restart

After sometime you can see the graphs in http://<server IP>/<Path to munin dir.>

Muninlite

Mininlite requires xinetd/inetd to work

General

These steps are common for both gentoo and debian systems.

cd /usr/local
wget http://downloads.sourceforge.net/project/muninlite/muninlite%20-%20stable/1.0.2/muninlite-1.0.2.tar.gz?use_mirror=biznetnetworks
tar -xvzf muninlite-1.0.2.tar.gz
mv muninlite-1.0.2 muninlite
cd muninlite
make
cp munin-node /usr/local/bin
chmod +x /usr/local/bin/munin-node
echo "munin           4949/tcp        lrrd            # Munin" >>/etc/services
With xinetd

add local ip of munin's servers ip to examples/xinetd.d/munin, by this you tell which is munin server to connect to

vi examples/xinetd.d/munin 
only_from       = 10.226.51.178 #local ip of irc.openbravo.com
cp examples/xinetd.d/munin /etc/xinetd.d
/etc/init.d/xinet.d restart
With inetd
cat examples/inetd.conf >> /etc/inetd.conf
/etc/init.d/inet.d restart

Monit and M/monit

Install And Configure munin (x86 specific).

Monit

cd /usr/local
wget http://mmonit.com/monit/dist/monit-5.0.3.tar.gz
tar -xvzf monit-5.0.3.tar.gz
cd monit-5.0.3
./configure
make
make install

Edit /usr/local/monit-5.0.3/monitrc according to what all services you need to monitor.

set mmonit http://monit:monit@<M/monit IP>:8080/collector
      set httpd port 2812 and use address <local IP>
          allow localhost
          allow <server IP>

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host www.example.com port 80 protocol http
      and request "/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

  check system localhost
    if loadavg (1min) > 4 then alert
    if loadavg (5min) > 2 then alert
    if memory usage > 75% then alert
    if cpu usage (user) > 70% then alert
    if cpu usage (system) > 30% then alert
    if cpu usage (wait) > 20% then alert

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@monitoring-server.com }
set alert email@domain.com
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /var/certs/monit.pem
     allow <user>:<password>


M/monit

cd /usr/local
wget wget http://mmonit.com/dist/mmonit-2.0.3-linux-x86.tar.gz
tar -xvzf mmonit-2.0.3-linux-x86.tar.gz
cd mmonit-2.0.3
./bin/mmonit start

Now you can browse through using http://<server IP>:8080/


Gentoo

Install And Configure munin (if using only monit and no M/monit).

emerge -va monit
/etc/init.d/monit start

Edit /etc/monitrc like this.

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@server1.example.com }
set alert root@localhost
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /var/certs/monit.pem
     allow <user>:<password>

check process proftpd with pidfile /var/run/proftpd.pid
   start program = "/etc/init.d/proftpd start"
   stop program  = "/etc/init.d/proftpd stop"
   if failed port 21 protocol ftp then restart
   if 5 restarts within 5 cycles then timeout

check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/ssh start"
   stop program  "/etc/init.d/ssh stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host www.example.com port 80 protocol http
      and request "/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

Debian

Install And Configure munin (if using only monit and no M/monit).

apt-get install monit

Edit vi /etc/monit/monitrc same as above (gentoo).

Configuring apache and secure connection.

mkdir /var/www/monit
echo "hello" > /var/www/monit/token
mkdir /var/certs
cd /var/certs

Create /var/certs/monit.cnf to look like this.

# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = IN

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = State

localityName                    = Locality Name (eg, city)
localityName_default            = Locality

organizationName                = Organization Name (eg, company)
organizationName_default        = Organization

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Dept.

commonName                      = Common Name (FQDN of your server)
commonName_default              = commonName

emailAddress                    = Email Address
emailAddress_default            = email@domain.com

[ cert_type ]
nsCertType = server
openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /var/certs/monit.pem -keyout /var/certs/monit.pem
openssl gendh 512 >> /var/certs/monit.pem
openssl x509 -subject -dates -fingerprint -noout -in /var/certs/monit.pem
chmod 700 /var/certs/monit.pem

Edit /etc/default/monit

startup=1
CHECK_INTERVALS=60
/etc/init.d/monit start

Retrieved from "http://wiki.openbravo.com/wiki/Release_Management/System_Monitoring_with_munin/monit"

This page has been accessed 9,727 times. This page was last modified on 20 April 2015, at 17:08. Content is available under Creative Commons Attribution-ShareAlike 2.5 Spain License.