Release Management/System Monitoring with munin/monit
![]() | This is an old document about monitoring, please go here for latest. |
Contents |
Introduction
This document gives a brief on why and how to use Monitoring tools to monitor a system/server.
Why
- It helps optimizing the usage of the sever (as it helped us to terminate some unused instances).
- It helps make servers more secure (as it helped us to detect the incoming SSH failure hits).
- It helps you manage your server effortlessly.
- etc.
Tools
Munin: It is the server node, which collects data from various munin-nodes or muninlite working on different servers and uses that data to generate the graphs. By default it runs after every 5 minutes (cron).
Munin-node: It is a client node which collects the data from the system (proc and other log files) as specified by its plugins when ever munin (server-node) requests for it. By default this runs on port 4949.
Muninlite: It is a lighter version of munin-node, this is lighter as this is built only using bash script. But it has a draw back too, it has less plugins and it does not works properly on ubuntu systems.
Monit: This tools has an upper hand as it deals with the services (apache/ssh/mysql etc) running on the system and is capable of restarting/stopping/starting/monitoring from the web interface. By default this runs on port 2812.
M/monit: This is a fancy dashboard which in terms of working is same as monit's web interface but is more presentable. By default it runs on port 8080.
Installation & Configuration
Munin and Munin-node
Gentoo
Install And Configure munin
emerge -av munin emerge --config net-analyzer/munin
Configuring Plugins
sudo -u munin munin-node-configure --shell
Best way to add plugins is to creating a symlink
ln -s /usr/libexec/munin/plugins/<plugin name> /etc/munin/plugins/ rc-update add munin-node default /etc/init.d/munin-node start
- Note: Before restarting edit /etc/munin/munin-node.conf to accept connections from server ip.
- Note: If using several nodes add them to /et/munin/munin.conf of server.
Debian
Install And Configure munin
apt-get install munin munin-node
Edit /etc/munin/munin.conf
dbdir /var/lib/munin htmldir /var/www/munin/ logdir /var/log/munin rundir /var/run/munin tmpldir /etc/munin/templates [server1.example.com] address 127.0.0.1 use_node_name yes # Note:/var/www/munin/ should be your apache root directory or somewhere inside that.
mkdir -p /var/www/munin chown munin:munin /var/www/munin /etc/init.d/munin-node restart
After sometime you can see the graphs in http://<server IP>/<Path to munin dir.>
Muninlite
Mininlite requires xinetd/inetd to work
General
These steps are common for both gentoo and debian systems.
cd /usr/local wget http://downloads.sourceforge.net/project/muninlite/muninlite%20-%20stable/1.0.2/muninlite-1.0.2.tar.gz?use_mirror=biznetnetworks tar -xvzf muninlite-1.0.2.tar.gz mv muninlite-1.0.2 muninlite cd muninlite make cp munin-node /usr/local/bin chmod +x /usr/local/bin/munin-node echo "munin 4949/tcp lrrd # Munin" >>/etc/services
With xinetd
add local ip of munin's servers ip to examples/xinetd.d/munin, by this you tell which is munin server to connect to
vi examples/xinetd.d/munin only_from = 10.226.51.178 #local ip of irc.openbravo.com
cp examples/xinetd.d/munin /etc/xinetd.d /etc/init.d/xinet.d restart
With inetd
cat examples/inetd.conf >> /etc/inetd.conf /etc/init.d/inet.d restart
Monit and M/monit
Install And Configure munin (x86 specific).
- Note: M/monit needs monit 5.0 or later.
Monit
cd /usr/local wget http://mmonit.com/monit/dist/monit-5.0.3.tar.gz tar -xvzf monit-5.0.3.tar.gz cd monit-5.0.3 ./configure make make install
Edit /usr/local/monit-5.0.3/monitrc according to what all services you need to monitor.
set mmonit http://monit:monit@<M/monit IP>:8080/collector set httpd port 2812 and use address <local IP> allow localhost allow <server IP> check process apache with pidfile /var/run/apache2.pid group www start program = "/etc/init.d/apache2 start" stop program = "/etc/init.d/apache2 stop" if failed host www.example.com port 80 protocol http and request "/monit/token" then restart if cpu is greater than 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 500 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if 3 restarts within 5 cycles then timeout check system localhost if loadavg (1min) > 4 then alert if loadavg (5min) > 2 then alert if memory usage > 75% then alert if cpu usage (user) > 70% then alert if cpu usage (system) > 30% then alert if cpu usage (wait) > 20% then alert set daemon 60 set logfile syslog facility log_daemon set mailserver localhost set mail-format { from: monit@monitoring-server.com } set alert email@domain.com set httpd port 2812 and SSL ENABLE PEMFILE /var/certs/monit.pem allow <user>:<password>
M/monit
cd /usr/local wget wget http://mmonit.com/dist/mmonit-2.0.3-linux-x86.tar.gz tar -xvzf mmonit-2.0.3-linux-x86.tar.gz cd mmonit-2.0.3 ./bin/mmonit start
Now you can browse through using http://<server IP>:8080/
Gentoo
Install And Configure munin (if using only monit and no M/monit).
emerge -va monit /etc/init.d/monit start
Edit /etc/monitrc like this.
set daemon 60 set logfile syslog facility log_daemon set mailserver localhost set mail-format { from: monit@server1.example.com } set alert root@localhost set httpd port 2812 and SSL ENABLE PEMFILE /var/certs/monit.pem allow <user>:<password> check process proftpd with pidfile /var/run/proftpd.pid start program = "/etc/init.d/proftpd start" stop program = "/etc/init.d/proftpd stop" if failed port 21 protocol ftp then restart if 5 restarts within 5 cycles then timeout check process sshd with pidfile /var/run/sshd.pid start program "/etc/init.d/ssh start" stop program "/etc/init.d/ssh stop" if failed port 22 protocol ssh then restart if 5 restarts within 5 cycles then timeout check process mysql with pidfile /var/run/mysqld/mysqld.pid group database start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed host 127.0.0.1 port 3306 then restart if 5 restarts within 5 cycles then timeout check process apache with pidfile /var/run/apache2.pid group www start program = "/etc/init.d/apache2 start" stop program = "/etc/init.d/apache2 stop" if failed host www.example.com port 80 protocol http and request "/monit/token" then restart if cpu is greater than 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 500 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if 3 restarts within 5 cycles then timeout check process postfix with pidfile /var/spool/postfix/pid/master.pid group mail start program = "/etc/init.d/postfix start" stop program = "/etc/init.d/postfix stop" if failed port 25 protocol smtp then restart if 5 restarts within 5 cycles then timeout
Debian
Install And Configure munin (if using only monit and no M/monit).
apt-get install monit
Edit vi /etc/monit/monitrc same as above (gentoo).
Configuring apache and secure connection.
mkdir /var/www/monit echo "hello" > /var/www/monit/token mkdir /var/certs cd /var/certs
Create /var/certs/monit.cnf to look like this.
# create RSA certs - Server RANDFILE = ./openssl.rnd [ req ] default_bits = 1024 encrypt_key = yes distinguished_name = req_dn x509_extensions = cert_type [ req_dn ] countryName = Country Name (2 letter code) countryName_default = IN stateOrProvinceName = State or Province Name (full name) stateOrProvinceName_default = State localityName = Locality Name (eg, city) localityName_default = Locality organizationName = Organization Name (eg, company) organizationName_default = Organization organizationalUnitName = Organizational Unit Name (eg, section) organizationalUnitName_default = Dept. commonName = Common Name (FQDN of your server) commonName_default = commonName emailAddress = Email Address emailAddress_default = email@domain.com [ cert_type ] nsCertType = server openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /var/certs/monit.pem -keyout /var/certs/monit.pem openssl gendh 512 >> /var/certs/monit.pem openssl x509 -subject -dates -fingerprint -noout -in /var/certs/monit.pem chmod 700 /var/certs/monit.pem
Edit /etc/default/monit
startup=1 CHECK_INTERVALS=60
/etc/init.d/monit start