Thursday, June 25, 2015

Metrics with CollectD and Graphite

CollectD is a daemon which periodically collects system performance statistics. It can write the data to different destinations including csv-files, rrdtool and Graphite.


The installation can be done with pkgutil:

# pkgutil -y -i collectd

The configuration file is located at /etc/opt/csw/collectd.conf. Make sure to comment out rrdtool as we want to use Graphite as destination:

#LoadPlugin rrdtool

Then activate the Graphite plugin:

LoadPlugin write_graphite

and make sure to configure the graphite plugin accordingly (there is an example down in the config file):

<Plugin write_graphite>
  <Node "stats">
    Prefix "server."
    Postfix ".collectd"

Then restart CollectD and the metrics should be flowing in to Graphite:

# svcadm restart cswcollectd

Make sure to reload the webpage of Graphite-Web to show the new metrics.

Collecting Metrics with Graphite

Graphite is a tool for storing and visualizing metrics. In this post we are installing Graphite from OpenCSW packages and do some basic configuration. The steps follow the tutorial presented in and focuses on the issues which differ, so reading up the original article after this brief tutorial is helpful.

Installing Graphite

There are three parts which make up a Graphite installation:

  1. A database library called Whisper (package CSWpy-whisper)
  2. A storage engine called Carbon (package CSWpy-carbon)
  3. A web interface (package CSWgraphite-web)

For simplicity we will install all packages on a single host. As the web interface automatically pulls in necessary dependencies it is sufficient to do

# pkgutil -y -i graphite_web

Configuring Graphite

Important locations:

  • configuration files: /etc/opt/csw/graphite
  • log files: /var/opt/csw/graphite/log and especially /var/opt/csw/graphite/log/webapp/error.log
  • data files: /var/opt/csw/graphite/whisper
First we need to edit /etc/opt/csw/graphite/ by changing the values for SECRET_KEY and TIME_ZONE. The secret key is used to salt hashes for the authentication. A decent secret key can be generated with

# cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 256 | head -1

The lines in should then read:
TIME_ZONE = 'Europe/Berlin'

Now the database can be populated with
# graphite-manage syncdb
Operations to perform:
  Synchronize unmigrated apps: account, cli, render, whitelist, metrics, url_shortener, dashboard, composer, events, browser
  Apply all migrations: admin, contenttypes, tagging, auth, sessions
Synchronizing apps without migrations:
  Creating tables...
    Creating table account_profile
    Creating table account_variable
    Creating table account_view
    Creating table account_window
    Creating table account_mygraph
    Creating table dashboard_dashboard
    Creating table events_event
    Creating table url_shortener_link
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying sessions.0001_initial... OK
  Applying tagging.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): 
Email address:
Password (again): 
Superuser created successfully.

Adjust the resolution and hold time for the carbon storage engine in /etc/opt/csw/graphite/storage-schemas.conf:

retentions = 10s:1h, 1m:7d, 15m:30d, 1h:2y

Then carbon-cache can be started with

# svcadm enable carbon-cache

We can now setup the Apache Virtual Host by adding this line to /opt/csw/apache2/etc/httpd.conf:

Include etc/extra/httpd-graphite-vhost.conf

Adjust permissions:
# chown -R nobody:nobody /var/opt/csw/graphite

Restart Apache:
# svcadm restart cswapache2

Everything is now running and it should now be possible to connect to Apache.