Metrics with CollectD and Graphite

CollectD is a daemon which periodically collects system performance statistics. It can write the data to different destinations including csv-files, rrdtool and Graphite.

Installation

The installation can be done with pkgutil:

# pkgutil -y -i collectd

The configuration file is located at /etc/opt/csw/collectd.conf. Make sure to comment out rrdtool as we want to use Graphite as destination:

#LoadPlugin rrdtool

Then activate the Graphite plugin:

LoadPlugin write_graphite

and make sure to configure the graphite plugin accordingly (there is an example down in the config file):

<Plugin write_graphite>

  <Node "stats">

    Prefix "server."

    Postfix ".collectd"

  </Node>

</Plugin>

Then restart CollectD and the metrics should be flowing in to Graphite:

# svcadm restart cswcollectd

Make sure to reload the webpage of Graphite-Web to show the new metrics.

Collecting Metrics with Graphite

Graphite is a tool for storing and visualizing metrics. In this post we are installing Graphite from OpenCSW packages and do some basic configuration. The steps follow the tutorial presented in http://kartar.net/2015/01/using-riemann-for-metrics/ and focuses on the issues which differ, so reading up the original article after this brief tutorial is helpful.

Installing Graphite

There are three parts which make up a Graphite installation:

1. A database library called Whisper (package CSWpy-whisper)
2. A storage engine called Carbon (package CSWpy-carbon)
3. A web interface (package CSWgraphite-web)

For simplicity we will install all packages on a single host. As the web interface automatically pulls in necessary dependencies it is sufficient to do

# pkgutil -y -i graphite_web

Configuring Graphite

Important locations:

• configuration files: /etc/opt/csw/graphite
• log files: /var/opt/csw/graphite/log and especially /var/opt/csw/graphite/log/webapp/error.log
• data files: /var/opt/csw/graphite/whisper

First we need to edit /etc/opt/csw/graphite/local_settings.py by changing the values for SECRET_KEY and TIME_ZONE. The secret key is used to salt hashes for the authentication. A decent secret key can be generated with

# cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 256 | head -1

-zAZa9a-A0zA99Z0-9-ZZZZAZ-ZAA9za09zZZ-9A00-zZ-A9ZZ99zzazzz-00za-Za-A-0z-z99aZaz0a0zaaAzAZZ0Za-a9z99-A0-AZ0-99A--zZaAz0a90zZ00-AZZ0aazzzZZ9A-0aA9aZ9-ZAZ0Z0AzAZ90aZ0a09AA9-a-zA9Aa-90aAAa-0ZA-090zZaAAa0ZZAzZza9a9a-A0-a-Za-09aaa-Zzz90Z0Z99a9azzzaa-0--zZzZA-ZA9

The lines in local_settings.py should then read:
SECRET_KEY='-zAZa9a-A0zA99Z0-9-ZZZZAZ-ZAA9za09zZZ-9A00-zZ-A9ZZ99zzazzz-00za-Za-A-0z-z99aZaz0a0zaaAzAZZ0Za-a9z99-A0-AZ0-99A--zZaAz0a90zZ00-AZZ0aazzzZZ9A-0aA9aZ9-ZAZ0Z0AzAZ90aZ0a09AA9-a-zA9Aa-90aAAa-0ZA-090zZaAAa0ZZAzZza9a9a-A0-a-Za-09aaa-Zzz90Z0Z99a9azzzaa-0--zZzZA-ZA9'
TIME_ZONE = 'Europe/Berlin'

Now the database can be populated with

# graphite-manage syncdb

Operations to perform:

  Synchronize unmigrated apps: account, cli, render, whitelist, metrics, url_shortener, dashboard, composer, events, browser

  Apply all migrations: admin, contenttypes, tagging, auth, sessions

Synchronizing apps without migrations:

  Creating tables...

    Creating table account_profile

    Creating table account_variable

    Creating table account_view

    Creating table account_window

    Creating table account_mygraph

    Creating table dashboard_dashboard

    Creating table events_event

    Creating table url_shortener_link

    Running deferred SQL...

  Installing custom SQL...

Running migrations:

  Rendering model states... DONE

  Applying contenttypes.0001_initial... OK

  Applying auth.0001_initial... OK

  Applying admin.0001_initial... OK

  Applying contenttypes.0002_remove_content_type_name... OK

  Applying auth.0002_alter_permission_name_max_length... OK

  Applying auth.0003_alter_user_email_max_length... OK

  Applying auth.0004_alter_user_username_opts... OK

  Applying auth.0005_alter_user_last_login_null... OK

  Applying auth.0006_require_contenttypes_0002... OK

  Applying sessions.0001_initial... OK

  Applying tagging.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.

Would you like to create one now? (yes/no): yes

Username (leave blank to use 'root'): 

Email address: me@mydomain.org

Password: 

Password (again): 

Superuser created successfully.

Adjust the resolution and hold time for the carbon storage engine in /etc/opt/csw/graphite/storage-schemas.conf:

retentions = 10s:1h, 1m:7d, 15m:30d, 1h:2y

Then carbon-cache can be started with

# svcadm enable carbon-cache

We can now setup the Apache Virtual Host by adding this line to /opt/csw/apache2/etc/httpd.conf:

Include etc/extra/httpd-graphite-vhost.conf

Adjust permissions:

# chown -R nobody:nobody /var/opt/csw/graphite

Restart Apache:
# svcadm restart cswapache2

Everything is now running and it should now be possible to connect to Apache.