Thursday, June 25, 2015

Metrics with CollectD and Graphite

CollectD is a daemon which periodically collects system performance statistics. It can write the data to different destinations including csv-files, rrdtool and Graphite.

Installation


The installation can be done with pkgutil:

# pkgutil -y -i collectd

The configuration file is located at /etc/opt/csw/collectd.conf. Make sure to comment out rrdtool as we want to use Graphite as destination:

#LoadPlugin rrdtool

Then activate the Graphite plugin:

LoadPlugin write_graphite

and make sure to configure the graphite plugin accordingly (there is an example down in the config file):

<Plugin write_graphite>
  <Node "stats">
    Prefix "server."
    Postfix ".collectd"
  </Node>
</Plugin>

Then restart CollectD and the metrics should be flowing in to Graphite:

# svcadm restart cswcollectd

Make sure to reload the webpage of Graphite-Web to show the new metrics.

Collecting Metrics with Graphite

Graphite is a tool for storing and visualizing metrics. In this post we are installing Graphite from OpenCSW packages and do some basic configuration. The steps follow the tutorial presented in http://kartar.net/2015/01/using-riemann-for-metrics/ and focuses on the issues which differ, so reading up the original article after this brief tutorial is helpful.

Installing Graphite


There are three parts which make up a Graphite installation:

  1. A database library called Whisper (package CSWpy-whisper)
  2. A storage engine called Carbon (package CSWpy-carbon)
  3. A web interface (package CSWgraphite-web)

For simplicity we will install all packages on a single host. As the web interface automatically pulls in necessary dependencies it is sufficient to do


# pkgutil -y -i graphite_web

Configuring Graphite


Important locations:

  • configuration files: /etc/opt/csw/graphite
  • log files: /var/opt/csw/graphite/log and especially /var/opt/csw/graphite/log/webapp/error.log
  • data files: /var/opt/csw/graphite/whisper
First we need to edit /etc/opt/csw/graphite/local_settings.py by changing the values for SECRET_KEY and TIME_ZONE. The secret key is used to salt hashes for the authentication. A decent secret key can be generated with


# cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 256 | head -1
-zAZa9a-A0zA99Z0-9-ZZZZAZ-ZAA9za09zZZ-9A00-zZ-A9ZZ99zzazzz-00za-Za-A-0z-z99aZaz0a0zaaAzAZZ0Za-a9z99-A0-AZ0-99A--zZaAz0a90zZ00-AZZ0aazzzZZ9A-0aA9aZ9-ZAZ0Z0AzAZ90aZ0a09AA9-a-zA9Aa-90aAAa-0ZA-090zZaAAa0ZZAzZza9a9a-A0-a-Za-09aaa-Zzz90Z0Z99a9azzzaa-0--zZzZA-ZA9

The lines in local_settings.py should then read:
SECRET_KEY='-zAZa9a-A0zA99Z0-9-ZZZZAZ-ZAA9za09zZZ-9A00-zZ-A9ZZ99zzazzz-00za-Za-A-0z-z99aZaz0a0zaaAzAZZ0Za-a9z99-A0-AZ0-99A--zZaAz0a90zZ00-AZZ0aazzzZZ9A-0aA9aZ9-ZAZ0Z0AzAZ90aZ0a09AA9-a-zA9Aa-90aAAa-0ZA-090zZaAAa0ZZAzZza9a9a-A0-a-Za-09aaa-Zzz90Z0Z99a9azzzaa-0--zZzZA-ZA9'
TIME_ZONE = 'Europe/Berlin'


Now the database can be populated with
# graphite-manage syncdb
Operations to perform:
  Synchronize unmigrated apps: account, cli, render, whitelist, metrics, url_shortener, dashboard, composer, events, browser
  Apply all migrations: admin, contenttypes, tagging, auth, sessions
Synchronizing apps without migrations:
  Creating tables...
    Creating table account_profile
    Creating table account_variable
    Creating table account_view
    Creating table account_window
    Creating table account_mygraph
    Creating table dashboard_dashboard
    Creating table events_event
    Creating table url_shortener_link
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying sessions.0001_initial... OK
  Applying tagging.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): 
Email address: me@mydomain.org
Password: 
Password (again): 
Superuser created successfully.


Adjust the resolution and hold time for the carbon storage engine in /etc/opt/csw/graphite/storage-schemas.conf:

retentions = 10s:1h, 1m:7d, 15m:30d, 1h:2y

Then carbon-cache can be started with

# svcadm enable carbon-cache

We can now setup the Apache Virtual Host by adding this line to /opt/csw/apache2/etc/httpd.conf:

Include etc/extra/httpd-graphite-vhost.conf

Adjust permissions:
# chown -R nobody:nobody /var/opt/csw/graphite

Restart Apache:
# svcadm restart cswapache2

Everything is now running and it should now be possible to connect to Apache.