No such thing as overkill

Pretty graphs, state-of-the-art backend system, easy deployment. Pick any two.

Graphite seems to be the current tool for taking care of heavy lifting, and of the three options, it has gone for first two. This is the story of making Graphite run on a Debian Sid system. To really add some gnarly twists, this is done with Lighttpd.

Some of the unexpected dependencies include Twisted (for graphite itself), and Django (for actually running Graphite).

Broken with Django 1.5 - investigation required. Upstream is aware, will be fixed eventually Fixed in Debian Unstable as #721085

Prepackaged

The Graphite stack is commonly available:

sudo apt-get install python-whisper graphite-carbon graphite-web

The tentacles...

Running Graphite with Postgres takes a few extra packages.


sudo apt-get install postgresql python-psycopg2 python-flup

Psycopg2 is a Postgres connector. Flup is required for FastCGI. And Postgres? Well, that's for Django.

Massaging Postgres into shape

Before we add any databases (for Django's benefit), there are a couple of changes we need to do:

New line in postgres's pg_ident.conf :


    # MAPNAME       SYSTEM-USERNAME         PG-USERNAME
    graphs          _graphite               graphite

This creates a new internal account-name map "graphs", telling that system name "_graphite" will be treated as Postgres user/role "graphite".

Then, we change access control for domain-socket connections:

Changes in postgres's pg_hba.conf


    # Locally mapped usernames in 'graphs'
    local   all             all               ident map=graphs
    # "local" is for Unix domain socket connections only
    local   all             all               ident

These changes mean that access control decisions are based on local identity (available to peers when connecting over domain socket), so these connections do not require passwords for database connections. We add the manually mapped account identification before the global default, so the map is checked first.

Restart Postgres.

No base without data

Let's create the database for our graph data.


    sudo -u postgres createuser -D -R -S graphite
    sudo -u postgres createdb -O graphite graphite

This gives us the Postgres user/role "graphite" as the owner of database "graphite". Thanks to mapping above, we can now do this:


    sudo -s
    su -l -s /bin/bash _graphite
    psql -d graphite graphite

So we can connect to the newly created graphite database as Postgres user "graphite", when our session user is "_graphite".

Configuring Graphite

The default configuration file /etc/graphite/settings.py corresponds to local_settings.py in Graphite documentation. However, the file provided in graphite-web packaging is from an older version. The database connection section in particular has changed. Instead of individual DATABASE_FOO tokens, the configuration section now allows multiple databases and has been changed into a dictionary. The options now reside in DATABASES['default'][FOO] - so the configuration for our setup looks like this:


DATABASES = {
  'default': {
    'NAME': 'graphite',
    'ENGINE': 'django.db.backends.postgresql_psycopg2',
    'USER': 'graphite',
    'PORT': 5432,
  }
}

The connection port is not generally needed, since we're connecting over domain socket. However, the port number is also used by Psycopg2: the listening port number is included in the name of the Postgresql socket. In case of non-standard Postgres port, the number will be necessary.

While editing settings.py, we also drop RRD_DIR from DATA_DIRS. We simply don't have any RRD files around at the moment, and Graphite insists that all data directories must exist. Thus, removing the bogus directory prevents a runtime error.

At this point, Graphite should be able to connect to Postgres. We'll initialise the database.


    _graphite% /usr/bin/graphite-manage syncdb

That takes care of the Django part. (Note that we did not create a Django project. Graphite IS a pre-packaged Django project, with its own manage.py wrapper. The file's just named differently and is placed in an unusual path.)

Connecting with FastCGI

We start by enabling FastCGI in Lighttpd. Individual modules are available in conf-available, and linked from conf-enabled just like with Apache.


/etc/lighttpd% ls -l conf-enabled/10-fastcgi.conf
lrwxrwxrwx 1 root 33 Jan 21 14:47 conf-enabled/10-fastcgi.conf -> ../conf-available/10-fastcgi.conf

Then, we need to choose the path for our FastCGI domain socket. In my case, I added the following block to /etc/lighttpd/lighttpd.conf:


# Graphite setup; FCGI+Django+Twisted+Carbon+GraphiteWeb
fastcgi.server = (
  "/graphite" => (
    "main" => (
      "socket" => "/run/graphite/graphite.sock",
      "check-local" => "disable",
    )
  ),
)

The socket is the communication point between Lighttpd and Graphite. Since the socket is created by Graphite, we need to ensure that Lighttpd processes can use the socket. We add www-data user to _graphite group:


% grep -e www-data -e _graphite /etc/group
www-data:x:33:
_graphite:x:122:www-data

Now, finally, with only slight preparations we can run the entire stack.

As root/sudo-capable user:


    sudo mkdir -m 775 /run/graphite
    sudo chown _graphite._graphite /run/graphite

And as _graphite:


/usr/bin/graphite-manage runfcgi socket=/run/graphite/graphite.sock
pidfile=/run/graphite/graphite-fcgi.pid daemonize=true umask=002

A particularly fun fact is that socket permissions are only applied when running in daemonize=true mode.

Taking it out for a spin

From the FastCGI server setup in lighttpd.conf we know that all queries to the path /graphite are forwarded to Graphite engine. We can test that Graphite is working correctly by manually entering the following URL in browser:

http://YOURHOST/graphite/render?target=carbon.*.*.*

If everything is working as it should, we should see a single graph showing how the storage backend (carbon) has been using resources.

Stapling the stack together

Visualised the stack looks something like this:

Graphite stack with lighty+fastcgi

Aivoituksia

Graphite - No Such Thing as Overkill