Skyline Architecture

Skyline is usually set to use Graphite metrics. This is done by placing a daemon call “carbon-relay” in front of the usual Graphite stack. Carbon-relay will repeat the metrics to multiple hosts. One of them will be the essay editing service traditional Graphite stack, and the other host will be the Skyline stack.

Skyline is composed of several parts:

• Horizon – Responsible for collecting, cleaning, and formatting incoming metrics before pushing to a Redis database.
• Analyzer – Fetches metrics from Redis and runs mathematical equations to detect anomalies
• Skyline-webapp – A Django webapp to display an anomaly graph when it occurs

In the above schema, CollectD is used to monitor and push server metrics to carbon-cache. Read more about Graphite architecture.

Install Graphite

Graphite Required

Skyline is commonly used with Graphite. That’s why you need to Install Graphite on CentOS.
You don’t have to run the Graphite full stack; running carbon-relay is enough. But, as carbon-relay is part of Graphite, you still have to install Graphite.

Configure Carbon-relay

Set the list of hosts carbon-relay needs to forward its metrics to.

sudo vim /opt/graphite/conf/relay-rules.conf

# change the file
[...]
destinations = 127.0.0.1:2004, 127.0.0.1:2024
[...]

With this configuration, carbon-relay will forward metrics to Skyline on port 2024 and to paper writers carbon-cache (Graphite) on port 2004. Both services run locally in this tutorial. Also, edit the carbon configuration:

sudo vim /opt/graphite/conf/carbon.conf

# change the file
[...]
DESTINATIONS = 127.0.0.1:2004, 127.0.0.1:2024
[...]

Start carbon-relay

Start carbon-relay using systemd:

sudo systemctl restart carbon-relay

Install Skyline

Required

Open a terminal and install a few tools. This includes an Apache server (which is probably already installed), the Redis database, and Python with few mathematical libraries and some compile tools:

sudo yum -y install httpd redis git
sudo yum -y install gcc gcc-c++ git pycairo mod_wsgi
sudo yum -y install python-pip python-devel blas-devel lapack-devel libffi-devel

Download and install Skyline

cd /opt
sudo git clone https://github.com/etsy/skyline.git
cd /opt/skyline

sudo pip install -U six
sudo pip install -r requirements.txt
sudo pip install numpy
sudo pip install scipy
sudo pip install pandas
sudo pip install patsy
sudo pip install statsmodels
sudo pip install msgpack-python

Some of the Python packages might take very long to compile. Be patient, maybe grab a coffee…

Configure Skyline

Don’t forget to create the required directory and configuration:

sudo cp /opt/skyline/src/settings.py.example /opt/skyline/src/settings.py
sudo mkdir /var/log/skyline
sudo mkdir /var/run/skyline
sudo mkdir /var/log/redis
sudo mkdir /var/dump/

Skyline requires some settings, so edit the file:

sudo vim /opt/skyline/src/settings.py

and replace the following with your own values:

• GRAPHITE_HOST = ‘YOUR_GRAPHITE_HOST‘
• HORIZON_IP = ‘0.0.0.0‘
• WEBAPP_IP = ‘YOUR_SKYLINE_HOST_IP‘

In my case I replace with the same IP as Skyline and Graphite run locally on the same host:

• GRAPHITE_HOST = ‘192.168.50.6‘
• HORIZON_IP = ‘0.0.0.0‘
• WEBAPP_IP = ‘192.168.50.6‘

Start Skyline

The Skyline stack is made up of a Redis database and three Python processes. Start all services:

cd /opt/skyline/bin

sudo redis-server redis.conf
sudo ./horizon.d start
sudo ./analyzer.d start
sudo ./webapp.d start

Access Skyline-WebApp

Open your browser at http://localhost:1500/
Warning! To access a remote IP such as http://remote_ip:1500/ you need to set rules into the CentOS default firewall. Or simply disable the firewall:

#disable firewall
sudo systemctl disable firewalld
sudo systemctl stop firewalld

Skyline-WebApp should be empty as no anomaly will have been detected at first. Don’t worry if some anomalies are listed. This will stabilize with time.

Send Metrics to Skyline and Graphite

Carbon-relay now forwards its data to Skyline and carbon-cache. Any metrics sent to carbon-relay should be available in Skyline and Graphite.

Install CollectD

To collect and push metrics to carbon-relay we like to use CollectD. If you haven’t installed CollectD yet, follow these easy instructions:

• install CollectD on CentOS / fedora
• install CollectD on Debian / Ubuntu

Configure CollectD

Carbon-cache listens on a different port. Make sure you edit /opt/collectd/etc/collectd.conf to send the data on port 2013.

sudo vim /opt/collectd/etc/collectd.conf

# edit the file
[...]

   Host "localhost"
   Port "2013"
   Prefix "collectd."
   Protocol "tcp"

[...]

Start CollectD

If you installed the systemd script simply run:

# start with SytemD
sudo systemctl restart collectd.service

# or manualy
sudo /opt/collectd/sbin/collectd

Test Skyline Integration

Let your Linux idle for at least 10 minutes. Skyline will train itself to recognize low CPU activity as being normal. To create an anomaly, suddenly create some high CPU activity:

# wait at least 10 minutes before creating the anomaly
timeout 40s dd if=/dev/zero of=/dev/null &
timeout 40s dd if=/dev/zero of=/dev/null &
timeout 40s dd if=/dev/zero of=/dev/null &
timeout 40s dd if=/dev/zero of=/dev/null &

Open Skyline-webApp in your browser on port 1500. It should detect the anomaly within 10 seconds.

Test Graphite

Graphite should still work the same as before, since it receives its metrics through carbon-relay. Wait a few seconds and open Graphite-WebApp. Same as previously, you should see some new metrics in the left panel. Open one of them to render it.
Warming! By default, Graphite-WebApp renders a 24 hour graph. To see some data points zoom in or wait a bit longer.

Troubleshooting

If you follow the above steps exactly, this should work just fine. But in case you don’t see your data in Graphite try the following.

Check Graphite Troubleshooting

Try the troubleshooting solutions provided in Graphite installation tutorial.

Check Skyline log files

less +F /var/log/skyline/horizon.log
less +F /var/log/skyline/analyzer.log
less +F /var/log/skyline/webapp.log

Check That Every Process is Running

ps aux | grep "carbon-ca\|httpd\|collectd\|(wsgi:graphite)\|horizon-agent.py\|analyzer-agent.py\|horizon-agent.py\|carbon-relay.py\|webapp.py"

# result somthing like:
# [...]
# sudo /opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf -f
# [...]
# python /opt/skyline/bin/../src/horizon/horizon-agent.py start
# [...]
# python /opt/skyline/bin/../src/analyzer/analyzer-agent.py start
# [...]
# python /opt/skyline/bin/../src/webapp/webapp.py restart
# /usr/sbin/httpd -DFOREGROUND
# [...]
# /bin/python bin/carbon-relay.py --instance=a start
# /bin/python bin/carbon-cache.py --instance=a start
# [...]

Conclusion

Skyline is the first of its kind: it detects anomalies and trigger alarms in real time. Skyline is definitively trying to build a better world, where DevOps don’t need to spend their time watching metrics!

But there are some drawbacks. It only detects very obvious anomalies. This isn’t too bad, as many anomalies are obvious, but it will still fail to detect complex anomalies. the idea was to build a solution you can extend with homemade detectors, but no third party detectors have yet been released, so we are stuck with the basic functionality.

Of course I’m biased, but for easier monitoring and detection of many other anomaly types, I recommend using our product “Anomaly.io”. We believe it beats Skyline in every way.

Graphite Architecture

Graphite is a set of monitoring tools composed of:

• carbon – A (few) daemons in charge of listening for data (TCP/UDP)
• whisper – A database library for storing time series data
• graphite webapp – A Django webapp that renders graphs

Install Graphite

Requirement

Open a terminal and install the required tools:

sudo yum -y update
sudo yum -y install httpd gcc gcc-c++ git pycairo mod_wsgi epel-release
sudo yum -y install python-pip python-devel blas-devel lapack-devel libffi-devel

Download and install Graphite

The following code snippet will download the last version of Graphite (carbon and web) and install writy my essay it all into /opt/graphite:

cd /usr/local/src
sudo git clone https://github.com/graphite-project/graphite-web.git
sudo git clone https://github.com/graphite-project/carbon.git

sudo pip install -r /usr/local/src/graphite-web/requirements.txt

cd /usr/local/src/carbon/
sudo python setup.py install

cd /usr/local/src/graphite-web/
sudo python setup.py install

sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
sudo cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
sudo cp /opt/graphite/conf/relay-rules.conf.example /opt/graphite/conf/relay-rules.conf
sudo cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
sudo cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf

sudo cp /usr/local/src/carbon/distro/redhat/init.d/carbon-* /etc/init.d/
sudo chmod +x /etc/init.d/carbon-*

Configure Graphite

This code snippet creates the local database (SQLite). It also imports some static content for Graphite-webapp, and write my essay creates some folders with the correct permissions.

cd /opt/graphite
#create database
sudo PYTHONPATH=/opt/graphite/webapp/ django-admin.py syncdb --settings=graphite.settings
#reply: "Yes" and enter
#user: root
#passwd: myPassword

#import static files
sudo PYTHONPATH=/opt/graphite/webapp/ django-admin.py collectstatic --settings=graphite.settings
# reply "yes"

#set permission
sudo chown -R apache:apache /opt/graphite/storage/
sudo chown -R apache:apache /opt/graphite/static/
sudo chown -R apache:apache /opt/graphite/webapp/

By default, the Apache package provided in CentOS is restrictive with regard to directory aliases. Edit graphite Apache conf to allow the static directory:

sudo vim /etc/httpd/conf.d/graphite.conf

[...]
 Alias /static/ /opt/graphite/static/

   Require all granted

   Order allow,deny
   Allow from all
   Require all granted

[...]

Start Graphite

Carbon’s daemons can be started with systemd. Graphite-WebApp is an apache website. To start both simply run:

sudo systemctl start carbon-cache

sudo systemctl enable httpd
sudo systemctl start httpd

Access Graphite-WebApp

Open your browser at http://localhost/

Warning! To access a remote IP such as http://remote_ip/ you need to set rules in the CentOS default firewall, or simply disable it:

#disable firewall
sudo systemctl disable firewalld
sudo systemctl stop firewalld

No data should be available yet as we haven’t sent any metrics to Graphite.

Send Metrics to Graphite

Install CollectD

To collect and push metrics to Graphite, we like to use CollectD. To install CollectD, follow these easy instructions:

• install CollectD on CentOS / fedora
• install CollectD on Debian / Ubuntu

Configure CollectD

If you followed the instructions to install CollectD on CentOS, edit the configuration file located in /opt/collectd/etc/collectd.conf. Using the write_graphite plugin, forward CollectD metrics to Graphite on port 2003:

sudo vim /opt/collectd/etc/collectd.conf

# edit file
[...]
LoadPlugin write_graphite
[...]

   Host "localhost"
   Port "2003"
   Prefix "collectd."
   Protocol "tcp"

Start CollectD

If you installed the systemd script, simply run:

sudo systemctl start collectd.service

 You can also start CollectD manually with:

sudo /opt/collectd/sbin/collectd

Wait a few seconds, then open Graphite-WebApp. You should see some new metrics in the left panel. Open one of them to render it.
Warming! by default Graphite-WebApp renders a 24 hour graph. To see some data points, zoom in or wait a bit longer.

Troubleshooting

If you follow the steps above exactly, everything should work just fine. If you don’t see any data in Graphite, try a few of these ideas.

Restart CollectD with Logging

Edit CollectD configuration and enable logging:

sudo vim /opt/collectd/etc/collectd.conf

# edit file
[...]
LoadPlugin syslog
LoadPlugin logfile

  LogLevel info
  File STDOUT
  Timestamp true
  PrintSeverity false

  LogLevel info

[...]

#start CollectD in foreground
/opt/collectd/sbin/collectd -f

You shouldn’t see any messages like “write_graphite plugin: Connecting to localhost:2013 via tcp failed. The last error was: failed to connect to remote host: Connection refused”

Check Carbon-cache

Check the carbon-cache log:

less +F /opt/graphite/storage/log/carbon-cache/carbon-cache-a/console.log

Check Graphite-WebApp log

less +F /opt/graphite/storage/log/webapp/error.log

Check that All Processes are Running

Make sure every necessary process is running:

ps aux | grep "carbon-ca\|httpd\|collectd\|(wsgi:graphite)"

#result somthing like:
# /usr/sbin/httpd -DFOREGROUND
# (wsgi:graphite) -DFOREGROUND
# [...]
# sudo /opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf -f
# /opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf -f
# /usr/sbin/httpd -DFOREGROUND
# [...]
# /bin/python bin/carbon-cache.py --instance=a start

Conclusion

Graphite is an incredible tool: easy to install, fast and flexible. Since its creation in 2006 it has seen the development of many plugins and tools. But it is getting a bit outdated by the next generation of monitoring tools, such as the trendy Grafana / Influx, which promise more flexibility and scalability.

The problem with graphite: it’s easy to monitor, but you need to spend your time watching all metrics to spot malfunctions. To solve this problem you could install Skyline to detect anomalies, but this will only detect (very) obvious problems. To monitor and detect complex anomalies in real time, sign up with Anomaly.io.

On our website, each time a user performs a pageview, edit or postcomment action we save it in the interaction table. A user can belong to one of the groups admin, user or guest. Using CollectD DBI Plugin and some custom SQL requests, we can monitor what is happening in our MySQL database.

SELECT event_type, count(user_id) AS value
FROM interaction GROUP BY event_type;
+-------------+-------+
| event_type  | value |
+-------------+-------+
| edit        |     1 |
| pageview    |     8 |
| postcomment |     2 |
+-------------+-------+

SELECT 'visitor.pageview' as c_key , count(*) AS c_value
FROM interaction WHERE event_type = "pageview";
+------------------+---------+
| c_key            | c_value |
+------------------+---------+
| visitor.pageview |       8 |
+------------------+---------+

Configure and Start the Plugin DBI

Edit the CollectD configuration, usually located at /opt/collectd/etc/collectd.conf

<Query "interaction_stats">
   Statement "SELECT event_type, count(user_id) AS value FROM interaction GROUP BY event_type"
   
   Type "gauge"
   InstancesFrom "event_type"
   ValuesFrom "value"
   
 
 <Query "pageview">
   Statement "SELECT 'visitor.pageview' as c_key ,count(*) AS c_value FROM interaction WHERE event_type = 'pageview'"
   
   Type "gauge"
   InstancesFrom "c_key"
   ValuesFrom "c_value"
   
 

 <Database "sessionmetrics">
   Driver "mysql"
   DriverOption "host" "localhost"
   DriverOption "username" "root"
   DriverOption "password" "mypassword"
   DriverOption "dbname" "mydb"
   Query "interaction_stats"
   Query "pageview"

Now restart CollectD:

$ # using systemd
$ service collectd restart
$ 
$ # or manually
$ /opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf

For troubleshooting, add this configuration:

LoadPlugin logfile

 LogLevel info
 File STDOUT

And start CollectD in the foreground:

$
$ /opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf -f
$

Metrics Dashboard

At this point, integrating with a monitoring dashboard should be easy. To record MySQL request results, we chose the excellent InfluxDB. For displaying the metrics from InfluxDB, we chose Grafana.

To graph the MySQL metrics you will need to:

1. Install CollectD on Debian or CentOS
2. Install InfluxDB from source (or from InfluxDB)
3. Send CollectD metrics to InfluxDB
4. Install and Configure Grafana with InfluxDB

Fake some user activity in MySQL:

INSERT INTO `interaction` (`user_id`, `group_name`, `event_type`) VALUES
(4, 'guest', 'pageview');

With more pageviews, the value should increase:

We can use InfluxDB DIFFERENCE aggregator to highlight events accruing in real time.

Open the Grafana web UI. Replicate the InfluxDB request:
SELECT difference(value) FROM dbi_value WHERE type_instance=’visitor.pageview’ AND $timeFilter GROUP BY time($interval) ORDER ASC;

Done! We can watch our pageviews per second update in real time. Proceed the exact same way for the postcomment, pageview and edit metrics.

Conclusion

Code monitoring with StatsD is easy to install and configure. It’s even easier since CollectD now embeds a StatsD implementation.
But making code changes is a bit painful.

With CollectD DBI Plugin, it becomes possible to monitor an application, or at least its output, without any code changes. Any SQL request that generate numbers can be used to produce new metrics.

It is definitively a good starting point into application monitoring since it requires no code changes. However, it can also be considered an advance technique for monitoring what’s really happening within the database.

Install StatsD Plugin for CollectD

Edit the CollectD configuration file usually located at /opt/collectd/etc/collectd.conf

Uncomment or add StatsD plugin and configuration:

LoadPlugin statsd

  Host "0.0.0.0"
  Port "8125"
  DeleteSets     true

Don’t forget to restart CollectD:

$ 
$ service collectd restart
$

• Host “0.0.0.0”: Listen to any IP. Change to “127.0.0.1” to only allow localhost to send metrics.
• port 8125: Where StatsD metrics will be received.
• DeleteSets true: The values of type sets metrics will not be sent if they didn’t change since the last interval.

Read more about StatsD plugin configuration.