Detecting Anomalies with Skyline

install skyline anomaly detection

Skyline is free and open source anomaly detection software. Simply plug Graphite metrics into Skyline to detect anomalous behaviour automatically without any configuration.

Skyline Architecture

Skyline is usually set to use Graphite metrics. This is done by placing a daemon call “carbon-relay” in front of the usual Graphite stack. Carbon-relay will repeat the metrics to multiple hosts. One of them will be the essay editing service traditional Graphite stack, and the other host will be the Skyline stack.

skyline with graphite

Skyline is composed of several parts:

  • Horizon – Responsible for collecting, cleaning, and formatting incoming metrics before pushing to a Redis database.
  • Analyzer – Fetches metrics from Redis and runs mathematical equations to detect anomalies
  • Skyline-webapp – A Django webapp to display an anomaly graph when it occurs

In the above schema, CollectD is used to monitor and push server metrics to carbon-cache. Read more about Graphite architecture.

Install Graphite

Graphite Required

Skyline is commonly used with Graphite. That’s why you need to Install Graphite on CentOS.
You don’t have to run the Graphite full stack; running carbon-relay is enough. But, as carbon-relay is part of Graphite, you still have to install Graphite.

Configure Carbon-relay

Set the list of hosts carbon-relay needs to forward its metrics to.

With this configuration, carbon-relay will forward metrics to Skyline on port 2024 and to paper writers carbon-cache (Graphite) on port 2004. Both services run locally in this tutorial. Also, edit the carbon configuration:

Start carbon-relay

Start carbon-relay using systemd:

Install Skyline

Required

Open a terminal and install a few tools. This includes an Apache server (which is probably already installed), the Redis database, and Python with few mathematical libraries and some compile tools:

Download and install Skyline

Some of the Python packages might take very long to compile. Be patient, maybe grab a coffee…

Configure Skyline

Don’t forget to create the required directory and configuration:

Skyline requires some settings, so edit the file:

and replace the following with your own values:

  • GRAPHITE_HOST = ‘YOUR_GRAPHITE_HOST
  • HORIZON_IP = ‘0.0.0.0
  • WEBAPP_IP = ‘YOUR_SKYLINE_HOST_IP

In my case I replace with the same IP as Skyline and Graphite run locally on the same host:

  • GRAPHITE_HOST = ‘192.168.50.6
  • HORIZON_IP = ‘0.0.0.0
  • WEBAPP_IP = ‘192.168.50.6

Start Skyline

The Skyline stack is made up of a Redis database and three Python processes. Start all services:

Access Skyline-WebApp

Open your browser at http://localhost:1500/
Warning! To access a remote IP such as http://remote_ip:1500/ you need to set rules into the CentOS default firewall. Or simply disable the firewall:

Skyline-WebApp should be empty as no anomaly will have been detected at first. Don’t worry if some anomalies are listed. This will stabilize with time.

skyline screenshot

Send Metrics to Skyline and Graphite

Carbon-relay now forwards its data to Skyline and carbon-cache. Any metrics sent to carbon-relay should be available in Skyline and Graphite.

Install CollectD

To collect and push metrics to carbon-relay we like to use CollectD. If you haven’t installed CollectD yet, follow these easy instructions:

Configure CollectD

Carbon-cache listens on a different port. Make sure you edit /opt/collectd/etc/collectd.conf to send the data on port 2013.

Start CollectD

If you installed the systemd script simply run:

Test Skyline Integration

Let your Linux idle for at least 10 minutes. Skyline will train itself to recognize low CPU activity as being normal. To create an anomaly, suddenly create some high CPU activity:

Open Skyline-webApp in your browser on port 1500. It should detect the anomaly within 10 seconds.

skyline anomaly detected

Test Graphite

Graphite should still work the same as before, since it receives its metrics through carbon-relay. Wait a few seconds and open Graphite-WebApp. Same as previously, you should see some new metrics in the left panel. Open one of them to render it.
Warming! By default, Graphite-WebApp renders a 24 hour graph. To see some data points zoom in or wait a bit longer.

graphite graph

Troubleshooting

If you follow the above steps exactly, this should work just fine. But in case you don’t see your data in Graphite try the following.

Check Graphite Troubleshooting

Try the troubleshooting solutions provided in Graphite installation tutorial.

Check Skyline log files

Check That Every Process is Running

Conclusion

Skyline is the first of its kind: it detects anomalies and trigger alarms in real time. Skyline is definitively trying to build a better world, where DevOps don’t need to spend their time watching metrics!

But there are some drawbacks. It only detects very obvious anomalies. This isn’t too bad, as many anomalies are obvious, but it will still fail to detect complex anomalies. the idea was to build a solution you can extend with homemade detectors, but no third party detectors have yet been released, so we are stuck with the basic functionality.

Of course I’m biased, but for easier monitoring and detection of many other anomaly types, I recommend using our product “Anomaly.io”. We believe it beats Skyline in every way.

Monitor & detect anomalies with Anomaly.io

SIGN UP
  • Rob

    Is anomaly.io using Skyline codebase?

    • martin magakian

      We don’t use Skyline codebase, but we provide the same functionality (and more).
      Skyline “only” use the Kolmogorov-Smirnov test, the histogram last data point and the least squares sigma. Anomaly.io implement those algorithm and many more…

help with term papers