Salesforce Anomaly Detection Using Anomaly.io

January 30, 2017 No Comments detection, influxdb

detect anomalies in salesforce

Monitoring Key Performance Indicators (KPIs) is essential to running a successful business. As one example, you should examine your lead generation KPI several times a day, to allow you to detect and correct problems as quickly as possible.

But you’re busyyou don’t have time to watch KPI indicators all day long. That’s where Anomaly.io comes in. By combining our detection algorithms with your Salesforce data, you can automatically detect problems and notify the appropriate personnel to ensure that speedy corrective action is taken.

From Salesforce to InfluxDB (Anomaly.io Protocol)

Let’s see how we can forward Salesforce data to Anomaly.io for automatic anomaly detection.

First, install the StreamSets Data Collector, which can easily subscribe to and stream Salesforce events. Next, log into your fresh StreamSets install with the default login/password combination “admin/admin”.
Now install the packages “Salesforce Library” and “InfluxDB 0.9+” from the PackageManager:

StreamSets package manager

Create a new Pipeline. Click the “Stage Library” (top right icon), select the origin “Salesforce”, then the processor “Groovy Evaluator”, and finally, the destination “InfluxDB”.

salesforce origin

Now link the nodes together:

Anomaly Flow Salesforce InfluxDB

Salesforce Node

This node collects historical “Opportunities” and subscribes to upcoming ones through the Salesforces Streaming API

Enter this data on the Salesforce tab:

  • Username: salesforceUserName
  • Password: MyPassword+securityToken
    e.g: MyPassword1FPeZD9z[…]
    Reset your salesforce security token
  • API version: 36.0
  • Query Existing Data: Yes
  • Subscribe for Notifications: Yes

IMPORTANT: Be sure to enter “36.0” for the API, as currently this is the only compatible version.

On the Query tab:

  • Use Bulk API: Yes
  • SOQL Query:
    SELECT Id, AccountId, CreatedDate, CloseDate, Amount
    FROM Opportunity WHERE Id > ‘${OFFSET}’ ORDER BY Id

And finally, on the Subscribe tab:

  • Push Topic: OpportunityCreate

Groovy Node

This node transforms Salesforce output into InfluxDB input. You need to do the following:

  • Set the required AccountId, if it is missing.
  • Translate Salesforce time into InfluxDB time.
  • Set a “measurement” to save the events into.

Example code for the Groovy tab:

InfluxDB Node

This node sends the data to Anomaly.io, which uses the InfluxDB protocol, for automatic anomaly detection.

Enter the following on the InfluxDB tab:

  • URL: http://events.anomaly.io:8086
  • User: yourAnomalyUser
  • Password: yourAnomalyPasswd
  • Database Name: salesforce
  • Auto-create Database: Yes
  • Record Mapping: Custom Mappings
  • Measurement Field: /measurement
  • Tag Fields: /AccountId
  • Value Fields: /Amount /CreatedDate /CloseDate

Register Salesforce PushTopic

In our StreamSets configuration, the Salesforce origin will request all historical data, then subscribe to a channel we called “OpportunityCreate“. To set this up, we need to create this PushTopic in Salesforce:

  1. Login to Salesforce
  2. Open the Developer Console.
  3. Select Debug | Open Execute Anonymous Window.
  4. In the Enter Apex Code window, paste in the following Apex code, then click Execute.

Run Streaming from Salesforce to InfluxDB (or Anomaly.io)

Click outside any node, then go to Configuration > Error Records and select “(Library: Basic)”.  Now we can finally run the pipeline:

Datasets run Anomaly Detection

You should see something like the following. Note that here a total of 32 “Opportunities” were processed, and StreamSets is waiting for new events to come in:

anomaly pipe

Salesforce historical data has now been imported into InfluxDB. What about future “Opportunities”? Let’s check to make sure the Salesforce Streaming API works:

  1.  Login to workbench.developerforce.com
  2. Go to Data > Insert | Object Type “Opportunity” | Next
  3. Enter:
    AccountId: 0010Y00000DX4EgQAL
    Amount: 100
    CloseDate: 2019-05-27T10:43:35.000Z
    Name: My Opportunity
    StageName: Open

Data should be processed within one second and then sent to InfluxDB or Anomaly.io.

Detect Anomalies in Salesforce

Using Anomaly.io

You can spend a lot of money on R&D, or you can save that money and use anomaly.io to detect exceptions and issues in your data. After a week of intense machine learning, our algorithm will pick the best anomaly detection method for your data. Our system can automatically apply techniques such as k-mean clustering, anomaly correlation, breakout detection and trend decomposition in real time, along with others.

Simply forward your data to anomaly.io; we’ll keep an eye on it.

Using Custom Anomaly Detection

Anomaly detection is complex, and especially difficult when you need to work with unknown problems. Your users and business will evolve, so you can’t rely on just setting simple thresholds.

Before you start, always check to make sure that your data is compatible with the algorithms you are using. For example, a common error is to use an algorithm intended for normally distributed time series when your data isn’t normally distributed.

You can also forward your Salesforce data into an open source solution such as Twitter Anomaly Detection, Twitter Breakout, or Skyline, or develop your own.

Monitor & detect anomalies with Anomaly.io

SIGN UP
help with term papers