NAV Navbar
Logo
shell python

Introduction

These pages document the HTTP API exposed by the OpsDash server. The API was first introduced in OpsDash version 1.6.

The API feature is disabled by default. To enable it, edit /etc/opsdash/server.cfg on your OpsDash server installation and uncomment or set api.enable = 1. The OpsDash server needs to be restarted (sudo service opsdash-serverd restart) for the change to take effect.

The APIs are available over HTTP at the same port as the web UI. That is, if your OpsDash web UI is running at http://your.server:port/, then all the API endpoints will be under http://your.server:port/pubapi/v1. Individual endpoints are documented below.

In general, the APIs are either GET requests that return information, or POST requests with a JSON body that modify the state. All responses are JSON. The standard responses are documented in their own section.

If you have enabled htpasswd in /etc/opsdash/server.cfg to require authentication when modifications are made, then this will be applicable for API access also. That is, the APIs will fail with 403 status unless a valid Authorization: Basic xx header is included with your HTTP request.

A particular version of the API (eg. v1) will be supported in future versions of OpsDash in a backward-compatible manner, although it might be deprecated.

General APIs

These APIs allow you to query the alert status, and to send in custom metric data.

Status

curl http://your.server:port/pubapi/v1/status
import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/status')
response = json.load(body)
{
  "down": [
    {
      "source": "web-node-43",
      "last": 1440305374
    },
    {
      "source": "web-node-44",
      "last": 1443889129
    }
  ],
  "breach": [
    {
      "dashid": 124,
      "dashname": "web-node-01",
      "graphid": 1608,
      "graphname": "Disk IOPS - /dev/sda",
      "metric": "sys.disk.ops.write:sda",
      "value": 6003.8,
      "threshold": 6000,
      "breachtype": 3
    },
    {
      "dashid": 149,
      "dashname": "web-node-02",
      "graphid": 1932,
      "graphname": "Processes",
      "metric": "sys.os.procs.running",
      "value": 1,
      "threshold": 3,
      "breachtype": 1
    }
  ]
}

The alert status of the servers and services monitored by OpsDash can be retrieved by this API. The information returned by this API corresponds to that displayed in the Open Alerts page of the OpsDash web UI.

Request

GET http://your.server:port/pubapi/v1/status

Response

The response is a JSON object having the following keys:

Key Type Description
down array An array of “down” objects, each representing a server or service that is down. See table below.
breach array An array of “breach” objects, each representing a metric that has breached an upper/lower warning/critical threshold. See table below.

Each “down” object has the following keys:

Key Type Description
source string The name of the source that is down.
last integer The time when the source was last heard from, as the number of seconds from 1 Jan 1970 UTC.

Each “breach” object has the following keys:

Key Type Description
dashid integer Unique identifier of the dashboard which contains the graph with the breached metric.
dashname string The name of the dashboard. For system-generated dashboards, this will be the name of the source or sourcegroup.
graphid integer Unique identifier of the graph.
graphtitle string The title of the graph.
metric string The metric which breached the threshold.
value float The current value of the metric.
threshold float The threshold value that was set for this metric.
breachtype integer The type of breach that occured. Values are: 0 = no breach, 1 = lower critical, 2 = lower warning, 3 = upper warning, 4 = upper critical

Report

curl -X POST \
 -d '{ "source": "pgbounce-prod3", "metric": "avg_query", "at": 1440304723, "value": 74529.3 }' \
 http://your.server:port/pubapi/v1/report
import urllib2, json

data = json.dumps({
    'source': 'pgbounce-prod3',
    'metric': 'avg_query',
    'at':     1440304723,
    'value':  74529.3
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/report', data)
response = json.load(body)
{
  "message": "enqueued"
}

Report the value of a (custom) metric that belongs to a (custom) source. If the source and/or metric does not exist already, it will be created.

Note that once a source is created, it will be monitored for uptime.

Request

POST http://your.server:port/pubapi/v1/report

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
source string Required. The name of the source to which this metric belongs to. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
metric string Required. The name of the metric. Must be within 128 characters not containing “<”, “>” or space.
at integer Required. Timestamp as number of seconds since 1 Jan 1970 UTC.
value float Required. The value of the metric as a floating point number.

Note: OpsDash v1.6 accepted only 32 character source names and did not allow underscore within it.

Response

See Standard Reponses section.

Note that the request is checked for validity and then queued to be stored. The API returns after enqueuing and does not wait until the store is complete.

Server APIs

These APIs allow you to list currently known servers, as well as delete them. Note that there is no explicit API to “create” a server – it is automatically created when OpsDash first sees a metric from an previously unknown source.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/server/list')
response = json.load(body)
curl http://your.server:port/pubapi/v1/server/list
{
  "servers": [
    {
      "name": "web-node-01",
      "last": 1440305374
    },
    {
      "name": "web-node-02",
      "last": 1443890757
    }
  ]
}

Get the list of all servers that are known to OpsDash.

Request

GET http://your.server:port/pubapi/v1/server/list

Response

The response is a JSON object having the following keys:

Key Type Description
servers array An array of “server” objects, each representing a server.

The “server” object has the following keys:

Key Type Description
name string The name of the server.
last integer The time when data was last reported for the server, as the number of seconds since Jan 1, 1970 UTC.

Delete

curl -X POST \
 -d '{ "name": "pgbounce-prod3" }' \
 http://your.server:port/pubapi/v1/server/delete
import urllib2, json

data = json.dumps({
    'name': 'pgbounce-prod3'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/server/delete', data)
response = json.load(body)
{
  "message": "enqueued"
}

Deletes a server and all associated data. Use with care!

Request

POST http://your.server:port/pubapi/v1/server/delete

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
name string Required. The name of the server to be deleted.

Service APIs

These APIs allow you to list, create, update and delete services.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/list')
response = json.load(body)
curl http://your.server:port/pubapi/v1/service/list
{
  "services": [
    {
      "name": "api-2",
      "type": "httpurl",
      "url": "http://queuesrv.int/",
      "method": "POST",
      "header": [
        {
          "key": "Content-Type",
          "value": "application/x-www-form-urlencoded"
        }
      ],
      "body": "ping=1"
    },
    {
      "name": "redis-stg-1",
      "type": "redis",
      "address": "10.3.0.12:6379"
    }
  ]
}

Get details of all configured services.

Request

GET http://your.server:port/pubapi/v1/service/list

Response

The response is a JSON object having the following keys:

Key Type Description
services array An array of “service” objects, each representing a service.

The “service” object has the following keys:

Key Type Description
name string The name of the service.
type string The type of service, one of: “mysql”, “postgresql”, “mongodb”, “memcached”, “redis”, “httpurl”, “elasticsearch”, “elasticsearch-index” or “elasticsearch-node”.
address string In case of mysql, postgresql, mongodb, memcached, redis, elasticsearch, elasticsearch-index or elasticsearch-node this will contain the host and port of the service, like “host:port”.
user string In case of mysql, postgresql or mongodb, this will contain the username used to connect to the service, if any.
ssl bool In case of postgresql, this will be true if the connection is SSL-enabled.
url string In case of httpurl, this will contain the HTTP URL of the service.
method string In case of httpurl, this will contain the HTTP method of the service.
header array In case of httpurl, if additional headers are configured, this will contain “field” objects that represent these additional headers. See table below.
body string In case of httpurl, if HTTP body is set, this will contain the HTTP body.
index string In case of elasticsearch-index, this will contain the name of the Elasticsearch index.
node string In case of elasticsearch-node, this will contain the hostname, IP or node ID (whichever was set) of the Elasticsearch node.

The “field” object has the following keys:

Key Type Description
key string The key of the header field.
value string The value of the header field.

Note that passwords will not be returned by the API for any service type.

Create

import urllib2, json

data = json.dumps({
    'name':    'redis-stg-3',
    'type':    'redis',
    'address': '10.2.30.4:6379'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/create', data)
response = json.load(body)
curl -X POST \
 -d '{ "name": "redis-stg-2", "type": "redis", "address": "10.2.30.4:6379" }' \
 http://your.server:port/pubapi/v1/service/create
{
  "message": "success"
}

Create a new service with the specified parameters.

Request

POST http://your.server:port/pubapi/v1/service/create

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
name string Required. The name of the service. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
type string Required. The type of service, one of: “mysql”, “postgresql”, “mongodb”, “memcached”, “redis”, “httpurl”, “elasticsearch”, “elasticsearch-index” or “elasticsearch-node”.
address string Required if type is mysql, postgresql, mongodb, memcached, redis, elasticsearch, elasticsearch-index or elasticsearch-node. This should contain the host and port of the service, like “host:port”. Port is not optional.
user string In case of mysql, postgresql, elasticsearch, elasticsearch-index, elasticsearch-node or mongodb, this can contain the username used to connect to the service, if one is needed.
pass string In case of mysql, postgresql, elasticsearch, elasticsearch-index, elasticsearch-node or mongodb, this can contain the password used to connect to the service, if one is needed.
ssl bool In case of postgresql, if this is set to true, the connection will be SSL-enabled.
https bool In case of elasticsearch, elasticsearch-index or elasticsearch-node, if this is set to true, the connection will be over https rather than http.
url string Required if type is httpurl. This should contain the HTTP URL of the service.
method string Required if type is httpurl. This should contain the HTTP Method of the service, one of “GET”, “POST”, “PUT”, “HEAD”.
header array If type is httpurl, additional headers can be specified by setting this to an array of “field” objects that represent these additional headers. See table below.
body string If type is httpurl, an HTTP body can be configured by setting this field to the contents of the HTTP body.
index string Required if type is elasticsearch-index. This should contain the name of the Elasticsearch index.
node string Required if type is elasticsearch-node. This should contain the hostname, IP or node ID of the Elasticsearch node.

The “field” object has the following keys:

Key Type Description
key string The key of the header field.
value string The value of the header field.

Note: OpsDash v1.6 accepted only 32 character service names and did not allow underscore within it.

Response

See Standard Reponses section.

Update

import urllib2, json
data = json.dumps({
    'name':    'redis-stg-3',
    'type':    'redis',
    'address': '10.2.9.8:6379'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/update', data)
response = json.load(body)
curl -X POST \
 -d '{ "name": "redis-stg-3", "type": "redis", "address": "10.2.9.8:6379" }' \
 http://your.server:port/pubapi/v1/service/update
{
  "message": "success"
}

Update the specified service. This takes exactly the same parameters as the service creation API. Changing the type of service is also possible.

Request

POST http://your.server:port/pubapi/v1/service/update

Body

The request body must contain a valid JSON object. The required keys are exactly the same as for the service creation API above.

Response

See Standard Reponses section.

Delete

curl -X POST \
 -d '{ "name": "mysql-t2" }' \
 http://your.server:port/pubapi/v1/service/delete
import urllib2, json

data = json.dumps({
    'name': 'mysql-t2'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/delete', data)
response = json.load(body)
{
  "message": "enqueued"
}

Deletes a service and all associated data. Use with care!

Request

POST http://your.server:port/pubapi/v1/service/delete

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
name string Required. The name of the service to be deleted.

Source Group APIs

These APIs allow you to list, create, update and delete source groups.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/list')
response = json.load(body)
curl http://your.server:port/pubapi/v1/sourcegroup/list
{
  "sourcegroups": [
    {
      "name": "elb-prod-5",
      "type": "list",
      "sources": [
        "ip-10-0-0-1",
        "ip-10-0-0-2",
        "ip-10-0-0-3"
      ]
    },
    {
      "name": "cache-prod-4",
      "type": "aws",
      "accessKeyId": "AKIAXXXXXXX",
      "region": "us-west-2",
      "tags": [
        {
          "key": "role",
          "value": "query-cache"
        }
      ]
    }
  ]
}

Get details of all configured source groups.

Request

GET http://your.server:port/pubapi/v1/sourcegroup/list

Response

The response is a JSON object having the following keys:

Key Type Description
sourcegroups array An array of “sourcegroup” objects, each representing a sourcegroup.

The “sourcegroup” object has the following keys:

Key Type Description
name string The name of the source group.
type string The type of source group, one of: “list”, “shellrx”, “posixrx” or “aws”.
sources array In case of type list, this will contain an array of strings, each string being the name of a source.
regex string In case of type shellrx or posixrx, this will contain the regular expression.
accessKeyId string In case of type aws, this will contain the AWS Access Key ID.
region string In case of type aws, this will contain the AWS region, like “us-east-1”.
tags array In case of type aws, and if tags have been configured, then this will contain an array of “tag” objects (see table below).

The “tag” object has the following keys:

Key Type Description
key string The key of the tag.
value string The value of the tag.

Note that the AWS Secret Access Key will not be returned by the API.

Create

import urllib2, json

data = json.dumps({
    'name':  'mc-cluster',
    'type':  'shellrx',
    'regex': 'mc*'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/create', data)
response = json.load(body)
curl -X POST \
 -d '{ "name": "mc-cluster", "type": "shellrx", "regex": "mc*" }' \
 http://your.server:port/pubapi/v1/sourcegroup/create
{
  "message": "success"
}

Create a new source group with the specified parameters.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/create

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
name string Required. The name of the source group. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
type string Required. The type of source group, one of: “list”, “shellrx”, “posixrx” or “aws”.
sources array Required if type is “list”. This should contain an array of strings, each string being the name of a source. Each source must be known to OpsDash.
regex string Required if type is “shellrx” or “posixrx”. This should contain the appropriate regular expression.
accessKeyId string Required if type is “aws”. This should contain the AWS Access Key ID.
secretAccessKey string Required if type is “aws”. This should contain the AWS Secret Access Key.
region string Required if type is “aws”. This should contain the AWS region, like “us-east-1”.
tags array In case of type aws, then can be set to an array of “tag” objects (see table below), which will be used to restrict the EC2 instances selected.

The “tag” object has the following keys:

Key Type Description
key string Required. The key of the tag.
value string The value of the tag.

Note: Currently regular expressions of type “shellrx” recognise only the wildcards * and ?.

Note: OpsDash v1.6 accepted only 32 character source group names and did not allow underscore within it.

Response

See Standard Reponses section.

Update

import urllib2, json

data = json.dumps({
    'name':  'mc-cluster',
    'type':  'posixrx',
    'regex': 'mc.*prod.*master'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/update', data)
response = json.load(body)
curl -X POST \
 -d '{ "name": "mc-cluster", "type": "posixrx", "regex": "mc.*prod.*master" }' \
 http://your.server:port/pubapi/v1/sourcegroup/update
{
  "message": "success"
}

Update the specified source group. This takes exactly the same parameters as the source group creation API. Changing the type of source group is also possible.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/update

Body

The request body must contain a valid JSON object. The required keys are exactly the same as for the source group creation API above.

Response

See Standard Reponses section.

Delete

curl -X POST \
 -d '{ "name": "elb-client5" }' \
 http://your.server:port/pubapi/v1/sourcegroup/delete
import urllib2, json

data = json.dumps({
    'name': 'elb-client5'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/delete', data)
response = json.load(body)

Deletes a source group and associated configuration. Note that the metric data is associated with individual sources, and deleting a source group does not affect that.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/delete

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
name string Required. The name of the source group to be deleted.

Metric APIs

These APIs allow you to manipulate individual metrics.

Delete

import urllib2, json

data = json.dumps({
    'source': 'custom.source.23',
    'metric': 'fuzz.factor'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/metric/delete', data)
response = json.load(body)
curl -X POST \
 -d '{ "source": "custom.source.23", "metric": "fuzz.factor" }' \
 http://your.server:port/pubapi/v1/metric/delete
{
  "message": "enqueued"
}

Deletes a metric and all associated data. Use with care!

Apart from the time series data for the metric, references to this metric will be removed from any graph that includes it. If there are open alerts against this metric, they may take up to a minute to clear. This API was introduced in v1.7.4.

Request

POST http://your.server:port/pubapi/v1/metric/delete

Body

The request body must contain a valid JSON object with the following keys:

Key Type Description
source string Required. The name of the source to which the metric belongs.
metric string Required. The name of the metric.

Response

See Standard Reponses section.

Webhook

OpsDash supports configuring webhooks that can receive real-time notifications. Currently, OpsDash supports webhooks for receiving alert notifications.

Alert Notifications

{
  "version": 1,
  "at": 1489907211,
  "breach": [
    {
      "dashid": 2,
      "dashname": "webhook-test-01",
      "graphid": 42,
      "graphname": "Load Average",
      "source": "webhook-test-01",
      "metric": "sys.os.loadavg",
      "breachtype": "warnub",
      "value": 2.4,
      "threshold": 2
    }
  ],
  "down": [
    {
      "source": "webhook-test-02",
      "down": true,
      "last": 1489907031
    }
  ],
  "status": "1 down, 1 warning"
}

Webhook alert notifications are sent out whenever:

A single Webhook notification may carry one or more such events.

The HTTP requests made by OpsDash are:

Request

POST https://your/webhook/url

Headers

The following extra HTTP headers will be set in the request:

Header Value Description
Content-Type application/json
X-Message-Version 1 the version of the JSON message schema, currently 1
User-Agent OpsDash/1.0 the user-agent and version
Cache-Control no-store

Body

The request body will contain a valid JSON object with the following keys:

Key Type Description
version number The message schema version, currently 1.
at number The time the notification was sent out, as the number of seconds from 1 Jan 1970 UTC.
breach array An array of “breach” objects, each representing a metric that has breached an upper/lower warning/critical threshold, or has cleared. See table below. May be absent.
down array An array of “down” objects, each representing a source that has just gone down, or has come up. See table below. May be absent.

Each “down” object has the following keys:

Key Type Description
source string The name of the source that has gone down or come up.
down boolean Will be true if the source has gone down, or false if the source has come up.
last integer The time when the source was last heard from, as the number of seconds from 1 Jan 1970 UTC.

Each “breach” object has the following keys:

Key Type Description
dashid integer Unique identifier of the dashboard which contains the graph with the breached metric.
dashname string The name of the dashboard. For system-generated dashboards, this will be the name of the source or sourcegroup.
graphid integer Unique identifier of the graph.
graphname string The title of the graph.
source string The source or the sourcegroup (see “combine” key below) involved in the breach.
metric string The metric involved in the breach.
breachtype string The type of breach that occured. Value is one of: “critlb”, “warnlb”, “warnub”, “critub” or an empty string, and indicates which threshold was breached. “lb” stands for lower bound, and “ub” for upper bound. If empty, the breach has cleared.
value float The value of the metric that exceeded the threshold.
threshold float The threshold, corresponding to the “breachtype”, that was set. This is valid only if “breachtype” is set.
status string A human-readable summary of the current overall alert status.

Note that the JSON request body may not be pretty-printed as shown in the example.

Response

The webhook must return an HTTP response with a status code of 200 within 3 seconds for the notification to be considered successful. No other parts of the response are examined.

Standard Responses

Generally, the OpsDash API will respond to requests with HTTP status code 200, 400 or 500. Some responses will include a JSON object as the HTTP body.

A status of 200 indicates success. If additional information is available, it will be included in the HTTP response as a JSON object like so:

{ "message": "queued" }

A status of 400 indicates some form of client error, like missing or incorrect parameters. The response will typically include a message indicating the reason:

{ "message": "name must be specified" }

A status of 500 indicates that the server encountered an error while performing the request. More information might be available in the OpsDash server log file at /var/log/opsdash/opsdash-server.log.

{ "message": "database error" }

Additionally, if htpasswd is set in /etc/opsdash/server.cfg, the API will fail with a 403 status code if a valid Authorization header is not present.

Feedback

Your feedback about these APIs, as well as questions, suggestions and comments are welcome! Drop us a mail at api-team@rapidloop.com.

© RapidLoop 2018 • All Rights Reserved.