Introduction

These pages document the HTTP API exposed by the OpsDash server. The API was first introduced in OpsDash version 1.6.

The API feature is disabled by default. To enable it, edit /etc/opsdash/server.cfg on your OpsDash server installation and uncomment or set api.enable = 1. The OpsDash server needs to be restarted (sudo service opsdash-serverd restart) for the change to take effect.

The APIs are available over HTTP at the same port as the web UI. That is, if your OpsDash web UI is running at http://your.server:port/, then all the API endpoints will be under http://your.server:port/pubapi/v1. Individual endpoints are documented below.

In general, the APIs are either GET requests that return information, or POST requests with a JSON body that modify the state. All responses are JSON. The standard responses are documented in their own section.

If you have enabled htpasswd in /etc/opsdash/server.cfg to require authentication when modifications are made, then this will be applicable for API access also. That is, the APIs will fail with 403 status unless a valid Authorization: Basic xx header is included with your HTTP request.

A particular version of the API (eg. v1) will be supported in future versions of OpsDash in a backward-compatible manner, although it might be deprecated.

General APIs

These APIs allow you to query the alert status, and to send in custom metric data.

Status

curl http://your.server:port/pubapi/v1/status

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/status')
response = json.load(body)

{
  "down": [
    {
      "source": "web-node-43",
      "last": 1440305374
    },
    {
      "source": "web-node-44",
      "last": 1443889129
    }
  ],
  "breach": [
    {
      "dashid": 124,
      "dashname": "web-node-01",
      "graphid": 1608,
      "graphname": "Disk IOPS - /dev/sda",
      "metric": "sys.disk.ops.write:sda",
      "value": 6003.8,
      "threshold": 6000,
      "breachtype": 3
    },
    {
      "dashid": 149,
      "dashname": "web-node-02",
      "graphid": 1932,
      "graphname": "Processes",
      "metric": "sys.os.procs.running",
      "value": 1,
      "threshold": 3,
      "breachtype": 1
    }
  ]
}

The alert status of the servers and services monitored by OpsDash can be retrieved by this API. The information returned by this API corresponds to that displayed in the Open Alerts page of the OpsDash web UI.

Request

GET http://your.server:port/pubapi/v1/status

Response

The response is a JSON object having the following keys:

Key	Type	Description
down	array	An array of “down” objects, each representing a server or service that is down. See table below.
breach	array	An array of “breach” objects, each representing a metric that has breached an upper/lower warning/critical threshold. See table below.

Each “down” object has the following keys:

Key	Type	Description
source	string	The name of the source that is down.
last	integer	The time when the source was last heard from, as the number of seconds from 1 Jan 1970 UTC.

Each “breach” object has the following keys:

Key	Type	Description
dashid	integer	Unique identifier of the dashboard which contains the graph with the breached metric.
dashname	string	The name of the dashboard. For system-generated dashboards, this will be the name of the source or sourcegroup.
graphid	integer	Unique identifier of the graph.
graphtitle	string	The title of the graph.
metric	string	The metric which breached the threshold.
value	float	The current value of the metric.
threshold	float	The threshold value that was set for this metric.
breachtype	integer	The type of breach that occured. Values are: 0 = no breach, 1 = lower critical, 2 = lower warning, 3 = upper warning, 4 = upper critical

Report

curl -X POST \
 -d '{ "source": "pgbounce-prod3", "metric": "avg_query", "at": 1440304723, "value": 74529.3 }' \
 http://your.server:port/pubapi/v1/report

import urllib2, json

data = json.dumps({
    'source': 'pgbounce-prod3',
    'metric': 'avg_query',
    'at':     1440304723,
    'value':  74529.3
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/report', data)
response = json.load(body)

{
  "message": "enqueued"
}

Report the value of a (custom) metric that belongs to a (custom) source. If the source and/or metric does not exist already, it will be created.

Note that once a source is created, it will be monitored for uptime.

Request

POST http://your.server:port/pubapi/v1/report

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
source	string	Required. The name of the source to which this metric belongs to. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
metric	string	Required. The name of the metric. Must be within 128 characters not containing “<”, “>” or space.
at	integer	Required. Timestamp as number of seconds since 1 Jan 1970 UTC.
value	float	Required. The value of the metric as a floating point number.

Note: OpsDash v1.6 accepted only 32 character source names and did not allow underscore within it.

Response

See Standard Reponses section.

Note that the request is checked for validity and then queued to be stored. The API returns after enqueuing and does not wait until the store is complete.

Server APIs

These APIs allow you to list currently known servers, as well as delete them. Note that there is no explicit API to “create” a server – it is automatically created when OpsDash first sees a metric from an previously unknown source.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/server/list')
response = json.load(body)

curl http://your.server:port/pubapi/v1/server/list

{
  "servers": [
    {
      "name": "web-node-01",
      "last": 1440305374
    },
    {
      "name": "web-node-02",
      "last": 1443890757
    }
  ]
}

Get the list of all servers that are known to OpsDash.

Request

GET http://your.server:port/pubapi/v1/server/list

Response

The response is a JSON object having the following keys:

Key	Type	Description
servers	array	An array of “server” objects, each representing a server.

The “server” object has the following keys:

Key	Type	Description
name	string	The name of the server.
last	integer	The time when data was last reported for the server, as the number of seconds since Jan 1, 1970 UTC.

Delete

curl -X POST \
 -d '{ "name": "pgbounce-prod3" }' \
 http://your.server:port/pubapi/v1/server/delete

import urllib2, json

data = json.dumps({
    'name': 'pgbounce-prod3'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/server/delete', data)
response = json.load(body)

{
  "message": "enqueued"
}

Deletes a server and all associated data. Use with care!

Request

POST http://your.server:port/pubapi/v1/server/delete

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
name	string	Required. The name of the server to be deleted.

Service APIs

These APIs allow you to list, create, update and delete services.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/list')
response = json.load(body)

curl http://your.server:port/pubapi/v1/service/list

{
  "services": [
    {
      "name": "api-2",
      "type": "httpurl",
      "url": "http://queuesrv.int/",
      "method": "POST",
      "header": [
        {
          "key": "Content-Type",
          "value": "application/x-www-form-urlencoded"
        }
      ],
      "body": "ping=1"
    },
    {
      "name": "redis-stg-1",
      "type": "redis",
      "address": "10.3.0.12:6379"
    }
  ]
}

Get details of all configured services.

Request

GET http://your.server:port/pubapi/v1/service/list

Response

The response is a JSON object having the following keys:

Key	Type	Description
services	array	An array of “service” objects, each representing a service.

The “service” object has the following keys:

Key	Type	Description
name	string	The name of the service.
type	string	The type of service, one of: “mysql”, “postgresql”, “mongodb”, “memcached”, “redis”, “httpurl”, “elasticsearch”, “elasticsearch-index” or “elasticsearch-node”.
address	string	In case of mysql, postgresql, mongodb, memcached, redis, elasticsearch, elasticsearch-index or elasticsearch-node this will contain the host and port of the service, like “host:port”.
user	string	In case of mysql, postgresql or mongodb, this will contain the username used to connect to the service, if any.
ssl	bool	In case of postgresql, this will be true if the connection is SSL-enabled.
url	string	In case of httpurl, this will contain the HTTP URL of the service.
method	string	In case of httpurl, this will contain the HTTP method of the service.
header	array	In case of httpurl, if additional headers are configured, this will contain “field” objects that represent these additional headers. See table below.
body	string	In case of httpurl, if HTTP body is set, this will contain the HTTP body.
index	string	In case of elasticsearch-index, this will contain the name of the Elasticsearch index.
node	string	In case of elasticsearch-node, this will contain the hostname, IP or node ID (whichever was set) of the Elasticsearch node.

The “field” object has the following keys:

Key	Type	Description
key	string	The key of the header field.
value	string	The value of the header field.

Note that passwords will not be returned by the API for any service type.

Create

import urllib2, json

data = json.dumps({
    'name':    'redis-stg-3',
    'type':    'redis',
    'address': '10.2.30.4:6379'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/create', data)
response = json.load(body)

curl -X POST \
 -d '{ "name": "redis-stg-2", "type": "redis", "address": "10.2.30.4:6379" }' \
 http://your.server:port/pubapi/v1/service/create

{
  "message": "success"
}

Create a new service with the specified parameters.

Request

POST http://your.server:port/pubapi/v1/service/create

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
name	string	Required. The name of the service. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
type	string	Required. The type of service, one of: “mysql”, “postgresql”, “mongodb”, “memcached”, “redis”, “httpurl”, “elasticsearch”, “elasticsearch-index” or “elasticsearch-node”.
address	string	Required if type is mysql, postgresql, mongodb, memcached, redis, elasticsearch, elasticsearch-index or elasticsearch-node. This should contain the host and port of the service, like “host:port”. Port is not optional.
user	string	In case of mysql, postgresql, elasticsearch, elasticsearch-index, elasticsearch-node or mongodb, this can contain the username used to connect to the service, if one is needed.
pass	string	In case of mysql, postgresql, elasticsearch, elasticsearch-index, elasticsearch-node or mongodb, this can contain the password used to connect to the service, if one is needed.
ssl	bool	In case of postgresql, if this is set to true, the connection will be SSL-enabled.
https	bool	In case of elasticsearch, elasticsearch-index or elasticsearch-node, if this is set to true, the connection will be over https rather than http.
url	string	Required if type is httpurl. This should contain the HTTP URL of the service.
method	string	Required if type is httpurl. This should contain the HTTP Method of the service, one of “GET”, “POST”, “PUT”, “HEAD”.
header	array	If type is httpurl, additional headers can be specified by setting this to an array of “field” objects that represent these additional headers. See table below.
body	string	If type is httpurl, an HTTP body can be configured by setting this field to the contents of the HTTP body.
index	string	Required if type is elasticsearch-index. This should contain the name of the Elasticsearch index.
node	string	Required if type is elasticsearch-node. This should contain the hostname, IP or node ID of the Elasticsearch node.

The “field” object has the following keys:

Key	Type	Description
key	string	The key of the header field.
value	string	The value of the header field.

Note: OpsDash v1.6 accepted only 32 character service names and did not allow underscore within it.

Response

See Standard Reponses section.

Update

import urllib2, json
data = json.dumps({
    'name':    'redis-stg-3',
    'type':    'redis',
    'address': '10.2.9.8:6379'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/update', data)
response = json.load(body)

curl -X POST \
 -d '{ "name": "redis-stg-3", "type": "redis", "address": "10.2.9.8:6379" }' \
 http://your.server:port/pubapi/v1/service/update

{
  "message": "success"
}

Update the specified service. This takes exactly the same parameters as the service creation API. Changing the type of service is also possible.

Request

POST http://your.server:port/pubapi/v1/service/update

Body

The request body must contain a valid JSON object. The required keys are exactly the same as for the service creation API above.

Response

See Standard Reponses section.

Delete

curl -X POST \
 -d '{ "name": "mysql-t2" }' \
 http://your.server:port/pubapi/v1/service/delete

import urllib2, json

data = json.dumps({
    'name': 'mysql-t2'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/service/delete', data)
response = json.load(body)

{
  "message": "enqueued"
}

Deletes a service and all associated data. Use with care!

Request

POST http://your.server:port/pubapi/v1/service/delete

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
name	string	Required. The name of the service to be deleted.

Source Group APIs

These APIs allow you to list, create, update and delete source groups.

List

import urllib2, json

body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/list')
response = json.load(body)

curl http://your.server:port/pubapi/v1/sourcegroup/list

{
  "sourcegroups": [
    {
      "name": "elb-prod-5",
      "type": "list",
      "sources": [
        "ip-10-0-0-1",
        "ip-10-0-0-2",
        "ip-10-0-0-3"
      ]
    },
    {
      "name": "cache-prod-4",
      "type": "aws",
      "accessKeyId": "AKIAXXXXXXX",
      "region": "us-west-2",
      "tags": [
        {
          "key": "role",
          "value": "query-cache"
        }
      ]
    }
  ]
}

Get details of all configured source groups.

Request

GET http://your.server:port/pubapi/v1/sourcegroup/list

Response

The response is a JSON object having the following keys:

Key	Type	Description
sourcegroups	array	An array of “sourcegroup” objects, each representing a sourcegroup.

The “sourcegroup” object has the following keys:

Key	Type	Description
name	string	The name of the source group.
type	string	The type of source group, one of: “list”, “shellrx”, “posixrx” or “aws”.
sources	array	In case of type list, this will contain an array of strings, each string being the name of a source.
regex	string	In case of type shellrx or posixrx, this will contain the regular expression.
accessKeyId	string	In case of type aws, this will contain the AWS Access Key ID.
region	string	In case of type aws, this will contain the AWS region, like “us-east-1”.
tags	array	In case of type aws, and if tags have been configured, then this will contain an array of “tag” objects (see table below).

The “tag” object has the following keys:

Key	Type	Description
key	string	The key of the tag.
value	string	The value of the tag.

Note that the AWS Secret Access Key will not be returned by the API.

Create

import urllib2, json

data = json.dumps({
    'name':  'mc-cluster',
    'type':  'shellrx',
    'regex': 'mc*'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/create', data)
response = json.load(body)

curl -X POST \
 -d '{ "name": "mc-cluster", "type": "shellrx", "regex": "mc*" }' \
 http://your.server:port/pubapi/v1/sourcegroup/create

{
  "message": "success"
}

Create a new source group with the specified parameters.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/create

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
name	string	Required. The name of the source group. Must be within 255 characters consisting only of alphanumeric characters, “_”, “.” and “-”.
type	string	Required. The type of source group, one of: “list”, “shellrx”, “posixrx” or “aws”.
sources	array	Required if type is “list”. This should contain an array of strings, each string being the name of a source. Each source must be known to OpsDash.
regex	string	Required if type is “shellrx” or “posixrx”. This should contain the appropriate regular expression.
accessKeyId	string	Required if type is “aws”. This should contain the AWS Access Key ID.
secretAccessKey	string	Required if type is “aws”. This should contain the AWS Secret Access Key.
region	string	Required if type is “aws”. This should contain the AWS region, like “us-east-1”.
tags	array	In case of type aws, then can be set to an array of “tag” objects (see table below), which will be used to restrict the EC2 instances selected.

The “tag” object has the following keys:

Key	Type	Description
key	string	Required. The key of the tag.
value	string	The value of the tag.

Note: Currently regular expressions of type “shellrx” recognise only the wildcards * and ?.

Note: OpsDash v1.6 accepted only 32 character source group names and did not allow underscore within it.

Response

See Standard Reponses section.

Update

import urllib2, json

data = json.dumps({
    'name':  'mc-cluster',
    'type':  'posixrx',
    'regex': 'mc.*prod.*master'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/update', data)
response = json.load(body)

curl -X POST \
 -d '{ "name": "mc-cluster", "type": "posixrx", "regex": "mc.*prod.*master" }' \
 http://your.server:port/pubapi/v1/sourcegroup/update

{
  "message": "success"
}

Update the specified source group. This takes exactly the same parameters as the source group creation API. Changing the type of source group is also possible.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/update

Body

The request body must contain a valid JSON object. The required keys are exactly the same as for the source group creation API above.

Response

See Standard Reponses section.

Delete

curl -X POST \
 -d '{ "name": "elb-client5" }' \
 http://your.server:port/pubapi/v1/sourcegroup/delete

import urllib2, json

data = json.dumps({
    'name': 'elb-client5'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/sourcegroup/delete', data)
response = json.load(body)

Deletes a source group and associated configuration. Note that the metric data is associated with individual sources, and deleting a source group does not affect that.

Request

POST http://your.server:port/pubapi/v1/sourcegroup/delete

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
name	string	Required. The name of the source group to be deleted.

Metric APIs

These APIs allow you to manipulate individual metrics.

Delete

import urllib2, json

data = json.dumps({
    'source': 'custom.source.23',
    'metric': 'fuzz.factor'
})
body = urllib2.urlopen('http://your.server:port/pubapi/v1/metric/delete', data)
response = json.load(body)

curl -X POST \
 -d '{ "source": "custom.source.23", "metric": "fuzz.factor" }' \
 http://your.server:port/pubapi/v1/metric/delete

{
  "message": "enqueued"
}

Deletes a metric and all associated data. Use with care!

Apart from the time series data for the metric, references to this metric will be removed from any graph that includes it. If there are open alerts against this metric, they may take up to a minute to clear. This API was introduced in v1.7.4.

Request

POST http://your.server:port/pubapi/v1/metric/delete

Body

The request body must contain a valid JSON object with the following keys:

Key	Type	Description
source	string	Required. The name of the source to which the metric belongs.
metric	string	Required. The name of the metric.

Response

See Standard Reponses section.

Webhook

OpsDash supports configuring webhooks that can receive real-time notifications. Currently, OpsDash supports webhooks for receiving alert notifications.

Alert Notifications

{
  "version": 1,
  "at": 1489907211,
  "breach": [
    {
      "dashid": 2,
      "dashname": "webhook-test-01",
      "graphid": 42,
      "graphname": "Load Average",
      "source": "webhook-test-01",
      "metric": "sys.os.loadavg",
      "breachtype": "warnub",
      "value": 2.4,
      "threshold": 2
    }
  ],
  "down": [
    {
      "source": "webhook-test-02",
      "down": true,
      "last": 1489907031
    }
  ],
  "status": "1 down, 1 warning"
}

Webhook alert notifications are sent out whenever:

• an alert threshold is breached
• an alert threshold that was breached has cleared
• a source is marked as down
• a source that was marked as down has cleared (has come back up)

A single Webhook notification may carry one or more such events.

The HTTP requests made by OpsDash are:

• posted to your URL via HTTP or HTTPS, versions HTTP/2 and HTTP/1.1 supported
• the request method is POST
• the body of the request will be a JSON object, see details below
• the notification is considered successful if an HTTP response with code 200 is received within 3 seconds
• if the request times out, it is retried upto 5 times, with a delay of 5 seconds between retries

Request

POST https://your/webhook/url

Headers

The following extra HTTP headers will be set in the request:

Header	Value	Description
Content-Type	application/json
X-Message-Version	1	the version of the JSON message schema, currently 1
User-Agent	OpsDash/1.0	the user-agent and version
Cache-Control	no-store

Body

The request body will contain a valid JSON object with the following keys:

Key	Type	Description
version	number	The message schema version, currently 1.
at	number	The time the notification was sent out, as the number of seconds from 1 Jan 1970 UTC.
breach	array	An array of “breach” objects, each representing a metric that has breached an upper/lower warning/critical threshold, or has cleared. See table below. May be absent.
down	array	An array of “down” objects, each representing a source that has just gone down, or has come up. See table below. May be absent.

Each “down” object has the following keys:

Key	Type	Description
source	string	The name of the source that has gone down or come up.
down	boolean	Will be true if the source has gone down, or false if the source has come up.
last	integer	The time when the source was last heard from, as the number of seconds from 1 Jan 1970 UTC.

Each “breach” object has the following keys:

Key	Type	Description
dashid	integer	Unique identifier of the dashboard which contains the graph with the breached metric.
dashname	string	The name of the dashboard. For system-generated dashboards, this will be the name of the source or sourcegroup.
graphid	integer	Unique identifier of the graph.
graphname	string	The title of the graph.
source	string	The source or the sourcegroup (see “combine” key below) involved in the breach.
metric	string	The metric involved in the breach.
breachtype	string	The type of breach that occured. Value is one of: “critlb”, “warnlb”, “warnub”, “critub” or an empty string, and indicates which threshold was breached. “lb” stands for lower bound, and “ub” for upper bound. If empty, the breach has cleared.
value	float	The value of the metric that exceeded the threshold.
threshold	float	The threshold, corresponding to the “breachtype”, that was set. This is valid only if “breachtype” is set.
status	string	A human-readable summary of the current overall alert status.

Note that the JSON request body may not be pretty-printed as shown in the example.

Response

The webhook must return an HTTP response with a status code of 200 within 3 seconds for the notification to be considered successful. No other parts of the response are examined.

Standard Responses

Generally, the OpsDash API will respond to requests with HTTP status code 200, 400 or 500. Some responses will include a JSON object as the HTTP body.

A status of 200 indicates success. If additional information is available, it will be included in the HTTP response as a JSON object like so:

{ "message": "queued" }

A status of 400 indicates some form of client error, like missing or incorrect parameters. The response will typically include a message indicating the reason:

{ "message": "name must be specified" }

A status of 500 indicates that the server encountered an error while performing the request. More information might be available in the OpsDash server log file at /var/log/opsdash/opsdash-server.log.

{ "message": "database error" }

Additionally, if htpasswd is set in /etc/opsdash/server.cfg, the API will fail with a 403 status code if a valid Authorization header is not present.

Feedback

Your feedback about these APIs, as well as questions, suggestions and comments are welcome! Drop us a mail at api-team@rapidloop.com.

© RapidLoop 2018 • All Rights Reserved.