We now offer a REST API to gather information from Elasticsearch. In this article we list what indices you have access to and how you can reach the REST API. We will also give you some PHP examples. The REST API is part of the ELK Stack. To find out more about the ELK Stack and Kibana, check out this article.

General

Naming convention

As a customer, you have access to the following indices:

  • logstash-nginx              [Hypernode Nginx access logs]
  • logstash-slowlogs        [Hypernode MySQL slow logs]
  • telegraf                          [Hypernode server metrics]
  • heartbeat                      [Hypernode availability data]

All indexes are suffixed with the creation date of the particular index. The format is “YEAR.MONTH.DAY”. As an example, the logstash-nginx index from new years day would be accordingly “logstash-nginx-2019.01.01”.

Every index contains multiple documents (docs), which relate to a single log line (nginx), metric (telegraf), availability check (heartbeat) or slow query (logstash-slowlogs). This doc contains multiple fields which might be different for each doc in an index.

We add one or two referencing fields to each document for the purpose of searching and filtering. Below we list the referencing fields for each index type.

Indices Referencing fields Content example
logstash*               hypernode_domain_id 123456
hypernode_domain_name               APPNAME.hypernode.io
telegraf* tag.hypernode_domain_id 123456
tag.hypernode_domain_name APPNAME.hypernode.io
heartbeat* monitor.host APPNAME.hypernode.io

Index rotation

Indexes are rotated between 00:45 and 01:00 at night (CET). We do this because servers can have long running processes and/or a shipping backlog. This means data shipped between 00:00 and 00:45 is included in the index of the previous day.

While this does not impact the desired date range in your search query, it is important to include the index of the previous day if you don’t want to miss out results.

REST API

You can reach the API at the URL: https://kibana.byte.nl/rest/

Documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/6.2/index.html

Rate limit

We limit the amount of API calls to 3 queries per second for each source IP address. This corresponds to one call every 333 ms. We allow a burst of up to 4 calls, which makes the maximum simultaneous calls 5. This means that after doing 5 calls, you have to wait 333 ms for the next call, or around 1.3 seconds before you can do 5 calls again.

PHP examples

Below we show two examples of how to gather information from Elasticsearch using the PHP-API. Installation instructions can be found on https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_quickstart.html

Example A: Query for the most recent availability document for a hypernode

Example B: Show the last 500 access log entries for two different hypernodes

02