Monitoring - Elastic Stack

OpenVidu Pro brings the power of Elastic Stack. Events and monitoring stats are sent to Elasticsearch and can be visualized through Kibana.



Accessing Kibana 🔗

OpenVidu Pro offers two different web dashboards:

  • Access OpenVidu Inspector through https://DOMAIN_OR_PUBLIC_IP/inspector
  • Access Kibana through https://DOMAIN_OR_PUBLIC_IP/kibana

The Elastic Stack deployed with OpenVidu is secured with Basic Auth by default. When connecting to it for the first time you will have to enter the user and password specified when deploying OpenVidu Pro.



Kibana Dashboards 🔗

By default, OpenVidu Pro imports to Kibana 5 different dashboards to monitor useful information about:

  • OpenVidu Sessions
  • OpenVidu Recordings
  • Monitoring OpenVidu Server Pro Node
  • Monitoring Media Nodes
  • NGINX

Let's see all the different dashboards:

OpenVidu Sessions dashboard 🔗

This dashboard presents a summary of your video sessions and their behavior. That includes:

  • The number of session per day
  • The location of your users
  • The number of publishers and subscribers you have at any time
  • The total streamed minutes over time
  • Some low-level interesting stats related to the media connections

The last point includes things like the average time to select a pair candidate during the negotiation process, the average milliseconds your clients take to complete the ICE gathering process with OpenVidu Server, the ratio of successful and failed connections, and some WebRTC stats for your published and subscribed streams (Jitter, packet loss, Round-Trip-Time, target bitrate...)

All in all, this information will help you understand better the behavior and performance of your sessions, and adapt your cluster accordingly.


OpenVidu Recordings dashboard 🔗

This dashboard presents at a glance the status of your recordings. It includes information such as:

  • The total number of recordings per day
  • Average duration and size of your recorded files
  • The distributions of you recordings by duration and size
  • The ratio of recordings according to their output mode (COMPOSED or INDIVIDUAL recordings) and their recorded tracks (audio/video recordings)


Monitoring OpenVidu Server Pro Node 🔗

This dashboard provides monitoring information about the OpenVidu Server Pro Node (see OpenVidu Pro architecture). It includes CPU, memory, disk, network, I/O and running threads.

Monitoring Media Nodes 🔗

This dashboard provides monitoring information about the Media Nodes (see OpenVidu Pro architecture). It includes CPU usage.

NGINX 🔗

This dashboard gives you information about the NGINX proxy of the OpenVidu Pro cluster, including workers, active connections, response time... This is actually the official NGINX dashboard of Kibana.



Creating your own visualizations and dashboards 🔗

The dashboards presented above, by default included in OpenVidu Pro, are just an example of what can be done thanks to Kibana. You can create your own visualizations, and set up your very own dashboards with them. To do so, you have available multiple events that OpenVidu Pro periodically stores in Elasticsearch, and you can then use them in Kibana to compose different types of graphs and other useful visual representations.

Each one of these events stored by OpenVidu Pro in Elasticsearch has an elastic_type field to identify the specific type of event. This field may be:

  • cdr: event of CDR/Webhook. Can take multiple forms according to the type of event (see OpenVidu CDR)
  • kms: Kurento Media Server event. These events are always associated to one WebRTC endpoint (a publisher or a subscriber). Can take multiple forms according to the type of event (see Kurento docs)
  • monitoringStats: event of CPU, memory and network statistics usage of OpenVidu Sever Pro Node
  • webrtcStats: event of WebRTC statistics for each media endpoint established in Media Nodes
  • networkQualityStats: event of network quality for a specific client. See Network quality
  • sessionSummary: summary of a session, stored once it is closed
  • recordingSummary: summary of a recording, stored once its session is closed
  • userSummary: summary of a user, stored once its session is closed
  • connectionSummary: summary of a connection, stored once its session is closed
  • publisherSummary: summary of a publisher, stored once its session is closed
  • subscriberSummary: summary of a subscriber, stored once its session is closed
{
  "sessionId": "TestSession",
  "timestamp": 1582277160836,
  "streamId": "str_CAM_AOIa_con_XZvrQOF5Du",
  "participantId": "con_XZvrQOF5Du",
  "connection": "OUTBOUND",
  "videoSource": "CAMERA",
  "videoFramerate": 30,
  "videoDimensions": "{\"width\":640,\"height\":480}",
  "audioEnabled": true,
  "videoEnabled": true,
  "event": "webrtcConnectionCreated",
  "elastic_type": "cdr"
}


NOTE 1: sessionSummary contains all the information available in the rest of summary documents, including an array of recordingSummary and an array of userSummary. In turn userSummary contains an array of connectionSummary, that finally contains an array of publisherSummary and other of subscriberSummary. To sum up, this is just a denormalization of the sessionSummary document, so Elasticsearch requests and Kibana visualizations are more flexible and easier to accomplish


NOTE 2: recordingSummary events may not contain the final information of the actual recordings (specifically properties size and duration). This is so because recordingSummary event is generated just after its session is closed, but since release 2.11.0 recordings may need a post-processing phase before being available for download and having these properties properly defined. To overcome this limitation, you can simply use the cdr event of type recordingStatusChanged and status ready corresponding to this recording (see event in CDR docs). There you will have all properties of the recording well defined


You can create powerful visualizations and dashboards by using these documents. Let's see a quick example. Imagine that you are interested in knowing how many users are connected to your OpenVidu sessions over time.

First thing is navigating to Visualize section in Kibana and clicking on the button to add a new visualization

Then we have to choose a visualization type. In this case a vertical bar graph might be a pretty good choice

We select openvidu index, because that's the index of every OpenVidu Pro event stored in Elasticsearch

The visualization page will be shown. Now we have to filter the desired events. In this case, we just want the userSummary event, as it gathers all the information about the final users connecting to our sessions. So, we make sure that field elastic_type.keyword is userSummary

Finally we have to configure the data passed to our graph. The metric we want (Y-axis) is simply Count, because there is one "userSummary" event for each final user connecting to a session. And as Bucket (X-axis) we configure a Date Histogram by using timestamp field of the event

To store the new visualization just click on Save button in the upper menu and give it a meaningful name

The example above is a very simple visualization, but you can apply any metric to any property (or set of properties) of any event (or set of events). You can explore pre-existing visualizations included by default in OpenVidu Pro, and for further info visit Kibana docs



Reviewing logs 🔗

Many of OpenVidu processes send their logs to Elasticsearch, so you can review their logs directly from Kibana. All the indexes which have logs from services are:

  • filebeat-kurento*: Kurento Media Server logs.
  • filebeat-coturn*: Coturn(TURN/STUN server) logs.
  • filebeat-redis*: Redis logs (This service is used to store TURN credentials).
  • openvidu-logs*: OpenVidu Server logs.

To search for logs you need to:

First of all, go to the Logs section of Kibana.

Go to settings, and configure from which index you want to search logs. In this particular example, we're searching Kurento Media Server logs by setting Log indexes to: filebeat-kurento*.

Add to Log Columns the attributes of the index you want to see in your logs stream. In this example we're adding the attribute log_level to see the type of log (INFO, WARN, ERROR, etc...). The attributes may change depending of the index you're reading. For example, in OpenVidu Server Pro, the log level is in the attribute severity for openvidu-logs* indexes.

Enter what you want to search. You can search for literals, attribute values, etc...

Searching Examples 🔗

  • Search for a literal in Kurento Media Server logs and see the context:
  • Search by log level in OpenVidu Server Pro logs:
  • Search for logs of one specific Media Node IP:



Configuring an external Elastic Stack 🔗

An external Elastic Stack can be configured with and without security. The table below shows a compatibility table between versions of OpenVidu Pro and external Elastic providers.

OpenVidu Pro version AWS Elasticsearch Service Elastic Cloud Your own Elastic Stack on premises
≤ 2.15.0 - - -
2.16.0 ≥ 7.3.0 ≥ 7.3.0 ≥ 7.3.0

OpenVidu Pro Configuration for external Elastic Stack 🔗

If you have an external Elastic Stack configured without security, you only need to specify these properties in the /opt/openvidu/.env file of your OpenVidu Server Pro Node. These properties are URLs that need to have specified a port, even if it is port 80 or 443. For example:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://elk.example.com:443
OPENVIDU_PRO_KIBANA_HOST=https://elk.example.com:443/kibana

If you have configured a user in your Elastic Stack, you need to also specify it in your /opt/openvidu/.env. Only Basic Auth is supported, which is used by Elastic Stack in their Basic License and is free to use. You need to modify these properties:

ELASTICSEARCH_USERNAME=<YOUR_ELK_USER>
ELASTICSEARCH_PASSWORD=<YOUR_ELK_USER_PASSWORD>

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server Pro. Just execute ./openvidu restart in /opt/openvidu.

  • Managed Users by OpenDistro Elastic Stack are compatible too. This is the distribution of Elastic Stack used by AWS.
  • You can see how to create a fine-grained user for OpenVidu in this section

Examples of managed Elastic Stack services 🔗

Any deployed Elastic Stack can be used, but to give a clear guide on how to configure an external Elastic Stack, we will explain how to configure a managed one in AWS and in Elastic Cloud.

Elastic Stack in AWS with OpenVidu Pro configuration 🔗

In the next section you will see how to configure a Managed Elastic Stack in AWS and how to configure it for OpenVidu:

First you need to go to the Elasticsearch Service in your AWS Account.

Select: Create new Domain.

Choose the Deployment Type you want.

After that you can configure your ELK as you want until security configuration, which is explained in the next steps.

Check Enable fine-grained access control and select Create master user. This user will have super user privileges but you can create after that a user with only necessary privileges.

Select Allow open access to the domain.

Click Next and Confirm and you're cluster will be created in a matter of seconds.

Get the Endpoint and Kibana URLs, which will be used in our OpenVidu Pro configuration

When the Domain Status of your Elasticsearch cluster is ready, let's configure OpenVidu Pro to use it. You will need to modify /opt/openvidu/.env and add into this file the username, password, Domain Endpoint and Kibana URL from previous steps. The .env file must have this parameters:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://xxxxxxxx.es.amazonaws.com:443
OPENVIDU_PRO_KIBANA_HOST=https://xxxxxxxx.es.amazonaws.com:443/_plugin/kibana/
ELASTICSEARCH_USERNAME=<YOUR_USERNAME>
ELASTICSEARCH_PASSWORD=<YOUR_PASSWORD>

Where:

  • OPENVIDU_PRO_ELASTICSEARCH_HOST: The Domain Endpoint URL visible in the Elasticsearch panel of AWS.
  • OPENVIDU_PRO_KIBANA_HOST: The Kibana URL also visible in the Elasticsearch panel of AWS.
  • ELASTICSEARCH_USERNAME: The master user you have created in previous steps.
  • ELASTICSEARCH_PASSWORD: The master password you have created in previous steps.

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server. Just execute ./openvidu restart in /opt/openvidu.

  • Note that you need to specify port 443 in OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST.
  • You can create a fine-grained user to just give access to OpenVidu to specific resources of Elasticsearch and Kibana

Elastic Stack in Elastic Cloud with OpenVidu Pro configuration 🔗

In this section you will see how to configure a Managed Elastic Stack in Elastic Cloud (The official Elastic Stack as a Service) and how to configure it for OpenVidu:

First of all, click on Create deployment.

Select: Elastic Stack.

Configure your Elastic Stack as you want and click Create deployment.

Get the Username and password generated by Elastic Cloud. This user will have super user privileges but you can create after that a user with only necessary privileges.

When the deployment is ready, get the Elasticsearch endpoint and Kibana endpoint to use it for OpenVidu Pro.

When the cluster is ready, let's configure OpenVidu Pro to use it. You will need to modify /opt/openvidu/.env and add into this file the Username and Password generated by Elastic Cloud and your ElasticSearch and Kibana endpoint URLs. The .env file must have this parameters:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://xxxxxxxx.elastic-cloud.com:9243/
OPENVIDU_PRO_KIBANA_HOST=https://yyyyyyyy.elastic-cloud.com:9243
ELASTICSEARCH_USERNAME=<YOUR_USERNAME>
ELASTICSEARCH_PASSWORD=<YOUR_PASSWORD>

Where:

  • OPENVIDU_PRO_ELASTICSEARCH_HOST: The Domain Endpoint URL visible in the Elastic Cloud panel.
  • OPENVIDU_PRO_KIBANA_HOST: The Kibana URL visible in the Elastic Cloud panel.
  • ELASTICSEARCH_USERNAME: The master user generated by Elastic Cloud.
  • ELASTICSEARCH_PASSWORD: The master password generated by Elastic Cloud.

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server. Just execute ./openvidu restart in /opt/openvidu.

  • Note that you need to specify port 9243 in OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST. But you can also use port 443, Elastic Cloud has configured both ports for both URLs.
  • OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST use the same https port, but both URLs has a different subdomain associated.
  • You can create a fine-grained user to let OpenVidu access to specific resources of Elasticsearch and Kibana

Create a fine-grained user 🔗

Configuring a user with all the privileges is not secure. To accomplish better security in your Elastic Stack cluster we will create a role and a user prepared to be used by OpenVidu Server Pro.

Create a fine-grained user using Kibana UI 🔗

First of all, go to the Stack Management section.

In the Stack Management section, go to Roles.

Click on Create role button.

Add into the role the Cluster privileges and Indexes privileges specified in the image. All these privileges are necessary for OpenVidu to send logs, metrics and Events related with OpenVidu.

Click on Add Space privilege.

Select Default space and Custom Privilege.

Select All in Saved Objects Management. In this way OpenVidu will be able to update, add and remove dashboards and other Kibana configurations.

Click on Create role button.

After the role is created, go to Users section.

Click on Create User button.

Configure an Username and a Password and the Role created before. Then just click on Create User button.

After creating the user you just need to configure it in your /opt/openvidu/.env file:

ELASTICSEARCH_USERNAME=<USER_NAME>
ELASTICSEARCH_PASSWORD=<USER_PASSWORD>

Create a fine-grained user using Elastic Stack REST API 🔗

If you want to create a role and a user without using the UI you can create both by using REST API requests.

  1. First you need to create a role using the name you want by calling Kibana REST API.
  2. Secondly, create a user which will use the role created before by calling Elasticsearch REST API.

You can see an example of both HTTP requests here:

  • ROLE_NAME: Name that you want for the role you are creating

$ curl -X PUT "localhost:5601/api/security/role/<ROLE_NAME>" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
   "elasticsearch":{
      "cluster":[
         "monitor",
         "manage_ilm",
         "manage_pipeline",
         "manage_ingest_pipelines",
         "manage_index_templates"
      ],
      "indexes":[
         {
            "names":[
               "metricbeat*",
               "openvidu",
               "openvidu-logs*",
               "filebeat*"
            ],
            "privileges":[
               "create_doc",
               "create_index",
               "manage",
               "read",
               "write",
               "manage_ilm"
            ]
         }
      ],
      "run_as":[

      ]
   },
   "kibana":[
      {
         "spaces":[
            "default"
         ],
         "base":[

         ],
         "feature":{
            "savedObjectsManagement":[
               "all"
            ]
         }
      }
   ]
}
'


After creating the user you just need to configure it in your /opt/openvidu/.env file:

ELASTICSEARCH_USERNAME=<USER_NAME>
ELASTICSEARCH_PASSWORD=<USER_PASSWORD>