Monitoring - Elastic Stack

OpenVidu Pro brings the power of Elastic Stack. Events and monitoring stats are sent to Elasticsearch and can be visualized through Kibana.

Accessing Kibana 🔗

OpenVidu Pro offers two different web dashboards:

  • Access OpenVidu Inspector through https://DOMAIN_OR_PUBLIC_IP/inspector
  • Access Kibana through https://DOMAIN_OR_PUBLIC_IP/kibana

The Elastic Stack deployed with OpenVidu is secured with Basic Auth by default. When connecting to it for the first time you will have to enter the user and password specified when deploying OpenVidu Pro.


Kibana Dashboards 🔗

By default, OpenVidu Pro imports to Kibana multiple dashboards to monitor useful information about your cluster, including metrics about performance, behavior and usage. Let's see all the different dashboards:

CPU vs Sessions/Connections/Streams/Recordings 🔗

This dashboard allows you to monitor the CPU usage of the system over time, presenting it against the total number of OpenVidu Sessions, Connections, Streams and Recordings.


OpenVidu Sessions 🔗

This dashboard presents a summary of your video sessions and their behavior. That includes:

  • The number of session per day
  • The location of your users
  • The number of publishers and subscribers you have at any time
  • The total streamed minutes over time
  • Some low-level interesting stats related to the media connections

The last point includes things like the average time to select a pair candidate during the negotiation process, the average milliseconds your clients take to complete the ICE gathering process with OpenVidu Server, the ratio of successful and failed connections, and some WebRTC stats for your published and subscribed streams (Jitter, packet loss, Round-Trip-Time, target bitrate...)

All in all, this information will help you understand better the behavior and performance of your sessions, and adapt your cluster accordingly.


OpenVidu Recordings 🔗

This dashboard presents at a glance the status of your recordings. It includes information such as:

  • The total number of recordings per day
  • Average duration and size of your recorded files
  • The distributions of you recordings by duration and size
  • The ratio of recordings according to their output mode (COMPOSED or INDIVIDUAL) and their recorded tracks (audio/video recordings)


Server Application Metrics 🔗

Statistics about OpenVidu Server Pro as a web server application. Among others:

  • Number of requests performed to OpenVidu Server Pro
  • Error rate of requests
  • Number of requests to each endpoint, and its response time
  • Http status codes returned


Server Monitoring Metrics 🔗

Statistics about OpenVidu Server Pro as a Spring application. It includes:

  • Start time and CPU usage
  • Stats about the Java Virtual Machine: heap, classes loaded, buffers, threads
  • Information about Tomcat server: active sessions, send and receive bytes
  • Number of logs by level: INFO, WARN, ERROR

[Metricbeat] Cluster Monitoring Metrics 🔗

Metrics overview of the entire cluster. This dashboard lets you see the state of your cluster globally, which includes:

  • Number of nodes
  • Average CPU usage
  • Average memory usage
  • Average disk usage
  • Inbound/Outbound networking metrics

Additionally you can check specific metrics of your nodes by clicking into one of the displayed hosts which will redirect you into the specific [Metricbeat] Node Monitoring Metrics of the selected host.

[Metricbeat] Node Monitoring Metrics 🔗

More detailed metrics about OpenVidu Pro Cluster Nodes. This dashboard includes information showed in [Metricbeat] Cluster Monitoring Metrics with additional information like:

  • Memory usage
  • Inbound/Outbound networking metrics
  • Network packet losses
  • Advanced charts about CPU/disk/network usage
  • Advanced information about disk usage

If you accessed this dashboard by selecting a node in [Metricbeat] Cluster Monitoring Metrics, you will see information about the selected node in this panel. Otherwise if you access this dashboard without any filter or by accessing the dashboard panel, this dashboard will show average information about the entire cluster.

[Metricbeat] Nginx Metrics 🔗

This dashboard contains metrics about the deployed NGINX proxy which handles all requests to the cluster. It includes:

  • Active connections: number of active client connections including waiting connections.
  • Request Rate: rate of client requests.
  • Drops Rate: dropped client connections.
  • Accepts and Handled Rate: number of accepted and handled client connections.
  • Reading/Writing/Waiting Rates: number of connections where Nginx is reading/writing/waiting.


Elasticsearch indexes 🔗

All events, logs and metrics of OpenVidu are stored within indexes in ElasticSearch. You can explore this data by going to:
Kibana Menu -> Discover -> Select a filebeat index.

OpenVidu logs and OpenVidu events are sent both by OpenVidu Pro Server in openvidu-logs-* and openvidu indexes respectively. On the other hand all the information about other logs services and metrics are sent by:

All data sent to Elasticsearch can be classified in this categories:

  • OpenVidu logs: This data, which contains all OpenVidu Pro Server logs, is sent to openvidu-logs-* indexes.
  • OpenVidu events: All events that occurs in OpenVidu Pro are sent to openvidu index. You can read more information in the OpenVidu Events section.
  • OpenVidu Application and Server metrics: All this data goes to server-metrics-* and session-metrics-*.
  • Filebeat data: This service is responsible for sending logs to filebeat-* indexes.
  • Metricbeat data: This service is responsible for sending metrics to metricbeat-* indexes.

All data sent by Filebeat and Metricbeat follows a common format named ECS (Elastic Common Schema). Here is a version table of the ECS format followed by each OpenVidu Pro Version.

OpenVidu Pro version Metricbeat / Filebeat Version ECS version - (More information about ECS here)
≤ 2.15.0 - -
≥ 2.16.0 7.8.0 1.5.0

Logs and metrics 🔗

All index patterns which start with filebeat-* refers to OpenVidu Pro services logs. On the other hand metrics are sent to indexes which start with metricbeat-*.

Logs and metrics indexes are:

  • filebeat-kurento*: Kurento Media Server logs.
  • filebeat-coturn*: Coturn(TURN/STUN server) logs.
  • filebeat-redis*: Redis logs (This service is used to store TURN credentials).
  • filebeat-media-node-controller*: Logs of the media-node-controller which manage orchestration in media nodes.
  • filebeat-openvidu-recording*: Logs of COMPOSED recording.
  • filebeat-nginx*: Logs of Nginx container.
  • openvidu-logs*: Logs of OpenVidu Server Pro.
  • openvidu-browser-logs*: Logs of clients/browsers using openvidu-browser related with OpenVidu itself. These are only sent if OPENVIDU_BROWSER_LOGS=debug in /opt/openvidu/.envfile (More info).
  • metricbeat-*: Metrics sent by all the nodes of the OpenVidu Pro Cluster.

Logs and metrics sent by Filebeat and Metricbeat include some properties to distinguish the origin of the data, in addition to the ECS attributes. These properties are present in all filebeat-*, metricbeat-* and openvidu-logs-* indexes:

  • node_role: Which can be masternode or medianode.
  • node_id: Which can be master_<id> or media_<id>. Depending on the deployed environment the <id> may change.
  • cluster_id: Defined Cluster Id in OPENVIDU_PRO_CLUSTER_ID. If no cluster id is defined, the DOMAIN_OR_PUBLIC_IP will be used instead.

OpenVidu Events 🔗

All events of OpenVidu are stored in the index openvidu, which have an elastic_type field to identify the specific type of event. This field may be:

  • cdr: event of CDR/Webhook. Can take multiple forms according to the type of event (see OpenVidu Webhook)
  • kms: Kurento Media Server event. These events are always associated to one WebRTC endpoint (a publisher or a subscriber). Can take multiple forms according to the type of event (see Kurento docs)
  • webrtcStats: event of WebRTC statistics for each media endpoint established in Media Nodes
  • webrtcDebug: event with further information about the WebRTC negotiation process, such as SDPs, for each media endpoint established in Media Nodes
  • networkQualityStats: event of network quality for a specific client. See Network quality
  • sessionSummary: summary of a session, stored once it is closed
  • recordingSummary: summary of a recording, stored once its session is closed. This object does not hold the final values of the recording entity and some properties may not be properly defined. Use WebHook event recordingStatusChanged with property status set to ready to get the full recording object
  • userSummary: summary of a user, stored once its session is closed
  • connectionSummary: summary of a connection, stored once its session is closed
  • publisherSummary: summary of a publisher, stored once its session is closed
  • subscriberSummary: summary of a subscriber, stored once its session is closed
{
  "elastic_type": "cdr",
  "sessionId": "TestSession",
  "uniqueSessionId": "TestSession_1614770057250",
  "event": "sessionDestroyed",
  "reason": "sessionClosedByServer",
  "startTime": 1614770057250,
  "duration": 32,
  "timestamp": 1614770090067,
  "cluster_id": "my_cluster",
  "master_node_id": "master_localhost"
}


NOTE 1: sessionSummary contains all the information available in the rest of summary documents, including an array of recordingSummary and an array of userSummary. In turn userSummary contains an array of connectionSummary, that finally contains an array of publisherSummary and other of subscriberSummary. To sum up, this is just a denormalization of the sessionSummary document, so Elasticsearch requests and Kibana visualizations are more flexible and easier to accomplish

NOTE 2: recordingSummary events may not contain the final information of the actual recordings (specifically properties size and duration). This is so because recordingSummary event is generated just after its session is closed, but since release 2.11.0 recordings may need a post-processing phase before being available for download and having these properties properly defined. To overcome this limitation, you can simply use the cdr event of type recordingStatusChanged and status ready corresponding to this recording (see event in WebHook docs). There you will have all properties of the recording well defined


Creating your own visualizations and dashboards 🔗

The dashboards presented above, by default included in OpenVidu Pro, are just an example of what can be done thanks to Kibana. You can create your own visualizations, and set up your very own dashboards with them. To do so, you have available multiple events that OpenVidu Pro periodically stores in Elasticsearch, and you can then use them in Kibana to compose different types of graphs and other useful visual representations.

You can create powerful visualizations and dashboards by using these documents. Let's see a quick example. Imagine that you are interested in knowing how many users are connected to your OpenVidu sessions over time.

First thing is navigating to Visualize section in Kibana and clicking on the button to add a new visualization

Then we have to choose a visualization type. In this case a vertical bar graph might be a pretty good choice

We select openvidu index, because that's the index of every OpenVidu Pro event stored in Elasticsearch

The visualization page will be shown. Now we have to filter the desired events. In this case, we just want the userSummary event, as it gathers all the information about the final users connecting to our sessions. So, we make sure that field elastic_type.keyword is userSummary

Finally we have to configure the data passed to our graph. The metric we want (Y-axis) is simply Count, because there is one "userSummary" event for each final user connecting to a session. And as Bucket (X-axis) we configure a Date Histogram by using timestamp field of the event

To store the new visualization just click on Save button in the upper menu and give it a meaningful name

The example above is a very simple visualization, but you can apply any metric to any property (or set of properties) of any event (or set of events). You can explore pre-existing visualizations included by default in OpenVidu Pro, and for further info visit Kibana docs


Reviewing logs 🔗

Reviewing logs with Kibana 🔗

Many of OpenVidu processes send their logs to Elasticsearch, so you can review their logs directly from Kibana. To search for logs you need to:

First of all, go to the Logs section of Kibana.

Enter what you want to search. You can search for literals, attribute values, etc...

Examples 🔗

  • Search for Kurento Media Server logs:
  • Search for Kurento Media Server logs of one specific node id:
  • Search for Kurento Media Server logs of one specific Media Node IP:
  • Search for OpenVidu Server logs:

Reviewing logs and metrics using OpenVidu Inspector 🔗

Searching logs in Kibana can be a little bit complicated when you want to debug specific logs for specific sessions. We offer you an easy way to search for these logs using the OpenVidu Inspector. You just need to:

Go to the section History section of OpenVidu Inspector.

Select the session you want to debug.

Choose what logs you want to review.


Configuring an external Elastic Stack 🔗

An external Elastic Stack can be configured with and without security. The table below shows a compatibility table between versions of OpenVidu Pro and external Elastic providers.

OpenVidu Pro version AWS Elasticsearch Service Elastic Cloud Your own Elastic Stack on premises
≤ 2.15.0 - - -
2.16.0 ≥ 7.3.0 ≥ 7.3.0 ≥ 7.3.0
≥ 2.17.0 ≥ 7.8.0 ≥ 7.8.0 ≥ 7.8.0

OpenVidu Pro Configuration for external Elastic Stack 🔗

If you have an external Elastic Stack configured without security, you only need to specify these properties in the /opt/openvidu/.env file of your Master Node. For example:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://elk.example.com
OPENVIDU_PRO_KIBANA_HOST=https://elk.example.com/kibana

If you have configured a user in your Elastic Stack, you need to also specify it in your /opt/openvidu/.env. Only Basic Auth is supported, which is used by Elastic Stack in their Basic License and is free to use. You need to modify these properties:

ELASTICSEARCH_USERNAME=<YOUR_ELK_USER>
ELASTICSEARCH_PASSWORD=<YOUR_ELK_USER_PASSWORD>

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server Pro. Just execute ./openvidu restart in /opt/openvidu.

  • Managed Users by OpenDistro Elastic Stack are compatible too. This is the distribution of Elastic Stack used by AWS.
  • You can see how to create a fine-grained user for OpenVidu in this section

Examples of managed Elastic Stack services 🔗

Any deployed Elastic Stack can be used, but to give a clear guide on how to configure an external Elastic Stack, we will explain how to configure a managed one in AWS and in Elastic Cloud.

Elastic Stack in AWS with OpenVidu Pro configuration 🔗

In the next section you will see how to configure a Managed Elastic Stack in AWS and how to configure it for OpenVidu:

First you need to go to the Elasticsearch Service in your AWS Account.

Select: Create new Domain.

Choose the Deployment Type you want.

After that you can configure your ELK as you want until security configuration, which is explained in the next steps.

Check Enable fine-grained access control and select Create master user. This user will have super user privileges but you can create after that a user with only necessary privileges.

Select Allow open access to the domain.

Click Next and Confirm and you're cluster will be created in a matter of seconds.

Get the Endpoint and Kibana URLs, which will be used in our OpenVidu Pro configuration

When the Domain Status of your Elasticsearch cluster is ready, let's configure OpenVidu Pro to use it. You will need to modify /opt/openvidu/.env and add into this file the username, password, Domain Endpoint and Kibana URL from previous steps. The .env file must have this parameters:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://xxxxxxxx.es.amazonaws.com
OPENVIDU_PRO_KIBANA_HOST=https://xxxxxxxx.es.amazonaws.com/_plugin/kibana/
ELASTICSEARCH_USERNAME=<YOUR_USERNAME>
ELASTICSEARCH_PASSWORD=<YOUR_PASSWORD>

Where:

  • OPENVIDU_PRO_ELASTICSEARCH_HOST: The Domain Endpoint URL visible in the Elasticsearch panel of AWS.
  • OPENVIDU_PRO_KIBANA_HOST: The Kibana URL also visible in the Elasticsearch panel of AWS.
  • ELASTICSEARCH_USERNAME: The master user you have created in previous steps.
  • ELASTICSEARCH_PASSWORD: The master password you have created in previous steps.

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server. Just execute ./openvidu restart in /opt/openvidu.

  • Note that you need to specify port 443 in OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST.
WARNING: Elasticsearch index deletion policy (which is responsible for delete indexes after the specified days in OPENVIDU_PRO_ELASTICSEARCH_MAX_DAYS_DELETE) does not work with AWS Elasticsearch. You must create your own deletion policy strategy in your AWS Elasticsearch service.

Elastic Stack in Elastic Cloud with OpenVidu Pro configuration 🔗

In this section you will see how to configure a Managed Elastic Stack in Elastic Cloud (The official Elastic Stack as a Service) and how to configure it for OpenVidu:

First of all, click on Create deployment.

Select: Elastic Stack.

Configure your Elastic Stack as you want and click Create deployment.

Get the Username and password generated by Elastic Cloud. This user will have super user privileges but you can create after that a user with only necessary privileges.

When the deployment is ready, get the Elasticsearch endpoint and Kibana endpoint to use it for OpenVidu Pro.

When the cluster is ready, let's configure OpenVidu Pro to use it. You will need to modify /opt/openvidu/.env and add into this file the Username and Password generated by Elastic Cloud and your ElasticSearch and Kibana endpoint URLs. The .env file must have this parameters:

OPENVIDU_PRO_ELASTICSEARCH_HOST=https://xxxxxxxx.elastic-cloud.com:9243/
OPENVIDU_PRO_KIBANA_HOST=https://yyyyyyyy.elastic-cloud.com:9243
ELASTICSEARCH_USERNAME=<YOUR_USERNAME>
ELASTICSEARCH_PASSWORD=<YOUR_PASSWORD>

Where:

  • OPENVIDU_PRO_ELASTICSEARCH_HOST: The Domain Endpoint URL visible in the Elastic Cloud panel.
  • OPENVIDU_PRO_KIBANA_HOST: The Kibana URL visible in the Elastic Cloud panel.
  • ELASTICSEARCH_USERNAME: The master user generated by Elastic Cloud.
  • ELASTICSEARCH_PASSWORD: The master password generated by Elastic Cloud.

Remember that any change you do in /opt/openvidu/.env will require you to restart your OpenVidu Server. Just execute ./openvidu restart in /opt/openvidu.

  • Note that you need to specify port 9243 in OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST. But you can also use port 443, Elastic Cloud has configured both ports for both URLs.
  • OPENVIDU_PRO_ELASTICSEARCH_HOST and OPENVIDU_PRO_KIBANA_HOST use the same https port, but both URLs has a different subdomain associated.
  • You can create a fine-grained user to let OpenVidu access to specific resources of Elasticsearch and Kibana

Create a fine-grained user 🔗

Configuring a user with all the privileges is not secure. To accomplish better security in your Elastic Stack cluster we will create a role and a user prepared to be used by OpenVidu Server Pro.

Create a fine-grained user using Kibana UI 🔗

First of all, go to the Stack Management section.

In the Stack Management section, go to Roles.

Click on Create role button.

Add into the role the Cluster privileges and Indexes privileges specified in the image. All these privileges are necessary for OpenVidu to send logs, metrics and Events related with OpenVidu.

Click on Add Space privilege.

Select Default space and Custom Privilege.

Select All in Saved Objects Management. In this way OpenVidu will be able to update, add and remove dashboards and other Kibana configurations.

Click on Create role button.

After the role is created, go to Users section.

Click on Create User button.

Configure an Username and a Password and the Role created before. Then just click on Create User button.

After creating the user you just need to configure it in your /opt/openvidu/.env file:

ELASTICSEARCH_USERNAME=<USER_NAME>
ELASTICSEARCH_PASSWORD=<USER_PASSWORD>

Create a fine-grained user using Elastic Stack REST API 🔗

If you want to create a role and a user without using the UI you can create both by using REST API requests.

  1. First you need to create a role using the name you want by calling Kibana REST API.
  2. Secondly, create a user which will use the role created before by calling Elasticsearch REST API.

You can see an example of both HTTP requests here:

  • ROLE_NAME: Name that you want for the role you are creating

$ curl -X PUT "localhost:5601/api/security/role/<ROLE_NAME>" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
   "elasticsearch":{
      "cluster":[
         "monitor",
         "manage_ilm",
         "manage_pipeline",
         "manage_ingest_pipelines",
         "manage_index_templates"
      ],
      "indices":[
         {
            "names":[
               "metricbeat*",
               "openvidu",
               "openvidu-logs*",
               "filebeat*",
               "server-metrics*",
               "session-metrics*",
               "loadtest*"
            ],
            "privileges":[
               "create_doc",
               "create_index",
               "manage",
               "read",
               "write",
               "manage_ilm"
            ]
         }
      ],
      "run_as":[

      ]
   },
   "kibana":[
      {
         "spaces":[
            "default"
         ],
         "base":[

         ],
         "feature":{
            "savedObjectsManagement":[
               "all"
            ]
         }
      }
   ]
}
'


After creating the user you just need to configure it in your /opt/openvidu/.env file:

ELASTICSEARCH_USERNAME=<USER_NAME>
ELASTICSEARCH_PASSWORD=<USER_PASSWORD>