Voice and video filters 🔗

OpenVidu API offers a simple way of applying filters to video and audio streams in the server side by making use of Kurento Media Server capabilities. This is the current status of filter support in OpenVidu:

  • Real time filters are only available for Kurento Media Server. OpenVidu Enterprise edition supports mediasoup as media server, and audio and video filters are not compatible with it.
  • You can apply one filter at a time to a published Stream. Every user subscribed to it will receive the modified stream.
  • You can remove an applied filter.
  • You can call any remote method offered by an applied filter
  • You can add and remove event listeners to any event dispatched by an applied filter.
  • You must configure the allowed filters the user will be able to apply when creating the Connection.

Check out openvidu-filters tutorial to see OpenVidu filters in action.

For OpenViduENTERPRISE voice and video filters are only available when using Kurento as media server. Currently mediasoup is not compatible with voice and video filters. See Kurento vs mediasoup.

Step by step 🔗


1) Generate a Connection with the filters the user will be able to apply 🔗

This is a simple way of securing the ability of applying filters from OpenVidu Browser, so that not every user is able to apply any filter at any time.

When creating a Connection, call ConnectionProperties.Builder#connectionProperties(KurentoOptions) to set allowedFilters value with method KurentoOptions.Builder#allowedFilters(String[]). This method receives a string array containing the name of the filters the user will be able to apply

ConnectionProperties connectionProperties = new ConnectionProperties.Builder()
    .type(ConnectionType.WEBRTC)
    .data("user_data")
    .role(OpenViduRole.PUBLISHER)
    .kurentoOptions(
        new KurentoOptions.Builder()
            .allowedFilters(new String[]{"GStreamerFilter", "FaceOverlayFilter"})
            .build())
    .build();
Connection connection = session.createConnection(connectionProperties);
String token = connection.getToken(); // Send this string to the client side


2.A) Initialize a Publisher object configured for using a filter from the beginning of the publishing ... 🔗

Use PublisherProperties, specifically property filter:

var OV = new OpenVidu();
var publisher = OV.initPublisher(
    targetElement,
    filter: {
        type: "GStreamerFilter",
        options: {
            command: "videoflip method=vertical-flip"
        }
    }
);

// ... user already connected to "session" with the appropriate token of the created Connection
session.publish(publisher);


2.B) ... or apply the filter dynamically after publishing the stream, whenever you want 🔗
// ... user already connected to the session with the appropriate token of the created Connection
// and successfully publishing the Publisher object

publisher.stream.applyFilter("GStreamerFilter", { command: "videoflip method=vertical-flip" })
    .then(() => {
        console.log("Video rotated!");
    })
    .catch(error => {
        console.error(error);
    });


3) You can execute any method offered by the filter 🔗
// ... user already connected to the session with the appropriate token of the created Connection,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.filter.execMethod("setElementProperty", {"propertyName":"method","propertyValue":"horizontal-flip"})
    .then(() => {
        console.log("Video rotation direction modified!");
    })
    .catch(error => {
        console.error(error);
    });


4) You can also subscribe to any filter event (if it dispatches any), and later unsubscribe from it 🔗
// ... user already connected to the session with the appropriate token of the created Connection,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.filter.addEventListener("FooFilterEvent", filterEvent => {
        console.log('Filter event received!. Data: ' + filterEvent.data);
    });

...

publisher.stream.filter.removeEventListener("FooFilterEvent");


4) To remove the filter 🔗
// ... user already connected to the session with the appropriate token of the created Connection,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.removeFilter()
    .then(() => {
        console.log("Filter removed");
    })
    .catch(error => {
        console.error(error);
    });

Moderators are not only able to call all of these methods over their Publisher.stream object, but also over any Subscriber.stream object. Also, they don't need any special Connection permissions to apply filters (no need of kurentoOptions configuration in their Connection) and can bypass any restriction set to other Connections in this regard.



Filter samples 🔗

GStreamer filters 🔗

These filters are set with type GStreamerFilter and an options parameter like this:

publisher.stream.applyFilter("GStreamerFilter", {"command": "GSTREAMER_COMMAND"})

A list of interesting values for GSTREAMER_COMMAND parameter is stated below. Replace GSTREAMER_COMMAND in the upper code snippet for any of the examples provided in the following list items:


Video overlay filters 🔗
  • gdkpixbufoverlay : overlays an image on top of the video. This is very useful, for example, to add a logo
    Example: gdkpixbufoverlay location=/images/img.png offset-x=10 offset-y=10 overlay-height=200 overlay-width=200
    Documentation: Link
  • textoverlay : overlays an embedded text on top of the video
    Example: textoverlay text="Embedded text" valignment=top halignment=right font-desc="Cantarell 25"
    Documentation: Link
  • timeoverlay : overlays the time the video stream has been playing
    Example: timeoverlay valignment=bottom halignment=right font-desc="Sans, 20"
    Documentation: Link
  • clockoverlay : overlays a clock with the local time (in OpenVidu deployment host)
    Example: clockoverlay valignment=bottom halignment=right shaded-background=true font-desc="Sans, 20"
    Documentation: Link


Video effect filters 🔗
  • videoflip : rotates the video stream
    Example: videoflip method=vertical-flip
    Documentation: Link
  • videocrop : crops the video steam
    Example: videocrop top=100 left=35 right=0 bottom=0
    Documentation: Link
  • videobox : resizes a video stream by adding borders or cropping
    Example: videobox fill=black top=20 bottom=20 left=-10 right=-10
    Documentation: Link
  • chromahold : removes all colors from the video stream except the RGB indicated one
    Example: chromahold target-r=0 target-g=0 target-b=255 tolerance=90
    Documentation: Link
  • coloreffects : applies different color filters to the video stream
    Example: coloreffects preset=heat
    Documentation: Link
  • videobalance : changes different properties of the video stream such as brightness, contrast, hue or saturation
    Example: videobalance saturation=0.0
    Documentation: Link
  • gamma : adjusts gamma level of the video stream
    Example: gamma gamma=5.0
    Documentation: Link
  • videomedian : adds a median filter to the video stream
    Example: videomedian filtersize=9 lum-only=false
    Documentation: Link
  • Many effects of effectv project : funny filters for the video stream like agingtv, dicetv, optv, quarktv, radioactv, revtv, rippletv, shagadelictv, streaktv, vertigotv, warptv
    Example: radioactv


Audio filters 🔗
  • audioecho : adds reverb to the audio stream
    Example: audioecho delay=50000000 intensity=0.6 feedback=0.4
    Documentation: Link
  • audioamplify : amplifies an audio stream by a given factor
    Example: audioamplify amplification=1.5 clipping-method=wrap-positive
    Documentation: Link
  • pitch : controls the pitch of an audio stream
    Example: pitch pitch=1.2
    Documentation: Link
  • Other audio filters: check them out in GStreamer site

All available GStreamer plugins can be found in GStreamer site.

Kurento filters 🔗


  • FaceOverlayFilter (overlay an image over detected faces)
    publisher.stream.applyFilter("FaceOverlayFilter")
        .then(filter => {
            filter.execMethod(
                "setOverlayedImage",
                {
                    "uri":"https://cdn.pixabay.com/photo/2013/07/12/14/14/derby-148046_960_720.png",
                    "offsetXPercent":"-0.2F",
                    "offsetYPercent":"-0.8F",
                    "widthPercent":"1.3F",
                    "heightPercent":"1.0F"
                });
        });
    


  • ChromaFilter (set a chroma background)
    publisher.stream.applyFilter(
        "ChromaFilter",
        {
            "window": {
                "topRightCornerX": 0,
                "topRightCornerY": 0,
                "width": 50,
                "height": 50
            },
            "backgroundImage": "https://www.maxpixel.net/static/photo/1x/Cool-Blue-Liquid-Lake-Abstract-Background-Clear-316144.jpg"
        });
    

You must install Kurento ChromaFilter library in the same host as Kurento Media Server: sudo apt-get install kms-chroma


  • ZBarFilter (detect and read bar codes information)
    publisher.stream.applyFilter("ZBarFilter")
        .then(filter => {
            filter.addEventListener("CodeFound", filterEvent => {
                console.log('Bar code found!. Data: ' + filterEvent.data);
            }
        });