Tuesday, 21 June 2016

qmlglsink - GStreamer and Qt's QML

Windows Support - NEW!!

Just recently, I added the missing piece for adding Windows support for qmlglsink for QtQuick 2 applications (tracked by this bug) with this commit which means that qmlglsink is now usable on all major platforms1. With the list of supported platforms (Android, iOS, OS X, Windows, Linux) for qmlglsink growing, I thought I'd write a little bit about it and why you should use it.

qmlglsink is a GStreamer element that will render a video stream into a QtQuick QML scene. As you can see from this example, qmlglsink does this by providing a GstGLVideoItem QML item that can be placed anywhere in the scene. The GstGLVideoItem is given a unique name to be able to link the element and the QML item. Without this link, the element cannot render the stream anywhere and will fail changing state. In the C++ code, we then retrieve the GstGLVideoItem by name like so. Then we set the pipeline to playing, start Qt's mainloop and watch the show.

Why Use qmlglsink?

The main reason that you should have a look at qmlglsink is if you need the full flexibility of GStreamer across multiple platforms. QtMultimedia only provides a small subset of the features possible with GStreamer and only uses GStreamer on unix systems. Only file-based playback or Buffer-based playback (depending on the backend) is supported by QtMultimedia and it does not come by default with zerocopy support for any platform.

GStreamer on the other hand supports a multitude of media sources from network streaming like rtp and rtsp, file and memory sources as well as recent GStreamer versions containing zerocopy support for Android, iOS2, the RPi, and some support for zerocopy on desktop Linux with VA-API. GStreamer is also extremely flexible and extensible as a multitude of different multimedia tasks are just a plugin away.

1 - Not all platforms are easy to build against Qt. Notably the recent Windows support requires manual intervention with qmake utilizing the added qtplugin.pro file. iOS has been tested against specially modified Qt pkg-config and libtool files.
2 - Unfortunately the iOS zerocopy support is not usable with Qt due to slight incompatibilities between OpenGL|ES 2 and OpenGL|ES 3 on the iOS platform. Qt currently defaults to OpenGL|ES 3 while GStreamer is still choosing OpenGL|ES 2 for zerocopy support. On all other platforms OpenGL ES 2 is compatible with OpenGL ES 3. Edit: This is no longer an issue with this and this commit

Tuesday, 16 February 2016

Vulkan in GStreamer

Today, Vulkan 1.0 as released. For those of you that don't what Vulkan is, the TL;DR is essentially that it's cross-platform/device lower level 3D API than OpenGL (both the desktop and the ES variants) of a similar scope to DX12, Metal or Mantle. As a result, the driver does less work and the application/engine takes over some of the management of resources which hopefully leads to a smaller driver divergence and therefore bugs while at the same time providing increased access to the capabilities of the GPU. Centricular graciously sponsored some of my time to have a look at Vulkan and implement a basic working prototype of a GStreamer sink element that would display video frames onto the screen. I hereby present to you, vulkansink, the result of that labour.
gst-launch-1.0 videotestsrc ! vulkanupload ! vulkansink

Disclaimer: this is still alpha-quality code and it hasn't had the same extensive testing as the OpenGL elements and there are a number of short fallings:
  1. It doesn't scale the input video to the output surface size (yet).
  2. There's very little locking (as required by the Vulkan specification, e.g. VkCommandBuffer's and VkCommandPool's).
  3. There aren't any effects/colour conversion implemented.
  4. Performance has not been a primary concern so there are some inefficiencies in the current design.
All that being said, my impressions of Vulkan are relatively good. It promises to fit infinitely better with GStreamer's extensive use of threads than an equivalent OpenGL implementation. That extra flexibility and functionality comes at the price of increased complexity however I think that is a necessary cost.

The hard API change also provides a higher baseline for implementations and applications to depend against which resets the amount of legacy code that would need to be maintained for older GPU's and systems.

Implementation Details

The current implementation of libgstgl was used extensively as a basis for the structure of how Vulkan is intended to be used in GStreamer.

Memory

Vulkan provides a high level overview of accessing GPU memory through VkDeviceMemory objects. These are simply wrapped in GstMemory objects as GstVulkanMemory with allocation and destruction routines for creating and destroying GPU memory as well as mapping with gst_memory_map() for CPU access to the data (if supported at creation).
Vulkan provides VkBuffer objects which are somewhat equivalent to OpenGL's Buffer objects which are used for storing vertices, draw indices, uniforms, texels, etc and is wrapped up in GStreamer into the GstVulkanBufferMemory object (a subclass of GstMemory). As a backing store, it allocates a GstVulkanMemory object.
GstVulkanImageMemory provides the same abstraction for VkImage objects which also have a backing store of a GstVulkanMemory.

Integrating with applications

The GstContext infrastructure is used extensively to not only share necessary state between elements but also providing application specific state into GStreamer. The current list of GstContext-able objects is, VkInstance, VkDevice, VkDisplay and the VkQueue which all have GStreamer wrappers to account for refcounting and in the future, Vulkan's locking requirements.

Friday, 4 September 2015

GStreamer 1.6 and OpenGL contexts

With the imminent release of GStreamer 1.6 there has been a change in the mechanism by which OpenGL contexts and Window System (X11, Wayland, Win32, OS X's Cocoa) specific resources are shared with a GStreamer pipeline for elements that need that information. Elements like sinks or any of the GL elements or any global application provided information that is shared between elements.

Previously, in 1.4, there was a property, 'external-opengl-context' available on some(!) elements that one could set on the current set of filter and mixer elements (not sink or source elements though!). While it mostly worked well enough, it was a behaviour difference to how window system handles are shared between the application and the GStreamer pipeline. Internal GL context sharing between elements was occurring in the ALLOCATION query as a value in a GstStructure of the meta:GstVideoGLTextureUploadMeta parameters. The main problem with this is 1. the ALLOCATION query is handled differently by different elements and more critically, 2. the tee element drops ALLOCATION queries entirely. This would result in a performance hit in a pipeline with GL elements containing a tee element like:
gltestsrc ! tee ! glimagesink
In the above pipeline, the GL context's would not be the same between or shared between gltestsrc and glimagesink, so the gltestsrc texture would be downloaded to system memory (a very slow operation) and reuploaded by glimagesink. Evidently we should always attempt to keep data on the GPU if at all possible to be as performant as possible.

GstContext provides a way of sharing not only between elements but also with the application using queries and messages. See https://developer.gnome.org/gstreamer/stable/gstreamer-GstContext.html for more details. Using GstContext we have the ability to share arbitrary data between elements (even across tee's) and the application which is what we want.

For the application side, the required activity is to call gst_element_set_context() with the needed/requested GstContext. This can be at application start up before the pipeline is set up or as a response to a message on the bus.

Example: sharing an X11 display with the bus callback
static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE) == 0) {
        Display *x11_display; /* get this from the application somehow */
        GstGLDisplay *gl_display = GST_GL_DISPLAY (gst_gl_display_x11_new_with_display (x11_display));

        context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
        gst_context_set_gl_display (context, gl_display);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}
Here's an example for passing a user provided GL context:
static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) {
        GstGLContext *gl_context; /* get this from the application somehow */
        GstStructure *s;

        context = gst_context_new ("gst.gl.app_context", TRUE);
        s = gst_context_writable_structure (context);
        gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}
You may have noticed that we don't specifiy how you retrieve these display's or application GL context's. That's very application specific and typically one is using some other library to perform the required setup for the platforms one is running on. However if you're using Gtk+, QML or CoreAnimation directly, there are ready made sink elements (gtkglsink, qmlglsink and caopengllayersink) that perform this for you that are also available in the upcoming GStreamer 1.6 release.

There are also some helpers for the OpenGL side of things if your library doesn't have a sink yet. See gst_gl_context_new_wrapped(), gst_gl_context_get_current_gl_context() and gst_gl_context_get_current_gl_version() in the GStreamer GL stack here and here.

happy GStreamer OpenGL-ing!

Sunday, 24 May 2015

GNOME Asia 2015

I've just arrived back from GNOME Asia and it's time to blog about the experience there.

The trip to Jakarta was relatively uneventful. An early train to Sydney started the day followed by the trip to Singapore. A tight connection (1 hour) combined with switching terminals made life a little interesting. The fact that one of the Changi terminal connection AirTrain's was out of service meant we were bussed across the tarmac. Then to find the gate and the aeroplane, and the short trip to Jakarta.

After landing in Jakarta, I was meant to meet up with a local Indonesian, however different terminal exit points meant that we missed each other and a phone call was necessary on another local's mobile phone. After meeting there was a bus trip followed by a KRL (train) trip followed by a ride on a Ojek (motorcycle) which made for an interesting trip to my sleeping quarters. A local meal was followed swiftly by some horizontal time.

Some early breakfast consisting of an interesting mix of bread, cheese and condensed milk was followed by a short walk to the conference venue where the workshops were to take place. As can be seen from the schedule The workshops started out with the a talk about the GNOME Indonesian translation. The workshop was presented in Bahasa Indonesia and unfortunately which I could not understand. He however had lots of pictures and gleaned some information from them. The following workshops by on contributing to GNOME and creating your first GNOME application were an interesting experience on how some people enter the GNOME contributor ecosystem. During the day, some fellow GStreamer hackers Arun and Nirbheek arrived and a mini reunion was held.

I attended a number of talks about using and promoting GNOME within the Indonesian shoe industry, the GNOME release cycle and a number of lightning talks. I also met a number of new people from Europe, David, Ekaterina, Alexandre and Andre. Originally I had planned on giving a rather technical talk on integrating GStreamer video and Gtk+ with respect to API use and best practices. However, after the first day, I felt a more general overview of the topic would fit better with the audience. So I adjusted the focus of my talk to include a more general overview of video handling within GStreamer

Overall, it was a time well spent catching up and meeting new people. I would like to thank the GNOME Foundation for sponsoring my travel.

Thursday, 19 February 2015

First blog post! Let's get straight into it.

While developing some cool new features for the GStreamer GL stack, I found the need to have GstBin contain the properties of a containing GstElement. Extra points if we can change the containing element and the GstBin properties react accordingly.

Option 1 - manual forwarding

The simplest and most boring solution, create subclasses with the necessary properties and forward to the relevant containing element. We should be able to do better than this.

Option 2 - GParamSpecOverride

The next option is to use g_param_spec_override which points the property to a property within another object. This comes with the advantage and pitfall that it's kinda supported by GLib directly but not thoroughly tested.

/* should only called from an object's class_init function */
void
install_properties_into_class_from_gtype (GObjectClass * self, GType type)
{
  GObjectClass *class;
  GParamSpec ** props;
  GParamSpec ** filtered_props;
  guint i, j, n;

  class = g_type_class_ref (type);
  if (!class)
    g_assert_not_reached ();

  props = g_object_class_list_properties (class, &n);
  filtered_props = g_new0 (GParamSpec *, n + 1);
  filtered_props[0] = NULL;

  for (i = 0, j = 1; i < n; i++) {
    /* don't forward element properties */
    if (!g_type_is_a (GST_TYPE_ELEMENT, props[i]->owner_type)) {
      filtered_props[j] = g_param_spec_override (name, props[i]);
      j++;
    }
  }
  g_object_class_install_properties (self, j, filtered_props);

  g_free (props);
  g_free (filtered_props);

  g_type_class_unref (class);
}

Then you also need to manually forward the set/get_property() to the appropriate object. Technically the documentation hints that the overridden property and the property to be overridden should be in the same class hierarchy, i.e. one be a subclass of the other. In practice, this seems to mostly works across different classes. You can provide an override for a property for a completely different object. However, I have noticed that g_object_class_list_properties seems to skip override properties altogether. Presumably because they are meant to be provided by another class lower down the hierarchy. This causes the properties to not show up in the gst-inspect output.

Option 3 - GBinding

Instead of overriding properties, bind them to each other. To do that we need equivalent properties on both classes. This is done by this hacky copy code for GParamSpec's

for (i = 0, j = 1; i < n; i++) {
  /* don't forward element properties */
  if (!g_type_is_a (GST_TYPE_ELEMENT, props[i]->owner_type)) {
    const gchar *name = g_param_spec_get_name (props[i]);
    const gchar *nick = g_param_spec_get_nick (props[i]);
    const gchar *blurb = g_param_spec_get_blurb (props[i]);
    GTypeQuery t_info;

    g_type_query (G_PARAM_SPEC_TYPE (props[i]), &t_info);

    printf ("prop %u size %u name %s nick %s blurb %s\n", i,
        t_info.instance_size, name, nick, blurb);

    /* the other option is to use g_param_spec_override() however the causes
     * the properties to not display in gst-inspect */
    filtered_props[j] = g_param_spec_internal (G_PARAM_SPEC_TYPE (props[i]),
        name, nick, blurb, props[i]->flags);
    j++;
  }
}
Then you need to call g_object_bind_property for each property after _class_init() (like _init()) like so.

GParamSpec **props;
guint i, n;

props = g_object_class_list_properties (G_OBJECT_CLASS (klass), &n);

for (i = 0; i < n; i++) {
  /* don't forward element properties */
  if (!g_type_is_a (GST_TYPE_ELEMENT, props[i]->owner_type)) {
    /* only bind properties that are in both objects */
    if (g_object_class_find_property (G_OBJECT_GET_CLASS (self->filter),
          props[i]->name)) {
      /* property binding only works for writable properties */
      if ((props[i]->flags & G_PARAM_WRITABLE) == G_PARAM_WRITABLE) {
        g_object_bind_property (self->filter, props[i]->name, self,
            props[i]->name, G_BINDING_BIDIRECTIONAL | G_BINDING_SYNC_CREATE);
      }
    }
  }
}

g_free (props);

As hinted by the code, this only seems to work for writable properties. Readable or construct-only properties will fail to bind correctly. Also, the ranges, defaults and some types don't seem to be copied correctly from the original property. If you try and set an invalid value however, it still warns appropriately from trying to set the value on the original property.

Option 4 - GstChildProxy

As the documentation says, GstChildProxy allows to set child properties from the containing element. The drawback is that it comes with a completely different API and any dependent code needs to be updated.

For this case, I think I'm going to start with GstChildProxy for the generic stuff and implement manual overrides for cases where we need actual properties on the containing elements like sinks with the last-sample or sync/async properties.