background Historically, Linux has lagged far behind other operating systems in terms of multimedia. Microsoft’s Windows and Apple’s MacOS already have great support for multimedia devices, multimedia creation, playback, and real-time processing. On the other hand, Linux’s low overall contribution to multimedia applications makes it difficult for Linux to compete with Windows and MacOS in professional-grade software. GStreamer is designed to address current issues in Linux multimedia.

What is GStreamer? GStramer is a framework for creating streaming media applications. GStreamer’s programming framework makes it possible to write any type of streaming media application. GStreamer is not limited to audio and video processing it can handle any type of data stream. The pipeline design approach has little load on the actual application of filters, and it can even be used to design high-end audio and video applications with high latency requirements.

The most obvious use of GStreamer is in building a player. GStreamer already supports files in many formats, including MP3, Ogg/ vorbis.mPEG-12, AVI, Quickime, mods, and more. From this perspective, GStreamer is more like a player. But its main advantage is that its pluggable components can be easily plugged into any pipe.

The GStreamer framework is based on plug-ins, some of which provide a wide variety of multimedia digital signal codecators, and some of which provide other functions. All plug-ins can be linked to any defined data flow pipeline.

Simple layering of GStreamer framework applications:

Media Applications

The top layer is for applications, such as GStreamer tools (GST-launch, GST-Inspect, etc.), and libraries based on GStreamer packaging (GST-Player, GST-RtSP-Server, GST -editing-services, etc.) according to the implementation of different scenarios.

The middle layer of Core Framework is Core Framework, which mainly provides:

  1. Interfaces required by upper-layer applications

  2. The framework of the Plugin

  3. The framework of Pipline

  4. Data transfer and processing between elements

  5. Syncing between multiple media streams (e.g. audio and video syncing)

  6. Various other tool libraries as required

The lowest layer of Plugins is a variety of plug-ins, which realize specific data processing and audio and video output. The application does not need to pay attention to the details of plug-ins, and the Core Framework layer is responsible for the loading and management of plug-ins. The main categories are:

1. Protocols: Responsible for the processing of various Protocols, such as FILE, HTTP, RTSP, etc.

2. Sources: Responsible for data source processing, alSA, V4L2, TCP/UDP, etc.

3. Formats: Handle media containers, avi, MP4, OGG, etc.

4. Codecs: Responsible for media Codecs, mp3, Vorbis, etc.

Filters: handle media streams, converters, mixers, effects, etc.

6. Sinks: Handle media stream output to specified equipment or destination, ALSA, XVIDEO, TCP/UDP, etc.

7. The GStreamer framework places core and plugins in different source packages, depending on the maturity of each module and the open source protocol used:

GStreamer: contains core framework and core Elements.

Gst-plugins-base: A necessary plug-in for GStreamer applications.

Gst-plugins-good: A high quality plugin that uses the LGPL license.

Gst-plugins-ugly: A high quality plugin that uses the GPL and other licensed libraries, such as X264, X265.

Gst-plugins-bad: plugins that need to be improved and can be moved to the good plugins list when they mature.

Gst-libav: Encapsulates libaV so it can be used in GStreamer framework.

The advantages of GStreamer

  1. The structure is clear and powerful: we can use a series of powerful tools to create media pipes without having to write a single line of code, making complex media control very simple. GStreamer provides a concise and simple API for plug-ins to create self-plugins, as well as integrating a number of debugging and tracing mechanisms and tools. GStreamer also provides a number of real-world examples.

  2. Flexible extensible performance: All GStreamer objects can be extended using GObject inheritance. All plug-ins can be loaded dynamically.

  3. The high performance is mainly reflected in the use of GLib’s G_ mem_ chunk and modular allocation algorithm to minimize memory allocation. The connection between plug-ins is very light-weight. Data passing through the pipe uses minimal overhead, and data passing between plug-ins in the pipe only involves pointer deprecation. Provides a mechanism to operate directly on target memory. For example, plug-ins can write directly to memory space shared by X Server, or buffers can point to arbitrary memory, such as the internal hardware buffers of a sound card. Refcounting and write copy minimize memcpy. Subbuffers effectively separate buffers into manageable chunks. Use cothreads to reduce thread consumption. Cothreads is a simple and fast way to switch subroutines as a measure of minimum consumption of 600 CPU cycles. Use special plug-ins to support hardware acceleration. Use plug-in registration with instructions so that the plug-in is loaded only when it is actually needed. All judgment data is not mutex.

We’ll learn more about GStreamer in the next installment

  • End –

With the rapid development of technology, AMu Lab will keep up with the pace of technology and constantly recommend the latest technology and hardware in the robot industry to everyone. The greatest value of our training is to see our trainees make rapid progress in technology. If you are in the robotics industry, please follow our official account, we will continue to publish the most valuable information and technology in the robotics industry. Amu Lab is committed to cutting-edge IT technology education and intelligent equipment, so that robot research and development more efficient!