GStreamer

GStreamer is a multimedia processing library which front end applications can leverage in order to provide a wide variety of functions such as audio and video playback, streaming, non-linear video editing and V4L2 capture support. It is mainly used as API working in the background of applications as Totem. But with the comman-line-tools gst-launch and entrans it is also possible to directly create and use gstreamer-piplines on the commandline.

Getting GStreamer

GStreamer, the most common Gstreamer-plugins and the most important tools like gst-launch are available through your disttribution's package management. But entrans and some of the plugins used in the examples below are not. You can find theirf sources bundled by the [GEntrans project] at sourceforge. Google may help you to find precompiled packages for your distro. There are the old GStreamer 0.10 and the new GStreamer 1.0. They are incompatible but can be installed parallel. Everything said below can be done with both GStreamer 0.10 and GStreamer 1.0. You simply have to use the appropriate commands, e.g. gst-launch-0.10 vs. gst-launch-1.0 (gst-launch is linked to one of them). Note the ffmpeg-plugins have been renamed to av. For example ffenc_mp2 of GStreamer 0.10 is called avenc_mp2 in GStreamer 1.0.

Using GStreamer for V4L TV capture

Why preferring GStreamer?

Despite the fact that GStreamer is much more flexible than other solutions most other tools especially those which are based on the ffmpeg library (e.g. mencoder) have by design difficulties in managing A/V-synchronisation: The common tools process the audio and the video stream independently frame by frame as frames come in. Afterwards they are muxed relying on their specified framerates. E.g. if you have a 25fps video stream and a 48,000kHz audio stream it simply takes 1 video frame, 1920000 audio frames, 1 video frame and so on. This probably leeds to sync issues for at least three reasons:

• If frames get dropped audio and video shift against each other. For example if your CPU is not fast enough and sometimes drops a video frame after 25 dropped audio frames the video is one second in advance. (Using mencoder this can be fixed by usindg the -harddup option in most situations. It causes mencoder to wath if video frames are dropped and to copy other frames to fill the gaps before muxing.)
• The audio and video devices need different time to start the streams. For example after you have started capturing the first audioframe is grabbed 0.01 seconds thereafter while the first videoframe is grapped 0.7 seconds thereafter. This leads to constant time-offset between audio and video. (Using mencoder this can be fixed by using the -delay option. But you have to find out the appropriate delay by try and error. It won't be very precise. Another problem is that things very often change if you update your software as for example the buffers and timeouts of the drivers and the audioframework change. So you have to do it again and again after each update.)
• If the clocks of your audio source and your video source are not accurate enough (very common on low-cost home-user-products) and your webcam sends 25.02 fps instead of 25 fps and your audio source delivers 47,999kHz instead of 48,000kHz audio and video are going to slowly shift by time. The result is that after an hour or so of capturing audio and video differ by a second or so. But that's not only an issue of inaccurate clocks. Analogue TV is not completely analogue but the frames are discrete. Therefore video-capturing devices can't rely on their internal clocks but have to synchronize to the incoming signal. That's no problem if capturing live TV, but if you capture the signal of a VHS recorder. These recorders can't instantly change the speed of the tape but need to slightly adjust the fps for accurate tracking, especially if the quality of the tape is bad. (This issue can't be addressed using mencoder.)

Entrans versus gst-launch

gst-launch is better documented and part of all distributions. But entrans is a bit smarter for the following reasons:

• It provides partly automatically composing of GStreamer pipelines
• It allows cutting of the streams; the most simple application of this feature is to capture for a distinct time. That allows the muxers to properly close the captured files writing correct headers which is not always given if you finissh capturing with gst-launch by simply typing Ctrl+C.

Sample commands

Record to ogg theora

gst-launch-0.10 oggmux name=mux ! filesink location=test0.ogg v4l2src device=/dev/video2 ! \  
video/x-raw-yuv,width=640,height=480,framerate=\(fraction\)30000/1001 ! ffmpegcolorspace ! \
theoraenc ! queue ! mux. alsasrc device=hw:2,0 ! audio/x-raw-int,channels=2,rate=32000,depth=16 ! \
audioconvert ! vorbisenc ! mux.

The files will play in mplayer, using the codec Theora. Note the required workaround to get sound on a saa7134 card, which is set at 32000Hz (cf. bug). However, I was still unable to get sound output, though mplayer claimed there was sound -- the video is good quality:

VIDEO:  [theo]  640x480  24bpp  29.970 fps    0.0 kbps ( 0.0 kbyte/s)
Selected video codec: [theora] vfm: theora (Theora (free, reworked VP3))
AUDIO: 32000 Hz, 2 ch, s16le, 112.0 kbit/10.94% (ratio: 14000->128000)
Selected audio codec: [ffvorbis] afm: ffmpeg (FFmpeg Vorbis decoder)

Record to mpeg4

Or mpeg4 with an avi container (Debian has disabled ffmpeg encoders, so install Marillat's package or use example above):

gst-launch-0.10 avimux name=mux ! filesink location=test0.avi v4l2src device=/dev/video2 ! \
video/x-raw-yuv,width=640,height=480,framerate=\(fraction\)30000/1001 ! ffmpegcolorspace ! \ 
ffenc_mpeg4 ! queue ! mux. alsasrc device=hw:2,0 ! audio/x-raw-int,channels=2,rate=32000,depth=16 ! \ 
audioconvert ! lame ! mux.

I get a file out of this that plays in mplayer, with blocky video and no sound. Avidemux cannot open the file.

Record to raw video

If you don't care for sound, this simple version works for uncompressed video:

gst-launch-0.10 v4l2src device=/dev/video5 ! video/x-raw-yuv,width=640,height=480 ! avimux ! \
filesink location=test0.avi

tcprobe says this video-only file uses the I420 codec and gives the framerate as correct NTSC:

$ tcprobe -i test1.avi
[tcprobe] RIFF data, AVI video
[avilib] V: 29.970 fps, codec=I420, frames=315, width=640, height=480
[tcprobe] summary for test1.avi, (*) = not default, 0 = not detected
import frame size: -g 640x480 [720x576] (*)
       frame rate: -f 29.970 [25.000] frc=4 (*)
   no audio track: use "null" import module for audio
           length: 315 frames, frame_time=33 msec, duration=0:00:10.510

The files will play in mplayer, using the codec [raw] RAW Uncompressed Video.

Using GStremaer to view pictures from a Webcam

gst-launch-0.10 v4l2src use-fixed-fps=false ! video/x-raw-yuv,format=\(fourcc\)UYVY,width=320,height=240 \
! ffmpegcolorspace ! ximagesink

gst-launch-0.10 v4lsrc autoprobe-fps=false device=/dev/video0 ! "video/x-raw-yuv, width=160, height=120, \
framerate=10, format=(fourcc)I420" ! xvimagesink  

Converting formats

To convert the files to matlab (didn't work for me):

mencoder test0.avi -ovc raw -vf format=bgr24 -o test0m.avi -ffourcc none 

For details, see gst-launch and google; the plugins in particular are poorly documented so far.

Further documentation resources

• Gstreamer project
• FAQ
• Documentation
  • Core plugins
  • FFMpeg plugins
  • Base plugins
  • Good plugins
  • Bad plugins
  • Ugly plugins
  • Entrans plugins
• man gst-launch

External Links

• GStreamer project page
• GStreamer Wikipedia page
• GEntrans project