V4L2 Userspace Library

From LinuxTVWiki
Revision as of 17:37, 7 June 2009 by CityK (talk | contribs) (rename file names (prep'ing for image merge))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

The need

What do we need in userspace, only for v4l (dvb is not concerned):

  • particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in kernelspace since it is of linear complexity but shall be moved to userspace)
  • colorspace translations
  • filters that be done in hardware if the selected hardware can, otherwise software plugin


The general constraints

  • Application shall still have access to the basic kernel driver. Thus, applications will have access to the 'raw' input data without software post-processing. Application developers love to develop optimized codecs :)
  • The original v4l driver infrastructure should not change. We do not want to break any drivers. However, the extended v4l driver will use common functionalities like video_ioctl2 if this contributes to simplify the architecture.
  • As a consequence, the v4l API does not change for applications.
  • v4l driver and extended v4l driver shall mutually exclusive since they share the same hardware
  • Limit as much as possible kernel memory and CPU consumption.
  • Elegant, flexible.This is why I specify before coding :)

The ideal solution

The ideal solution would be to create a v4l2 library provided to applications. They would access the v4l2 drivers via this library that would manage decompression if needed. If this library existed from the v4l project start, applications would have linked to it forcibly. The v4l project has a long history and it is more and more difficult to break compatibility now and force applications to link to a new library.

A solution

I thought it would be funny to create userspace drivers like FUSE based projects do. Applications would see nothing but a new device driver to access, all decompression algorithm would be in userspace. The path from the application to the v4l2 kernel driver: [Application]<->[/dev/video1 (created by FUSE-like driver)]<->[/dev/video0 (created by kernel v4l2 driver)] For applications, this changes nothing... For v4l drivers for hardware that can fullfill all application requested formats, this changes... nothing too. This architecture is interesting for v4l drivers that interface hardware that provide proprietary compressed video pictures or that are not able to fullfill the application needs by hardware.

A general overview of the thing:

V4L2 Userspace Library General.png

As a consequence:

The only change for the application is only a new device to access (/dev/video1) so there is no extension of the v4l API/ABI. The only change for the user is to launch another daemon in the starting scripts (/etc/rc.d/...). The v4l driver does not change, only extension points will be implemented if necessary.

The extended v4l driver is an instance of a v4l driver with some entry points for the helper daemon


Specification

First, some obvious assumptions:

  • The API available to the applications will be named v4l public API (headers in path /usr/include/linux/)
  • All extensions will be made by internal APIs. I will name it v4l extension private API (headers in path /usr/src/linux/drivers/media/video/...)
  • I will name uncompressed frames formats the ones defined in the v4l public API.
  • I will name compressed frames formats the ones not defined in the v4l public API.


General use case view of the extended v4l driver:

V4L2 Userspace Library Diagramme1.png

Open/close access the hardware

Summary

This step corresponds to the first and the last access to hardware via the extended driver. Context The application opens the extended v4l driver. The scenario will assume that /dev/video0 is the base device entry point and /dev/video1 is the extended entry point.

Description

  1. The application opens /dev/video1.
  2. The extended device driver asks the base driver to lock the device.
  3. The application controls/gets incoming frames from the hardware (see further).
  4. The application closes /dev/video1.
  5. The extended device driver wipes out all internal allocations and asks the v4l helper to do so...
  6. The extended device driver asks the base driver to unlock the device.

V4L2 Userspace Library OpenCloseAccessTheHardware.png

Exceptions

  • At open, the base v4l driver is already locked. The extended v4l driver returns -EBUSY.


Control the hardware

Summary

This step corresponds mainly to the ioctl() entry points' behaviour. The extended v4l driver will have to interact with the v4l base driver and with the v4l helper daemon with specific ioctls. The scenario will show the typical controls the application will address to the extended v4l driver. In most cases, the extended v4l driver will ask the base v4l driver to do the job. The v4l helper daemon will be used on specific control steps.

Context

The application needs to get frames from the v4l peripheral, this task is fulfilled by the extended v4l driver with the help of the base v4l driver. Pre-conditions

The extended driver is opened properly on /dev/video1.

Description

  1. The application gets the device hardware capabilities with VIDIOC_QUERYCAP.
  2. The application enumerates inputs with VIDIOC_ENUMINPUT.
  3. The application sets desired input with VIDIOC_S_INPUT.
  4. The application enumerates supported video standards with VIDIOC_ENUMSTD.

For all these previous steps, the extended video driver transmits the request to the base driver without interaction.

  1. The application negociates with the extended v4l driver the video data format, with VIDIOC_ENUM_FMT, VIDIOC_S_FMT.
  2. If the v4l helper is here, see the extension point named "Provide compatible video formats". Otherwise, see alternative A1

V4L2 Userspace Library ControlTheHardware.png

Alternatives

A1 - the v4l helper daemon is not running (the /dev/v4lctl driver is not opened by the helper daemon) the extended v4l driver transmits the request to the base driver without interaction...


Get video frames

Summary

As the driver is correctly configured, the hardware captures and streams data from the device. This step will show how to managed the incoming data and provide them to the Application. This is the most important part to understand from this specification.

Pre-conditions

The extended v4l driver is properly configured on /dev/video1.

Description

  1. The application requests the driver to allocate frames with VIDIOC_REQBUFS.
  2. The extended v4l driver requests the v4l helper daemon to allocate video buffers
  3. The application requests each allocated buffer characteristics with VIDIOC_QUERYBUF.
  4. The application requests the driver to enqueue each frame with VIDIOC_QBUF.
  5. The application starts the acquisition chain with VIDIOC_STREAMON.
  6. The application waits and dequeues a frame with VIDIOC_DQBUF.
  7. The extended v4l driver waits and dequeues a frame from the base v4l driver. If the frame needs to be uncompressed, see the extension point named "Uncompress frame".

Otherwise the extended v4l driver transmits the request to the base driver without interaction.

  1. As soon as the frame is processed, the application calls VIDIOC_QBUF to re-enqueue this buffer.
  2. The VIDIOC_DQBUF/QBUF mechanism is repeated to get each frame sequentially.
  3. The application stops the acquisition with VIDIOC_STREAMOFF.

V4L2 Userspace Library GetVideoFrames.png

Exceptions

EX1 The v4l helper daemon is not active or does not respond correctly to the extended v4l device driver request The extended v4l device driver returns -EFAULT (internal error) to the application, this ends the process.

Provide compatible video data formats

Summary

This step specifies how the extended v4l driver is able to provide to the application the hardware and software compatible video data formats

Pre-conditions

The v4l helper daemon has been registered

Description

  1. Either the application has requested to enumerate all supported video formats,
    1. The extended v4l driver ask the v4l helper daemon all possible formats for this hardware.
    2. The v4l helper daemon provides all additional software video format.
  2. Or the applications has requested to set a specific video format,
    1. The extended v4l driver asks the v4l helper daemon if the requested video format is possible for this hardware
    2. If the helper daemon is able to process by software the video format, it acknowledges with the negociated video format. Otherwise it returns a NACK


Uncompress frames

Summary

This part concerns the interactions with the userland v4l helper in order to uncompress frames.

Context

In order to precise the context of a frame decompression, here is a collaboration overview of all components involved.

V4L2 Userspace Library UncompressFrames.png

Pre-conditions

The extended v4l driver is properly configured on /dev/video1. The v4l helper is present and available (has answered correctly to all steps before) The extended v4l driver received a new compressed frame from the base v4l driver

Description

  1. The extended v4l driver prepares data for the v4l helper daemon. This implies some generic preprocessing such as eventual memcopies.
  2. The extended v4l driver asks the v4l helper daemon to uncompress the newly available frame
  3. The helper daemon answers as soon as the processing is done
Retrieved from "https://www.linuxtv.org/wiki/index.php?title=V4L2_Userspace_Library&oldid=23323"