In addition to the official Edinburgh's Media Summit, there was an extra meeting to discuss Stateless Codecs implementation. The meeting notes are enclosed here.

Stateless codec discussion

Hans presented a summary of this in his stateless codec status presentation, here are a bit more details.

We had a discussion (first in the Microsoft sponsor suite, then at the bar) on how to support user space for the stateless codecs better. The expected outcome of that would be a rough understanding how a stateless codec user space library would look like.

Attendees

• Alexandre Courbot
• Chris Healy
• Ezequiel Garcia
• Hans Verkuil
• Kieran Bingham
• Laurent Pinchart
• Maxime Ripard
• Mauro Carvalho Chehab
• Nicolas Dufresne
• Niklas Söderlund
• Philip Zabell
• Sakari Ailus
• Tomasz Figa
• Victor Jáquez

Buffer management

Nicolas reported an issue in V4L2 buffer management. The V4L2 decouples the buffers from the format, and assumes all queued buffers (at a given point of time) have the same format. (Ed. note: the request API could be used to address this, but that particular features is not yet supported.)

User space library

The existing projects generally integrate their own bitstream parsers for codecs. There are subtle reasons why that tends to be the case, instead of using more generic parsers. There are differences in error handling, for instance, or other matters of policy, the variation which could be difficult to fully offer using a generic API.

Maxime noted that VLC recently released a new parser meant to be used as a library, and that could be useful. Nicolas believes that we’d need a parser library independent of any other code base to avoid pulling in extra libraries and this parser would need to be maintained. It could be difficult to find the volunteers to do that.

Does ChromeOS have its own parser? Alexandre believes it does, but little was known beyond that.

There’s also the language problem: ffmpeg and gstreamer are written in C, the ChomeOS parser in C++, VLC is moving to Rust. What do we pick, how do we ensure interoperability?

libVA re-use

As a short-term solution, implementing a generic wrapper using the V4L2 stateless codec API to offer libVA API would enable generic applications to use the V4L2 stateless codec drivers as most applications already support libVA.

70 % of the applications use FFMPEG, which has a software codec API that is nearly identical to the V4L2 statless codec API. It would be trivial for applications to switch to V4L2 natively.

We need to explain our plans to libVA maintainers to better coordinate libVA API development in a way the V4L2 stateless codecs are taken into account.

Source code hosting

libva is hosted on freedesktop. Should we host the libva-v4l2-codec backend there, or host it on linuxtv.org? Hans would prefer linuxtv.org as it’s “closer to our kernel implementation”.

Backend support in libva

libva loads backends in order, and picks the first one that reports it can support the platform. There is also an environment variable that can specify a backend. Ezequiel enquired how to support platforms that would have multiple hardware codecs. libva doesn’t seem to support this at the moment. Nicolas reported that there’s an Intel SoC that have both an Intel graphics core and a Vega64 graphics core that both have a codec.

Hans said that a platform that expose multiple codecs will likely be used for specialized applications, and requiring those to implement codec support directly is acceptable. Our main focus should be to support the common case.

Vendor support

NVidia is following our progress and is interested in using the V4L2 stateless API. On the userspace side, vdpau is pretty much dead, they have moved to nvdec. OMX is being phasing out, in particular that is taking place for RaspberryPi now.

Tooling

bootlin has developed a debugging tool called v4l2-request-test that has been very useful to debug the codec driver without going through the full userspace stack. This is worth mentioning and integrating.

API discussions

Using buffer indices as handles to reference frames

This has been proposed by Tomasz, and Hans has serious concerns, he believes that having userspace predict what buffer indices will be used in the future is very fragile and would prefer using a separate 64-bit cookie associated with v4l2_buffers.

Using capture buffer indices as reference frame handles requires predicting the buffer index on the capture queue which the output queue frames will be decoded into. We could use the output queue buffer index instead, but that wouldn’t work with multi-slice decoding (multiple output buffers for a single capture buffer). Using a cookie set by userspace on the output side, then copied to the capture queue by the driver, solves that problem. All slices queued on the output queue for the same decoded picture will have the same cookie value (userspace will have to ensure that).

Tomasz would prefer a buffer index-based solution, to avoid keeping a cookie-index map in userspace. Due to how V4L2 works, enqueuing a new dmabuf handle on the capture side for a V4L2 buffer with a given index will effectively delete the corresponding cookie, so userspace would need to ensure it doesn’t overwrite buffers; (Tomasz: To clarify, I don’t see the significant benefit of using cookies over indices. It makes it easier for user space, because it doesn’t have to predict the CAPTURE buffers, but still is error prone because of the buffer requeuing problem. For now it would be good to see how it translates into real code, though. In the meantime I can try to find a better idea.)

sailus