Re: [media-workshop] Etherpad used on the mini summit

A few small corrections:

On 10/17/2014 06:41 PM, Mauro Carvalho Chehab wrote:

Em Fri, 17 Oct 2014 14:46:32 +0200 Ricardo Ribalda Delgado ricardo.ribalda@gmail.com escreveu:

...

https://etherpad.fr/p/media-summit-2014

Just took a snapshot with all that it is there currently at Etherpad. I also added there the email/company for Nicolas Dufresne and fixed some things related to the Friday's DVB discussions, but didn't review the rest yet.

Please check if everything is ok, and please fix what's needed, as I suspect that some texts may require some corrections/additions/changes.

Regards, Mauro

=

Media summit October 2014 Düsseldorf

Attendees

Mauro Carvalho Chehab - mchehab@osg.samsung.com - Samsung Hans Verkuil - hverkuil@xs4all.nl, , hverkuil - Cisco Laurent Pinchart - laurent.pinchart@ideasonboard.com, , pinchartl - Ideas on board Hans de Goede - Red hat Sakari Ailus - sakari.ailus@linux.intel.com, sailus@retiisi.org.uk, sailus - Intel Kamil Debski - k.debski@samsung.com - Samsung Johannes Stezenbach - js@linuxtv.org - LinuxTV Julien Beraud - Parrot (www.parrot.com) julien.beraud@parrot.com Pawel Osciak - Google Henning Garbers - PCTV Chris Kohn - Xilinx Philipp Zabel - p.zabel@pengutronix.de - Pengutronix Guennaddi Liakhovetski - g.liakhovetski@gmx.de, , lyakh - Intel Ricardo Ribalda Delgado - @ribalda ricardo@qtec.com -Qtechnolgy A/S Lucas Stach - l.stach@pengutronix.de - Pengutronix Ralph Metzler - Meltzler Brothers Mike Krufky - Samsung Friedrich Schwittay - PCTV Nicolas Dufresne - nicolas.dufresne@collabora.com - Collabora Peter Griffin - peter.griffin@linaro.org - Linaro/ST

Configuration stores - Hans Verkuil

• Need to report which settings/controls were applied
• Drivers needs to be able to report full/limited Android CameraHAL v3 support (full means full synchronization support)
• We probably need a mechanism to be able to remove (free) config stores
• Can we use APPLY(0) instead of CLOSE?
• Per buffer or per frame configuration? Note: there is no notion of frame IDs yet in V4L2.
• Would be nice to be able to disable APPLY, perhaps a flag for the core similar to HAS STORE

Android camera HAL v3 - Sakari Ailus

http://www.retiisi.org.uk/v4l2/foil/android-camera-hal-v3-v4l2.pdf

• Capture requests queued lead to returned buffers in order, best effort to make per-buffer configuration apply correctly needed
• Capture requests/returned buffers do not map directly to QBUF/DQBUF because multiple buffers in different formats may be produced from a single request. Some outputs could be implemented in software, e.g. multiple YUV outputs when hardware simply does not contain as many.
• Should statistical data be returned in a separate video buffer queue? Might be associated via sequence number, probably separate frame ID better
• 3A library input: frame metadata, depends on sensor, might be in-frame or out-of band. If possible, separate buffer queue prepared to get to it early.
• HAL v3 needs Start Of Frame and End Of Frame events
• If there is no hardware mechanism for synchronization, apply sensor settings as soon as possible after SOF and hope for the best. It is possible to verify the timing has been successful in the user space.
• Sensor settings and ISP settings applied at different points in time, as sensor settings usally take effect on the following frame
• Sensors may implement all or parts of the ISP themselves. It may not be feasible to implement hal v3 full profile for such sensors.
• Start of Exposure events not to be implemented in the kernel. HAL must calculate the event time based on other available information, such as end of frame (or start of frame event) and other frame timing parameters.
• Some metadata tags could be mapped to V4L2 (compound) controls, they are associated with a single capture request. Some metadata tags (statistics) are produced from video buffer queues. Other metadata tags originate from HAL.
• Needs a guarantee that a single capture request will produce all buffers requested by it with the request's parameters, needs per-buffer configuration, associating buffers with a certain frame.
• A working halv3 implementation for UVC (limited profile) would be nice to have
• A working halv3 implementation on top of some hardware (full profile) is needed to test assumptions
• ISP configuration may be implemented by controls or via an output video buffer queue
• V4L2: independent queues (possibly multiple video device nodes), frame synchronization needed when queueing the buffers, drivers are expected to always return those buffers, possibly with an error flag, HALv3: multiple streams are synchronized via capture requests
• Requests may contain buffers for every queue or only some queues.
• All buffers captured as part of a request must originate from the same sensor frame. Other behaviour is not allowed.
• All buffers have start of exposure timestamps (approximated by start of frame timestamp - exposure time)
• Frames from different capture queues can be associated using sequence numbers.
• Further discussion
  • How are buffers and controls from different device nodes associated as a single request by the driver?
  • Applying parameters from sub-device nodes

DVB API improvements - Mauro Carvalho Chehab

• New ioctl needed for PLP (physical layer pipe) readback from frontend, or get_property (might be too big for that)
• How to support tuners with many demodulators?
• To add mmap(), DMABUF support: use videobuf2 for buffer delivery - how to avoid V4L2 dependencies? Reuse memops but not videobuf2-core, reimplement queue/dequeue, state handling, see what should be shared.
• TBD: How to improve integration of DVB frontends, demuxes and CAM drivers

Work plan:

     - Make a more detailed list of requirements for DVB - to be done by the DVB developers;

     - Send patches for PLP table retrival by userspace - Hauppauge/PCTV;

     - Send a proposal for DVB-C2 - Ralph Meltzler;

       - After the above steps is completed, we should discuss between the DVB developers about who will be doing what.

Media development process - Hans Verkuil

• Fixes are written for an upcoming release typically based on findings from rc1
• Fixes should get higher priority
• Fixes must be submitted to Mauro by rc3

Should be:

- Fixes must be upstreamed by rc3

• Patches that go to the fixes tree are not applied to master. This is because the fixes sent to Linus will reappear from upstream, eventually to master branch as well.
• Use 'GIT FIXES' for urgent bug fixes (Hans will send V4L2 submitting patches text (after updating) to Pawel for wiki inclusion)
• Consider github as git repo for better failover.
• Co-locating with other conferences creates schedule conflicts
• Too costly to move before/after conferences (bosses will only pay for the meeting, not for the conference)
• Making the mini-summit open for all would be an interesting experiment.
• But we shouldn't charge for the mini-summit.
• Next year: kernel summit in South Korea, so the media mini-summit will be in Korea as well.

V4L2 raw codec API - Pawel Osciak

• The current codec API is suitable for codecs which perform all the stages of encoding or decoding.
• Newer codec hardware implementations expect the CPU to perform less computing intensive parts of the codec.
• VA-API like model: codec split between user and kernel spaces
• Some vendors provide closed user space codec libraries. Yuck.
• Bit stream formats are vendor independent, so the parsing of the bit stream should be as well.
• Processing must be performed per-slice. This allows lower latency.
• Decoding to be implemented in libv4l. Multiple hardware devices may benefit from the stream parsing functionality this way.
• GStreamer plugins already implements stream parsing. License?

Multiple timestamps - Ricardo Ribalda

-Some Industrial Machines require metadata about the state of the machine when the image is taken -When this data is small enough (~32 bits) and it is similar to a timestamp (ex. encoder data) we might use the timecode struct, already part of vb2_buffer -Sometimes the metadata can be much bigger. An example has been shown regarding an optical grader of potatoes. In this sytem, the spinoflex (conveyor belt with fingers) has a state of around 1000bits -This information does not fit the event data structure and share the same problems as the camera statistics, therefore common solutions should be used. -The proposal of creating a new ioctl for this (GET_META) has been generally rejected. Instead two options have been propossed depending of the size: -A) Configuration stores are an option as well. A single array control could be used for this. -B)Video buffers could be used. This is very much analogous to sensor metadata after all. -Laurent pointed out that A single interface to pass statistics to the user space is preferred, independently of the size of the statistics. The common consensous seems to be that:

• If the data is transferred to the memory, video buffers are probably the right solution.
• For 1000 bits, compound controls are very likely the best option.
• Image statistics of tens or hundreds of kilobytes should use a video buffer queue.

Work plan: - Hans will make sure that the configuration store can support this usercase - We will define a generic fourcc for statistics - Libv4l will be in charge of converting the vendor specific statistics into this generic format

Friday:

Runtime reconfiguration of pipelines - Chris & Laurent

• Xilinx need to support adding any type of daughterboard to their FPGA pipeline: userspace needs to load a DT overlay dynamically for that daughterboard. Note that the Xilinx driver won't know which devices are on that daughterboard since it is the customer/thirdparty that makes it.
• Partial reconfiguration: ideally only the IP blocks that will be replaced need to be removed in the MC: this is complex.
• Can we make an initial implementation to just tear down everything and rebuild it with the new IP instead of the old? Less flexible, but much easier to do until we can change the MC on the fly.
• Option: perhaps just disable entities instead of deleting them. But it's a bit of a hack, and we should just bite the bullet and add proper support.
• Result: implement everything tear down first. Later work on how to delete entities on the fly.

Create a virtual MC device driver: what should be there? - Hans Verkuil

• sensor, at least two DMA engines giving two different formats
• Bayer support?
• different input ports (csi/parallel), cropping/scaler block
• configurable pipeline (including m2m device)
• flash, lens

Hierarchical media devices - Philipp Zabel

Highly reconfigurable hardware - Julien Beraud

• 44 sub-devices connected with an interconnect.
• As long as formats match, any sub-device could be connected to any other sub-device through a link.
• The result is 44 * 44 links at worst.
• A switch sub-device proposed as the solution to model the interconnect. The sub-devices are connected to the switch sub-devices through the hardware links that connect to the interconnect.
• The switch would be controlled through new IOCTLs S_ROUTING and G_ROUTING.

. Patches available: http://git.linuxtv.org/cgit.cgi/pinchartl/media.git/log/?h=xilinx-wip

libdvbv5 - Mauro Carvalho Chehab

• Part of libv4l v1.6.0
• Most applications still use DVBv3 API.
• Contains MPEG-TS decoders for program information.
• Compilation options to choose different parts of the library to be built. Not everyone needs everything.
• The user may want to use custom replacements for parts of the library, especially for parsing tables like EPG.
• The library still needs the MPEG-TS parsers for channel parsing. If the MPEG-TS parser isn't part of the library, the functionality would need to be available elsewhere using the same interface.
• libdvbv5 license right now GPL, agreement on license change to LGPL exists from all contributors.

DVB tuners - Mauro Carvalho Chehab

• Explained the old hybrid model and the new model used by the tuners and the odds with the current approach
• It was also explained that I2C MUX is a need, in order to help fixing the problems with suspend/resume inside the subsystem
• Some concerns arrised if the new model actually works or not for analog TV, as currently there's no tuner using it for analog
• It was agreed that the new model is the way to go, but it requires a carefully review, and changes should not be done on all tuners at once, but, instead, in steps

Other DVB issues - Mauro Carvalho Chehab

• Better information on device capabilities is needed in the user space, as, currently, there are very few bits available for DVB caps at fe_caps enum, used by FE_GET_INFO. Also, some standards can have lots of options (ISDB-T has 28), and reporting there to userspace using bits is actually not a good option.

. Ralph reminded that there are DVB-C/C2 devices that only support automatic bitrate detection, and he is even unsure if it is possible to retrieve the detected bitrate from such frontends. So, we need a way to allow "CAN_AUTO_SYMBOL_RATE". That's bad, as applications like w_scan may run several times over the same symbol rates, and may actually be detecting the same transponder several times.

• Mkruky proposed to add a new ioctl (FE_GET_PROPS_INFO - or something like that) that would return all the valid values for a given property for the current delivery system.
• Old DVBv3 APIs that weren't used anymore should be clearly documented as deprecated

Dead pixels: Ricardo Ribalda

• Position of dead pixels + classification of dead pixels (dead/lazy/others)
• Store this on a flash: common format?
• No problem with the dead pixel proposal, but it needs to be used upstream somewhere before it can be used. Ricardo's driver is unlikely to be upstreamed. Not sure if adding it to vivid is sufficent.
• Suggestion: omap3 should have it, try getting a beagleboard/bone + sensor and implementing it there.
• sensor/module metadata: needs an RFC first.

Multiple selections - Ricardo Ribalda

• Hans's patch "videodev2.h: add v4l2_ctrl_selection compound control type" implements selection controls. This replaces the selection interface, and existing drivers should be converted.
• Pad information missing from the RFC patchset.
• Multiple selections will not be available on selections interface.

Add this note:

- Hans stopped working on this, but welcomes someone else continuing with it.

Regards,

Hans