media-workshop

media-workshop@linuxtv.org

113 discussions

new uvc driver Quanta 4035 :4035
by Giuliano Lotta 13 Jan '23

13 Jan '23

hi Just to inform you that the fix proposed by Laurent for Quanta 4035 :4035, has been reported to work ALSO on Quanta 4035 :4033. See https://bugs.launchpad.net/bugs/2000947 Br Giuliano ---------- Forwarded message --------- Da: Kolbjorn Blix Dahle <2000947(a)bugs.launchpad.net> Date: ven 13 gen 2023 alle ore 16:40 Subject: [Kernel-packages] [Bug 2000947] Re: UVC Quanta 0408:4035 camera PROBLEM To: <giuliano.lotta(a)gmail.com> Just to inform you that the patched also works with ID 0408:4033 Quanta Computer, Inc. ACER HD User Facing after changing 4035 to 4033.

1 0

Edinburgh Media Summit 2018 meeting report v2
by Sakari Ailus 08 Apr '19

08 Apr '19

Hello everyone, -------8<------------- I've changed the original report according to the comments received on the list so there's nothing major new here; just a small update. Mauro: I've made the changes discussed over e-mail. Could you split off the latter part of the report into a separate article, starting with the line "Tuesday's stateless codec discussion"? Thanks. -------8<------------- Here's the report on the Media Summit held on 25th October in Edinburgh. The report is followed by the stateless codec discussion two days earlier. Note: this is bcc'd to the meeting attendees plus a few others. I didn't use cc as the list servers tend to reject messages with too many recipients in cc / to headers. Most presenters used slides some of which are already available here (expect more in the near future): <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/> The original announcement for the meeting is here: <URL:https://www.spinics.net/lists/linux-media/msg141095.html> The raw notes can be found here: <URL:http://www.retiisi.org.uk/~sailus/v4l2/notes/osseu18-media.html> Attendees --------- Brad Love Ezequiel Garcia Gustavo Padovan Hans Verkuil Helen Koike Hidenori Yamaji Ivan Kalinin Jacopo Mondi Kieran Bingham Laurent Pinchart Mauro Chebab Maxime Ripard Michael Grzeschik Michael Ira Krufky Niklas Söderlund Patrick Lai Paul Elder Peter Griffin Ralph Clark Ricardo Ribalda Sakari Ailus Sean Young Seung-Woo Kim Stefan Klug Vinod Koul CEC status - Hans Verkuil ------------------------- Hans prensented an update on CEC status. - cec-gpio error injection support - tda998x (including BeagleBoard Bone support after gpiolib changes) - ChromeOS EC CEC - In progress: omap5/dra7xx/am57xx TI (waiting for DSS redesign to land) - In progress: SECO cec driver (for UDOO x86 boards, expected for 4.21) - DisplayPort CEC-Tunneling-over-AUX for i915, nouveau, amdgpu - MegaChips 2900 chipset based adapters seems to support this protocol very well - Continuing work on CEC utilities, esp. the compliance test: it is in continuous use at Cisco. The current CEC status is available here: <URL:https://hverkuil.home.xs4all.nl/cec-status.txt> Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> rc-core status report - Sean Young ---------------------------------- In the last year all staging lirc drivers have been either removed or ported to rc-core. Decoding of the more obscure IR protocols and protocol variants can now be done with BPF, with support in both the kernel and ir-keytable (which is in v4l-utils). Generally we're in a good situation wrt IR support. There is some more ancient hardware (serial or usb-serial) that does not have support but not sure if anyone cares. kernel-doc is a little sparse and does not cover BPF IR decoding, so that needs improving. There was a discussion on enabling builds with CONFIG_RC_CORE=n. Sean suggested we could have rc_allocate_driver() return NULL and have the drivers deal with this gracefully, i.e. their probe functions should continue without IR. Mauro said there should be a per-driver config option (as is done for saa7134 for example). No conclusion was reached on this. Persistent storage of controls - Ricardo Ribalda ------------------------------------------------ Ricardo gave a presentation on a proposed solution for using the V4L2 control framework as an interface for updating control value defaults on sensor EEPROM. Sensors commonly come with device specific tuning information that's embedded in the device EEPROM. Whereas this is also very common for raw cameras on mobile devices, the discussion this time was concentrated on industrial cameras. The EEPROM contents may be written by the sensor vendor but occasionally may need to be updated by customers. Setting the control default value was suggested as the exact mechanism to do this. The proposal was to use controls as the interface to update sensor tuning information in the EEPROM. There were arguments for and against the approach: + Drivers usually get these things right: relying on an user space program to do this is an additional dependency. + Re-use of an existing interface (root priviledge check may be added). - Partial solution only: EEPROM contents may need to be updated for other reasons as well, and a "spotty" implementation for updating certain EEPROM locations seems very use case specific. - Changes required to the control framework for this --- defaults are not settable at the moment. - The need is very use case specific, and adding support for that in a generic framework does not seem to fit very well. The general consensus appears to be not to change the control framework this way, but to continue to update the EEPROM using a specific user space program. Tooling for sub-system tree maintenance - Laurent Pinchart ---------------------------------------------------------- Laurent talked about the DRM tree maintenance model. The DRM tree has switched to co-maintainer model. This has made it possible to share the burden of tree maintenance, removing bottlenecks they've had. The larger number of people having (and using) their commit rights has created the need for a more strict rules for the tree maintenance, and subsequently a tool to implement it. It's called "DIM", the DRM Inglorious Maintenance tool. This is a command line tool that works as a front-end to execute the workflow. <URL:https://01.org/linuxgraphics/gfx-docs/maintainer-tools/dim.html> In particular what's worth noting: - The conflicts are resolved by the committer, not by the tree maintainer. - DIM stores conflict resolutions (as resolved by developers) to a shared cache. - DIM makes doing common mistakes harder by using sanity checks. There are about 50 people who currently have commit rights to the DRM tree. There are no reports of commit rights having been forcibly removed as of yet. This strongly suggests that the model is workable. The use of the tool puts additional responsibilities as well as some burden to the committers. Before the patches may be pushed, they are first compiled on developer's machine. That requires time, and without special arrangements such as having a second local workspace, and that time is away from productive work. The discussion that followed was concentrated on the possibility of using a similar model for the media tree. While the suggestion was initially met by mostly favourable reception, there were concerns as well. V4L2 *was* maintained generally according to the suggested model --- albeit without the proposed tools or process that needed to be strictly followed. There was once an incident which involved merging around 9000 lines of unreviewed code in a lot of places. What followed was not pretty, and this eventually lead to loss of multiple developers. Could this happen again? The DRM tree has not suffered such incidents, and generally it understood such incident could be addressed by simply reverting such a patch and removing commit rights if necessary. (Editor note: we have reverted the media tree master state to an earlier commit many times for various reasons. Version control was based on Mercurial back then and just reverting a single patch wasn't that easy back then?) Some opined that we do not have a bottleneck in reviewing patches and getting them merged whilst others thought this was not the case. It is certainly true that a very large number of patches (around 500 in the last kernel release) went in through the media tree. It still appears that there would be more patches and more drivers to get in if the throughput was higher. It was also agreed to try to improve tooling at linuxtv.org to streamline the workflow. (Ed. note: also see testing related topics below.) Current status of testing on the media tree - Sakari ---------------------------------------------------- The common practice in media subsystem development is that developers do test their patches before submitting them. This is an unwritten rule: sometimes patches end up not being tested after making slight changes to them, or they have been tested on a different kernel version. The developer may also simply forget to test the patch. Besides this, it is not uncommon that changing the kernel configuration or switching to a different architecture will cause a compilation warning or an error. The 0-day bot will catch some of these errors before the patches are merged, but that testing does not fully cover all the possible cases. There are some common pain points in V4L2-related Kconfig options (plain V4L2, MC or MC + subdev uAPI); newly submitted drivers may in fact require one of these, but the developer may not have realised that and so this ends up not being taken into account in Kconfig. Once the review is done, and after being applied to the sub-maintainer tree, a patch is applied to Mauro's local tree and Mauro performs additional tests on it. These tests currently prevent a fair number of problems reaching a wider audience than the media developers. On the other hand, whenever an issue is found, the patch will have to be fixed by the sub-maintainer or the developer. This is hardly ideal, as the problem has existed usually for a month or two before being spotted --- by a program. These checks should be instead performed on the patch when it's submitted. Automated testing - Ezequiel Garcia ----------------------------------- Ideal Continuous Integration process consists of the following steps: 1. patch submission 2. review and approval 3. merge The core question is "what level of quality standards do we want to enforce". The maintenance process should be modelled around this question, and not the other way around. Automated testing can be a part of enforcing the quality standards. There are three steps: 1. Define the quality standard 2. Define how to quantify quality in respect to the standard 3. Define how to enforce the standards On the tooling side, an uAPI test tool exists. It's called v4l2-compliance, and new drivers are required to pass the v4l2-compliance test. It has quite a few favourable properties: - Complete in terms of the uAPI coverage - Quick and easy to run - Nice output format for humans & scripts There are some issues as well: - No codec support (stateful or stateless) - No SDR or touch support - Frequently updated (distribution shipped v4l2-compliance useless) - Only one contributor Ezequiel noted that some people think that v4l2-compliance is changing too often but Hans responded that this is a necessity. The API gets amended occasionally and the existing API gets new tests. Mauro proposed moving v4l2-compliance to the kernel source tree but Hans preferred keeping it separate. That way it's easier to develop it. To address the problem of only a single contributor, it was suggested that people implementing new APIs would need to provide the tests for v4l2-compliance as well. To achieve this, the v4l2-compliance codebase needs some cleanup to make it easier to contribute. The codebase is larger and there is no documentation. V4l2-compliance also covers MC, V4L2 and V4L2 sub-device uAPIs. DVB will require its own test tooling; it is not covered by v4l2-compliance. In order to facilitate automated testing, a virtual DVB driver would be useful as well. The task was added to the list of projects needing volunteers: <URL:https://linuxtv.org/wiki/index.php/Media_Open_Source_Projects:_Looking_…> There are some other test tools that could cover V4L2 but at the moment it seems somewhat far-fetched any of them would be used to test V4L2 in the near future: - kselftest - kunit - gst-validate - ktf (https://github.com/oracle/ktf, http://heim.ifi.uio.no/~knuto/ktf/) KernelCI is a test automation system that supports automated compile and boot testing. As a newly added feature, additional tests may be implemented. This is what Collabora has implemented, effectively the current demo system runs v4l2-compliance on virtual drivers in a virtual machines (LAVA slaves). A sample of the current test report is here: <URL:https://www.mail-archive.com/linux-media@vger.kernel.org/msg135787.html> The established way to run KernelCI tests is off the head of the branches of the stable and development kernel trees, including linux-next. This is not useful as such to support automated testing of patches for the media tree: the patches need to be tested before they are merged, not after merging. In the discusion that followed among a slightly smaller group of people, it was suggested that tests could be run from select developer kernel trees, from any branch. If a developer needs long-term storage, (s)he could have another tree which would not be subject automated test builds. Alternatively, the branch name could be used as a basis for triggering an automated build, but this could end up being too restrictive. Merging the next rc1 by the maintainer would be no special case: the branch would be tested in similar way than the developer branches containing patches, and tests should need to pass before pushing the content to the media tree master branch. Ezequiel wished that people would reply to his e-mail to express their wishes on the testing needs (see sample report above). Stateless codecs - Hans Verkuil ------------------------------- Support for stateless codecs will be merged for v4.20 with an Allwinner staging codec driver. The earlier stateless codec discussion ended up concluding that the bitstream parsing is application specific, so there will be no need for a generic implementation that was previously foreseen. The question that remains is: should there be a simple parser for compliance testing? All main applications support libva which was developed as the codec API to be used with Intel GPUs. A libVA frontend was written to support the Cedrus stateless V4L2 decoder driver. It remains to be seen whether the same implementation could be used as such for the other stateless codec drivers or whether changes, or in the worst case a parallel implementation, would be needed. Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> New versions of the old IOCTLs - Hans Verkuil --------------------------------------------- V4L2 is an old API with shifting focus in terms of functionality and hardware supported. While there has been lots of changes to the two during the existence of V4L2, some of the API is unchanged since the old times. While the API is usable for the purpose, it is needlessly clunky: it is often not obvious how an IOCTL is related to the task at hand (such as using S_PARM to set the frame interval) or the API does not use year 2038-safe timestamps (struct v4l2_buffer). These APIs deserve to be updated. * VIDIOC_*_PARM In the case of VIDIOC_G_PARM and VIDIOC_S_PARM, the IOCTLs are only used to set and get the frame interval. In this case, what can be done, is to add a new IOCTL definition, with the same IOCTL number and with binary-equivalent IOCTL argument struct that only contains the field for the frame rate itself. This is binary-compatible with the existing code and no compatibility code will be needed. The new IOCTLs will be called VIDIOC_G_FRAME_INTERVAL and VIDIOC_S_FRAME_INTERVAL. * VIDIOC_ENUM_FRAME_INTERVALS Besides discrete set of supported frame intervals, VIDIOC_ENUM_FRAME_INTERVALS has stepwise frame interval as well. Stepwise could be removed as the Qualcomm venus codec and uvc (100 ns units) are the only users. Additionally, the buffer type should be added to struct v4l2_frmivalenum. There was also a discussion related to enumerating frame intervals in units of ns vs. fractional seconds. The reasoning using a fraction is that this way the frame interval on many standards can be conveyed precisely. Somebody recalled "flick", that is is the common denominator of the frame rates on all TV standards. Drivers could simply move to use the flick as the denominator, to make frame interval reporting uniform across the drivers. * struct v4l2_buffer struct v4l2_buffer is an age-old struct. There are a few issues in it: - The timestamp is not 2038-safe. - The multi-plane implementation is a mess. - Differing implementation for the end single-plane and multi-plane APIs is confusing for both applications and drivers. The proposal is to create a new v4l2_buffer struct. The differences to the old one would be: - __u64 timestamps. These are 2038-safe. The timestamp source is maintained, i.e. the type remains CLOCK_MONOTONIC apart from certain drivers (e.g. UVC) that lets the user choose the timestamp. - Put the planes right to struct v4l2_buffer. The plane struct would also be changed; the new plane struct would be called v4l2_ext_plane. - While at it, the plane description can be improved: - The start of data from the beginning of the plane memory. - Add width and height to the buffer? This would make image size changes easier for the codec. (Ed. note: pixel format as well. But this approach could only partially support what the request API is for.) - Unify single- and multi-planar APIs. The new struct could be called v4l2_ext_buffer. As the new IOCTL argument struct will have different syntax as well as semantics, it deserves to be named differently. Compatibility code will be needed to convert the users of the old IOCTLs to the new struct used internally by the kernel and drivers, and then back to the user. * struct v4l2_create_buffers Of the format, only the pix.fmt.sizeimage field is effectively used by the drivers supporting VIDIOC_CREATE_BUFS. This could be simplified, by just providing the desired buffer size instead of the entire v4l2_format struct. The user would be instructed to use TRY_FMT to obtain that buffer size. The need to delete buffers seems to have eventually surfaced. That was expected, but it wasn't known when this would happen. As the buffer index range would become non-contiguous, it should be possible to create buffers one by one only, as otherwise the indices of the additional buffers would no longer be communicated to the user unambiguously. So there would be new IOCTLs: - VIDIOC_CREATE_BUF - Create a single buffer of given size (plus other non-format related aspects) - VIDIOC_DELETE_BUF - Delete a single buffer - VIDIOC_DELETE_ALL_BUFS - Delete all buffers The naming still requires some work. The opposite of create is "destroy", not "delete". * struct v4l2_pix_format vs. struct v4l2_pix_format_mplane Working with the two structs depending on whether the format is multi-planar or not is painful. While we're doing changes in the area, the two could be unified as well. (Editor note: this could be still orthogonal to the buffers, so it could be done separately as well. We'll see.) Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> Fault tolerant V4L2 - Kieran Bingham ------------------------------------ Kieran presented a system where the media hardware complex consisted of eight more or less independent camera sensors that naturally end up being within a single media device. The current implementation, as well as the API, necessitates that all devices in a media device probe successfully before the entire media device is exposed to the user. Otherwise the user would see with a partial view of the device, without the knowledge it is such. To address the problem, additional information need to be provided to the user space. In particular: - Events on the media device to tell the graph has changed. - Graph version number is incremented at graph change (already implemented). - The property API could be applicable --- placeholders for entities that have not yet appeared? - Alternative: known entities that have failed to probe created in the media graph and marked "disable" or "failed". - Query the state of media graph completeness. That way, even when the devices in a media controller device appear one by one, the user space will be able to have all the necessary information on the registration state of the device. Complex cameras - Mauro Chehab ------------------------------ Some new laptops integrate a raw Bayer camera + ISP instead of a USB webcam. This is expected to increase, as the solution is generally cheaper and results in better quality images --- as long as all the pieces of the puzzle are in place, including the proprietary 3A library. Still, such devices need to be supported. (Ed. note: there were two talks related to this topic given in the ELc-E.) <URL:https://www.youtube.com/watch?v=KpaNNJr92CY&index=31&list=PLbzoR-pLrL6q…> <URL:https://www.youtube.com/watch?v=GIhV7tiUji0&index=60&list=PLbzoR-pLrL6q…> Development process - All ------------------------- Topic-wise this is continuation of the "Tooling for sub-system tree maintenance", "Current status of testing on the media tree" and "Automated testing" topics above. The question here is whether there's something that could be improved in the media development process and if so, how could that be done. What came up was a suggestion to have multi-committer tree in a similar manner as the DRM developers do. This was seen to be more interesting for developers than simply being asked to review patches. It certainly does raise the need for more precise rules for what may be committed to the multi-committer tree, when etc. It was also requested that experienced driver maintainers would send pull requests on patches to their drivers instead of going through a sub-maintainer (pre-agreed with the relevant (sub)maintainer). This would take some work away from sub-maintainers, but not the maintainer. No firm decisions were reached in this topic. Perhaps this could be tried out? There was also a request to document the sub-maintainer names in the wiki so that it'd be easier for people to figure out who to ping if their patches do not get merged. linuxtv.org hosting - All ------------------------- Mauro noted that linuxtv.org is currently hosted in a virtual machine somewhere in a German university. The administrator of the virtual machine has not been involved with Video4Linux for some time but has been kind to provide us the hosting over the years. It has been recognised that there is a need to find a new hosting location for the virtual machine. There is also a question of the domain name linuxtv.org. Discussion followed. What could be agreed on rather immediately was that the domain name should be owned by "us". "Us" is not a legal entity at the moment, and a practical arrangement to achieve that could be to find a new association to own the domain name. Mauro will discuss with LF to find out what they can offer. Concerns were expressed over other organisations providing us with hosting we are not in charge of ourselves. Other options related to domain ownership and hosting will be researched as well. No decisions were reached on the topic. Tuesday's stateless codec discussion ------------------------------------ Hans presented a summary of this in his stateless codec status presentation, here are a bit more details. We had a discussion (first in the Microsoft sponsor suite, then at the bar) on how to support user space for the stateless codecs better. The expected outcome of that would be a rough understanding how a stateless codec user space library would look like. The raw notes are available here: <URL:http://www.retiisi.org.uk/~sailus/v4l2/notes/osseu18-codecs.html> * Attendees Alexandre Courbot Chris Healy Ezequiel Garcia Hans Verkuil Kieran Bingham Laurent Pinchart Maxime Ripard Mauro Carvalho Chehab Nicolas Dufresne Niklas Söderlund Philip Zabell Sakari Ailus Tomasz Figa Victor Jáquez * Buffer management Nicolas reported an issue in V4L2 buffer management. The V4L2 decouples the buffers from the format, and assumes all queued buffers (at a given point of time) have the same format. (Ed. note: the request API could be used to address this, but that particular features is not yet supported.) * User space library The existing projects generally integrate their own bitstream parsers for codecs. There are subtle reasons why that tends to be the case, instead of using more generic parsers. There are differences in error handling, for instance, or other matters of policy, the variation which could be difficult to fully offer using a generic API. Maxime noted that VLC recently released a new parser meant to be used as a library, and that could be useful. Nicolas believes that we'd need a parser library independent of any other code base to avoid pulling in extra libraries and this parser would need to be maintained. It could be difficult to find the volunteers to do that. Does ChromeOS have its own parser? Alexandre believes it does, but little was known beyond that. There's also the language problem: ffmpeg and gstreamer are written in C, the ChomeOS parser in C++, VLC is moving to Rust. What do we pick, how do we ensure interoperability? * libVA re-use As a short-term solution, implementing a generic wrapper using the V4L2 stateless codec API to offer libVA API would enable generic applications to use the V4L2 stateless codec drivers as most applications already support libVA. 70 % of the applications use FFMPEG, which has a software codec API that is nearly identical to the V4L2 statless codec API. It would be trivial for applications to switch to V4L2 natively. We need to explain our plans to libVA maintainers to better coordinate libVA API development in a way the V4L2 stateless codecs are taken into account. * Source code hosting libva is hosted on freedesktop. Should we host the libva-v4l2-codec backend there, or host it on linuxtv.org? Hans would prefer linuxtv.org as it's "closer to our kernel implementation". * Backend support in libva libva loads backends in order, and picks the first one that reports it can support the platform. There is also an environment variable that can specify a backend. Ezequiel enquired how to support platforms that would have multiple hardware codecs. libva doesn't seem to support this at the moment. Nicolas reported that there's an Intel SoC that have both an Intel graphics core and a Vega64 graphics core that both have a codec. Hans said that a platform that expose multiple codecs will likely be used for specialized applications, and requiring those to implement codec support directly is acceptable. Our main focus should be to support the common case. * Vendor support NVidia is following our progress and is interested in using the V4L2 stateless API. On the userspace side, vdpau is pretty much dead, they have moved to nvdec. OMX is being phasing out, in particular that is taking place for RaspberryPi now. * Tooling bootlin has developed a debugging tool called v4l2-request-test (https://github.com/bootlin/v4l2-request-test) that has been very useful to debug the codec driver without going through the full userspace stack. This is worth mentioning and integrating. * API discussions Using buffer indices as handles to reference frames This has been proposed by Tomasz, and Hans has serious concerns, he believes that having userspace predict what buffer indices will be used in the future is very fragile and would prefer using a separate 64-bit cookie associated with v4l2_buffers. Using capture buffer indices as reference frame handles requires predicting the buffer index on the capture queue which the output queue frames will be decoded into. We could use the output queue buffer index instead, but that wouldn't work with multi-slice decoding (multiple output buffers for a single capture buffer). Using a cookie set by userspace on the output side, then copied to the capture queue by the driver, solves that problem. All slices queued on the output queue for the same decoded picture will have the same cookie value (userspace will have to ensure that). Tomasz would prefer a buffer index-based solution, to avoid keeping a cookie-index map in userspace. Due to how V4L2 works, enqueuing a new dmabuf handle on the capture side for a V4L2 buffer with a given index will effectively delete the corresponding cookie, so userspace would need to ensure it doesn't overwrite buffers; (Tomasz: To clarify, I don't see the significant benefit of using cookies over indices. It makes it easier for user space, because it doesn't have to predict the CAPTURE buffers, but still is error prone because of the buffer requeuing problem. For now it would be good to see how it translates into real code, though. In the meantime I can try to find a better idea.) -- Sakari Ailus e-mail: sakari.ailus(a)iki.fi

1 0

[ANN] Edinburgh Media Summit 2018 meeting report
by Sakari Ailus 11 Mar '19

11 Mar '19

Hello everyone, Here's the report on the Media Summit held on 25th October in Edinburgh. The report is followed by the stateless codec discussion two days earlier. Note: this is bcc'd to the meeting attendees plus a few others. I didn't use cc as the list servers tend to reject messages with too many recipients in cc / to headers. Most presenters used slides some of which are already available here (expect more in the near future): <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/> The original announcement for the meeting is here: <URL:https://www.spinics.net/lists/linux-media/msg141095.html> The raw notes can be found here: <URL:http://www.retiisi.org.uk/~sailus/v4l2/notes/osseu18-media.html> Attendees --------- Brad Love Ezequiel Garcia Gustavo Padovan Hans Verkuil Helen Koike Hidenori Yamaji Ivan Kalinin Jacopo Mondi Kieran Bingham Laurent Pinchart Mauro Chebab Maxime Ripard Michael Grzeschik Michael Ira Krufky Niklas Söderlund Patrick Lai Paul Elder Peter Griffin Ralph Clark Ricardo Ribalda Sakari Ailus Sean Young Seung-Woo Kim Stefan Klug Vinod Koul CEC status - Hans Verkuil ------------------------- Hans prensented an update on CEC status. Besides the slides, noteworthy information is maintained here: <URL:https://hverkuil.home.xs4all.nl/cec-status.txt> Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> rc-core status report - Sean Young ---------------------------------- (Contributed by Sean Young) In the last year all staging lirc drivers have been either removed or ported to rc-core. Decoding of the more obscure IR protocols and protocol variants can now be done with BPF, with support in both the kernel and ir-keytable (which is in v4l-utils). Generally we're in a good situation wrt IR support. There is some more ancient hardware (serial or usb-serial) that does not have support but not sure if anyone cares. kernel-doc is a little sparse and does not cover BPF IR decoding, so that needs improving. There was a discussion on enabling builds with CONFIG_RC_CORE=n. Sean suggested we could have rc_allocate_driver() return NULL and have the drivers deal with this gracefully, i.e. their probe functions should continue without IR. Mauro said there should be a per-driver config option (as is done for saa7134 for example). No conclusion was reached on this. Persistent storage of controls - Ricardo Ribalda ------------------------------------------------ Ricardo gave a presentation on a proposed solution for using the V4L2 control framework as an interface for updating control value defaults on sensor EEPROM. Sensors commonly come with device specific tuning information that's embedded in the device EEPROM. Whereas this is also very common for raw cameras on mobile devices, the discussion this time was concentrated on industrial cameras. The EEPROM contents may be written by the sensor vendor but occasionally may need to be updated by customers. Setting the control default value was suggested as the exact mechanism to do this. The proposal was to use controls as the interface to update sensor tuning information in the EEPROM. There were arguments for and against the approach: + Drivers usually get these things right: relying on an user space program to do this is an additional dependency. + Re-use of an existing interface (root priviledge check may be added). - Partial solution only: EEPROM contents may need to be updated for other reasons as well, and a "spotty" implementation for updating certain EEPROM locations seems very use case specific. - Changes required to the control framework for this --- defaults are not settable at the moment. - The need is very use case specific, and adding support for that in a generic framework does not seem to fit very well. The general consensus appears to be not to change the control framework this way, but to continue to update the EEPROM using a specific user space program. Tooling for sub-system tree maintenance - Laurent Pinchart ---------------------------------------------------------- Laurent talked about the DRM tree maintenance model. The DRM tree has switched to co-maintainer model. This has made it possible to share the burden of tree maintenance, removing bottlenecks they've had. The larger number of people having (and using) their commit rights has created the need for a more strict rules for the tree maintenance, and subsequently a tool to implement it. It's called "DIM", the DRM Inglorious Maintenance tool. This is a command line tool that works as a front-end to execute the workflow. <URL:https://01.org/linuxgraphics/gfx-docs/maintainer-tools/dim.html> In particular what's worth noting: - The conflicts are resolved by the committer, not by the tree maintainer. - DIM stores conflict resolutions (as resolved by developers) to a shared cache. - DIM makes doing common mistakes harder by using sanity checks. There are about 50 people who currently have commit rights to the DRM tree. There are no reports of commit rights having been forcibly removed as of yet. This strongly suggests that the model is workable. The use of the tool puts additional responsibilities as well as some burden to the committers. Before the patches may be pushed, they are first compiled on developer's machine. That requires time, and without special arrangements such as having a second local workspace, and that time is away from productive work. The discussion that followed was concentrated on the possibility of using a similar model for the media tree. While the suggestion was initially met by mostly favourable reception, there were concerns as well. V4L2 *was* maintained generally according to the suggested model --- albeit without the proposed tools or process that needed to be strictly followed. There was once an incident which involved merging around 9000 lines of unreviewed code in a lot of places. What followed was not pretty, and this eventually lead to loss of multiple developers. Could this happen again? The DRM tree has not suffered such incidents, and generally it understood such incident could be addressed by simply reverting such a patch and removing commit rights if necessary. (Editor note: we have reverted the media tree master state to an earlier commit many times for various reasons. Could it be one of the reasons the 9000 line patch was not reverted was that the version control wasn't based on git??) Some opined that we do not have a bottleneck in reviewing patches and getting them merged whilst others thought this was not the case. It is certainly true that a very large number of patches (around 500 in the last kernel release) went in through the media tree. It still appears that there would be more patches and more drivers to get in if the throughput was higher. Current status of testing on the media tree - Sakari ---------------------------------------------------- The common practice in media subsystem development is that developers do test their patches before submitting them. This is an unwritten rule: sometimes patches end up not being tested after making slight changes to them, or they have been tested on a different kernel version. The developer may also simply forget to test the patch. Besides this, it is not uncommon that changing the kernel configuration or switching to a different architecture will cause a compilation warning or an error. The 0-day bot will catch some of these errors before the patches are merged, but that testing does not fully cover all the possible cases. There are some common pain points in V4L2-related Kconfig options (plain V4L2, MC or MC + subdev uAPI); newly submitted drivers may in fact require one of these, but the developer may not have realised that and so this ends up not being taken into account in Kconfig. Once the review is done, and after being applied to the sub-maintainer tree, a patch is applied to Mauro's local tree and Mauro performs additional tests on it. These tests currently prevent a fair number of problems reaching a wider audience than the media developers. On the other hand, whenever an issue is found, the patch will have to be fixed by the sub-maintainer or the developer. This is hardly ideal, as the problem has existed usually for a month or two before being spotted --- by a program. These checks should be instead performed on the patch when it's submitted. Automated testing - Ezequiel Garcia ----------------------------------- Ideal Continuous Integration process consists of the following steps: 1. patch submission 2. review and approval 3. merge The core question is "what level of quality standards do we want to enforce". The maintenance process should be modelled around this question, and not the other way around. Automated testing can be a part of enforcing the quality standards. There are three steps: 1. Define the quality standard 2. Define how to quantify quality in respect to the standard 3. Define how to enforce the standards On the tooling side, an uAPI test tool exists. It's called v4l2-compliance, and new drivers are required to pass the v4l2-compliance test. It has quite a few favourable properties: - Complete in terms of the uAPI coverage - Quick and easy to run - Nice output format for humans & scripts There are some issues as well: - No codec support (stateful or stateless) - No SDR or touch support - Frequently updated (distribution shipped v4l2-compliance useless) - Only one contributor Ezequiel noted that some people think that v4l2-compliance is changing too often but Hans responded that this is a necessity. The API gets amended occasionally and the existing API gets new tests. Mauro proposed moving v4l2-compliance to the kernel source tree but Hans preferred keeping it separate. That way it's easier to develop it. To address the problem of only a single contributor, it was suggested that people implementing new APIs would need to provide the tests for v4l2-compliance as well. To achieve this, the v4l2-compliance codebase needs some cleanup to make it easier to contribute. The codebase is larger and there is no documentation. V4l2-compliance also covers MC, V4L2 and V4L2 sub-device uAPIs. DVB will require its own test tooling; it is not covered by v4l2-compliance. In order to facilitate automated testing, a virtual DVB driver would be useful as well. The task was added to the list of projects needing volunteers: <URL:https://linuxtv.org/wiki/index.php/Media_Open_Source_Projects:_Looking_…> There are some other test tools that could cover V4L2 but at the moment it seems somewhat far-fetched any of them would be used to test V4L2 in the near future: - kselftest - kunit - gst-validate - ktf (https://github.com/oracle/ktf, http://heim.ifi.uio.no/~knuto/ktf/) KernelCI is a test automation system that supports automated compile and boot testing. As a newly added feature, additional tests may be implemented. This is what Collabora has implemented, effectively the current demo system runs v4l2-compliance on virtual drivers in a virtual machines (LAVA slaves). A sample of the current test report is here: <URL:https://www.mail-archive.com/linux-media@vger.kernel.org/msg135787.html> The established way to run KernelCI tests is off the head of the branches of the stable and development kernel trees, including linux-next. This is not useful as such to support automated testing of patches for the media tree: the patches need to be tested before they are merged, not after merging. In the discusion that followed among a slightly smaller group of people, it was suggested that tests could be run from select developer kernel trees, from any branch. If a developer needs long-term storage, (s)he could have another tree which would not be subject automated test builds. Alternatively, the branch name could be used as a basis for triggering an automated build, but this could end up being too restrictive. Merging the next rc1 by the maintainer would be no special case: the branch would be tested in similar way than the developer branches containing patches, and tests should need to pass before pushing the content to the media tree master branch. Ezequiel wished that people would reply to his e-mail to express their wishes on the testing needs (see sample report above). Stateless codecs - Hans Verkuil ------------------------------- Support for stateless codecs will be merged for v4.20 with an Allwinner staging codec driver. The earlier stateless codec discussion ended up concluding that the bitstream parsing is application specific, so there will be no need for a generic implementation that was previously foreseen. The question that remains is: should there be a simple parser for compliance testing? All main applications support libva which was developed as the codec API to be used with Intel GPUs. A libVA frontend was written to support the Cedrus stateless V4L2 decoder driver. It remains to be seen whether the same implementation could be used as such for the other stateless codec drivers or whether changes, or in the worst case a parallel implementation, would be needed. Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> New versions of the old IOCTLs - Hans Verkuil --------------------------------------------- V4L2 is an old API with shifting focus in terms of functionality and hardware supported. While there has been lots of changes to the two during the existence of V4L2, some of the API is unchanged since the old times. While the API is usable for the purpose, it is needlessly clunky: it is often not obvious how an IOCTL is related to the task at hand (such as using S_PARM to set the frame interval) or the API does not use year 2038-safe timestamps (struct v4l2_buffer). These APIs deserve to be updated. * VIDIOC_*_PARM In the case of VIDIOC_G_PARM and VIDIOC_S_PARM, the IOCTLs are only used to set and get the frame interval. In this case, what can be done, is to add a new IOCTL definition, with the same IOCTL number and with binary-equivalent IOCTL argument struct that only contains the field for the frame rate itself. This is binary-compatible with the existing code and no compatibility code will be needed. The new IOCTLs will be called VIDIOC_G_FRAME_INTERVAL and VIDIOC_S_FRAME_INTERVAL. * VIDIOC_ENUM_FRAME_INTERVALS Besides discrete set of supported frame intervals, VIDIOC_ENUM_FRAME_INTERVALS has stepwise frame interval as well. Stepwise could be removed as the Qualcomm venus codec and uvc (100 ns units) are the only users. Additionally, the buffer type should be added to struct v4l2_frmivalenum. There was also a discussion related to enumerating frame intervals in units of ns vs. fractional seconds. The reasoning using a fraction is that this way the frame interval on many standards can be conveyed precisely. Somebody recalled "flick", that is is the common denominator of the frame rates on all TV standards. Drivers could simply move to use the flick as the denominator, to make frame interval reporting uniform across the drivers. * struct v4l2_buffer struct v4l2_buffer is an age-old struct. There are a few issues in it: - The timestamp is not 2038-safe. - The multi-plane implementation is a mess. - Differing implementation for the end single-plane and multi-plane APIs is confusing for both applications and drivers. The proposal is to create a new v4l2_buffer struct. The differences to the old one would be: - __u64 timestamps. These are 2038-safe. The timestamp source is maintained, i.e. the type remains CLOCK_MONOTONIC apart from certain drivers (e.g. UVC) that lets the user choose the timestamp. - Put the planes right to struct v4l2_buffer. The plane struct would also be changed; the new plane struct would be called v4l2_ext_plane. - While at it, the plane description can be improved: - The start of data from the beginning of the plane memory. - Add width and height to the buffer? This would make image size changes easier for the codec. (Ed. note: pixel format as well. But this approach could only partially support what the request API is for.) - Unify single- and multi-planar APIs. The new struct could be called v4l2_ext_buffer. As the new IOCTL argument struct will have has different syntax as well as semantics, it deserves to be named differently. Compatibility code will be needed to convert the users of the old IOCTLs to the new struct used internally by the kernel and drivers, and then back to the user. * struct v4l2_create_buffers Of the format, only the pix.fmt.sizeimage field is effectively used by the drivers supporting VIDIOC_CREATE_BUFS. This could be simplified, by just providing the desired buffer size instead of the entire v4l2_format struct. The user would be instructed to use TRY_FMT to obtain that buffer size. The need to delete buffers seems to have eventually surfaced. That was expected, but it wasn't known when this would happen. As the buffer index range would become non-contiguous, it should be possible to create buffers one by one only, as otherwise the indices of the additional buffers would no longer be communicated to the user unambiguously. So there would be new IOCTLs: - VIDIOC_CREATE_BUF - Create a single buffer of given size (plus other non-format related aspects) - VIDIOC_DELETE_BUF - Delete a single buffer - VIDIOC_DELETE_ALL_BUFS - Delete all buffers The naming still requires some work. The opposite of create is "destroy", not "delete". * struct v4l2_pix_format vs. struct v4l2_pix_format_mplane Working with the two structs depending on whether the format is multi-planar or not is painful. While we're doing changes in the area, the two could be unified as well. (Editor note: this could be still orthogonal to the buffers, so it could be done separately as well. We'll see.) Slides: <URL:https://www.linuxtv.org/downloads/presentations/media_summit_2018/media…> Fault tolerant V4L2 - Kieran Bingham ------------------------------------ Kieran presented a system where the media hardware complex consisted of eight more or less independent camera sensors that naturally end up being within a single media device. The current implementation, as well as the API, necessitates that all devices in a media device probe successfully before the entire media device is exposed to the user. Otherwise the user would see with a partial view of the device, without the knowledge it is such. To address the problem, additional information need to be provided to the user space. In particular: - Events on the media device to tell the graph has changed. - Graph version number is incremented at graph change (already implemented). - The property API could be applicable --- placeholders for entities that have not yet appeared? - Alternative: known entities that have failed to probe created in the media graph and marked "disable" or "failed". - Query the state of media graph completeness. That way, even when the devices in a media controller device appear one by one, the user space will be able to have all the necessary information on the registration state of the device. Complex cameras - Mauro Chehab ------------------------------ Some new laptops integrate a raw Bayer camera + ISP instead of a USB webcam. This is expected to increase, as the solution is generally cheaper and results in better quality images --- as long as all the pieces of the puzzle are in place, including the proprietary 3A library. Still, such devices need to be supported. (Ed. note: there were two talks related to this topic given in the ELc-E.) <URL:https://www.youtube.com/watch?v=KpaNNJr92CY&index=31&list=PLbzoR-pLrL6q…> <URL:https://www.youtube.com/watch?v=GIhV7tiUji0&index=60&list=PLbzoR-pLrL6q…> Development process - All ------------------------- Topic-wise this is continuation of the "Tooling for sub-system tree maintenance", "Current status of testing on the media tree" and "Automated testing" topics above. The question here is whether there's something that could be improved in the media development process and if so, how could that be done. What came up was a suggestion to have multi-committer tree in a similar manner as the DRM developers do. This was seen to be more interesting for developers than simply being asked to review patches. It certainly does raise the need for more precise rules for what may be committed to the multi-committer tree, when etc. It was also requested that experienced driver maintainers would send pull requests on patches to their drivers instead of going through a sub-maintainer (pre-agreed with the relevant (sub)maintainer). This would take some work away from sub-maintainers, but not the maintainer. No firm decisions were reached in this topic. Perhaps this could be tried out? There was also a request to document the sub-maintainer names in the wiki so that it'd be easier for people to figure out who to ping if their patches do not get merged. linuxtv.org hosting - All ------------------------- Mauro noted that linuxtv.org is currently hosted in a virtual machine somewhere in a German university. The administrator of the virtual machine has not been involved with Video4Linux for some time but has been kind to provide us the hosting over the years. It has been recognised that there is a need to find a new hosting location for the virtual machine. There is also a question of the domain name linuxtv.org. Discussion followed. What could be agreed on rather immediately was that the domain name should be owned by "us". "Us" is not a legal entity at the moment, and a practical arrangement to achieve that could be to find a new association to own the domain name. The hosting of the virtual machine could possibly be handled by the same association. In practice this would likely mean a virtual machine on a hosting provider. Ideally this would be paid for by a company or a group of companies. No decisions were reached on the topic. Tuesday's stateless codec discussion ------------------------------------ Hans presented a summary of this in his stateless codec status presentation, here are a bit more details. We had a discussion (first in the Microsoft sponsor suite, then at the bar) on how to support user space for the stateless codecs better. The expected outcome of that would be a rough understanding how a stateless codec user space library would look like. The raw notes are available here: <URL:http://www.retiisi.org.uk/~sailus/v4l2/notes/osseu18-codecs.html> * Attendees Alexandre Courbot Chris Healy Ezequiel Garcia Hans Verkuil Kieran Bingham Laurent Pinchart Maxime Ripard Mauro Carvalho Chehab Nicolas Dufresne Niklas Söderlund Philip Zabell Sakari Ailus Tomasz Figa Victor Jáquez * Buffer management Nicolas reported an issue in V4L2 buffer management. The V4L2 decouples the buffers from the format, and assumes all queued buffers (at a given point of time) have the same format. (Ed. note: the request API could be used to address this, but that particular features is not yet supported.) * User space library The existing projects generally integrate their own bitstream parsers for codecs. There are subtle reasons why that tends to be the case, instead of using more generic parsers. There are differences in error handling, for instance, or other matters of policy, the variation which could be difficult to fully offer using a generic API. Maxime noted that VLC recently released a new parser meant to be used as a library, and that could be useful. Nicolas believes that we'd need a parser library independent of any other code base to avoid pulling in extra libraries and this parser would need to be maintained. It could be difficult to find the volunteers to do that. Does ChromeOS have its own parser? Alexandre believes it does, but little was known beyond that. There's also the language problem: ffmpeg and gstreamer are written in C, the ChomeOS parser in C++, VLC is moving to Rust. What do we pick, how do we ensure interoperability? * libVA re-use As a short-term solution, implementing a generic wrapper using the V4L2 stateless codec API to offer libVA API would enable generic applications to use the V4L2 stateless codec drivers as most applications already support libVA. 70 % of the applications use FFMPEG, which has a software codec API that is nearly identical to the V4L2 statless codec API. It would be trivial for applications to switch to V4L2 natively. Mauro would like us to explain our plans to Intel to avoid surprises later on. * Source code hosting libva is hosted on freedesktop. Should we host the libva-v4l2-codec backend there, or host it on linuxtv.org? Hans would prefer linuxtv.org as it's "closer to our kernel implementation". * Backend support in libva libva loads backends in order, and picks the first one that reports it can support the platform. There is also an environment variable that can specify a backend. Ezequiel enquired how to support platforms that would have multiple hardware codecs. libva doesn't seem to support this at the moment. Nicolas reported that there's an Intel SoC that have both an Intel graphics core and a Vega64 graphics core that both have a codec. Hans said that a platform that expose multiple codecs will likely be used for specialized applications, and requiring those to implement codec support directly is acceptable. Our main focus should be to support the common case. * Vendor support NVidia is following our progress and is interested in using the V4L2 stateless API. On the userspace side, vdpau is pretty much dead, they have moved to nvdec. OMX is being phasing out, in particular that is taking place for RaspberryPi now. * Tooling bootlin has developed a debugging tool called v4l2-request-test (https://github.com/bootlin/v4l2-request-test) that has been very useful to debug the codec driver without going through the full userspace stack. This is worth mentioning and integrating. * API discussions Using buffer indices as handles to reference frames This has been proposed by Tomasz, and Hans has serious concerns, he believes that having userspace predict what buffer indices will be used in the future is very fragile and would prefer using a separate 64-bit cookie associated with v4l2_buffers. Using capture buffer indices as reference frame handles requires predicting the buffer index on the capture queue which the output queue frames will be decoded into. We could use the output queue buffer index instead, but that wouldn't work with multi-slice decoding (multiple output buffers for a single capture buffer). Using a cookie set by userspace on the output side, then copied to the capture queue by the driver, solves that problem. All slices queued on the output queue for the same decoded picture will have the same cookie value (userspace will have to ensure that). Tomasz would prefer a buffer index-based solution, to avoid keeping a cookie-index map in userspace. Due to how V4L2 works, enqueuing a new dmabuf handle on the capture side for a V4L2 buffer with a given index will effectively delete the corresponding cookie, so userspace would need to ensure it doesn't overwrite buffers; (Tomasz: To clarify, I don't see the significant benefit of using cookies over indices. It makes it easier for user space, because it doesn't have to predict the CAPTURE buffers, but still is error prone because of the buffer requeuing problem. For now it would be good to see how it translates into real code, though. In the meantime I can try to find a better idea.) -- Sakari Ailus e-mail: sakari.ailus(a)iki.fi

6 8

[ANN] Prague Media summit report October 2017
by Sakari Ailus 04 Jun '18

04 Jun '18

Hello everyone, Here's the Prague Media summit report from 26th and 27th October 2017. It took long but finally it's here! If you feel something important is missing or incorrect, feel free to reply. First day ********* The first day was mostly concentrated on discussing the media development process. No notes were done at the time of the meeting, so any report here is based on my own recollection may be unrepresentative of the entire meeting. Attendees Sakari Ailus - Mauro Carvalho Chehab Shuah Khan Mike Krufky Gustavo Padovan Laurent Pinchart Niklas Söderlund Hans Verkuil Media development statistics ============================ Mauro gave a presentation on media tree development statistics he's been collecting. The slides can be found here: <URL:https://linuxtv.org/downloads/presentations/media_summit_2017/media_sum…> Media device sharing ==================== Media device sharing was discussed during the first day as well. The use case is related to a USB device that in Linux has several interfaces (?) which in Linux are devices. The purpose of this would be to show entities backed by different devices from the same USB device, without either of the drivers being necessarily aware of each other. The USB device is also hot unpluggable, meaning that the devices backing the media entities in the media graph may disappear, and do so in any order as they could appear. Media device sharing requires improvements to the Media controller framework. Especially: - The Media controller framework assumes the media device is backed by a single struct device. There's also an ops struct, also related to a single driver. There may no longer be 1:1 relation between these two and the media device. - Object lifetime management. Object lifetime management has been found to be not designed well enough in Media controller in general. The original use case was non-hotpluggable devices and the early efforts address the problem were focussed in preventing unbinding of drivers from devices in various ways. This need to be properly addressed, with refcounts, so that the necessary memory objects for various operations will stay where needed as long as needed. (I'm not going to details here. Work has been done to this end but it is not complete yet.) Second day ********** The report on the second day is based on the notes written during the meeting. The raw notes can be found here: <URL:http://www.retiisi.org.uk/v4l2/notes/media-summit-2017.html> Attendees Jacopo Mondi Mauro Chehab Niklas Söderlund Benoit Parrot Pavel Machek Ricardo Ribalda Dimitrios Katasaros Laurent Pinchart Hans Verkuil Gustavo Padovan Alexandre Courbot Sakari Ailus Mike Krufky Brad Love Tomas Dratva CEC status ========== Hans gave a presentation on CEC status. The slides are available here: <URL:https://linuxtv.org/downloads/presentations/media_summit_2017/cec-statu…> Latest CEC status is always available here: <URL:https://hverkuil.home.xs4all.nl/cec-status.txt> Side note: should we use the new serial bus (serdev) for the RC serial driver? Explicit synchronization in V4L2 ================================ Gustavo gave a presentation on explicit synchronisation in V4L2. It's available here: <URL:https://linuxtv.org/downloads/presentations/media_summit_2017/2017-Medi…> In-fences are easy, there is no large issue in the RFC. Out-fences, however, are subject to buffer reordering issues. A new event has been created to report buffer order to userspace. Fences are created at QBUF time, but can't be used by userspace until an event reports which buffer will be available next. Not all drivers perform reordering. As an optimization a capability flag could tell userspace that buffers are always ordered and that events are not needed. The capability flag should likely happen at/after setting format, as out-of-order depends on the video format Reordering is mostly useful for codecs, and can also occur from buffer recycling internally in drivers when capture errors happen. While recycling is allowed by V4L2, we could forbid it when fences are used. In that case all buffers would be returned to userspace in order, and errors be reported with the error flag set. We may need to make capture queues not reorder buffers as well, so basically if we are using in-fences we don't let the bufffers reorder. So when there is no reorder we can return the buffer's fence at the QBUF time and not need the OUT_FENCE event for these cases. If the user doesn't pickup the outstanding OUT_FENCE event before DQBUF just set the fence to -1 at DQBUF time. Discussion if timestamps are enough for audio/video synchronization... and what kind of timestamps to use. Wall clock timestamps are not usable for synchronisation, monotonic is the way to go. Converting device timestmaps to monotonic in the kernel can be troublesome. Time stamping with fences --- time stamps related to buffers are only available at dqbuf time (unless SoF events are being used, and they're supported by only some hardware). This is too late in some use cases, if the timestamp is used to determine when the frame should be displayed. Capture to networking (a.k.a. partial fences) - We need to move from a frame-based API to a slice-based API. The problem isn't limited to fences, the whole V4L2 API is based on frames. Latency could be lowered in this way, but it isn't clear whether frame-based handling is the main source of latency in any use case. To support very low latency (mostly for professional video use cases) we would need to redesign all buffer handling in V4L2. We'll postpone this for now until someone comes with a convincing use case and enough resources to implement it. Media Subsystem Maintainance ============================ Some years ago V4L2 sub-maintainers were introduced to address recognised bottlenecks (there's only one Mauro, for instance). Today we have more contributors but the number of V4L2 and Media core developers who actively review patches submitted to the list remains small. The current model appears to have its limits as well, which is also why we're looking how others are doing their work. Gustavo explained how the DRM subsystem is maintained. The DRM subsystem consists of the core of the subsystem and individual drivers, a large part of which are maintained in the drm-misc tree. The drm-misc tree is also the target for small core changes and DRM-wide changes. The drm-misc tree has a number of committers (37 at the time of the presentation was given) who also review each others patches. The model switch was done as there were many contributors and bottlenecks due to that. Each drm-misc committer has the ability to push patches to the same, common branch, naturally requiring the patches to be reviewed on the mailing list. Once a consensus is eached among the drm-misc developers reviewing the patch, a patch may be merged. A pull request is sent, by one or the drm-misc maintainers (of which there are three), to David Arlie from the common branch weekly. Dave merges this pull request to his tree. Committers of the drm-misc tree take great care in ensuring the quality of their code base. This has lead to improved developer skills among the existing developers due to more challenging and responsible work. This, in turn, makes it harder for patches to fall through the cracks as there are many reviewers and committers capable of doing the required work to get the code merged. Such model has attracted more contributors and solved process bottlenecks. There is a policy that would lead on removing committer rights if abused. However, there wasn't any incident so far that justified the removal of committer right. Bad patches getting merged will be reverted, or hopefully, not getting merged in the first place. Could media subsystem switch to this model? ------------------------------------------- The DRM guys have a set of tools that support their work flow. These tools are nor DRM specific as such but they currently do contain DRM specific configuration that would need to be moved to a configuration file before the tools were usable for media tree development. Right now, the media subsystem is rather complex, containing multiple different kernel APIs and frameworks (e.g. V4L2, DVB, Media controller) and a large number of drivers using them. Mauro, the media tree maintainer, does have checks in place to catch patches that could potentially cause havoc if merged to Linus's tree in form of breaking the user space API. If the media tree would be maintained by a group of committers instead, similar checks would have to be in place in order to avoid breakages that are visible outside the subsystem. Developers often test the patches and may perform compile tests (e.g. through 01.org's kbuild test robot) but they may be missed just after the final, trivial changes that sometimes end up breaking things. For instance, before pushing new patches to the master branch, the following checks should be automatically performed so that developers don't end up pushing stuff that's not good to go there: - Perform checkpatch.pl checks, ensures the branch has the correct baseline and does check for many errors - Compile the new patches. - Destructive push to the master branch need to be prevented. - Question to Mauro: what kind of checks are you actually performing on the patches you apply or pull requests you pick the patches from? Could Jenkins or 0day be helpful here? Jenkins would require more infra on our side whereas 0day already exists. 0day requires explicit instructions to check a kernel git tree periodically, this is managed by Fengguang Wu (fengguang.wu at intel.com). Security -------- Having more committers places more strict requirements on developer machine security. This is because many more people have push access to the master branch, essentially there are many more points where development tree integrity may be compromised. (Note: nothing will tell you this has happened; push done by an attacker on a developer's machine looks exactly like a developer would have done that.) Konstantin Ryabitsev gave an excellent presentation on securing developer workstations (outside Media summit); I couldn't find a video (I guess there was no camera) but the slides are online: <URL:https://mricon.com/talks/osseu17.pdf> Patchwork --------- - No information in patchwork e-mails on who acted on a patch. This would be important to add. - patchwork.linuxtv.org is using 1.x version of Patchwork. Switch to 2.x should be done, but there are database incompatibilities between the two. Is there a migration script? - The alternative is to use kernel.org patchwork. This has been troublesome in the past and it is not known it'd be less so now. (Why so escaped me.) Media subsystem documentation ============================= Mauro made a presentation on the state of the documentation of the media subsystem. It is available here: <URL:https://linuxtv.org/downloads/presentations/media_summit_2017/media_sum…> In the following discussion it was pointed out that videobuf2 has been mostly undocumented but this hasn't been a problem in practice: the API is sensible, and the drivers using videobuf2 are mostly doing the right thing. This does not excuse the lack of documentation however. There's a push to split kAPI and uAPI documentation in the kernel. V4L2 now has them combined in a single document. The reason for the push is that the audiences to the two APIs are largely different. Related to the above Mauro gave this presentation in the KS documentation session: <URL:https://linuxtv.org/downloads/presentations/media_summit_2017/ks_docume…> While the V4L2 reference documentation is in a very good shape, how-to-type documentation is not available for application developers who have little idea of how to use the API. The reference documentation does not help here. Where should KernelDoc function documentation go to? ---------------------------------------------------- It was discussed whether KernelDoc documentation should be placed in headers vs. in .c files. The points raised were (the list may be inconclusive): - Keeping the documentation in headers may result in changing the function but missing changing the documentation. This could happen as the developer misses the documentation in header file, which is entirely plausible if the function prototype is not changed. - Some prefer to have the documentation with the interface definition, on the grounds it is easier to browse it that way. Tools such as C-scope do find definitions of functions based on the function definition itself, not the prototype in the header. - Some saw extensive documentation fitting better to the header. The .c files could grow significantly due to documentation, making it more difficult to work with the code in them. It was decided to keep KernelDoc function documentation in the header files. SoC camera removal ------------------ There are not many drivers left using the SoC camera framework. sh-ceu is being converted by Jacopo to a proper V4L2 driver. This work is ongoing. V4L2 clock used by SoC camera, pxa_camera and two i2c sensor drivers only. One option for the two sensor drivers could be to just remove them; they can be brought back if needed. Videobuf1 removal ----------------- atomisp staging driver and more than 10 non-staging drivers still use videobuf1. One of the most notable ones are bttv. Work would be needed to convert these drivers to videobuf2. Request API =========== Alexandre Courbot gave a presentation on the V4L2 jobs API, which is effectively intended to address the same problem area than the Media request API. There is interest in getting request API to upstream. Several classes of hardware need it, including cameras supporting Android camera HAL v3 and stateless video codecs, the two main use cases at hand at the moment. The request API involves, in principle, all API elements and aspects of the V4L2 and Media controller APIs. This requires forward-looking approach in the implementation which will unlikely to be a complete one in the beginning. The Request API approach uses the media controller to create request objects. Note that this does not work if there are multiple media controllers involved that need to coordinate. As a consequence, devices managed by different drivers cannot be part of the same request. The MC approach requires that stateless codec drivers implement a media graph which is just a video node with two pads and a codec subdev with two pads. An old version of the Request API is in use by ChromeOS for several HW codecs for several years now. Support for this should be done first to prevent the risk that vendors will drop V4L2 support and find other solutions. The API should be extensible to the complex video pipeline use-case, though. In order to support stateless codecs we add support for the request API to the V4L2 ioctls that need this (ioctls operating on struct v4l2_buffer and v4l2_ext_controls). Creating, cloning, applying and deleting request objects is done through the media controller. In the long run we want to be able to set the request data for all bridge and subdev drivers with a single MC ioctl. As part of this we could make the control framework more generic: i.e. no longer V4L2 specific but a generic media framework. This would be similar to the DRM atomic framework. Note that Media device complexes may have devices that are out of scope of requests such as slow i2c devices where you still have to hit the frame related to the request. So what do you do if you miss? This would require additional complex cross-driver mechanisms in kernel. At least for now make user space responsible for applying configurations to such devices. A request object is essentially created empty from the point of view of the user. Only values that are changed in the request are part of the request, other values remain unchanged (unless those values change due to a side-effect of setting a request value). When a request completes, make a copy of the volatile controls (since that's the value at that point in time). This is needed for auto-<whatever> functions that need to know such volatile values. It should be possible to poll on the request's file handle and be woken up when the request is done. When dequeuing a buffer the kernel will set the request field to a request filehandle if the buffer is associated with a request. The kernel knows this information, so userspace doesn't have to set this field. Todo: Implement polling for request file handles. This allows polling for specific requests, e.g. as get woken up when the last request in a set of requests is done. Actions ------- Laurent: check the suitability of DRM tooling for V4L2 Mauro: check if kernel.org improved patchwork support. Hans: tentative (not this year) convert old videobuf documentation to videobuf2. Alexandre: MC Request API (REQUEST_CMD), with request components through V4L2 devices, request creation and queueing through MC Hans: control framework API: sync with Alexandre by 2nd week of November at the latest. -- Sakari Ailus sakari.ailus(a)linux.intel.com

2 1

[ANNOUNCE] Linux Media Summit 2016 Report – San Diego (draft)
by Mauro Carvalho Chehab 20 Apr '16

20 Apr '16

This is the draft of the media summit report. I'll be soon posting an online version of it, with the group photo, at linuxtv.org. Please review. Also, for the ones that presented a slide deck there, please send me the slides, for me to post there too. Thanks! Mauro --- Linux Media Summit 2016 – San Diego =================================== Attendees list: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> (Renesas) Shuah Khan <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> (Intel) Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) 1. CEC Framework Status update ============================== First version of the framework close to completion. Will likely be submitted for Kernel 4.7, although it could be merged only for 4.8. Independent framework, allows consumers in V4L2, DRM, ALSA, … Driver support WIP for Pulse Eight USB dongle, omap4, adv7604/7842/7511. Future plans: ------------- - ARC/CDC hotplug support - Implement high level protocol constraints (resend, timeout, rate limiting of messages). Whether those constraints can be implemented in the kernel remains to be analyzed, as the kernel to userspace API might be at a much lower level than the constraints. A userspace library might be another option. 2. Quick demo of the new qdisp utility that is in development ============================================================= The qdisp utility is a simpler alternative to qv4l2 that handles video capture and show the captured video only. Currently, qdisp requires OpenGL, OpenGLES support is planned. Color space and format conversion code is based on GPU shaders. It will be split into a library to be shared with qv4l2. A CPU-based alternative would be feasible but isn’t planned at the moment. The qdisp code is currently available here: <http://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=cec>. 3. Request API Status update ============================ At the moment: Allows to chain multiple of the existing IOCTLs into a request which will either be applied atomicly or not at all Initial proposal ---------------- - New IOCTL to start a commit - APPLY operation will apply changes in a request immediately. Useful to change multiple controls at the same time - QUEUE operation will queue changes with a buffer and will be applied when the buffer is processed Alternative proposal -------------------- One new IOCTL that takes all state and applies it atomicly (like DRM atomic modesetting) Open question ------------- How to perform atomic operations across subssytems? V4L2, DRM, ALSA, MC Action item ----------- Hans will contact Pawel to see what/when needs to be upstreamed for e.g. the rockchip driver. 4. Stream multiplexing (Guennadi Liakhovetski) ============================================== CSI-2 has up to 4 virtual channels (2 bits), 6 bits for Data Type Virtual channels do not have to be in sync with one another, so different virtual channels can carry different framerates. Within a virtual channel each line is tagged with a data type as well, so it can be used to pass metadata + videodata in one virtual channel as well. Requirements ------------ - Pipeline validation - Format validation - Bandwidth/QoS - Routing (related to muxing/demuxing) - Sounds interesting to have a non-V4L2 specific solution to solve cases where there a multiple entities linked into a bus or needing switch. For example, the ALSA subsystem supports TDM. Proposed solution ----------------- Introduce the concept of virtual channels which are routed on the top of the physical links. A virtual channel has a route that goes through multiple physical entities, with routing information at each entity on how the data is forwarded. Action item ----------- Laurent will dig up old router entity code he posted in the past and re-post it or provide a link to that code. 5. DT Bindings for flash & lens controllers =========================================== There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don’t transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? Sensor DT node add phandle to focus controller.: add generic v4l binding properties to reference such devices. 6. How to improve the linux-media patch and review process? =========================================================== Submaintainership status ------------------------ Currently, sub-maintainership is not working as expected. Also, we’re currently lacking DVB and RC sub-maintainers. Reviewers --------- We also lack reviewers. Shuah offered help with Media Controller patch reviews Proposals --------- Idea from Daniel Vetter to handle public APIs that might need some tweaking is to have it depend on a debug config option so enabling that api would make the kernel tainted. Action items ------------ Post a RFC asking for volunteers for sub-maintainership for DVB and Remote Controllers at the linux-media ML. In the case of Remote Ccontrollers: we could also post RFC to linux-input. Mauro: contact Kamil to ask status of codec sub-maintainership. If no time, then Hans can take over. Push on Intel (Sakari, Guennadi) (perhaps talk to Dirk Hohndel), Samsung (Marek and team), Google (Pawel) to give them time for upstreaming/reviewing. Next media workshop ------------------- Should we organize the next media workshop at ELC-E or LPC/KS ? Let’s try with a quick survey of who plans to go where. As this workshop is held in the US Europe would be good to attract more European developers. 7. Fix broken media_device alloc/remove – Media Device Allocator API ===================================================================== Slides: <https://drive.google.com/open?id=0B0NIL0BQg-AlTUQxdG1zWW15Tmc> The media module (media.ko) needs to be owner of the media devnode.cdev, and not the first driver that registers it. With the such change, `media_devnode_register()` and `__media_device_register()` don’t need struct module passed in. This can be cleaned up. Setting cdev parent should be set correctly. It was suggested to look on how looking howiio-core handles it. All drivers that use the media conroller should use the Media Device Allocator API. Dynamic changes ate the graph (with happens when two drivers bind to MC) doesn’t produce an event to use space. So, application would need to poll to find changes to the topology. This is not ideal and require more work. 8. Media Controller connection Entities ======================================= MC currently lacks a way to expose how external sources and outputs (RF, S-Video, composite, etc) are connected. Mauro explained what are the userspace needs that are not covered by our APIs today and that could benefit from the MC API : basically one of the goals for the MC, back on the 2009 discussions, were to be able to show the device nodes that related and should used together to capture analog TV, digital TV and ALSA, and to be able to prevent the related drivers to stream on unexpected ways. So, when an analog TV connector is in use, the DVB API can’t be enabled and vice-versa. The device nodes issue is unrelated to connectors, but supporting the connectors is needed to be able to prevent using two incompatible paths at the same time. In other words, Analog and digital paths are usually exclusive, and without MC and MC connectors, there’s no way for userspace to know what the constraints are. Physical x Logical representation --------------------------------- While media cards/boards have physical connectors, the chips managed by the Kernel driver sees a “logical” connection, in the sense that each different input/output is mapped via a pin set with a corresponding register setup. Due to that, the representation used by V4L2 (VIDIOC_ENUM_INPUT, VIDIOC_G_INPUT, VIDIOC_S_INPUT) is to represent the logical connection. Another desired feature of represent MC connections is to present information about connectors to the user to make it easier to know where to plug the cables. Such representation is physical connector based. A connection-based representation in MC would require properties to map them to physical connectors. A connector-based representation in MC would require properties to map them to logical connections. Provided that both mappings are possible, the MC representation could either use logical connections or physical connectors. This problem is similar to virtual channels over a logical link (like CSI-2, see “4. Stream multiplexing”), in the sense that logical connectors can be thought of as a specific routing of one or more signals from the physical connector through some fabric (e.g. switch, crossbar) to the demodulator. For RF and Composite, the physical and logical representation are the same, as those connectors have just one analog signal. S-Video has two signals on it (Y+C). When a S-Video signal is sent to the connector, the physical and logical representation matches. However, some devices allow to use the S-Video connector to send a composite signal, by using a Composite->S-Video cable, but requiring a different setup at the chipset. On such cases, the physical and logical representation is different. There’s also one case that it was not covered yet: how to handle the cases where one logical connection is mapped via several different connectors? This is common on ALSA, where the stereo input/output can be either a single jack or two RCAs. Proposal for complex connections -------------------------------- Model physical connectors and support a routing ioctl for the entities that they are connected to. For existing drivers that use S_INPUT, we can either not show the logical connectors at all, or we just show the logical connectors in the absence of knowledge about the actual physical connectors. How that routing ioctl should look like is unknown, but this might be done the same way as the routing as discussed in the context of CSI-2. Some subdevs that have complex routing are saa7115, msp34xx and adv7842. Action items: ------------- It was decided that, for now, we’ll map via MC only the cases where the physical and logical representation is the same, e. g. RF, Composite and S-Video signals over a S-Video connector, postponing the other cases to where we have a routing ioctl. - For S-Video, use 2 pads. - Use only one sink PAD per input at the analog demod; - RF and Connections are OK – just one pad; - not map complex cases like Composite over S-Video. 9. Pad identification ===================== Currently, a pad is identified by index and whether it is a input or output pad. If the index changes between kernel versions, the userspace ABI breaks. While we don’t have the properties API, we could use an enum to give a type to the pad. Don’t expose this to userspace yet, as exposing it to userspace requires more discussions, plus the properties API. Adding pad names exposed to userspace can also be useful. However, there’s a risk of userspace relying on specific string names to identify the pad. So, in order to avoid repeating the same mistake we did with subdev and entity names, the API should define in details what the name should contain and how it should be constructed. What userspace can expect from the name, including information that can be extracted from the name (e.g. can I sscanf(“%u”) to get the pad number ?) also needs to be defined in the API. Action item ----------- Create a pad type enum. 10. Other topics ================ When adding a new API that is still experimental, the DRM/KMS subsystem covers the API with a configuration option and taints the kernel when the option is selected. This lasts for a few kernel versions until the API stabilizes and is then removed. This could be experimented in V4L2, effectively giving us back CONFIG_EXPERIMENTAL. Should we avoid reserved fields in new ioctls, or use API versioning (as in DRM/KMS) ? Versioning is a bit more work in userspace, but avoids issues with application that don’t zero reserved fields and break when the API is extended. Other action items ------------------ - Review MC virtual driver v3 patch (vimc) from Hellen (MC developers); - Submit/review the “dynamically allocates pad config” patch (Laurent); - Split off legacy defines in media.h to media-legacy.h (Hans); - User versioning for CEC ioctls instead of reserved fields (Hans). Cheers, Mauro -- Thanks, Mauro

1 2

Etherpad notes
by Mauro Carvalho Chehab 09 Apr '16

09 Apr '16

Agenda for the Media Workshop 2016 - San Diego Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> (Renesas) Shuah Khan <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> (Intel) Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) 1. CEC Framework Status update (Hans Verkuil) 09:30-09:40 First version of the framework close to completion. Will likely be submitted for Kernel 4.7, although it could be merged only for 4.8. Independent framework, allows consumers in V4L2, DRM, ALSA, ... Driver support WIP for Pulse Eight USB dongle, omap4, adv7604/7842/7511. Future plans: ARC/CDC hotplug support Implement high level protocol constraints (resend, timeout, rate limiting of messages). Whether those constraints can be implemented in the kernel remains to be analyzed, as the kernel to userspace API might be at a much lower level than the constraints. A userspace library might be another option. 2. Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:40-09:50 qdisp is a simpler alternative to qv4l2 that handles video capture and show the captured video only. qdisp requires OpenGL, OpenGLES is planned. Color space and format conversion code is based on GPU shaders. It will be split into a library to be shared with qv4l2. A CPU-based alternative would be feasible but isn't planned at the moment. qdisp is available here: http://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=cec 3. Request API Status update (Laurent Pinchart) 09:50-10:00 At the moment: Allows to chain multiple of the existing IOCTLs into a request which will either be applied atomicly or not at all New IOCTL to start a commit APPLY operation will apply changes in a request immediately. Useful to change multiple controls at the same time QUEUE operation will queue changes with a buffer and will be applied when the buffer is processed Alternative: One new IOCTL that takes all state and applies it atomicly (like DRM atomic modesetting) Open question: How to perform atomic operations across subssytems? V4L2, DRM, ALSA, MC Action item Hans: contact Pawel to see what/when needs to be upstreamed for e.g. the rockchip driver. 4. Stream multiplexing (Guennadi Liakhovetski) 10:15-11:00 CSI-2 has up to 4 virtual channels (2 bits), 6 bits for Data Type Virtual channels do not have to be in sync with one another, so different virtual channels can carry different framerates. Within a virtual channel each line is tagged with a data type as well, so it can be used to pass metadata + videodata in one virtual channel as well. Requirements: - Pipeline validation - Format validation - Bandwidth/QoS - Routing (related to muxing/demuxing) One idea: Introduce the concept of virtual channels which are routed ontop of the physical links. A virtual channel has a route that goes through multiple physical entities, with routing information at each entity on how the data is forwarded. Sounds interesting to have a non-V4L2 specific solution to solve cases where there a multiple entities linked into a bus or needing switch. TDM for ALSA Action item Laurent: dig up old router entity code and post it or provide a link to that code. 5. DT Bindings for flash & lens controllers 11:00-11:15 There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don't transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? (Hans) Sensor DT node add phandle to focus controller.: add generic v4l binding properties to reference such devices. 6. How to improve the linux-media patch and review process? 11:00-12:15 Action item Mauro: contact Kamil to ask status of codec sub-maintainership. If no time, then Hans can take over. No interest in DVB/RC from other developers. Shuah can help with Media Controller patch reviews Idea from Daniel Vetter to handle public APIs that might need some tweaking is to have it depend on a debug config option so enabling that api would make the kernel tainted. Action item Mauro: Push on Intel (Sakari, Guennadi) (perhaps talk to Dirk Hohndel), Samsung (Marek and team), Google (Pawel) to give them time for upstreaming/reviewing. For rc maintainers: also post rfc to linux-input. Should we organize the next media workshop at ELC-E or LPC/KS ? Let's try with a quick survey of who plans to go where. As this workshop is held in the US Europe would be good to attract more European developers. 7. Fix broken media_device alloc/remove - Media Device Allocator API (Shuah Khan) 12:15-13:00 https://drive.google.com/open?id=0B0NIL0BQg-AlTUQxdG1zWW15Tmc Media module needs to be owner of the media devnode - not the first driver that registers it With the above change, media_devnode_register() and __media_device_register() don't need struct module passed in. This can be cleaned up. Setting cdev parent should be set correctly - reference iio-core Media Device Allocator API and change all drivers to use it. Dynamically changing the graph in a two driver model isn't exposed to use space - (application needs to poll to find changes to the topology) - considered unsupported at the moment. 8. Connector Entity discussion (Mauro) 14:00-15:30 What are userspace needs that are not covered by our APIs today and that could benefit from the MC API ? What device nodes are related and are to be used together to capture analog TV? This is unrelated to connectors. Analog and digital paths are usually exclusive, without a way for userspace to know what the constraints are. The representation is logical connection based. Present information about connectors to the user to make it easier to know where to plug cables. The representation is physical connector based. A connection-based representation in MC would require properties to map them to physical connectors. A connector-based representation in MC would require properties to map them to logical connections. The two last use cases above can thus be implemented with an MC representation based on either logical inputs or physical connectors, they don't require one specific representation. Problem is similar to virtual channels over a logical link (like CSI-2, see "4. Stream multiplexing"). Logical connectors can be thought of as a specific routing of one or more signals from the physical connector through some fabric (e.g. switch, crossbar) to the demodulator. Idea: Model physical connectors and support a routing ioctl for the entities that they are connected to. For existing drivers that use S_INPUT, we can either not show the logical connectors at all, or we just show the logical connectors in the absence of knowledge about the actual physical connectors. How that routing ioctl should look like is unknown, but this might be done the same way as the routing as discussed in the context of CSI-2. Some subdevs that have complex routing are saa7115, msp34xx and adv7842. Action items: - Code S-Video with 2 pads; - Onesink PAD per input at the analog demod; - RF and Connections are OK - just one pad; - For now, not map Composite over S-Video. 9. Pad identification (Mauro) 15:45-16:30 A pad is identified by index and whether it is a input or output pad. If the index changes between kernel versions the userspace ABI breaks. Use an enum to give a type to the pad. Don't expose this to userspace yet. Adding pad names exposed to userspace can also be useful. To avoid repeating the same mistake we did with subdev and entity names, the API should define in details what the name should contain and how it should be constructed. What userspace can expect from the name, including information that can be extracted from the name (e.g. can I sscanf("%u") to get the pad number ?) also needs to be defined in the API. 10. Anything else? 16:45- When adding a new API that is still experimental, the DRM/KMS subsystem covers the API with a configuration option and taints the kernel when the option is selected. This lasts for a few kernel versions until the API stabilizes and is then removed. This could be experimented in V4L2, effectively giving us back CONFIG_EXPERIMENTAL. Should we avoid reserved fields in new ioctls, or use API versioning (as in DRM/KMS) ? Versioning is a bit more work in userspace, but avoids issues with application that don't zero reserved fields and break when the API is extended. 11. Other action items - Review MC virtual driver v3 patch (vimc) form Hellen - Submit/review subdev patch that dynamically allocate patches (Laurent) - Split off legacy defines in media.h to media-legacy.h (Hans) - User versioning for CEC ioctls instead of reserved fields (Hans) -- Cheers, Mauro

2 1

Media Workshop: agenda
by Hans Verkuil 07 Apr '16

07 Apr '16

Agenda for the Media Workshop Date: April 7 Location: Bankers Hill - 3rd floor Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> Shuah Kahn <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) Agenda ====== 09:00-09:15: Setup laptops, drink tea/coffee, etc. etc. 09:15-09:30: Introduction 09:30-09:40: CEC Framework Status update (Hans Verkuil) 09:40-09:50: Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:50-10:00: Request API Status update (Laurent Pinchart) 10:00-10:15: Break 10:15-11:00: Stream multiplexing (Guennadi Liakhovetski) 11:00-11:15 There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don't transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? (Hans) 11:00-12:15: linux-media patch and review process: how to improve this? 12:15-13:00: Fix broken media_device alloc/remove (Shuah Kahn) 13:00-14:00: Lunch 14:00-15:30: Connector Entity discussion (Mauro) 15:30-15:45: Break 15:45-16:30: Pad identification (Mauro) 16:30-16:45: Break 16:45- Anything else? These times are tentative and typically will change during the day. Regards, Hans

1 0

Media Workshop: (final?) agenda
by Hans Verkuil 07 Apr '16

07 Apr '16

Agenda for the Media Workshop Date: April 7 Location: TBC Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> Shuah Kahn <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) Others? Regarding the agenda: I put the non-MC topics first since MC discussions tend to fill all available time. We don't have that many non-MC topics anyway. The first three topics are a demo and two status updates and are just informative and not intended to cause a lot of discussion. Should that happen, then those discussions should probably be moved to later in the day. Agenda ====== 09:00-09:15: Setup laptops, drink tea/coffee, etc. etc. 09:15-09:30: Introduction 09:30-09:40: CEC Framework Status update (Hans Verkuil) 09:40-09:50: Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:50-10:00: Request API Status update (Laurent Pinchart) 10:00-11:00: Stream multiplexing (Guennadi Liakhovetski) 11:00-11:15: Break 11:00-12:00: linux-media patch and review process: how to improve this? 12:00-12:45: Fix broken media_device alloc/remove (Shuah Kahn) 12:45-13:45: Lunch 13:45-15:15: Connector Entity discussion (Mauro) 15:15-15:30: Break 15:30-16:15: Pad identification (Mauro) 16:15- Anything else? These times are tentative and typically will change during the day. If you think you need more/less time, please let me know. I've guessed at some of the duration of these topics. Guennadi, I understand you'll have some slides prepared for your topic, right? Regards, Hans _______________________________________________ media-workshop mailing list media-workshop(a)linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/media-workshop

3 7

Media Workshop: draft agenda
by Hans Verkuil 29 Mar '16

29 Mar '16

Draft Agenda for the Media Workshop Date: April 7 Location: TBC This is a draft agenda, please reply with corrections. Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Magnus Damm <magnus.damm(a)gmail.com> Shuah Kahn <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) Others? Regarding the agenda: I put the non-MC topics first since MC discussions tend to fill all available time. We don't have that many non-MC topics anyway. The first three topics are a demo and two status updates and are just informative and not intended to cause a lot of discussion. Should that happen, then those discussions should probably be moved to later in the day. Agenda ====== 09:00-09:15: Setup laptops, drink tea/coffee, etc. etc. 09:15-09:30: Introduction 09:30-09:40: CEC Framework Status update (Hans Verkuil) 09:40-09:50: Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:50-10:00: Request API Status update (Laurent Pinchart) 10:00-11:00: Stream multiplexing (Guennadi Liakhovetski) 11:00-11:15: Break 11:00-12:00: linux-media patch and review process: how to improve this? 12:00-12:45: Fix broken media_device alloc/remove (Shuah Kahn) 12:45-13:45: Lunch 13:45-15:15: Connector Entity discussion (Mauro) 15:15-15:30: Break 15:30-16:15: Pad identification (Mauro) 16:15- Anything else? These times are tentative and typically will change during the day. If you think you need more/less time, please let me know. I've guessed at some of the duration of these topics. Links to discussions that will help with understanding these topics will be welcome. My apologies for not doing that as I have little time today, but I wanted to get this first draft out now. Guennadi, I understand you'll have some slides prepared for your topic, right? Regards, Hans

3 2

workshop topics
by Hans Verkuil 29 Mar '16

29 Mar '16

I have these topics: - Continue MC connector entity discussion - CEC status update (to be confirmed) - A short demo of the qdisp utility I am working on Laurent, would it make sense to discuss your work on the request API? Another possible topic: media-ctl vs mc_nextgen_test: having two utilities makes me very unhappy! I think we have a fairly small group this time, so I am not sure if we need a full fledged agenda. If the group will be larger than expected, then please let me know and I will prepare a proper agenda. Regards, Hans

6 18

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

media-workshop