media-workshop April 2016

media-workshop@linuxtv.org

5 participants
4 discussions

[ANNOUNCE] Linux Media Summit 2016 Report – San Diego (draft)
by Mauro Carvalho Chehab 20 Apr '16

20 Apr '16

This is the draft of the media summit report. I'll be soon posting an online version of it, with the group photo, at linuxtv.org. Please review. Also, for the ones that presented a slide deck there, please send me the slides, for me to post there too. Thanks! Mauro --- Linux Media Summit 2016 – San Diego =================================== Attendees list: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> (Renesas) Shuah Khan <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> (Intel) Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) 1. CEC Framework Status update ============================== First version of the framework close to completion. Will likely be submitted for Kernel 4.7, although it could be merged only for 4.8. Independent framework, allows consumers in V4L2, DRM, ALSA, … Driver support WIP for Pulse Eight USB dongle, omap4, adv7604/7842/7511. Future plans: ------------- - ARC/CDC hotplug support - Implement high level protocol constraints (resend, timeout, rate limiting of messages). Whether those constraints can be implemented in the kernel remains to be analyzed, as the kernel to userspace API might be at a much lower level than the constraints. A userspace library might be another option. 2. Quick demo of the new qdisp utility that is in development ============================================================= The qdisp utility is a simpler alternative to qv4l2 that handles video capture and show the captured video only. Currently, qdisp requires OpenGL, OpenGLES support is planned. Color space and format conversion code is based on GPU shaders. It will be split into a library to be shared with qv4l2. A CPU-based alternative would be feasible but isn’t planned at the moment. The qdisp code is currently available here: <http://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=cec>. 3. Request API Status update ============================ At the moment: Allows to chain multiple of the existing IOCTLs into a request which will either be applied atomicly or not at all Initial proposal ---------------- - New IOCTL to start a commit - APPLY operation will apply changes in a request immediately. Useful to change multiple controls at the same time - QUEUE operation will queue changes with a buffer and will be applied when the buffer is processed Alternative proposal -------------------- One new IOCTL that takes all state and applies it atomicly (like DRM atomic modesetting) Open question ------------- How to perform atomic operations across subssytems? V4L2, DRM, ALSA, MC Action item ----------- Hans will contact Pawel to see what/when needs to be upstreamed for e.g. the rockchip driver. 4. Stream multiplexing (Guennadi Liakhovetski) ============================================== CSI-2 has up to 4 virtual channels (2 bits), 6 bits for Data Type Virtual channels do not have to be in sync with one another, so different virtual channels can carry different framerates. Within a virtual channel each line is tagged with a data type as well, so it can be used to pass metadata + videodata in one virtual channel as well. Requirements ------------ - Pipeline validation - Format validation - Bandwidth/QoS - Routing (related to muxing/demuxing) - Sounds interesting to have a non-V4L2 specific solution to solve cases where there a multiple entities linked into a bus or needing switch. For example, the ALSA subsystem supports TDM. Proposed solution ----------------- Introduce the concept of virtual channels which are routed on the top of the physical links. A virtual channel has a route that goes through multiple physical entities, with routing information at each entity on how the data is forwarded. Action item ----------- Laurent will dig up old router entity code he posted in the past and re-post it or provide a link to that code. 5. DT Bindings for flash & lens controllers =========================================== There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don’t transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? Sensor DT node add phandle to focus controller.: add generic v4l binding properties to reference such devices. 6. How to improve the linux-media patch and review process? =========================================================== Submaintainership status ------------------------ Currently, sub-maintainership is not working as expected. Also, we’re currently lacking DVB and RC sub-maintainers. Reviewers --------- We also lack reviewers. Shuah offered help with Media Controller patch reviews Proposals --------- Idea from Daniel Vetter to handle public APIs that might need some tweaking is to have it depend on a debug config option so enabling that api would make the kernel tainted. Action items ------------ Post a RFC asking for volunteers for sub-maintainership for DVB and Remote Controllers at the linux-media ML. In the case of Remote Ccontrollers: we could also post RFC to linux-input. Mauro: contact Kamil to ask status of codec sub-maintainership. If no time, then Hans can take over. Push on Intel (Sakari, Guennadi) (perhaps talk to Dirk Hohndel), Samsung (Marek and team), Google (Pawel) to give them time for upstreaming/reviewing. Next media workshop ------------------- Should we organize the next media workshop at ELC-E or LPC/KS ? Let’s try with a quick survey of who plans to go where. As this workshop is held in the US Europe would be good to attract more European developers. 7. Fix broken media_device alloc/remove – Media Device Allocator API ===================================================================== Slides: <https://drive.google.com/open?id=0B0NIL0BQg-AlTUQxdG1zWW15Tmc> The media module (media.ko) needs to be owner of the media devnode.cdev, and not the first driver that registers it. With the such change, `media_devnode_register()` and `__media_device_register()` don’t need struct module passed in. This can be cleaned up. Setting cdev parent should be set correctly. It was suggested to look on how looking howiio-core handles it. All drivers that use the media conroller should use the Media Device Allocator API. Dynamic changes ate the graph (with happens when two drivers bind to MC) doesn’t produce an event to use space. So, application would need to poll to find changes to the topology. This is not ideal and require more work. 8. Media Controller connection Entities ======================================= MC currently lacks a way to expose how external sources and outputs (RF, S-Video, composite, etc) are connected. Mauro explained what are the userspace needs that are not covered by our APIs today and that could benefit from the MC API : basically one of the goals for the MC, back on the 2009 discussions, were to be able to show the device nodes that related and should used together to capture analog TV, digital TV and ALSA, and to be able to prevent the related drivers to stream on unexpected ways. So, when an analog TV connector is in use, the DVB API can’t be enabled and vice-versa. The device nodes issue is unrelated to connectors, but supporting the connectors is needed to be able to prevent using two incompatible paths at the same time. In other words, Analog and digital paths are usually exclusive, and without MC and MC connectors, there’s no way for userspace to know what the constraints are. Physical x Logical representation --------------------------------- While media cards/boards have physical connectors, the chips managed by the Kernel driver sees a “logical” connection, in the sense that each different input/output is mapped via a pin set with a corresponding register setup. Due to that, the representation used by V4L2 (VIDIOC_ENUM_INPUT, VIDIOC_G_INPUT, VIDIOC_S_INPUT) is to represent the logical connection. Another desired feature of represent MC connections is to present information about connectors to the user to make it easier to know where to plug the cables. Such representation is physical connector based. A connection-based representation in MC would require properties to map them to physical connectors. A connector-based representation in MC would require properties to map them to logical connections. Provided that both mappings are possible, the MC representation could either use logical connections or physical connectors. This problem is similar to virtual channels over a logical link (like CSI-2, see “4. Stream multiplexing”), in the sense that logical connectors can be thought of as a specific routing of one or more signals from the physical connector through some fabric (e.g. switch, crossbar) to the demodulator. For RF and Composite, the physical and logical representation are the same, as those connectors have just one analog signal. S-Video has two signals on it (Y+C). When a S-Video signal is sent to the connector, the physical and logical representation matches. However, some devices allow to use the S-Video connector to send a composite signal, by using a Composite->S-Video cable, but requiring a different setup at the chipset. On such cases, the physical and logical representation is different. There’s also one case that it was not covered yet: how to handle the cases where one logical connection is mapped via several different connectors? This is common on ALSA, where the stereo input/output can be either a single jack or two RCAs. Proposal for complex connections -------------------------------- Model physical connectors and support a routing ioctl for the entities that they are connected to. For existing drivers that use S_INPUT, we can either not show the logical connectors at all, or we just show the logical connectors in the absence of knowledge about the actual physical connectors. How that routing ioctl should look like is unknown, but this might be done the same way as the routing as discussed in the context of CSI-2. Some subdevs that have complex routing are saa7115, msp34xx and adv7842. Action items: ------------- It was decided that, for now, we’ll map via MC only the cases where the physical and logical representation is the same, e. g. RF, Composite and S-Video signals over a S-Video connector, postponing the other cases to where we have a routing ioctl. - For S-Video, use 2 pads. - Use only one sink PAD per input at the analog demod; - RF and Connections are OK – just one pad; - not map complex cases like Composite over S-Video. 9. Pad identification ===================== Currently, a pad is identified by index and whether it is a input or output pad. If the index changes between kernel versions, the userspace ABI breaks. While we don’t have the properties API, we could use an enum to give a type to the pad. Don’t expose this to userspace yet, as exposing it to userspace requires more discussions, plus the properties API. Adding pad names exposed to userspace can also be useful. However, there’s a risk of userspace relying on specific string names to identify the pad. So, in order to avoid repeating the same mistake we did with subdev and entity names, the API should define in details what the name should contain and how it should be constructed. What userspace can expect from the name, including information that can be extracted from the name (e.g. can I sscanf(“%u”) to get the pad number ?) also needs to be defined in the API. Action item ----------- Create a pad type enum. 10. Other topics ================ When adding a new API that is still experimental, the DRM/KMS subsystem covers the API with a configuration option and taints the kernel when the option is selected. This lasts for a few kernel versions until the API stabilizes and is then removed. This could be experimented in V4L2, effectively giving us back CONFIG_EXPERIMENTAL. Should we avoid reserved fields in new ioctls, or use API versioning (as in DRM/KMS) ? Versioning is a bit more work in userspace, but avoids issues with application that don’t zero reserved fields and break when the API is extended. Other action items ------------------ - Review MC virtual driver v3 patch (vimc) from Hellen (MC developers); - Submit/review the “dynamically allocates pad config” patch (Laurent); - Split off legacy defines in media.h to media-legacy.h (Hans); - User versioning for CEC ioctls instead of reserved fields (Hans). Cheers, Mauro -- Thanks, Mauro

1 2

Etherpad notes
by Mauro Carvalho Chehab 09 Apr '16

09 Apr '16

Agenda for the Media Workshop 2016 - San Diego Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> (Renesas) Shuah Khan <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> (Intel) Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) 1. CEC Framework Status update (Hans Verkuil) 09:30-09:40 First version of the framework close to completion. Will likely be submitted for Kernel 4.7, although it could be merged only for 4.8. Independent framework, allows consumers in V4L2, DRM, ALSA, ... Driver support WIP for Pulse Eight USB dongle, omap4, adv7604/7842/7511. Future plans: ARC/CDC hotplug support Implement high level protocol constraints (resend, timeout, rate limiting of messages). Whether those constraints can be implemented in the kernel remains to be analyzed, as the kernel to userspace API might be at a much lower level than the constraints. A userspace library might be another option. 2. Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:40-09:50 qdisp is a simpler alternative to qv4l2 that handles video capture and show the captured video only. qdisp requires OpenGL, OpenGLES is planned. Color space and format conversion code is based on GPU shaders. It will be split into a library to be shared with qv4l2. A CPU-based alternative would be feasible but isn't planned at the moment. qdisp is available here: http://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=cec 3. Request API Status update (Laurent Pinchart) 09:50-10:00 At the moment: Allows to chain multiple of the existing IOCTLs into a request which will either be applied atomicly or not at all New IOCTL to start a commit APPLY operation will apply changes in a request immediately. Useful to change multiple controls at the same time QUEUE operation will queue changes with a buffer and will be applied when the buffer is processed Alternative: One new IOCTL that takes all state and applies it atomicly (like DRM atomic modesetting) Open question: How to perform atomic operations across subssytems? V4L2, DRM, ALSA, MC Action item Hans: contact Pawel to see what/when needs to be upstreamed for e.g. the rockchip driver. 4. Stream multiplexing (Guennadi Liakhovetski) 10:15-11:00 CSI-2 has up to 4 virtual channels (2 bits), 6 bits for Data Type Virtual channels do not have to be in sync with one another, so different virtual channels can carry different framerates. Within a virtual channel each line is tagged with a data type as well, so it can be used to pass metadata + videodata in one virtual channel as well. Requirements: - Pipeline validation - Format validation - Bandwidth/QoS - Routing (related to muxing/demuxing) One idea: Introduce the concept of virtual channels which are routed ontop of the physical links. A virtual channel has a route that goes through multiple physical entities, with routing information at each entity on how the data is forwarded. Sounds interesting to have a non-V4L2 specific solution to solve cases where there a multiple entities linked into a bus or needing switch. TDM for ALSA Action item Laurent: dig up old router entity code and post it or provide a link to that code. 5. DT Bindings for flash & lens controllers 11:00-11:15 There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don't transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? (Hans) Sensor DT node add phandle to focus controller.: add generic v4l binding properties to reference such devices. 6. How to improve the linux-media patch and review process? 11:00-12:15 Action item Mauro: contact Kamil to ask status of codec sub-maintainership. If no time, then Hans can take over. No interest in DVB/RC from other developers. Shuah can help with Media Controller patch reviews Idea from Daniel Vetter to handle public APIs that might need some tweaking is to have it depend on a debug config option so enabling that api would make the kernel tainted. Action item Mauro: Push on Intel (Sakari, Guennadi) (perhaps talk to Dirk Hohndel), Samsung (Marek and team), Google (Pawel) to give them time for upstreaming/reviewing. For rc maintainers: also post rfc to linux-input. Should we organize the next media workshop at ELC-E or LPC/KS ? Let's try with a quick survey of who plans to go where. As this workshop is held in the US Europe would be good to attract more European developers. 7. Fix broken media_device alloc/remove - Media Device Allocator API (Shuah Khan) 12:15-13:00 https://drive.google.com/open?id=0B0NIL0BQg-AlTUQxdG1zWW15Tmc Media module needs to be owner of the media devnode - not the first driver that registers it With the above change, media_devnode_register() and __media_device_register() don't need struct module passed in. This can be cleaned up. Setting cdev parent should be set correctly - reference iio-core Media Device Allocator API and change all drivers to use it. Dynamically changing the graph in a two driver model isn't exposed to use space - (application needs to poll to find changes to the topology) - considered unsupported at the moment. 8. Connector Entity discussion (Mauro) 14:00-15:30 What are userspace needs that are not covered by our APIs today and that could benefit from the MC API ? What device nodes are related and are to be used together to capture analog TV? This is unrelated to connectors. Analog and digital paths are usually exclusive, without a way for userspace to know what the constraints are. The representation is logical connection based. Present information about connectors to the user to make it easier to know where to plug cables. The representation is physical connector based. A connection-based representation in MC would require properties to map them to physical connectors. A connector-based representation in MC would require properties to map them to logical connections. The two last use cases above can thus be implemented with an MC representation based on either logical inputs or physical connectors, they don't require one specific representation. Problem is similar to virtual channels over a logical link (like CSI-2, see "4. Stream multiplexing"). Logical connectors can be thought of as a specific routing of one or more signals from the physical connector through some fabric (e.g. switch, crossbar) to the demodulator. Idea: Model physical connectors and support a routing ioctl for the entities that they are connected to. For existing drivers that use S_INPUT, we can either not show the logical connectors at all, or we just show the logical connectors in the absence of knowledge about the actual physical connectors. How that routing ioctl should look like is unknown, but this might be done the same way as the routing as discussed in the context of CSI-2. Some subdevs that have complex routing are saa7115, msp34xx and adv7842. Action items: - Code S-Video with 2 pads; - Onesink PAD per input at the analog demod; - RF and Connections are OK - just one pad; - For now, not map Composite over S-Video. 9. Pad identification (Mauro) 15:45-16:30 A pad is identified by index and whether it is a input or output pad. If the index changes between kernel versions the userspace ABI breaks. Use an enum to give a type to the pad. Don't expose this to userspace yet. Adding pad names exposed to userspace can also be useful. To avoid repeating the same mistake we did with subdev and entity names, the API should define in details what the name should contain and how it should be constructed. What userspace can expect from the name, including information that can be extracted from the name (e.g. can I sscanf("%u") to get the pad number ?) also needs to be defined in the API. 10. Anything else? 16:45- When adding a new API that is still experimental, the DRM/KMS subsystem covers the API with a configuration option and taints the kernel when the option is selected. This lasts for a few kernel versions until the API stabilizes and is then removed. This could be experimented in V4L2, effectively giving us back CONFIG_EXPERIMENTAL. Should we avoid reserved fields in new ioctls, or use API versioning (as in DRM/KMS) ? Versioning is a bit more work in userspace, but avoids issues with application that don't zero reserved fields and break when the API is extended. 11. Other action items - Review MC virtual driver v3 patch (vimc) form Hellen - Submit/review subdev patch that dynamically allocate patches (Laurent) - Split off legacy defines in media.h to media-legacy.h (Hans) - User versioning for CEC ioctls instead of reserved fields (Hans) -- Cheers, Mauro

2 1

Media Workshop: agenda
by Hans Verkuil 07 Apr '16

07 Apr '16

Agenda for the Media Workshop Date: April 7 Location: Bankers Hill - 3rd floor Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> Shuah Kahn <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Benoit Parrot <bparrot(a)ti.com> (Texas Instruments) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) Agenda ====== 09:00-09:15: Setup laptops, drink tea/coffee, etc. etc. 09:15-09:30: Introduction 09:30-09:40: CEC Framework Status update (Hans Verkuil) 09:40-09:50: Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:50-10:00: Request API Status update (Laurent Pinchart) 10:00-10:15: Break 10:15-11:00: Stream multiplexing (Guennadi Liakhovetski) 11:00-11:15 There are drivers that create their MC topology using the device tree information, which works great for entities that transport data, but how to detect entities that don't transport data such as flash devices, focusers, etc.? How can those be deduced using the device tree? (Hans) 11:00-12:15: linux-media patch and review process: how to improve this? 12:15-13:00: Fix broken media_device alloc/remove (Shuah Kahn) 13:00-14:00: Lunch 14:00-15:30: Connector Entity discussion (Mauro) 15:30-15:45: Break 15:45-16:30: Pad identification (Mauro) 16:30-16:45: Break 16:45- Anything else? These times are tentative and typically will change during the day. Regards, Hans

1 0

Media Workshop: (final?) agenda
by Hans Verkuil 07 Apr '16

07 Apr '16

Agenda for the Media Workshop Date: April 7 Location: TBC Attendees: Mauro Carvalho Chehab <mchehab(a)osg.samsung.com> (Samsung) Lars-Peter Clausen <lars(a)metafoo.de> (Analog Devices) Magnus Damm <magnus.damm(a)gmail.com> Shuah Kahn <shuahkh(a)osg.samsung.com> (Samsung) Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Kuninori Morimoto <kuninori.morimoto.gx(a)renesas.com> (Renesas) Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> (Ideas on Board) Niklas Söderland <niklas.soderlund(a)ragnatech.se> (Ragnatech) Hans Verkuil <hverkuil(a)xs4all.nl> (Cisco) Others? Regarding the agenda: I put the non-MC topics first since MC discussions tend to fill all available time. We don't have that many non-MC topics anyway. The first three topics are a demo and two status updates and are just informative and not intended to cause a lot of discussion. Should that happen, then those discussions should probably be moved to later in the day. Agenda ====== 09:00-09:15: Setup laptops, drink tea/coffee, etc. etc. 09:15-09:30: Introduction 09:30-09:40: CEC Framework Status update (Hans Verkuil) 09:40-09:50: Quick demo of the new qdisp utility that is in development (Hans Verkuil) 09:50-10:00: Request API Status update (Laurent Pinchart) 10:00-11:00: Stream multiplexing (Guennadi Liakhovetski) 11:00-11:15: Break 11:00-12:00: linux-media patch and review process: how to improve this? 12:00-12:45: Fix broken media_device alloc/remove (Shuah Kahn) 12:45-13:45: Lunch 13:45-15:15: Connector Entity discussion (Mauro) 15:15-15:30: Break 15:30-16:15: Pad identification (Mauro) 16:15- Anything else? These times are tentative and typically will change during the day. If you think you need more/less time, please let me know. I've guessed at some of the duration of these topics. Guennadi, I understand you'll have some slides prepared for your topic, right? Regards, Hans _______________________________________________ media-workshop mailing list media-workshop(a)linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/media-workshop

3 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

media-workshop April 2016