media-workshop October 2013

media-workshop@linuxtv.org

16 participants
9 discussions

[ANNOUNCE] Notes on the Media summit 2013-10-23
by Mauro Carvalho Chehab 04 Nov '13

04 Nov '13

Notes on the Media summit 2013-10-23 List of Participants: Ricardo Ribalda - Qtechnology Hans Verkuil - Cisco Sakari Ailus - Intel Mauro Carvalho Chehab - Samsung Kieran Kunhya - Open Broadcast Systems Sylwester Nawrocki - Samsung Guennadi Liakhovetski Benjamin Gaignard - ST Microelectronics Hugues Fruchet - ST Microelectronics Andrzej Hajda - Samsung Peter Senna Tschudin - Coccinelle? Hans de Goede - Red Hat Michael Krufky - Samsung Paul Walmsley - Nvidia Laurent Pinchart - Ideas on board 1. Mauro Chehab: is the submaintainer arrangement working? General consensus is that it is working. Hans pointed that the commits ML is not working. Mauro will check what's happening at LinuxTV website after returning back. Hans also pointed that patches for the rcX kernel aren't always picked up quickly enough. Mauro says this is due to his move to Samsung which took a lot of his time. This should improve. 2. Ricardo Ribalda Delgado: Multiple selections For a more flexible cropping selection a new extended rectangle struct has to be added with possibly negative top and left offsets. This would allow to explore capabilities of industry grade sensors, that can be configured to crop multiple rectangles from the input area. Sylwester: an alternative approach would be to use indexed windows. The issue of atomicity should also be addressed. Ricardo is proposing to add the capability of adding more than one selection areas - No objections so far. - Helper functions to detect if rectangles overlap or are contained on other rectangles are needed. - Documentation changes for V4L2 subdev selection API. Special care needs to be taken for too few or too many rectangles. In both cases the rectangles field should be set to the actual number of allowed rectangles. In the too few case an error should be returned, in the too many case the extra rects should just be ignored. Mauro thinks that it will be useful to see if the width/height fields in v4l2_rect can be changes to u32 instead of s32: will require code review of drivers. 3. Hans Verkuil: colorspaces - Colorspace: limited/full range Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp Proposal accepted, although the "_SRGB_LIM" name might need to be improved, although no alternatives were given. ALso mention in the docs that S_FMT can overwrite the colorspace for output devices if the selected one isn't supported. 4. Kieran Kunhya: SDI The biggest problem is separate audio and video file descriptors. Audio data is transfered in the video blanking areas, where should they be separated - in the kernel, using multiple planes, or in the user-space? One possibility is to consider this data as similar to an "uncompressed MPEG" datastream. It might be possible to use hardware-specific time-stamps to synchronise the data. Separate ALSA and V4L2 nodes are inherently problematic since they do not keep the audio and video together. Another possibility is to offer two APIs: for professional applications, where data is perfectly synchronised, and separate audio and video for "human consumption." A format for professional applications were VBI/HBI/Video is all captured in one big frame is OK (provided an open source lib is created to parse it). The proposal for using a multiplanar API where the audio is in a separate plane ran into opposition: the alsa devs should be asked whether it is possible to return the audio data in such a way that it can be exactly associated with the corresponding video buffer. This also requires that the audio can be variable length since for NTSC the size of the audio data will alternate between frames. If this is possible, then the alsa API can be used and things can still remain in sync. UPDATE: Mauro and Hans had some discussions with Takashi (ALSA Maintainer), in order to have some shed about the possibilities. We're planning to discuss more on this Friday's lunch. 5. Paul Walmsley on behalf of Bryan Wu: LED / flash API integration Currently functionality is accessible from two APIs: LED and flash. A proposal is presented to put V4L2 flash subdev on top of LED core. It would be better not to go via the LED trigger layer. - One to one mapping between LED chips and V4L2 sub-devices. This probably means the LED flash driver needs to register the sub-device. - Two interfaces: should the sysfs interface be disabled if an application chose the flash mode on V4L2 API? The flash driver has no knowledge of streaming state or capturing frames. - Same requirements as for the LED API as the V4L2 flash API needs (LED flashes only). - Currently there's no V4L2 API to put a sensor into a flash mode to automatically trigger a flash pin. This would have to be handled. It is also possible, that a V4L2 application uses a sensor, but only puts it into a "flash mode" (single-shot mode) for several frames. The LED might therefore have to be made busy always when the sensor is used. Situations, where an LED is busy should already happen, e.g. if one application wants to use it for a torch and another one for notifications. Results: a. There seemed to be broad consensus that there's no need to use the ledtrig-camera.c layer. Instead it's probably better for the v4l2 flash subdev code to call directly into the LED framework core. b. How to register LED devices initially? How to associate LED chips to v4l2? In particular, how should this be modeled in DT data? There needs to be some link to connect sensors to flash chips in the DT data. The media controller API has the notion of groups; perhaps that can be expressed in DT with phandles. c. How should hardware triggers be implemented? Some sensor hardware has the ability to automatically fire the flash when the sensor activates, without the involvement of any Linux-side software (beyond the initial configuration of the feature). There needs to be some knowledge in the LED core that can enable this. d. Several concerns over races and multiple users: should the sysfs LED interface always be exposed, or not? For example, if a sysfs request comes in to the LED core while a hardware capture is occurring, should it return -EBUSY? Sakari felt that it was better to arbitrate this in userspace, since it was a matter of policy, while some others wanted it to be handled in the kernel. There was a debate and divergence of opinion on this. If the kernel will be responsible for this mutal exclusion, then it should also be possible to disable the use of other LED triggers once v4l2 is using it. 6. Mauro Chehab: DVB / V4L2 / MC integration Mauro wants to extend Media Controller/Subdevice functionality to DVB, in order to allow setting a complete pipeline that would cover V4L2/ALSA/DVB/DRM as a hole. This is needed by Consumers Electronics producs like digital TV and STB. Problem: multiple subsystems might be using the MC API with or without connections between them. How to set such a topology up? Possibilities: a. To have just one global media controller device, not all graphs inside it would need to be connectable. b. The first user creates an MC device. Any further users detect that their MC device is already present and will reuse it c. a separate driver creates an MC device instance, e.g. a driver for links between subsystems One problem to be discussed with DT people is how to express the hardware connections between the different subsystems. This information should be provided by the open firmware, in order for the media controller to create the proper bus links internally on their graphs. 7. Hans Verkuil: try_fmt compliance Problem: VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but, in practice, video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Also, some applications rely on the -EINVAL error code, with is wrong, as not all drivers currently returns it: some just return 0, and fills the pixelformat data with a supported format. Resolution: fix applications, schedule driver change to comply with the spec and not return -EINVAL for an unsupported pixel format. New drivers should never return -EINVAL and should comply to the spec instead. 8. Hans Verkuil: V4L2 support libraries - libv4l2util library does not offer a public API and has a funny name. To be renamed as libv4l2misc. - create a new libv4l2misc (a libv4l2util already exists) where all the various useful bits of functionality can be placed. Posted publicly to properly discuss the API. - Doxygen to be used to document the library API. 9. Hans Verkuil: Interaction between selection API, ENUM_FRAMESIZES and S_FMT - Hans proposed to add G/S_FRAMESIZE ioctls in order to make clearer about what size is being changed, on scenarios that have both scaling and cropping. - E.g. UVC webcams have an interface to specify the image size. The camera achieves that using cropping and scaling, but the driver does not have information on which one. - sn9c102 does some weird non-standard things. This driver were obsoleted by gspca (except for a few really old devices that gspca developers were unable to use). This driver were removed on Fedora several versions ago, and nobody ever complained about it. So, it should be moved to staging and removed in a few kernel cycles' time. - Most of the developers didn't buy the idea of creating a new pair of ioctls. - Hans will write a new proposal on the topic that, instead of adding new ioctls (G/S_FRAMESIZE),it will add a new selection target and try to avoid the use of flags (in opposite to the current proposal). 10. Laurent Pinchart: ALSA and Media controller - ALSA folks thought of using Media controller for pipeline control but miss features: - dynamic creation/removal of MC entities; - lack of a way to store value pairs. - ALSA entities would need to be added and removed at runtime: DSP firmware decides what kind of entities exist - E.g. equalisers could be created as needed - ALSA already has APIs for that - Detailed sub-system specific data is needed: - ALSA: what kind of features the entity has - Proposal: binary blob specific to entity that contains key: value pairs --- this could be used in V4L2 as well 11. Laurent Pinchart: sharing I2C sub-device drivers between V4L2 and KMS - Dynamic assignment of devices between display and capture pipelines - The DT won't contain information on which driver should be used - Two separate drivers are currently facing such issue, but it seems that lot more will be added; - Something similar to the V4L2 sub-device is required on KMS. Without this feature, sharing sub-devices will be hard if not impossible. - We need something similar for DVB/V4L tuners, but it's much easier to push that upstream, as the media subsystem already uses MC subdevs. 12. Sakari Ailus: Multi format streams / metadata - metadata has to be separated from the image, but kept in sync with frames - some hardware is able to separate metadata automatically - Frame format descriptors to be extended - Set operation should be removed and replaced with a different solution on existing drivers - Format index to be added to struct v4l2_mbus_framefmt - No conclusion reached on how other streams are captured in the user space, requires further discussion and time - Multiple video nodes or multiplexing by the type field? - The upper 16 bits in the type field could be used to index the buffer queue - No need to split the field; macros can be used to access different parts of it - Stealing bytes from struct v4l2_format may have issues: the struct is not packed, and at least the alignment would need to be verified on supported archs - Multiple buffer types are not supported by videobuf2 currently with the exception of memory-to-memory devices and capture/overlay - Changes could be required in videobuf2 - Events could be used to pass the metadata to the user space - Extended events required - Does not help with sensors that produce multiple images of the same frame - Causes two copies of the data: one to the event buffer, the other to the user space when the event is dequeued - Unwanted especially on embedded systems where copying costs: there's a measurable effect from e.g. copying 4 kiB 60 times per second from two sensors - DQBUF takes the buffer type as an argument, and the user would need to try different queues once select(2) returns - Multi-format buffers are not an answer for metadata since metadata must be in the user space as fast as possible - Multi-format buffers could be implemented by extending v4l2_format - The name of the IOCTLs taking that as an argument will stay the same, as well as the IOCTL number - The old definition is still kept around for binary compatibility - The UVC 1.5 might be a use case for multi format streams, too - The SDI has multiple streams, too, but could work on multi-format buffers - The user shouldn't have to choose whether to use multi-format buffers or multiple video nodes - Painful for drivers to handle, duplicate code in drivers - Drivers must only implement one - No consensus was reached. Sakari has to update the RFC to include all three topics 13: Hugues Fruchet: Video codecs - There's a need to parse bitstream fields for those codecs, but that requires complex code (10K lines). Moving it to kernel could make it unstable, as it is harsh to write those parsers without any risk of causing crashes. - It seems to be better to put those parsers inside libv4l, using an open source license. Results: - Drivers that require proprietary user space components should stay out of mainline - Multi-format buffers could be useful here - The hardware/firmware needs a lot of data extracted from the bitstream next to the bitstream itself. This is a custom format, so it is OK to add a new pixelformat for each of those formats. Such complex parsing should be done in userspace in libv4l2. - If very little parsing is required (MPEG), then that can be done in the kernel instead. - Recommendation is to start simple with e.g. just an MPEG implementation. -- Cheers, Mauro -- Cheers, Mauro

2 2

Fw: Draft summary of v4l/dvb mini-summit
by Mauro Carvalho Chehab 28 Oct '13

28 Oct '13

Hi Takashi, As I commented to you earlier today, we had some discussions related to ALSA and media (see topics 4 an 10 below). It would be great if we could find some time during the rest of this week for an informal discussion for the interested parties. What do you think? Regards, Mauro Forwarded message: Date: Wed, 23 Oct 2013 21:17:17 +0000 From: Hans Verkuil <hverkuil(a)xs4all.nl> To: media-workshop(a)linuxtv.org Cc: Paul Walmsley <paul(a)pwsan.com> Subject: [media-workshop] Draft summary of v4l/dvb mini-summit Hi all, These are the draft notes from today's media mini-summit. These notes combine the etherpad notes with my own notes. I'm sure I've forgotten somethings, so just reply to add your improvements. I've made those presentations that I have available here: http://hverkuil.home.xs4all.nl/presentations/summit-2013/ Regards, Hans Notes on the Media summit 2013-10-23 List of Participants: Ricardo Ribalda - Qtechnology Hans Verkuil - Cisco Sakari Ailus - Intel Mauro Carvalho Chehab - Samsung Kieran Kunhya - Open Broadcast Systems Sylwester Nawrocki - Samsung Guennadi Liakhovetski Benjamin Gaignard - ST Microelectronics Hugues Fruchet - ST Microelectronics Andrzej Hajda - Samsung Peter Senna Tschudin - Coccinelle? Hans de Goede - Red Hat Michael Krufky - Samsung Paul Walmsley - Nvidia Laurent Pinchart - Ideas on board 1. Mauro Chehab: is the submaintainer arrangement working? General consensus is that it is working. Hans pointed that the commits ML is not working. Also patches for the rcX kernel aren't always picked up quickly enough. Mauro says this is due to his move to Samsung which took a lot of his time. This should improve. 2. Ricardo Ribalda Delgado: Multiple selections For a more flexible cropping selection a new extended rectangle struct has to be added with possibly negative top and left offsets. This would allow to explore capabilities of industry grade sensors, that can be configured to crop multiple rectangles from the input area. Sylwester: an alternative approach would be to use indexed windows. The issue of atomicity should also be addressed. Ricardo is proposing to add the capability of adding more than one selection areas - No objections so far. - Helper functions to detect if rectangles overlap or are contained on other rectangles - Documentation changes for V4L2 subdev selection API. Special care needs to be taken for too few or too many rectangles. In both cases the rectangles field should be set to the actual number of allowed rectangles. In the too few case an error should be returned, in the too many case the extra rects should just be ignored. Mauro thinks that it will be useful to see if the width/height fields in v4l2_rect can be changes to u32 instead of s32: will require code review of drivers. 3. Hans Verkuil: colorspaces - Colorspace: limited/full range Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp Proposal accepted, although the "_SRGB_LIM" name might need to be improved, although no alternatives were given. ALso mention in the docs that S_FMT can overwrite the colorspace for output devices if the selected one isn't supported. 4. Kieran Kunhya: SDI The biggest problem is separate audio and video file descriptors. Audio data is transfered in the video blanking areas, where should they be separated - in the kernel, using multiple planes, or in the user-space? One possibility is to consider this data as similar to an "uncompressed MPEG" datastream. It might be possible to use hardware-specific time-stamps to synchronise the data. Separate ALSA and V4L2 nodes are inherently problematic since they do not keep the audio and video together. Another possibility is to offer two APIs: for professional applications, where data is perfectly synchronised, and separate audio and video for "human consumption." A format for professional applications were VBI/HBI/Video is all captured in one big frame is OK (provided an open source lib is created to parse it). The proposal for using a multiplanar API where the audio is in a separate plane ran into opposition: the alsa devs should be asked whether it is possible to return the audio data in such a way that it can be exactly associated with the corresponding video buffer. This also requires that the audio can be variable length since for NTSC the size of the audio data will alternate between frames. If this is possible, then the alsa API can be used and things can still remain in sync. 5. Paul Walmsley on behalf of Bryan Wu: LED / flash API integration Currently functionality is accessible from two APIs: LED and flash. A proposal is presented to put V4L2 flash subdev on top of LED core. It would be better not to go via the LED trigger layer. - One to one mapping between LED chips and V4L2 sub-devices. This probably means the LED flash driver needs to register the sub-device. - Two interfaces: should the sysfs interface be disabled if an application chose the flash mode on V4L2 API? The flash driver has no knowledge of streaming state or capturing frames. - Same requirements as for the LED API as the V4L2 flash API needs (LED flashes only). - Currently there's no V4L2 API to put a sensor into a flash mode to automatically trigger a flash pin. This would have to be handled. It is also possible, that a V4L2 application uses a sensor, but only puts it into a "flash mode" (single-shot mode) for several frames. The LED might therefore have to be made busy always when the sensor is used. Situations, where an LED is busy should already happen, e.g. if one application wants to use it for a torch and another one for notifications. 6. Mauro Chehab: DVB / V4L2 / MC integration Problem: multiple subsystems might be using the MC API with or without connections between them. How to set such a topology up? Possibilities: - global media controller device, not all entities have to be connectable - first user creates an MC device, any further users detect, that their MC device is already present and also begin using it - a separate driver creates an MC device instance, e.g. a driver for links between subsystems 7. Hans Verkuil: try_fmt compliance Problem: VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. Resolution: fix applications, schedule driver change to comply with the spec and not return -EINVAL for an unsupported pixel format. New drivers should never return -EINVAL and should comply to the spec instead. 8. Hans Verkuil: V4L2 support libraries - libv4l2util library does not offer a public API and has a funny name. To be renamed as libv4l2misc. - create a new libv4l2misc (a libv4l2util already exists) where all the various useful bits of functionality can be placed. Posted publicly to properly discuss the API. - Doxygen to be used to document the library API. 9. Hans Verkuil: Interaction between selection API, ENUM_FRAMESIZES and S_FMT - E.g. UVC webcams have an interface to specify the image size. The camera achieves that using cropping and scaling, but the driver does not have information on which one. - sn9c102 to be moved to staging and removed in a few kernel cycles' time. - Hans to write a new proposal on the topic. Instead of adding new ioctls (G/S_FRAMESIZE) add a new selection target and try to avoid the use of flags (as introduced in the current proposal). 10. Laurent Pinchart: ALSA and Media controller - ALSA folks thought of using Media controller for pipeline control but miss features - ALSA entities would need to be added and removed at runtime: DSP firmware decides what kind of entities exist - E.g. equalisers could be created as needed - ALSA already has APIs for that - Detailed information which is sub-system specific - ALSA: what kind of features the entity has - Proposal: binary blob specific to entity that contains key: value pairs --- this could be used in V4L2 as well 11. Laurent Pinchart: sharing I2C sub-device drivers between V4L2 and KMS - Dynamic assignment of devices between display and capture pipelines - The DT won't contain information on which driver should be used - Two separate drivers are an issue - Something similar to the V4L2 sub-device required on KMS. Without this feature sharing sub-devices will be hard if not impossible. - We need something similar for DVB/V4L tuners, but it's much easier to push that through since we have control of the media subsystem. 12. Sakari Ailus: Multi format streams / metadata - metadata has to be separated from the image, but kept in sync with frames - some hardware is able to separate metadata automatically - Frame format descriptors to be extended - Set operation should be removed and replaced with a different solution on existing drivers - Format index to be added to struct v4l2_mbus_framefmt - No conclusion reached on how other streams are captured in the user space, requires further discussion and time - Multiple video nodes or multiplexing by the type field? - The upper 16 bits in the type field could be used to index the buffer queue - No need to split the field; macros can be used to access different parts of it - Stealing bytes from struct v4l2_format may have issues: the struct is not packed, and at least the alignment would need to be verified on supported archs - Multiple buffer types are not supported by videobuf2 currently with the exception of memory-to-memory devices and capture/overlay - Changes could be required in videobuf2 - Events could be used to pass the metadata to the user space - Extended events required - Does not help with sensors that produce multiple images of the same frame - Causes two copies of the data: one to the event buffer, the other to the user space when the event is dequeued - Unwanted especially on embedded systems where copying costs: there's a measurable effect from e.g. copying 4 kiB 60 times per second from two sensors - DQBUF takes the buffer type as an argument, and the user would need to try different queues once select(2) returns - Multi-format buffers are not an answer for metadata since metadata must be in the user space as fast as possible - Multi-format buffers could be implemented by extending v4l2_format - The name of the IOCTLs taking that as an argument will stay the same, as well as the IOCTL number - The old definition is still kept around for binary compatibility - The UVC 1.5 might be a use case for multi format streams, too - The SDI has multiple streams, too, but could work on multi-format buffers - The user shouldn't have to choose whether to use multi-format buffers or multiple video nodes - Painful for drivers to handle, duplicate code in drivers - Drivers must only implement one - Sakari to update the RFC to include all three topics 13: Hugues Fruchet: Video codecs - Drivers that require proprietary user space components will stay out of mainline - Multi-format buffers could be useful here - The hardware/firmware needs a lot of data extracted from the bitstream next to the bitstream itself. This is a custom format, so it is OK to add a new pixelformat for each of those formats. The parsing should be done in userspace in libv4l2. - If very little parsing is required (MPEG), then that can be done in the kernel instead. - Recommendation is to start simple with e.g. just an MPEG implementation.

2 1

Draft summary of v4l/dvb mini-summit
by Hans Verkuil 28 Oct '13

28 Oct '13

Hi all, These are the draft notes from today's media mini-summit. These notes combine the etherpad notes with my own notes. I'm sure I've forgotten somethings, so just reply to add your improvements. I've made those presentations that I have available here: http://hverkuil.home.xs4all.nl/presentations/summit-2013/ Regards, Hans Notes on the Media summit 2013-10-23 List of Participants: Ricardo Ribalda - Qtechnology Hans Verkuil - Cisco Sakari Ailus - Intel Mauro Carvalho Chehab - Samsung Kieran Kunhya - Open Broadcast Systems Sylwester Nawrocki - Samsung Guennadi Liakhovetski Benjamin Gaignard - ST Microelectronics Hugues Fruchet - ST Microelectronics Andrzej Hajda - Samsung Peter Senna Tschudin - Coccinelle? Hans de Goede - Red Hat Michael Krufky - Samsung Paul Walmsley - Nvidia Laurent Pinchart - Ideas on board 1. Mauro Chehab: is the submaintainer arrangement working? General consensus is that it is working. Hans pointed that the commits ML is not working. Also patches for the rcX kernel aren't always picked up quickly enough. Mauro says this is due to his move to Samsung which took a lot of his time. This should improve. 2. Ricardo Ribalda Delgado: Multiple selections For a more flexible cropping selection a new extended rectangle struct has to be added with possibly negative top and left offsets. This would allow to explore capabilities of industry grade sensors, that can be configured to crop multiple rectangles from the input area. Sylwester: an alternative approach would be to use indexed windows. The issue of atomicity should also be addressed. Ricardo is proposing to add the capability of adding more than one selection areas - No objections so far. - Helper functions to detect if rectangles overlap or are contained on other rectangles - Documentation changes for V4L2 subdev selection API. Special care needs to be taken for too few or too many rectangles. In both cases the rectangles field should be set to the actual number of allowed rectangles. In the too few case an error should be returned, in the too many case the extra rects should just be ignored. Mauro thinks that it will be useful to see if the width/height fields in v4l2_rect can be changes to u32 instead of s32: will require code review of drivers. 3. Hans Verkuil: colorspaces - Colorspace: limited/full range Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp Proposal accepted, although the "_SRGB_LIM" name might need to be improved, although no alternatives were given. ALso mention in the docs that S_FMT can overwrite the colorspace for output devices if the selected one isn't supported. 4. Kieran Kunhya: SDI The biggest problem is separate audio and video file descriptors. Audio data is transfered in the video blanking areas, where should they be separated - in the kernel, using multiple planes, or in the user-space? One possibility is to consider this data as similar to an "uncompressed MPEG" datastream. It might be possible to use hardware-specific time-stamps to synchronise the data. Separate ALSA and V4L2 nodes are inherently problematic since they do not keep the audio and video together. Another possibility is to offer two APIs: for professional applications, where data is perfectly synchronised, and separate audio and video for "human consumption." A format for professional applications were VBI/HBI/Video is all captured in one big frame is OK (provided an open source lib is created to parse it). The proposal for using a multiplanar API where the audio is in a separate plane ran into opposition: the alsa devs should be asked whether it is possible to return the audio data in such a way that it can be exactly associated with the corresponding video buffer. This also requires that the audio can be variable length since for NTSC the size of the audio data will alternate between frames. If this is possible, then the alsa API can be used and things can still remain in sync. 5. Paul Walmsley on behalf of Bryan Wu: LED / flash API integration Currently functionality is accessible from two APIs: LED and flash. A proposal is presented to put V4L2 flash subdev on top of LED core. It would be better not to go via the LED trigger layer. - One to one mapping between LED chips and V4L2 sub-devices. This probably means the LED flash driver needs to register the sub-device. - Two interfaces: should the sysfs interface be disabled if an application chose the flash mode on V4L2 API? The flash driver has no knowledge of streaming state or capturing frames. - Same requirements as for the LED API as the V4L2 flash API needs (LED flashes only). - Currently there's no V4L2 API to put a sensor into a flash mode to automatically trigger a flash pin. This would have to be handled. It is also possible, that a V4L2 application uses a sensor, but only puts it into a "flash mode" (single-shot mode) for several frames. The LED might therefore have to be made busy always when the sensor is used. Situations, where an LED is busy should already happen, e.g. if one application wants to use it for a torch and another one for notifications. 6. Mauro Chehab: DVB / V4L2 / MC integration Problem: multiple subsystems might be using the MC API with or without connections between them. How to set such a topology up? Possibilities: - global media controller device, not all entities have to be connectable - first user creates an MC device, any further users detect, that their MC device is already present and also begin using it - a separate driver creates an MC device instance, e.g. a driver for links between subsystems 7. Hans Verkuil: try_fmt compliance Problem: VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. Resolution: fix applications, schedule driver change to comply with the spec and not return -EINVAL for an unsupported pixel format. New drivers should never return -EINVAL and should comply to the spec instead. 8. Hans Verkuil: V4L2 support libraries - libv4l2util library does not offer a public API and has a funny name. To be renamed as libv4l2misc. - create a new libv4l2misc (a libv4l2util already exists) where all the various useful bits of functionality can be placed. Posted publicly to properly discuss the API. - Doxygen to be used to document the library API. 9. Hans Verkuil: Interaction between selection API, ENUM_FRAMESIZES and S_FMT - E.g. UVC webcams have an interface to specify the image size. The camera achieves that using cropping and scaling, but the driver does not have information on which one. - sn9c102 to be moved to staging and removed in a few kernel cycles' time. - Hans to write a new proposal on the topic. Instead of adding new ioctls (G/S_FRAMESIZE) add a new selection target and try to avoid the use of flags (as introduced in the current proposal). 10. Laurent Pinchart: ALSA and Media controller - ALSA folks thought of using Media controller for pipeline control but miss features - ALSA entities would need to be added and removed at runtime: DSP firmware decides what kind of entities exist - E.g. equalisers could be created as needed - ALSA already has APIs for that - Detailed information which is sub-system specific - ALSA: what kind of features the entity has - Proposal: binary blob specific to entity that contains key: value pairs --- this could be used in V4L2 as well 11. Laurent Pinchart: sharing I2C sub-device drivers between V4L2 and KMS - Dynamic assignment of devices between display and capture pipelines - The DT won't contain information on which driver should be used - Two separate drivers are an issue - Something similar to the V4L2 sub-device required on KMS. Without this feature sharing sub-devices will be hard if not impossible. - We need something similar for DVB/V4L tuners, but it's much easier to push that through since we have control of the media subsystem. 12. Sakari Ailus: Multi format streams / metadata - metadata has to be separated from the image, but kept in sync with frames - some hardware is able to separate metadata automatically - Frame format descriptors to be extended - Set operation should be removed and replaced with a different solution on existing drivers - Format index to be added to struct v4l2_mbus_framefmt - No conclusion reached on how other streams are captured in the user space, requires further discussion and time - Multiple video nodes or multiplexing by the type field? - The upper 16 bits in the type field could be used to index the buffer queue - No need to split the field; macros can be used to access different parts of it - Stealing bytes from struct v4l2_format may have issues: the struct is not packed, and at least the alignment would need to be verified on supported archs - Multiple buffer types are not supported by videobuf2 currently with the exception of memory-to-memory devices and capture/overlay - Changes could be required in videobuf2 - Events could be used to pass the metadata to the user space - Extended events required - Does not help with sensors that produce multiple images of the same frame - Causes two copies of the data: one to the event buffer, the other to the user space when the event is dequeued - Unwanted especially on embedded systems where copying costs: there's a measurable effect from e.g. copying 4 kiB 60 times per second from two sensors - DQBUF takes the buffer type as an argument, and the user would need to try different queues once select(2) returns - Multi-format buffers are not an answer for metadata since metadata must be in the user space as fast as possible - Multi-format buffers could be implemented by extending v4l2_format - The name of the IOCTLs taking that as an argument will stay the same, as well as the IOCTL number - The old definition is still kept around for binary compatibility - The UVC 1.5 might be a use case for multi format streams, too - The SDI has multiple streams, too, but could work on multi-format buffers - The user shouldn't have to choose whether to use multi-format buffers or multiple video nodes - Painful for drivers to handle, duplicate code in drivers - Drivers must only implement one - Sakari to update the RFC to include all three topics 13: Hugues Fruchet: Video codecs - Drivers that require proprietary user space components will stay out of mainline - Multi-format buffers could be useful here - The hardware/firmware needs a lot of data extracted from the bitstream next to the bitstream itself. This is a custom format, so it is OK to add a new pixelformat for each of those formats. The parsing should be done in userspace in libv4l2. - If very little parsing is required (MPEG), then that can be done in the kernel instead. - Recommendation is to start simple with e.g. just an MPEG implementation.

5 4

Lunch on Friday
by Mauro Carvalho Chehab 25 Oct '13

25 Oct '13

As I talked with several developers at yesterday's KS dinner, most of the developers will only departure on Sat/Sunday. So, I'm thinking on having a media developers dinner this night. This would be nice for us to meet together on a more informal environment, have fun together and perhaps discussing some other random subjects. I'm also inviting Takashi (ALSA maintainer), as we may use this opportunity for some informal discussions on the joint media/ALSA related topics. My suggestion is for us to meet at the Sheraton's 1st Floor lobby by 7pm. I'll talk with KS local organizers in order to get some suggestions for a place were we can sit all together at the same table. As we need to know in advance how many people will be present, could you please reply back to this e-mail, if you're planning to be there or not? Thanks! Mauro

6 9

[ANN] Final agenda for the V4L/DVB mini-summit
by Hans Verkuil 22 Oct '13

22 Oct '13

Here is the final schedule. I moved a few topics around at the request of Laurent and to make it a better fit for the break times. Please note that the LED topic has moved to a slightly later timeslot. We plan to start at 9:00 and we have the Glamis room until 18:00. The schedule is as follows: 9:00 Welcome/Introduction 9:10 Mauro Carvalho Chehab - Better integration between DVB and V4L2, including starting using the media controller API on DVB side too. 30 min 9:40 Kieran Kunhya - (3G/HD-)SDI multiplexed raw format This is a professional interface used in broadcasting and CCTV. The most serious issue is that many vendors provide Linux drivers with V4L2 and ALSA - which is not acceptable for maintaining lipsync, let alone maintaining the exact relationship between audio samples and video that SDI provides. Some other issues are mentioned here: https://wiki.videolan.org/SDI_API/ The wiki page has a very loose proposal for an API, though perhaps the per-line idea is ambitious at this stage. Field or frame capture is more realistic. 10:00 Break 10:30 Ricardo Ribalda Delgado, Sylwester Nawrocki - Support for multiple rectangle cropping See thread: http://www.spinics.net/lists/linux-media/msg67824.html 10:50 Sylwester Nawrocki - LED flash support 11:10 Hans Verkuil - Colorspace: limited/full range Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp 11:20 Hans Verkuil - VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. We need to decide how to adjust the spec. I propose to just say that some drivers will map it silently and others will return -EINVAL and that you don't know what a driver will do. Also specify that an unsupported pixelformat is the only reason why TRY_FMT might return -EINVAL. Alternatively we might want to specify explicitly that EINVAL should be returned for video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all others. 11:40 Hans Verkuil - Decide on how v4l2 support libraries should be organized. There is code for handling raw-to-sliced VBI decoding, ALSA looping, finding associated video/alsa nodes and for TV frequency tables. We should decide how that should be organized into libraries and how they should be documented. The first two aren't libraries at the moment, but I think they should be. The last two are libraries but they aren't installed. Some work is also being done on an improved version of the 'associating nodes' library that uses the MC if available. 12:00 Lunch 13:00 Hans Verkuil - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See this thread for all the nasty details: http://www.spinics.net/lists/linux-media/msg65137.html Also see my draft presentation (link above) for more info and proposals. 14:00 Laurent Pinchart - Status update regarding the media controller API usage on ALSA (Mauro likes to know) and sharing i2c transmitters between V4L2 and DRM (Hans V. likes to know). 14:10 Sakari Ailus - Multi-format frames and metadata. Support would be needed on video nodes and V4L2 subdev nodes. I'll prepare the RFC for the former; the latter has an RFC here: http://www.spinics.net/lists/linux-media/msg67295.html 15:00 Break 15:30 Hugues Fruchet - How to handle codecs where part of the processing is done in HW and part in SW? 17:30 End Note: these times are tentative. Some topics will take less time, some will take more time. Regards, Hans

4 3

V2: Agenda for the Edinburgh mini-summit
by Hans Verkuil 22 Oct '13

22 Oct '13

I have collected all the ideas up to now in a V2 of the agenda. The items are grouped by the person(s) that suggested them. As done in the past those who suggested a topic and who will attend the mini-summit are expected to prepare for it, perhaps making a (very) small presentation if necessary. Hans Verkuil: - Discuss ideas/use-cases for a property-based API. An initial discussion appeared in this thread: http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/6… - What is needed to share i2c video transmitters between drm and v4l? Hopefully we will know more after the upcoming LPC. - Decide on how v4l2 support libraries should be organized. There is code for handling raw-to-sliced VBI decoding, ALSA looping, finding associated video/alsa nodes and for TV frequency tables. We should decide how that should be organized into libraries and how they should be documented. The first two aren't libraries at the moment, but I think they should be. The last two are libraries but they aren't installed. Some work is also being done on an improved version of the 'associating nodes' library that uses the MC if available. - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See this thread for all the nasty details: http://www.spinics.net/lists/linux-media/msg65137.html - VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. We need to decide how to adjust the spec. I propose to just say that some drivers will map it silently and others will return -EINVAL and that you don't know what a driver will do. Also specify that an unsupported pixelformat is the only reason why TRY_FMT might return -EINVAL. Alternatively we might want to specify explicitly that EINVAL should be returned for video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all others. Mauro Carvalho Chehab: - Better integration between DVB and V4L2, including starting using the media controller API on DVB side too. - Get the status about the media controller API usage on ALSA. Oliver Schinagl, Benjamin Gaignard, Hugues Fruchet, Laurent Pinchart, Pawel Osciak: - How to handle codecs where part of the processing is done in HW and part in SW? Sakari Ailus: - Multi-format frames and metadata. Support would be needed on video nodes and V4L2 subdev nodes. I'll prepare the RFC for the former; the latter has an RFC here: http://www.spinics.net/lists/linux-media/msg67295.html Ricardo Ribalda Delgado, Sylwester Nawrocki: - Support for multiple rectangle cropping See thread: http://www.spinics.net/lists/linux-media/msg67824.html Feel free to add suggestions to this list. As it stands I don't think it will be possible to handle it all in one day. In particular the codec problem as mentioned by Oliver et al needs a lot of time. Should we set aside a separate day for just this? October 21 or 22 would work for me. I would really like to get some feedback on this. If we decide to go for a second day for this topic, then I can see if I can get a room. It looks like there is a lot of interest in getting this sorted, so brainstorming for a day might be quite useful. Note: my email availability will be limited in the next two weeks, especially next week, as I am abroad (LinuxCon and LPC). Regards, Hans _______________________________________________ media-workshop mailing list media-workshop(a)linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/media-workshop

12 33

[ANN] Final (?) version of the agenda
by Hans Verkuil 19 Oct '13

19 Oct '13

The ARM kernel agenda came in, and Laurent expects to be available after 12:00 (perhaps even earlier). So I moved all the topics that do not require his presence to the morning. Laurent, please check if this works for you. We plan to start at 9:00 and we have the room until 17:30 or 18:00 (to be decided). There are breaks from 10:00-10:30, 12:00-13:00, 15:00-15:30. So we have realistically at best around 6 1/2 hours for actual discussions. And that's filled up completely. Regards, Hans Topics for the V4L2/DVB mini-summit: Mauro Carvalho Chehab: - Better integration between DVB and V4L2, including starting using the media controller API on DVB side too. 30 min Ricardo Ribalda Delgado, Sylwester Nawrocki: - Support for multiple rectangle cropping See thread: http://www.spinics.net/lists/linux-media/msg67824.html 20 min Sylwester Nawrocki - LED flash support 20 min Kieran Kunhya - (3G/HD-)SDI multiplexed raw format This is a professional interface used in broadcasting and CCTV. The most serious issue is that many vendors provide Linux drivers with V4L2 and ALSA - which is not acceptable for maintaining lipsync, let alone maintaining the exact relationship between audio samples and video that SDI provides. Some other issues are mentioned here: https://wiki.videolan.org/SDI_API/ The wiki page has a very loose proposal for an API, though perhaps the per-line idea is ambitious at this stage. Field or frame capture is more realistic. 20 min Hans Verkuil: Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp - VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. We need to decide how to adjust the spec. I propose to just say that some drivers will map it silently and others will return -EINVAL and that you don't know what a driver will do. Also specify that an unsupported pixelformat is the only reason why TRY_FMT might return -EINVAL. Alternatively we might want to specify explicitly that EINVAL should be returned for video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all others. 20 min - Colorspace: limited/full range 10 min - (LP?) Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See this thread for all the nasty details: http://www.spinics.net/lists/linux-media/msg65137.html Also see my draft presentation (link above) for more info and proposals. 60 min - (LP) Decide on how v4l2 support libraries should be organized. There is code for handling raw-to-sliced VBI decoding, ALSA looping, finding associated video/alsa nodes and for TV frequency tables. We should decide how that should be organized into libraries and how they should be documented. The first two aren't libraries at the moment, but I think they should be. The last two are libraries but they aren't installed. Some work is also being done on an improved version of the 'associating nodes' library that uses the MC if available. 30 min Laurent Pinchart: - Status update regarding the media controller API usage on ALSA (Mauro likes to know) and sharing i2c transmitters between V4L2 and DRM (Hans V. likes to know). 10 min Hugues Fruchet: - (LP) How to handle codecs where part of the processing is done in HW and part in SW? 2 hours Sakari Ailus: - (LP) Multi-format frames and metadata. Support would be needed on video nodes and V4L2 subdev nodes. I'll prepare the RFC for the former; the latter has an RFC here: http://www.spinics.net/lists/linux-media/msg67295.html 50 min In principle the agenda is now closed for new topics. Regards, Hans

2 2

Kernel Summit Media Mini-summit attendees on Oct 23 in Edinburgh
by Mauro Carvalho Chehab 11 Oct '13

11 Oct '13

Hi, I'm trying to consolidate the list of interested people on participating at this year's the media mini-summit. From what I got from the discussions, we have, so far: Benjamin Gaignard <benjamin.gaignard(a)linaro.org> Guennadi Liakhovetski <g.liakhovetski(a)gmx.de> Hans Verkuil <hverkuil(a)xs4all.nl> Hugues FRUCHET <hugues.fruchet(a)st.com> Laurent Pinchart <laurent.pinchart(a)ideasonboard.com> Mauro Carvalho Chehab <m.chehab(a)samsung.com> Michael Krufky <mkrufky(a)kernellabs.com> Oliver Schinagl <oliver+list(a)schinagl.nl> Pawel Osciak <posciak(a)chromium.org> Peter Senna Tschudin <peter.senna(a)gmail.com> Ricardo Ribalda Delgado <ricardo.ribalda(a)gmail.com> Sakari Ailus <sakari.ailus(a)iki.fi> Please let me know if I'm missing someone, or if one of the above won't be able to go to the meeting, as my plan is to send the invitations tomorrow. Thank you! Mauro

9 12

V3: Agenda for the Edinburgh mini-summit
by Hans Verkuil 07 Oct '13

07 Oct '13

Hi all, I have collected all the ideas up to now in a V3 of the agenda and included time allocated for each topic. The items are grouped by the person(s) that suggested them. As done in the past those who suggested a topic and who will attend the mini-summit are *EXPECTED TO PREPARE FOR IT*. If you present a topic, then make a (small?) presentation if applicable. Those attending should read at least those documents from the presenters that are of special interest to them. We have a full schedule, so let's not waste time. If you have a presentation ready before hand, then just post a link to your presentation so people can read it before the summit. Please check the time I have allocated for you and let me know if you think it is too short or too long. We plan to start at 9:00 and we have the room until 17:30 or 18:00 (to be decided). There are breaks from 10:00-10:30, 12:00-13:00, 15:00-15:30. So we have realistically at best around 6 1/2 hours for actual discussions. And that's filled up completely. The order of the topics is still up in the air. For those marked with 'LP' I would like to have Laurent Pinchart present, but since he is also attending the ARM summit we do not yet know when he will be available. Once I know that I'll shuffle things around to accommodate Laurent. Laurent, please double check if I marked topics with LP where I shouldn't or vice versa. I have one topic with a 'LP?' prefix where I wasn't really sure. Topics for the V4L2/DVB mini-summit: Hans Verkuil: Draft presentation for all these topics: http://hverkuil.home.xs4all.nl/presentations/summit2013.odp - (LP) Decide on how v4l2 support libraries should be organized. There is code for handling raw-to-sliced VBI decoding, ALSA looping, finding associated video/alsa nodes and for TV frequency tables. We should decide how that should be organized into libraries and how they should be documented. The first two aren't libraries at the moment, but I think they should be. The last two are libraries but they aren't installed. Some work is also being done on an improved version of the 'associating nodes' library that uses the MC if available. 30 min - (LP?) Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See this thread for all the nasty details: http://www.spinics.net/lists/linux-media/msg65137.html Also see my draft presentation (link above) for more info and proposals. 60 min - VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided, but in practice video capture board tend to do that, while webcam drivers tend to map it silently to a valid pixelformat. Some applications rely on the -EINVAL error code. We need to decide how to adjust the spec. I propose to just say that some drivers will map it silently and others will return -EINVAL and that you don't know what a driver will do. Also specify that an unsupported pixelformat is the only reason why TRY_FMT might return -EINVAL. Alternatively we might want to specify explicitly that EINVAL should be returned for video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all others. 20 min - Colorspace: limited/full range 10 min Mauro Carvalho Chehab: - Better integration between DVB and V4L2, including starting using the media controller API on DVB side too. 30 min Laurent Pinchart: - Status update regarding the media controller API usage on ALSA (Mauro likes to know) and sharing i2c transmitters between V4L2 and DRM (Hans V. likes to know). 10 min Hugues Fruchet: - (LP) How to handle codecs where part of the processing is done in HW and part in SW? 2 hours Sakari Ailus: - (LP) Multi-format frames and metadata. Support would be needed on video nodes and V4L2 subdev nodes. I'll prepare the RFC for the former; the latter has an RFC here: http://www.spinics.net/lists/linux-media/msg67295.html 50 min Ricardo Ribalda Delgado, Sylwester Nawrocki: - Support for multiple rectangle cropping See thread: http://www.spinics.net/lists/linux-media/msg67824.html 20 min Sylwester Nawrocki - LED flash support 20 min Kieran Kunhya - (3G/HD-)SDI multiplexed raw format This is a professional interface used in broadcasting and CCTV. The most serious issue is that many vendors provide Linux drivers with V4L2 and ALSA - which is not acceptable for maintaining lipsync, let alone maintaining the exact relationship between audio samples and video that SDI provides. Some other issues are mentioned here: https://wiki.videolan.org/SDI_API/ The wiki page has a very loose proposal for an API, though perhaps the per-line idea is ambitious at this stage. Field or frame capture is more realistic. 20 min In principle the agenda is now closed for new topics. Regards, Hans

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

media-workshop October 2013