[linux-dvb] Re: Alternative v4 demux API

[Holger Waechtler <holger@convergence.de>]

Michael Hunold wrote:
> Hello Holger,
>
> > I like the idea of a source based routing in Michael's approach (we've 
> > been discussing this issue before in another thread some months ago), 
> > the idea not to specify input and output device of the demux but 
> > instead specify a stream source in the decoder. This removes some 
> > ambingiuties.
>
> Yes.
>
> > If we want to remove even more ambinguities we should distinguish in 
> > the demux API between general purpose filter banks and 
> > decoder-attached filter banks.
>
> Yes, or separate this to separate devices. The latter has the advantage, 
> that adding specialized IOCTLs for example is really bound to a special 
> device.
Isn't it simpler to make these a flag in the filter specification struct?

> I'd really like to know what stuff is not covered by my approach and why 
> you think it's too high level.
It makes abstractions that don't match any hardware I know. I don't know 
any hardware that has real stream filters in the decoder units.

> > Can you please specify the term 'special recording unit' a little more 
> > in detail? -- Are you simply talking about hardware that does just 
> > DMA's a demuxed stream into a dedicated memory area and allows 
> > simultanous DMA transfers from this area to the harddisk?
>
> For example. Or the hardware automatically sets up IDE DMA transfers if 
> told so. I admit that this does not make sense for systems like Linux, 
> where you usally don't write to any drive outside of the ide layer.
>
> But if you really like to do that, having a separate recording device is 
> handy: as described above, you can add whatever IOCTLs you like, even if 
> your device driver, not just in the dvb-core.
>
> This is way more complicated if you do this through the one-in-all demux 
> approach.
It's more the question whether you want to implement raw disk accesses 
or rather want to use the filesystem layer for your video recordings, not?

> > Would it not be sufficient to add a output route into a file 
> > descriptor so that the kiobufs of the filesystem layers can get feeded 
> > using the demux DMA pointers directly (don't know whether 2.6 still 
> > uses kiobufs, maybe this has changed, but the underlying concepts are 
> > probably the same).
>
> It's indeed a problem that for a recording, all data is copied from DVB 
> to user space, just to be pushed to the IDE layer afterwards. But this 
> is a Linux problem in general.
No, the approach I described below would be a zero-copy approach. The 
demux DMA's directly into the memory area that is mapped using mmap() by 
the filesystem layer. As soon the demux layer releases the kiobuf page 
lock and and flushes the kiobuf the filesystem layer will start the DMA 
transfer to disk.

>  > Or even simpler: Make the demux output ringbuffer double-buffered and
>  > allow to relocate the backbuffer from userspace using a demux-ioctl. A
>  > userspace program would now mmap() the destination file, then relocate
>  > the backbuffer. As soon front- and backbuffers get flipped the demux
>  >  API
>  > faults this page in and then DMA's into this area. Maybe one needs
>  > more
>  > than 2 buffers, maybe a quad-buffered architecture is required to get
>  > fluent transfers.
>
> In theory, you can write to user space memory from device drivers 
> easily. In practise, most hardware has some sort of DMA limitations (for 
> example 16k aligned memory), so this is not an option unfortunately, 
> because you cannot control memory allocation for userspace.
If these limitations exist the filesystem layer should mmap() the file 
memory pages apprpriatly, not? In any case kiobufs contain entire pages, 
these are always aligned to page boundaries.

> As you pointed out, another approach is to let the user application 
> mmap() the kernel dvb ringbuffer.
>
> But I don't know if this works well with the Linux IDE / SCSI layer, 
> though, ie. if these layers can handle the mmap()ed pointers afterwards.
I have only the 2.4 reference here, maybe this has changed in 2.6. Take 
a closer look on /usr/src/linux/fs/buffers.c, function 
generic_direct_IO() and brw_kiovec(). These are the actual 
implementations of the direct_IO() callback in struct 
address_space_operations in the file system layer. This function is 
basically the bridge between inodes in the file systems and IDE/SCSI 
blocks. Here the mapped kiobufs of the inode get written to disk and new 
requested buffers get allocated.

The demux driver would have to lock the kiobuf, start the DMA transfer, 
then mark the page as dirty and flush it.

The good thing about an approach like this would be that you could use 
any filesystem you like for your video data partition and get things 
like journalling for free (if you want it and your processor has the 
headroom to handle it).

> But before thinking about this, someone should really investigate if 
> these memcpy()s are *really* a problem for system performance. Sure, 
> there is a lot of copying going on, but I guess that most of the *real* 
> performance is eaten somewhere else.
This actually depends on the platform you are talking about, not? In any 
case you want to use accelerating and power-saving hardware on embedded 
platforms where available, the CPU can probably always do better things 
than just copying bytes around. It's the same as for your harddisk - you 
don't want to use PIO if DMA is available, right? PIO is nothing else 
than copying memory byte-by-byte using the CPU to the IDE bus.

Holger



--
Info:
To unsubscribe send a mail to ecartis@linuxtv.org with "unsubscribe linux-dvb" as subject.