B) Writing PS/AVPES
IMHO you simply can't and shouldn't do that. Writing anything else except TS
to the demux doesn't make any sense to me. What do you expect the demux to
do with PES? The only job of the demux is to route / demultiplex / convert
TS into someting else (different queues, PES, sections, ...)
If you write non-TS data the only reason could be playback. So it is a lot
more straight forward to write that data directly to the decoder. And that
is excatly what /dev/dvb/adapter0/{audio0,video0} are for. Lets keep the
demux TS only.
What about DVD Program Streams? When your hardware can handle these
streams directly you probably don't want to demultiplex the stream in
userspace...