Приветствую, VDR user
would you show please your modeline in xorg.conf please do you use 1080i or 1080p in it ?
I don't know if this will help you any but I use xine-vdpau for both sdtv & hdtv. I haven't changed anything in .xine/config so the buffer settings and so on are whatever they are as default. My box runs debian with vdr-1.7.7 and xine-0.9.2 in case that matters. Currently using xine-vdpau r266 released earlier today and nvidia driver 185.18.14 with an 8400GS (with spdif connected to it). I don't think it matters but I'm outputting to a tv using dvi->hdmi cable.
I start xine with: xine -A alsa -V vdpau --post vdr_video --post vdr_audio vdr://tmp/vdr-xine/stream#demux:mpeg_pes --verbose=2 --fullscreen --no-gui --no-mouse --deinterlace --no-logo --no-splash