Hello,
I has similar problem with jerky video on vdpau. After some investigation I hope that I found solution. I was try increase volume of output surfaces.
In file xine-lib-1.2/src/video_out/video_out_vdpau.c
change line: #define NOUTPUTSURFACE 2
I am trying value 6 and it is perfect on nVidia G100 or G130 graphic, and almost perfect on 8300/shared
I can confirm it on 3 machines, and I have confirmation from 2 other people.
Jiri