Hi Stefan,
I tried several of your suggestions with little luck. You can see the results here:
http://www.flickr.com/photos/16371732@N00
The pictures: "softdevice_wrongoverlay1_software", and "softdevice_wrongoverlay_software" show the OSD problem with software mode rendering, no changes to any softdevice sourcefiles.
The main problem is the OSD (in software mode) has banding across it, as if every 4 or 5 lines are not drawn. This occurs no matter what the color space setting is or swapping the U & V components.
Using xineliboutput the OSD appears without banding, as can be seen in the second picture with the DVD progress bar.
Using the "hardware" setting the OSD appears correctly but "blinks" momentarily when there is video (it is not visible when video is playing). I have experienced this problem with a TNT2 as well using the X.org "nv" driver.
Stefan Lucke wrote:
A) If you get wrong colors (that should be both: video and osd with and without video) you could try to change FOURCC_YV12 to FOURCC_I420 in file softdevice.c (2 times).
When I did this both OSD and video colors were wrong, and the banding in the OSD still appeared.
If there are still troubles, please provide something like table for each fourrcc type.
B) Another try is to choose the "Video Blitter" instead of "Video Texture". For this you'll have to change variable use_xv_port = 54 (around line 740 in file video-xv.c). I don't know whats the difference between them.
I tried with FOURCC_YV12 and no change in the banding problem for OSD.
C) In case you have a problem like OSD-colors ok and video-colors bad OSD-colors bad and video-colors ok you may try to change in video-xv.c the OSD u and v components, line 1643, and in line 1652 (OsdPu -> OsdPv): AlphaBlend(pixels[1]+i*xvWidth/2+cutLeft, OsdPv+i*OSD_FULL_WIDTH/2+cutLeft, //<-- !! change to OsdPu Pv+ i * UVstride+cutLeft, OsdPAlphaUV+i*OSD_FULL_WIDTH/2+cutLeft, fwidth/2-(cutLeft+cutRight));
This made the OSD colors all wrong, video colors were find but the banding issue still there.
Do you have any other suggestions?
Thank you, CR.