OK, I spent a bit of time looking at this today as there doesn't seem to be much movement on FFMPEG and H264 at the moment.
I have managed to get xine to now detect the H264 video size prior to starting up the CoreAVC decoder and set the size within the initialisation function: -
memset(&bih, 0x00, sizeof(bih)); bih.biWidth = sps.width; bih.biHeight = sps.height; bih.biPlanes = 1; bih.biBitCount = 24; bih.biCompression = 0x34363248; //31435641; //AVC1 bih.biSizeImage = 0; bih.biXPelsPerMeter=10000; bih.biYPelsPerMeter=10000; bih.biClrUsed=0; bih.biClrImportant=0; bih.biSize = sizeof(bih); buf->content = malloc(sizeof(bih));
It detects BBC HD as the right video size, i.e. 1440 x 1080 - however this now results in a squased image on the display with black bars down the left and right sides. Any idea how you tell CoreAVC what the video size is and to scale it to the size that xine is using? It detects other video sizes such as 1920x1088 and displays them all correctly on the screen so I know that we are now very close!
Once I get this bit right, I'll release a patch to xine's demux_mpeg_pes.c that will properly detect the source video size prior to starting CoreAVC.
Thanks for any help.