Have a look at the hdtsplay-demo in the hdshm3-folder. It is a simple demo that plays back TS files over the DeCypher. If that works, your basic driver setup (+ the HW) is OK, and something in the vdr (or the patches) itself is wrong.
Playing back a ts file with hdtsplay is actually even worse. I am using a patched hdshm.c file that I downloaded. The one in the reelbox SVN testing used 'pci_get_device_reverse', which I think has been removed from the newer kernels.
I will SVN a completely fresh reelbox tommorrow and see if maybe I have made some mistakes when installing the drivers.
Josce