DxTory with the default codec DOES work fine in the current experimental branch 0.54.126545
The only other codec that "sort of" worked was "Cinepak codec by Radius" however it did not seem to record video, only a still frame. A video file was produced.
All other codecs seem to lock at "INIT" indicated when the FPS monitor switches to orange. It never initializes and indicates the split FPS monitor to display the game fps and file write fps.