[tests] use "h264" instead of "libx264" for tests (fixes: #951)

This allows running the test suite against FFmpeg builds which do not have libx264 enabled.