The question came up to me the other day about the Xbox's ability to play back 5.1, and I thought I would revisit the issue. I found an existing video on the net the said it was an AVI for the Xbox with 5.1 sound. Sure enough, I threw it on a usb drive, plugged it in, and had 5.1 surround sound.
I looked at the specs of the file, and after a few trial and error attempts, I had a dvd converted and playing back on my Xbox 360 in full glorious 5.1 surround sound.
The magic command I use is:
mencoder dvd://1 -ovc lavc -oac copy -channels 6 -lavcopts vcodec=mpeg4:threads=2:vbitrate=4500:autoaspect:turbo:mbd=2:trell=1:vpass=1 -o test.avi -ffourcc xvid -mc 0 -noskip -vf scale=854:480,softskip,harddup -ofps 24000/1001
This command produces an avi file with a variable-rate mpeg4 video and a 5.1 AC3 audio channel.
A few interesting points on the command. I use two-pass encoding. So, you'd have to run the command 2x changing the vpass to 2 on the second run. Even though I told it to copy the stream, unless I specifically added the "-channels 6" command, mencoder only encoded 2 channels. Another issue is that you have to force the FOURCC to xvid. The final issue I had was the aspect ratio. Without forcing the scale to 854x480 ( 16/9 DVD ), the Xbox would not scale it to 16/9. mencoder allows the -aspect flag, however, as the mencoder man page states, it is only recognized by mplayer. So, scale the video while encoding it, and it plays back fine.
There are lots of other options to mencoder--it's got to be the most complex command I've ever seen. The documentation for mplayer is great, so check there if you have questions.
If anyone knows how to or can figure out if you can encode H264 videos using avi or mpeg format with a 5.1 audio channel, drop me a line.
Handbrake is another great tool for converting videos. They have an XBox 360 preset, but it looks like it only supports faac two channel audio with x264 video. Handbrake is great since it auto-crops the video. Using mplayer/mencoder, I have to play the video first to figure out if it needs cropping and to get the correct cropping parameters to pass into mencoder.
Now, if only I had a huge server farm to encode my DVD collection.