I'm no expert in the field of video compression or recording... but here are my tips:
Use FRAPS for recording (make sure you have version 2.1)
Make sure you have a fast hard drive for captures.
I only have a Western Digital 200GB, but i'm upgrading to the Raptor 10,000RPM, which should be able to handle the overhead.
For compression, do one of two things:
Either, for AVI or WMV or quicktime compression, scale it down to about 400X300. It looks better than larger video, trust me. Things like WMV and DivX don't like "large" videos.
For an extra high quality video, use Bink. It's the same tool used for the Half life 2 previews, which if you've seen them are excellent quality.
It's about DVD-size, but the quality is worth it and it looks amazing.