@d00p, there will be a difference in audio, as you will not only have less dedicated audio channels, but the stereo audio may also be lower quality format compared with the format you're getting 5.1 from (stereo PCM vs Dolby Digital 5.1). Assuming they're the same quality stream, the phantom center channel will likely be pretty similar, as what happens is the left and right sides will both play the audio signal for either case.
If you're also adding other channels, the differences will be more noticeable.
Mark good posts by pressing the like button, and select the best answer on questions you've asked to help others find solutions.