Why does music written in a minor key sound "sad" or "downbeat," while music in a major key sounds "happy" or "upbeat"?

I think most of it is just cultural conditioning, we play major key songs when we're happy and minor key songs when we're sad (though this is changing in modern music). Another part of it could be that minor chords have their midtones clash a bit more with there tonic notes than major chords, it sounds slightly "wrong" to our ears and takes away from the happy effect.