Why do electrons transition between energy levels within the atom, and how do we detect these transitions?

Since excited atoms can't be excited for long, the electrons jump down to a lower energy level, and as this occurs, the electron emits a photon with energy (and thus wavelength) equal to the difference in energy levels between the two levels the electron jumps in between.

Explanation:

Electrons in atoms and molecules can change (make transitions in) energy levels by emitting or absorbing a photon (of electromagnetic radiation), whose energy must be exactly equal to the energy difference between the two levels.

Electronic transitions involve exciting an electron from one principle quantum state to another. Without incentive, an electron will not transition to a higher level. Only by absorbing energy, can an electron be excited.