What factor led to the growth of towns and cities in the medieval ages?

After the fall of the Roman Empire, many people fled to wealthy people who could protect them from outside invaders such as barbarians, and bandits. The Wealthy owned castles and manors which helped keep the ones in need to be safe.