Why is the cooling system in a nuclear power station important? what will happen if the system malfunction?

Cooling is needed to condense the after-turbine steam in the internal circuit and recycle it. As the steam condenses back to water, the surplus heat must be discharged to the air or a body of water.

Cooling Power Plants

The amount of cooling required by any steam-cycle power plant (of a given size) is determined by its thermal efficiency. It has essentially nothing to do with whether it is fuelled by coal, gas or uranium.
However, currently operating nuclear plants often do have slightly lower thermal efficiency than coal counterparts of similar age, and coal plants discharge some waste heat with combustion gases, whereas nuclear plants rely on water.
Nuclear power plants have greater flexibility in location than coal-fired plants due to fuel logistics, giving them more potential for their siting to be determined by cooling considerations.
The most common types of nuclear power plants use water for cooling in two ways:
To convey heat from the reactor core to the steam turbines.
To remove and dump surplus heat from this steam circuit. (In any steam/ Rankine cycle plant such as present-day coal and nuclear plants there is a loss of about two-thirds of the energy due to the intrinsic limitations of turning heat into mechanical energy.)