If the coolant in a refrigerator was not compressed back into a liquid after it flowed through the refrigerator what would happen

The system would stop cooling, and simply recycle warm vapor through the refrigeration loop.

From the Mollier diagram we see that one part of the “refrigeration” loop is not complete. The step from 1 to 2 in the diagram would not occur.
This may occur due to a lack of sufficient refrigerant in the system or a faulty compressor. In any case, this results in warm “coolant” vapor passing into the receiver without any condensation (Step 3-4 cannot happen).
Without any liquid coolant available, the cooling effect of expansion of the liquid through the evaporator is lost (step from 5-6), and the system will simply use energy (and generate more heat) while recirculating warm vapor.