Your windows are a strong line of defense in your home against the outside elements, but windows are usually more permeable than walls. Because of that, they let in more heat or cold from the world outside and leak your expensive air conditioning through tiny cracks, gaps, poor seal, openings, or simple conduction.
This leakage can happen even if your windows are unbroken and still sealed, much less if your windows are broken. So how bad can it be if your windows are damaged? It depends.
Taking a look at your utility bills may offer some insight into the status of your windows. Are your bills going up every month? Do you feel like your bills used to be lower than they currently are? Do you feel a draft around your windows? How do you know if you should replace or repair damaged windows?