Why does the water from my fridge taste weird?

On my fridge, which name I don't remember, there's a display on the front which lights up whenever it's time to change the filter. If I go more than a few weeks without changing the filter, when the display is lit that is, the water starts tasting funny.