Man, that's the best question I have read in a long time.

The answer is "You don't know."  

All things wear out with time and use.  With a filter unless you are doing a microanalyses (specifically a colony count), there's no way to tell.  Some filters, specifically the ceramics, if not cracked with very limited use on clean water may last a lifetime while others like a charcoal filter best to replace every six months after use.  

Now that's scary first looking at it, but there is a some good news.  People typically don't get infected by exposure to one or two micro-organizams  as our bodies immune system can fight them off.  It typically takes "X" amount, another unknown and variable person to person, to cause infection.  What this means is for most people if you start seeing suspended solids (cloudy water) then that's time to change the filter.  

A recent (last couple of years) study on the consumption of alcohol in other countries was very enlightening.  People who drank had liqueur once a day reduced contamination chances by 90% while those who drank a couple beers did so by 50%.  This is another very good reason to take a flask of hooch on your field adventures.  A little drink by the fire, will warm you up a tad while improving your contamination odds.  

Now that being said in my five decades of drinking found water, I believe water selection is still 90% of the battle.  A slow moving stangnant pond may be easier to get to and to pump water from but a fast moving cold stream is far likely to have contamination to begin with.   Even within a given body of water where is very important opting for the least biological matter, best color/clarity, and coldest spot.  It seriously worth a few extra steps.  

As a bit of trivia, in the 60s we had neither EPA or water filters.  With raw sewage going into streams as well as chemicals, water was far more foul than today.  It didn't stop us but it damn sure made us careful where we got our drinking water and when in doubt, boil or treat with iodine.  

Tj