There's a story I like to tell about this involving a guy who, one morning on the bus I was taking to my grad school lab, attempted to convince me that π is exactly equal to 3, as implied by the Bible (1 Kings 7:23). I pointed out that π is demonstrably not exactly equal to 3, and his response has stuck with me: "Scientists have wasted billions of our tax dollars trying to find π, and millions of digits later they still don't have an exact number, just a lot of blather about how it's infinite or whatever. So how can you tell me it's not 3 when you don't know what it is?"
Popular portrayals of science deserve some blame, for throwing the word "prove" around.
"Prove" (and alternate forms) is the right word, but understood wrongly. To prove is essentially to put to the test/trial, and while it can often be read as "the positive answer" it more rightly often refers to the question being asked regardless of the answer that may then have followed.
"The proof of the pudding is in the eating", you must taste the food to confirm its nature. "Galley proof"/"Proof sheet", the initial test-print to confirm layout/content. "Prove all things, hold fast to that which is true.", analyse for veracity before then taking that which has it for your informed position. "(100, or other value)% proof", a concentration of alcohol as empirically established. "Proving bread", allowing the yeast/equivalent to properly demonstrate its ability, before actually baking it. "Not proven", not sufficiently tested to establish a result (especially of guilt, in Scottish law).
And of course the classic misunderstood "The exception that proves the rule". There's an apparent rule and then along comes an exception to the rule that explores the limitations of the assumption, breaking the assuredness of the rule and/or forcing alteration so that the original rule is not confirmed (as often mis-stated) but actually shown to be not valid. Finding an exception shows where the rule is wrong.
(Although some say "the exception that proves the rule" is more about that when there is a noted exception it indicates that somewhere
outwith that exception there must yet be rule-following of the kind the exception does not obey, or else that exception would not be notable. But it does rely on fully countable and testable non-exceptions, or else all you're really implying is that the exception is the true rule and anything going by the old rule is an exception to the exception. And if not all exceptions to the exception's rule may themselves follow the old-rule, then you have to consider a rule for the old-rule candidate(s), a rule regarding the exception(s) and who knows how many further rules you need to cover exceptions to tje exceptions that are themselves exceptions to the old-rule. e.g. Possessives, in English¹.)
¹. The basic rule is to add apostrophe-'s' to all words that don't end in an 's' already as a plural, otherwise just append the apostrophe after the existing final 's'. This comes pre-catering for the exception that a non-plural s-ending word ought to have apostrophe-s. "The mass's gravitational pull is..." "The masses' gravitational pulls are..." "Professor Jones's experiments on gravitational attraction demonstrated that..." "Each LIGO installation around the world is set up in the shape of a giant L, the L's placement being flat to the local ground level." "If each of the Ls' orientations are in differing planes then we can ultimately determine the direction from which a detected signal came."
But there are exceptions. Biblical names don't "s's" themselves, for some archaic reason likely to originate with calligraphising monks not laboriously inking out "Jesus's" when they might otherwise have done so. (Or at least once they started to do it in English rather than Latin or whichever other precusror.) But that exception doesn't prove (confirm) the rule, because you need to consider the possessive (and determiner) pronouns that are all archaic exceptions, with "its" being the one people stumble over most because there is an "it's" that is in the separate group of regular contractions. Except for the pronoun "one" which
isn't an exception like its brethren and sistren and othren. Something belonging to one is one's. Unless it is the very self of one, which is oneself. None of these
confirm the aformentioned rules, merely highlighting limitations to the generality of the rule, letting a couple more 'rule-breaking' examples that I haven't yet mentioned exist, so none of those exceptions
justify the more naïve versions of the rule, they merely show that a more complex version of the rule needs to be established to cope with the additional rigor of all these exceptions.