What are you on about?
Lets use bigger numbers for the sake of comprehension.
Suppose you have 50% damage reduction mastery:
You have Ashe with 0 armour take 100 damage. That damage was reduced to 50 damage.
You have a Rammus with 100 armour (and thus, 50% damage reduction). His armour will reduce the damage to 50. The effective damage reduction from the mastery then is only 25 of the remaining 50 damage.
Thus the actual damage being reduced from just the mastery is halved compared to if you had no resistances. It appears more because the armour is doing work but if you discounted the gold used to obtain the armour, then Rammus is actually worse of. This is why old Alistar with his 75% damage reduction ult could afford to be crazy even without any resistances.
It's a matter of utility. I'll give another example to illustrate.
You now have the 50% damage reduction mastery and 2000 gold:
Ashe buys 70 AD and takes 100 damage. She reduces it to 50 damage. Her total is 50 reduced damage and 70 AD.
Rammus buys 100 armour and takes 100 damage. He reduces it to 25 damage. His total is 75 reduced damage.
As you can see, if you calculate by worth of gold, Ashe is far better off even though her damage reduced is 33% less than Rammus. She spent her money more efficiently than Rammus.
Sure, at the end of the day it makes Rammus tankier and honestly Ashe wouldn't be taking such a mastery but the point should be pretty clear. When you have a %-based stat, you are statistically making it less cost-efficient by buying flat-based stats. You still get the benefit, but the benefit would be worth less gold than what it could normally be.
Edit: Just to emphasize. Just because it makes it less cost efficient doesn't mean you shouldn't be buying resistances on a tank... The mastery only gives a tiny amount after all, not 50%.
Have you seen a tank Alistar ulting? He's
absolutely invulnerable. Old AP alistar wasn't ridiculous because % reduction is better without masteries, he was rediculous because he could effectively get 300 MR and armor. If you stack that bonus on top of armor, you get ridiculous tankiness, but the reason it's not done is because
you only need to be so tanky. At a certain point, you're hard as hell to kill and the enemy team doesn't really care about specifics.
And tankiness is not measured in how much of an individual hit you're reducing, it's measured in how much damage you can soak before you die. Let's look at your bigger mastery again. The 50% one.
Ashe here has 1000 health and 25 armor. This gives her 20% reduction, so she effectively has 1250 health against physical damage. If she gets an additional 50% reduction, she can take 2500 unmitigated damage before dying. The 50% reduction mastery here is basically letting her take another 1250 damage.
Then we have rammus here with 2000 health and 100 armor. This gives him a 50% reduction so he effectively has 4000 health against physical damage. If he gets an additional 50% reduction, he can take 8000 unmitigated damage before dying. The 50% reduction mastery here is basically letting him take another 4000 damage.