What happens is supermarkets sell milk as a loss leader, as an enticement for consumers to shop with them. It's good for milk drinkers, but with the CAP providing the shortfall it's essentially giving people cheap milk at the tax payer's expense.
You say more farms could close. In fact they already have. In 1950 there were 196,000 dairy farms in Britain. Now it's closer to 14,000. What's happened is cows have become massively more productive in yield (through selective breeding I'd guess), but the cost of distributing milk has increased vastly: where once the milk was sold locally almost straight from the cow, now it must be refridgerated in huge tanks, packaged and delivered to supermarkets etc. There are also health and safety requirements that raise the cost in equipment and labour. Also, large farms have to pay their workers a daily wage, while small family farms typically don't.
In other countries (France, Poland, Ukraine, India and so on), dairy small holdings are widely distributed throughout the country and can sell locally more cheaply. What they might lack in economies of scale (Polish farms have on average 3 and a bit cows), they make up for in economies of distribution.
Further, in Britain despite the rich variety of farmhouse cheeses, there's been a historical reliance on importing cheese and butter (esp. Danish butter)— this means that the excess of production can't be as easily funneled into cheese and butter.
All this to say, the solution isn't necessarily fewer farms.