Note: the following post is based of my knowledge as a mathematician, and what I have picked up from my brother, a physisist.
Maths can be crudely divided into two area - pure and applied. Pure maths can be thought of as a study of logic - that is, if A and B are true, can you deduce C? Applied maths is finding solutions and manipulating equations.
Consider the example of an object whose height h is a function of time given by h=ut-gt2/2. A pure mathematician would tell you that h take some values none of the time, one value once and some values multiple times. If you wanted to find a solution for when h is a specific value, they could tell you if a solution exists or not, and if it exists they could tell you some facts about it (e.g. it would be less than zero). Actually finding the solution is the realm of applied maths.
A physisist, on the other hand, would consider the problem of firing an object into the air and modelling its height. The physisist would consider the situation, consider the forces and physical laws involved, eliminate negligible forces such as air resistance, and find that it follows the model above. The physisist can then use that to find solutions.
Physics and applied maths are extremely similar. Physically analysing a situation can produce a mathematical model that approximates it, and that model can the be solved to produce an answer. The pure mathematicians develop theorems, which the applied mathematicians then turn into solutions. The physisists then take these solutions and adapt them to fit their needs.
Taking both physics and maths is a very good choice. You'll find them both a hell of a lot easier, as there is so much overlap. Maths is a hard course, as is physics, but due to the enormous amounts of overlap, a maths/physics course is easier than almost, if not all other two subject combinations.
University level physics contains a lot of math. More than you think, by a long way, and more complicated than the math classes you'd experience at a pre-university level.
Also, programming is a very useful skill, for both mathematicians and physisists. Not only does programming enable one to better create complicated models (for the physisists), but the pure maths theorems can provide incredibly efficient solutions to otherwise extremely difficult problems. Furthermore, the mental processes and structure that are developed by programming are very helpful in both maths and physics.
I would encourage you not to think of maths and physics as seperate subjects. Instead, imagine a three step process for solving equations: Pure maths, where the concepts are rules are developed, Applied maths, where the concepts are applied in theoretical situations and solutions found, and Physics, where the theoretical solutions are adapted and made to fit real problems. A physics degree involves applied math and physics, whereas a math degree involves pure and applied math (though the proportions can change depending on what modules you do).