Seeing as I haven't heard of it, what would slamming the sword into the ground do? Transmute it?
Yes, I know it's a cop-out, but it is still theoretically possible, though not remotely likely for weird shit to occur. Very well, I shall present a different argument. How would you know it's the past? You can only observe the area you are in, not the rest of the universe, even if you could do so, his supposed past may have a differing past to the past of your own past.
Because it looks like the past.
Except one only has an idea of what the past is, a summation of memory and knowledge. To say it looks like the past would thus be to say that it looks like what one thinks what the past was.
You can make that same argument about any location. How do I know I'm at the house I grew up in? Because it looks like the house I grew up in. How do I know I'm at the White House? It looks like the White House. How do I know I'm in Low-Earth Orbit? Because it looks like Low-Earth Orbit. And because I'm suffocating, but that's just another sense so it fits.
In other words, your argument is fundamentally meaningless. You can travel to the past, but you just can't be 100% certain of it (just like you can't be 100% certain of anything).
At the end of the day, since this is using multiple worlds. Say you travel to the past, what it actually means is that you travel to what matches your interpretation of the past which is a pointless venture since you have chosen to observe what agrees with your viewpoint.
Yes, you cannot be 100% sure of it, hence why I may argue it is the result of memory and knowledge, there's no concrete past that one may know of. Devil's Proof, this "true past" may as well be God, or the aether with no way to prove it nor any method to disprove it. Unfair, I admit but it nonetheless sustains the matter.
Sorry, I can't really form any concrete argument since the magic here doesn't so much have any rules as much as it has vague suggestions with little to no known limitations.
And that's why you shouldn't declare anything as being impossible.
It is impossible as the GM said it is, the issue is, in regards to magic, I do not know what the precise reason is. Presumably, to record data on all particles and determine previous states is just as impossible with magic as it is without.