A dorf can carry the cart. You need to have a track stop constructed beside the tile the magma should be dumped in, and it has to be one level above the water in order not to just get everything evaporated into steam (or possibly deadly magma mist), I think. The track stop should be set up with the dumping direction on construction, unless DFHack is used, in which case it can be changed later.
You then need to set up a Route with a Route Stop on top of the Track Stop (don't forget to remove the default departing conditions), and then assign the filled mine cart to that Route Stop.
The mine cart has to be magma safe (iron/steel/candy, but NOT nether-cap). To fill it you can either use the route through magma technique (which I don't use), or the pumping technique, where a magma safe screw pump (all parts, typically gabbro block and green glass tube and screw) is created on top of a dug magma filled tunnel with magma safe grates covering two acess holes with two tiles in between. The pump is placed on these tiles and manually pumping up from the first grate and out above the second on (draining downwards). I always enclose the pump with walls and magma safe doors, which may or may not be needed. A second (or rather, first) Route is created with a Route Stop at the pump's output tile and the cart is assigned to the Route Stop. When the cart has been placed, the pump is hand cranked briefly, and then the pump is stopped and the cart ordered to be placed on the destination stop.