Programaticaly I would solve this issue by generating an walkable area by just Djikstraic-ing outward from the chain a number of spaces equal to its length (done only once when the chain is built or when their is terrain change in the constructed area, or a length change to the chain). The creature moves inside this area in the normal manor and simply treats areas outside that as impassible (like being inside a small room), it can use doors and stairwells normally. Thus the chain is not actually simulated, though a graphic could be placed on the shortest path between the animal and the anchor point to convey to the player the fact that the animal is chained (a different character could appear when the chain is taught "\" vs when it is slack "S" to convey more information). This would eliminate all tangling or wrapping of chains which frankly are not worth having in my opinion.
Another great idea would be the creation of a 'Winch' created as a building from a Block a pipe section and a mechanism the winch can installed only on solid ground. Multiple chains can then be installed to the winch. The winch alone allows a menu option for 'setting' the effective length of the chain from 1 to the total length of chains attached to it but a dwarf with the pump-operator (perhaps renamed to machine operator) must actually visit the chain and 'reel/unreel' the winch to make it take effect, at which time the creatures walkable area is recomputed.
Connecting a lever to a winch activates only one function, extending the chain to its maximum length when flipped to ON the chain is released to full length. This dose not over-write or erase the 'set' length for the winch, it just temporarily ignores it. Flipping to OFF dose not shorten the chain, but it dose re-instate the effect of the set length. The winch will now 'call out' for a dwarf to shorten it or it will accept power applied via some other method to shorten the chain. Still this is a very effective way to unleash chain animals onto a well defined 'kill zone' and is not a one off event like the lever-2-chain linkage.
A winch can receive power by Gears/Axles just like anything else and this power is applied just like that provided by a dwarf. A winch that receives power from a gear will consume 10 power but only if it's desired length is shorter then its current length. Once the correct length is reached the winch stops consuming power. Setting up a lever with one linkage to the winch to releasing the chain and a second linkage to the gear to reel in the chain will create a convenient one switch solution. As with all power devices placing them adjacent to each other allows them to transmit power to each other.
A creature standing outside the newly desired area of movement (aka the chain is taught and pulling the creature towards the winch) will trigger a 'tug-of-war' with the dwarf that is operating the winch and if the creature wins the reeling in attempt fails and the chain remains at its previous effective length, the dwarf will try repeatedly to pull the creature in and this will make for excellent exercise.
On each successfully reeling in the chain shortens 1 unit and the tug-of-war must be repleted to shorten each unit of chain. Tug-of-war is triggered only if the chain is taught so for example if a length 10 chain is attached to a creature standing 3 units from the winch is being shortened to 2 then the winch operator will rapidly shorten the chain to length 3 (very little exertion or strength gain to do this) and will have to win one tug-of-war with the creature to shorten the last bit.
Even an exceedingly strong creatures can be slowly reeled in by the winch operating dwarf stubbornly repleting the attempt until by chance he exploits a time when the creature of its own volition goes near the winch and the chain is slack which allows him to avoid the unwindable tug-of-war. The creature can never lengthen the chain, they can only break it, naturally the chain is only as strong as the weakest link and in the case of the winch the winch mechanism, pipe and block count as links so an Adamantium chain had better be connected to a full Adamantium winch. A creature can break the chain leaving the winch intact or break the winch which will deconstruct the winch and probably ruin or degrade the failed part, some or all of the chain might remain attached to the creature and be dragged around.
If you really MUST reel in that chained mega-beast (for example to pull it back inside a smaller room that can be shut off from the main pit so that your dwarfs can loot the items from the charred corpses) their are a number of ways to do that as well. The simplest is to install additional mechanisms on the winch, each additional mechanism doubles the effective strength of the dwarf but also doubles the time necessary to conduct all length changing actions both tug-of-war and the reeling in of slack chains or lengthening the chain, the effect is cumulative so 4 additional mechanisms means 5x force but 5x slower operation. Ideally their would be no limit to the number of mechanisms attachable in this way other then what is practical.
Another method is to create a second winch any ware in the fort and command it to be linked by the standard double mechanism method to the first winch, this second winch can no longer accept chains of its own and any chains it previously had are detached (but other winches may still attach too it). The second winch will transmit all the force applied to it to the task of the first AND issuing a command to the first winch will summon dwarfs to work both winches, in essence the second winch becomes a second 'slot' for operators slaved to the first winch. Any number of winches may be linked too a winch but each winch can have only one outgoing link, thus 'trees' of winches can be created all commanded from the 'master' or root winch.
This technique can be combined with the one above, the mechanism based multiplier effect. When doing so the count of the highest winch in the system will be used for the combined power output of all dwarfs on all the winches. Programaticaly the master which inherits the highest mechanism count of its slaves if one is higher then its own count.
For example winch A is the master winch to which the chain is attached and it has 1 extra mechanism, attached to it are winch B which has 3 extra mechanisms and winch C which has none. The mechanism count of winch B is inherited by the master winch and the combined dwarf power of operators irrespective of which winch they operate are combined and then modified by the mechanisms, specifically multiplying power by the extra mechanism count + 1, which is 3 extra mechanisms in this case, for 4x power and 1/4th speed. If Winch B were disconnected from A the system would drop to a 1 mechanism and 2x power and 1/2 speed (in addition their would be a max of only 2 dwarfs operating the winches rather then 3). Thus linking/delinking whole winches will allow a system to easily gear up or down.