Can an act of war be declared by a non-state actor? War is legally recognized as occurring between states. And yes, civilian courts can work. They convicted and executed Timothy McVeigh, they convicted and imprisoned the guys behind the first WTC bombing in 1993.
Except this was on much more massive a scale than the WTCB.
At the time I would say that this could be an act of war, Taliban and Al-Qaida, both are kinda joined at the hip.
Except when they're not. For all their anti-American rhetoric after the fact, the Taliban were unconcerned with the US prior to 9/11. They were an Afghan movement, dedicated to establishing law and order under strict Sharia law in Afghanistan. The first part was laudable (especially considering what a giant clusterf**k the place had been in the early '90s when the warlords were going at it). The second part was where it got problematic.
Had they given up Osama bin Laden right away, they'd still be in charge, because frankly it wasn't worth our time or effort to dislodge them. They weren't a threat to us.
As to the initial point, scope isn't necessarily relevant. Up until 9/11, we had tried mass murderers and terrorists in civil court, and continue to do so. Despite the elevated damage, I see no fundamental reason that KSM cannot be tried in a civil court. This wasn't something on the scale of Nazi Germany.