Here's the GravSim file:
https://docs.google.com/file/d/0B6hJly6aYB0ZVXV5NzY5b1N4NTg/edit?usp=sharingAnd here's the accompanying spreadsheet listing the values used:
https://docs.google.com/spreadsheet/ccc?key=0AqhJly6aYB0ZdDlzVWM3aFhvYzBFRlFlNkxYSjlMS2c&usp=sharingAs for your own setup and what was wrong with it, it's hard to say, as the state in which it was uploaded was already highly destabilised. It's possible that you've set the orbits of the satellites too high, too high eccentricities, and/or the objects were too close to each other.
It is also possible that you have simply gone overboard with the simulation speed. You see, the program is cutting corners at high speeds, which is best observed in Solar System simulation. Set it too high and soon you're left with one or two planets, the rest having been ejected into the interstellar space due to accumulating errors. It can even happen with purely two-body system, which is not possible in reality.
The system as it is has been tested at speeds of 2048 for up to 1500 years. The same time span at 8192 leads to the orbit of the second moon of the fourth planet(4-2) to oscillate wildly. Usually, this trend tends to continue, leading to ejection in the near future.
Generally, the bits most prone to unstability are two of the smaller moons(4-2 and 5-1), the rest can take a bit more time-rate abuse, but set it high enough and everything
will break down.
My advice would be to stick to simulation speeds no exceeding 2048.
About the system and the reasons it is what it is:
Eccentricities and inclinations were set low to help stabilise the system.
1 is a slightly larger than Earth planet of similar composition. Mostly silicates with nickel-iron core.
1-1 is a large nickel-iron planetoid, or a small dwarf planet, 1200 km across. Microgravity.
2 has 2/3rds Earth's radius, and very high density of 14g/cm^3, to conform to the high gravity requirement. It simply has to be largely composed of heavy metals in the neighbourghood of iridium and lead, most of it in a large core, with less dense stuff in a relatively thin surface layer. Perhaps a core of a gas giant that migrated close to the star early in the system's evolution?
You'll have to come up with some handwave'y reason for the required lack of minable resources(or change the system sheet).
It's at the inner edge of the habitable zone, so keep the atmosphere relatively free of greenhouse gases. Perhaps high oxygen content, to help justify the existence of large animal life in thin atmosphere?
Gravity is 150% of Earth's. Going any higher would make megafauna that much less plausible, and compound the problems with mineral abundance.
3 is a bit larger than Uranus, with density a bit lower than Saturn's.
4 is 30% larger than Earth, of similar composition(a bit denser), probably with methane ice surface and highly differentiated(heavy metals in the core, silicates in the crust).
4-1 is the size of Ganymede, surface probably looks like Europa's. Overall composition of lighter stuff than on Earth(which is still heavy as moons go, mind you), with extra load of heavy radioactives. The system sheet says gravity should be normal, but here it's half the Earth's. Upping it to 1g would require turning it into another very dense planet, with the same problems regarding the general scarcity of resources as with 2.
4-2 is another large planetoid-like body, made of silicates and ice.
5 is almost twice the size of Earth, close to the theoretical limit for rocky planets. Lighter compounds are trapped on the surface in the form of ices, with thin atmosphere of sublimated methane. Relatively high density helps justify high industrial metal content.
5-1 is a small nickel-iron planetoid, perhaps not even spherical. High density to help keep microgravity for the colony(~5% of g), despite tiny size.
5-2 is similar to 4-2. Perhaps the colonists mine for water ice for sustenance and trade.