Usually a float is a "single" as defined in IEEE 754, which is 32-bits.
Usually double is exactly what it says on the tin, two singles in size, 64 bits.
Usually a long double is some size bigger than a double. On GCC it is 80-bits I believe. Visual C++ just treats it as another double. This is why I say usually, the standard doesn't specify exact sizes or exact implementations so different compilers will do things differently.
The basic representation is the highest bit is the sign. You then have a series of exponent bits, followed by the fraction. 32-bit typically will have an exponent of 8-bits, and a fraction of 23 bits. 64-bit has 11 exponents and 52 fractions, hence the greater accuracy. Rounding errors can occur as not every number can be represented using this notation, like how we might have to simplify 1423.423543674574574564 to 1.4234e10^3. The more digits we can store in that representation, the closer to the true value we become.
You don't really ever need to worry about this until it bites you in the arse. Personally in C++ I'll always typedef the floats I'm using, that way if I ever encounter a problem with accuracy I can just change one line and see if that fixes things. You know how sometimes a physics simulation will just go crazy and explode? It's surprising how often that's due to a rounding error...
There are some things to be aware of in those situations, the main one being: Mixing very big and very small numbers together doesn't usually end well. 0.0000232342324 * 2423423524653 = omfgwtfisgoingonwhyiseverythingexploding? If you're doing this, revisit the scales and find some way to stop it.
Fixing your timestep is always good. I personally have found it works well to ensure the timestep is always a multiple of a specific step, capping that multiple at a specific value and passing the remainder over to the next frame.