As an aside before I start: Don't approximate stuff until the end. It makes things really hard to follow and causes a lot of error in the long run.
You can't just use cos(theta) and sin(theta) to find the X and Y there. Those should always come out to your whole squared. The only reason your Z didn't come out to 0 there is because of calculator funniness.
Step one is to find the magnitude of A along the direction of B, so you rearrange your dotproduct formula for |A|*cos(theta), which leaves you with A·B/|B|. At this point you now have the answer to A, although usually questions would ask for the full components of Aproj so you won't be able to stop at this point.
The next step to what you need to do is called the unit vector. You're trying to find out what 1 unit in the direction of B is. This means you find the magnitude of B, then divide each component of B by it. You already know the magnitude is root(13), so now you find the unit vector of B. (-1/root(13),2/root(13),3/root(13)).
Now that you know the direction, you multiply the magnitude of the projection of A onto it and you're able to find out the components of Aproj.
To find B, since you now have done A properly, you subtract A-Aproj (not |A|-|Aproj|), and you're left with the rejection. Find the magnitude of that and you're done.
It's worth noting that if all you need is the magnitude of the rejection you could in theory find theta with the dotproduct, then |A|*sin(theta) would be the magnitude of the rejection, then you could find the normal to B in the plane defined by AB and multiply that by the magnitude of Arej to find the components, then A-Arej would be Aproj, but at that point you're just going in an ugly circle and could've solved it the sane way already and are just having fun.