First of all, you definitely have the moral high ground here.
Second, these sound like the common problem of not being assertive enough.
Being assertive is somewhere between being aggressive and submissive. You want to have a high chance of getting what you want while giving them a chance to voice their opinion and have a good quality conversation. You don't want to beat them up, yell at them, or give them no chance to do anything about it. You also don't want to let them walk all over you and continue to annoy you whenever they like, but rather you just want to have your needs addressed.
To be assertive: state your position, state the consequences so they are well known, and then follow through with the consequences. Building up the courage to say it plainly "It still bothers me that you groped me" is a good step. Following through with a consequence is the harder part. If you say "If you ever attempt that again, I will not be your friend", and then you have to follow through with it. You can still feel guilty, but you can't let up.
If you have nothing relevant to use as a consequence, then you just have to be firm. Tell the one guy you don't want him coming over while he's grounded (because you like the rules, you respect his parents, you don't want to get caught in the middle of it, whatever). If he comes over pleading that he's locked out or something, say your very sorry but you don't want him there until he's not grounded. Then keep him out of your house while you proudly and happily go drink some milk and eat a cookie. He'll eventually leave, you might feel a bit of guilt, and then you are completely successful. Repeat as necessary.
By the way, a 4 year old coming to a neighbors house by himself for a whole day... that is cause to lock the doors. Also, its a cause to put things on shelves and in cabinets while he's over. Toddlers are dangerous, and especially to themselves when they're not supervised. They very well might play with bleach, or stuff a crayon up their nose. Then, while they're at your house, you are liable for the child hurting themselves. Unless someone agreed to watching them, that would be a cause to call child services.