I hate these kind of people who are physically violent. It is often because they experienced and suffered violence themselves.
In some form or another, people suffer abuse, physically, mentally or spiritually. What we can do is to not perpetuate the cycle, but to end it.
The truth is, you have some form of power over him, or he feels powerless, and he hates it. No one likes to feel trapped.
Still, the fact that he's around seems to indicate he feels some sense of duty, even if he feels unable to confront it directly due to lack of self-belief.
That he spends time and attention harassing you shows that he thinks of you as a part of his life, for good or bad, and that he can't or won't escape from you.
In a weird way, it's a way of confronting his problems and a form of commitment for him, and a form of 'faithfulness' to show up routinely and wait for hours (although I don't know if he is strictly monogamous)
As a male, he probably suffers from a need to 'mark his territory' and 'earn respect' from you and your family, and that's probably all he wants. He wants to be acknowledged, even though he has problems acknowledging his own role (probably because no one believed in him).
If you deny his role and place in your life, you would crush his self-esteem/manhood. It is difficult to take the role of fatherhood, when everyone is putting you down (or when you put yourself down) because it is a formative event in a man's life.
Almost certainly he suffers from an identity crisis as to the point of his existence with regard to the babies life (is there a point in me being around? am I a good/trustworthy partner? will the baby love/care for me? will I love the child? am I good/strong/tough enough? can I handle this? will my future be with this girl and baby? will I have a 'proper' relationship with this girl and my child? can I make up for the past? is there a point to, or should I just smoke weed?)
Smoking bud is probably a form of escapism because he doesn't feel capable of taking such an arduous/demanding role.
It is a form of escapism and a way to avoid our problems. But in other ways, much like alcohol, it gives a feeling of relief. The abuse of it is probably the worst thing, not the actual weed-smoking itself. It will stop naturally once he fills his life with other things more important.
He probably doubts many things, but almost certainly all to do with his own worth/value/goodness/strength/toughness. It is a shame that society routinely crushes the strong and able manfolk until they are unable to believe in themselves.
(If you wanted to change this person, you could start by detailing his good points and what kind of things he's good at. However, most effective would be to find out what he considers his own good qualities and nurture/encourage them.)
Babies grow up quickly. Soon the situation will change. You will begin to have a responsible little adult (depending in how you raise them) by your side to help you and look after you. More than that, human beings are extraordinarily resilient and can go through most any trauma. As long as you are honest and sincere to your child, they will grow up more-or-less well adjusted, even if the world is falling apart around them.
If you love this person, then there can be resolution and happiness. All things can be done through love.
If you do not love this person, it may be better to talk, negotiate or flee. Nothing much can be done with the unwilling.
The fact that he shows up shows in some way, he is willing, to an extent. Without pushing it, you can coax more out of him. It requires trust.
In any case, this person doesn't have the conviction, belief or self-determination to change their own destiny, most likely because someone told him that he couldn't.
You should, in some way, make him realise that he has the power to change his life, be a caring father, a good partner, a good person, work hard, earn money, be respected, be a man, be loved(!) and cared for etc.
People need to feel certain things in order to be self-fulfilled. When we are broken by life, we lose our hope. When our dreams are crushed, we stop dreaming.
Most important is
1. Be respected
2. Be loved
for women, it's probably the other way around
1. Be loved
2. Be respected
Either way, this person obviously wants respect, and a role in the babies/your life. Failing that, just to be acknowledged is not too much to ask for.
More than that this person wants to respect their own self as a person, as a father, as a partner, as a family member.
Whether you decided to broach the topic of a full/proper relationship is up to you, but if you think it is possible, I would try.
Men can and do change for love, family and children. It's a part of growing up and becoming a responsible, mature adult. No one is born knowing how to do this, it's something we have to learn, and as long as we are willing, we can do our best and not give up.
Having a child/partner can help us become a new person, if we want to become a new person.
Good luck, sincerely;
-Sevenforce