Why is it so hard to say no
Hiding and ignoring messages is pretty easy at first, but in the end it feels even shittier.
I can relate to this. This was me for many years after I moved out, and I still fall into these habits sometimes without realizing it.
I love my parents and they love me, but they have their issues. I moved out in a hurry when we were on pretty bad terms, when my Stepdad escalated from argument to violence. My Mom and he are both good people, but they're frustrated by things about their lives, and often vent it by picking at the people around them rather than addressing the cause of the frustration. Being around both of them when they're in a negative mood (which can strike whenever) is incredibly frustrating, takes a lot of mental/emotional energy, and I can't always handle it.
For a while, I dodged calls. I didn't know how to say no, and I hated myself for doing it that way, but I had a lot to do to get my life in order. Being around them all the time was dredging up a lot of old bad experiences and generally taking the piss out of me. Eventually, they started showing up at my apartment unannounced, which I did my best to deal with. They did that a few times, until one time when I said that I understood they drove all that way to see me, but that I needed them to leave and not show up like that without at least clearing it with me first. I felt like garbage for that too, and we didn't talk for a while after that, but eventually things did get better. They learned my boundaries, and I started to figure out how to set boundaries with people too. It's still hard to be around them some days, but we've all grown a bit and can get along well enough.
If this relates to your experience, try not to hate yourself for needing to set boundaries. Also, don't avoid setting those boundaries when you have to. You're probably feeling beat up by being around them for good reason, and aren't in the wrong for how you feel. You also don't have any responsibility to subject yourself to their behavior, but if you want to learn to be around them, you can do that too. I'm not a therapist, but I'd bet that setting gentle but clear boundaries and such will help with this in the long term.
TL;DR - If you only want to have lunch, you can let them know that. If they press, you can be nice and still stand your ground. If they're upset, they're going to have to learn to deal with that; you're your own dude.