Some relevant data is from that drug testing for Florida welfare recipients, that didn't turn up more than a couple of percent of recipients using illegal drugs, and almost all of those were marijuana. It actually cost more money to administer the tests than the taxpayer saved in people thrown off welfare xD
The bottom line is that yeah, maybe a few people are swapping the food stamps food for other items, but it's a almost certainly very small minority who are doing that, and the stamps are still being spent in the economy, which is already proven to be the biggest possible GDP positive effect of any government program ever conceived, and not (on the whole) inflationary, since it creates more new GDP than the amount spent.
"feeding the poor" is just a beneficial side-effect. The stamps are justified by the economic modelling alone. Don't forget all that spending creates a lot of jobs, thus decreasing the total # unemployed. Some of the people serving the food stamps people would have been unemployed themselves otherwise.
Anyway, they almost all go to families with kids. That has benefits for the kids - or do we hold children fiscally responsible for their parents being poor? Feeding kids improves their school performance and health, which has benefits for the whole economy years down the track, the short-term fiscal multiplier don't take these benefits into account.