First: decide if you want a discreet graphics card right away, or if you might consider waiting till later to keep your current build cheap.
I will assume you want to keep your current build cheap. Look at the AMD A series processors, they include a decent on chip graphics package at an affordable price.
Badass! $120:
http://www.newegg.com/Product/Product.aspx?Item=N82E16819113280Decent! $75:
http://www.newegg.com/Product/Product.aspx?Item=N82E16819113282Next, pick out a motherboard that works. Socket FM2 for the above processors. The decision for motherboard really makes a difference in the long run. Ignore everything that does not have at least one PCIE 2.0 x16 or better slot. This means you can upgrade your shit with better GPU if you decide you want to stomp some pixels.
Form factor will affect things. If you are going for affordability, just stick with the nearly universal ATX.
A cheap board meeting the above qualifications might cost around $50 in either ATX or micro-ATX.
http://www.newegg.com/Product/Product.aspx?Item=N82E16813130661Do you want more than 2 memory slots? that will cost around $85 in an ATX or micro-ATX.
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157334A mini ITX will be closer to $90.
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157340Add some ram, DDR3 is cheap as hell, 8 gigs for less than $35, lots of choices from value and performance. This is only one of many to choose from:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148543Note: verify that your memory is on the tested compatibility list for your motherboard! This will save you some headaches.
Now you have the core of a computer. select a case/power supply, make sure it will fit your motherboard, make sure it has at least 1 6/8 pin GPU rail and 500 watts (this is future proofing for an eventual GPU upgrade), make sure it has power connectors for whatever type of drives you are using, be they PATA or SATA.
This will cost you somewhere between $30 and $100 depending on how fancy you want your stuff.
By now, you probably spent around $200-300.
Hard drive? doesn't matter much for a budget build, 500gigs is a-ok.
Optical drive? those are also cheap or can be scavanged.
Windows 7? better than windows 8.
By now you probably spent around $400-500.
This gives you a decent mid range PC good enough for even modern games (at low/medium detail settings)
Adding a graphics card capable of pushing medium/high detail settings for foreseeable future games) is a modest proposal that might set you back another $100-$200.