That is a good suggestion there Nobody.
I used to use that same approach approach Untill I finally bought a Mill.
Here is a very cheep way to improve on the drill press mill.
for a mere 40 bucks you can get a machinists cross slide vise that you can crank back and forth like a (mini) mill table. The vise gets secured to the drill table and you feed the part in by cranking the handles. Check out this vice at Harbor Fright!!
http://www.harborfreight.com/cpi/ctaf/displayitem.taf?Itemnumber=538
For 40 bucks you have basically a milling machine. Next step would be to buy a couple of end mills and away you go.