you can use the end mills for holes mostly in a drill press. Your biggest issues with the drill press using as a mill are vibrations, deflection in the quill and deflection in the post. I have a milling machine that is 700 pounds heavy, 20''x10'' table with a geared head. but is column type - and even though the column is massive there is still deflection in it, about .005 - the dovetail type mills are the desired ones and the best are the knee type but they are damn heavy + most of them require 3 phase power not to mention the price.
for small jobs you could use a cross slide table but don't expect any precision out of it.