I am pretty conservative with my cuts since I am not in a production mode. I don’t try to take more than .05" off per pass and use a 16000 RPM and a 10-30 IPM plunge/feed rate. I use a I/8" bit with a short cut length and a 1/4" shank to keep deflection down. I am sure you can use a 1/4" bit as you have with great success also but my applications have not required a lot of material removal. If I had to do a deep pocket where a lot of material must be removed I have a 1/4" roughing bit that I would probably use followed by a finishing pass with my 1/8" or possibly a 1/4" bit for large pockets. Face milling or side milling are another applications where a 1/4" or larger bits would have a lot of value but I haven’t tried to do either in metal.
I have a project coming up where I will need to drill a large grid of 6mm threaded holes in a 1" thick aluminum plate. To keep the cut time to something reasonable in this case I plan to use a 5mm spiral bit to peck drill the holes followed by a multi-form thread mill to cut the 6mm threads.