Max, does your chuck have a tapered arbor?
That's the $1000 question. I expect Max understands the implications of this, but for those that don't I will explain.
Many (most/all?) hobby shop sized drill presses use a tapered arbour that the chuck is mounted to and inserted up into the column, relying only on the friction of the tapered fit to stay in place. This works great for centering the chuck and carries the torque loads from drilling very well. As a bonus, the axial thrust forces involved in drilling continuously push the arbour up into the column, keeping it tight and re-seating it if it were to come loose for any reason.
If you put a side load on the chuck without an accompanying thrust load, such as when using an end mill as shown above, it is quite possible for the tapered arbour to work loose and have the chuck fall out while the press is running. This is dangerous and hard on the equipment.
Lots of people use their press in this fashion without an issue, but it can lead to a dangerous situation and it is always good to be informed of the possible consequences of your actions before getting surprised in an unpleasant way. After that the choice is up to you.
Mark