Its a problem when the work piece is at the limit of the collet. If it needs any more than a good push on the bench, the collet is too small. You need to go up a size. Problem then is you are at the other end of the limit for grip.
You can get 0.5mm increment collets (not cheap) that will hold the piece better than the next whole size up.
(I alway tap the end of the workpiece as I’m tightening the collet. Seats better giving better concentricity)
