I would use a 4mm pilot as an 11mm drill will find it hard to make a start and start rubbing. Thereby blunting the drill. I am guessing you are pecking rather than holding constant pressure. A drill should be cutting or it will be blunting itself.Done, I used a block of wood on the table and lowered the chuck onto it a few times, the jaws were withdrawn inside the chuck body.
I am undecided if I should just "go for it" and try the 11mm hole without 4mm pilot or go back to the 4mm pilot scaling up to 11mm now that I have cleaned up the taper and chuck. The 540rpm lowest speed restrictance with the drill is leaning me towards the 4mm pilot then the 11mm. I think if I "go for it" at 540rpm for the 11mm hole without 4mm pilot hole I have a high chance of blunting the bit?
Rgds