As far as I can tell, the issue is if you work with it all the time. You could probably (it's not like I'm advising it) spend a weekend smashing it up in a sealed basement with no ppe, and you would still die of something else first.
Have a job smashing up aesbestos in basements with no ppe (or, for example, lagging pipework in ships, for a more real world example) and you're going to suffer.
But (as I understand it) it's not really like poison or high dose radiation, where you get got once and you're a gonner. Unless you work with it on a repeat basis, somthing else is going to have you first.
Unless you're real unlucky. But then with luck that bad, the bus will probably hit you first anyway.
Ymmv. No recommendations given.
I dont believe it has ever been proven or disproven that 1 single fibre can/ cant cause asbestosis. So one weekend might well be the end of you. Just depends on your luck.
I spent 5/6 years playing/ climbing on/ through our old broken asbestos roof shed at our old house when i was a wee lad
But as you say its more than likely il be a goner with something else!