HSE have no jurisdiction over a business that has no employees.That is a business which is not limited and all people who work there are partners or a sole trader.
But what if a 'sole trader' business was working in a public area doing whatever job, and injured a member of the public due to a negligent act or omission.
HSE would surely interested in him then?
I've met HSE inspectors on a few occasions. Twice when i worked for someone else, and once while i've been on my own.
They make it clear that they are there to help, but it's also pretty obvious they will hammer you if you don't get things right.
Any Prohibition notice they issue to you (and can be for quite simple things) is permanently registered on their website for all prospective clients to view)