Stanley knife or a sharp scraper will take care of most, if not all of it.
The silicone eater described above really only works best on thin layers of silicone
What you may have might not be silicone, but Acrylic frame sealant. Only scraping will get that off..
Not sure if it would work on brick, but I cut the excess off with a blade, then soften with silicone eater, then wipe off. Works a treat on non porous surfaces.
spured on by this, i have used the screwfix cheapo WD 40 clone and can confirm it removed the manky silicone round my sink...ill give it a couple of days then re do the joint.