There is a small difference between 'cast iron' as continuously cast round, square or rectangular bar and 'a casting' that happens to be made of iron.
The first will be as you say providing it is from a reputable source. If it is an old window weight or weight-lifting weight, the quality can be more variable.
In principle, a casting should be the same but if it has thin and thick sections on it, they can cool at different rates, so the skin thickness can vary. If the casting is done badly, parts can be 'chilled' which would be very hard.
The skin is formed as part of the cooling process when the iron goes from liquid to solid, so once removed it is gone forever (it is not like the mould on cheese where even if you scrape it off, it will come back).
If the casting has been cast at the right temperature there should be no hard spots and the skin is basically sand fusing which is abrasive . I knock it off with a grinder
leave them outside to go rusty for a few month if the casting is crappy and has hard spots you can get it heat treated if its worth doing . a good tip tool will get through the skin no ptobs