5Dropped by
always crack them
Never sell one intact. Even the worst possible roll of gems inside is worth several times what a merchant offers for the unopened rock, and a normal roll is far past that. Throw it at a floor, sweep up what scatters, and you come out ahead every single time. Do it in an enclosed room rather than over a chasm so nothing drops into lava.
farming them
They come off the two granite enemies, which live in the granite caves and are easy to automate because the golem has a statue. Wire one to a timer, park a platform in front of it, and you have a gem supply for hooks, phaseblades and staffs without swinging a pickaxe at a single ore vein.