Chaos Fish

The reason to keep these in a chest instead of cooking them is the Teleportation Potion. Being able to jump somewhere random on the map is worth more than a meal, especially for exploring a fresh world or bailing out of a fight. If you fish the Underground Hallow at all, put every one you pull into a dedicated stack and forget about the food recipe.

You do not need Hardmode. Bring Hallowed Seeds in from another world and grow yourself an underground pocket of the biome, and the fish will show up in it well before you have broken a single altar. Some of the special world seeds start with the biome already there. Shimmer is a third route, since a Teleportation Potion thrown in reverts to the fish it was made from.

Either way they are a rare catch, so bring your best fishing power and go during the weather that helps rather than casting a hundred times with a wooden pole.