Diamond Robe

Diamond tops the gem robe ladder, but the distance between it and the cheap versions is small enough that mining specifically for it is a waste of a trip. The real cost is competition: diamonds also go into the strongest gem staff, and early on a better weapon beats a slightly better shirt every time. Make this one out of leftovers.

The hat is the actual decision. A Wizard Hat raises your critical rate, which suits a cheap spell you fire without stopping. A Magic Hat deepens the mana pool instead, which keeps a heavy spell going through a long fight. Neither bonus cares about your leg armor, so there is no reason to downgrade your pants for it.

Wearing any gem robe without a Wizard Hat pulls Tim into the caverns far more often than normal. That is the deliberate route to the Magic Hat. Once it is yours, put a hat back on or the spawns keep coming for no reward.