Fast feedback is key, but I'm skeptical of the $100 figure for training nanoGPT. If you use spot instances on Lambda or RunPod you can train a model that size for less than a dollar. I've been running similar experiments recently and the compute cost is basically a rounding error.
1 comments