Overfit a single batch first
Before trusting a training run on the full dataset, check that the model can drive its loss to near zero on a tiny handful of examples. If it cannot even memorize ten examples, the bug is in the code, not the data.
The image of a researcher inventing a clever new algorithm in a flash of insight is mostly wrong. Most days are spent running experiments that fail in uninformative ways, reading what a hundred other labs published this month, and trying to tell whether a promising result is real or an artifact of a lucky random seed.
The craft passed down inside the field is less about which architecture to memorize and more about experimental discipline: how to isolate what actually caused an improvement, when to trust a benchmark and when not to, and how to keep a negative result from being quietly thrown away.
What the work demands
Mathematical foundations88
Programming and systems74
Experimental rigor85
Literature synthesis66
Research taste84
Writing and communication62
Mathematical foundations
Linear algebra, probability, calculus and optimization are the language nearly every model and paper is written in; without them, new methods are unreadable rather than just unfamiliar.
Programming and systems
Writing efficient code in Python and a deep-learning framework, and understanding enough distributed-systems engineering to run training jobs across dozens or thousands of processors.
Experimental rigor
Designing controlled comparisons, running enough random seeds to trust a result, and isolating which specific change actually caused an improvement rather than assuming the obvious explanation.
Literature synthesis
Keeping up with a field publishing tens of thousands of new preprints a year, and telling which handful actually matter, is now a skill in its own right.
Research taste
Choosing which of many plausible directions is worth months of compute and attention — the hardest skill to teach directly, usually absorbed by working alongside an experienced advisor.
Writing and communication
A result that cannot be written up clearly enough to survive peer review, or explained to a non-specialist funder, struggles to have any impact regardless of how real it is.
A day in the life
- 8–10
Checking runs and reading
Reviewing overnight experiment results and skimming new arXiv preprints and lab chat discussion before the day's focused work begins.
- 10–13
Deep work: experiments and code
The best-protected block of the day, spent writing training code, debugging a model, or designing the next experiment in a series.
- 13–14
Lunch
Often eaten with labmates, and frequently the setting for the kind of informal idea-swapping that a scheduled meeting rarely produces.
- 14–16
Meetings and reading group
Lab meetings, one-on-ones with an advisor or manager, and a rotating reading group where someone presents a paper the group needs to understand.
- 16–19
Analysis and writing
Digging into why an experiment did or did not work, updating a shared results document, and drafting sections of a paper ahead of a deadline.
- 19–8
Off the clock (mostly)
Personal time — except in the weeks before a major conference deadline, when evenings and weekends routinely disappear into finishing experiments.
The know-how
Craft knowledge practitioners actually pass on — not motivation.
Tune the baseline as hard as the new idea
A surprising share of published 'improvements' shrink or disappear once the comparison baseline gets the same hyperparameter search budget as the new method — a pattern documented across GANs and language models alike.
Ablate one change at a time
When a new method with five changes bundled together beats an old one, remove each change independently and re-test before crediting any single idea — otherwise the paper's story and the actual cause of the gain can quietly diverge.
Distrust a benchmark you're also optimizing against
A leaderboard number stops measuring general ability once researchers repeatedly tune directly against it — a dynamic long known as Goodhart's law, and cited directly in critiques of benchmark saturation across NLP and vision.
Write the claims before the last experiment
Drafting a paper's core claims early forces a concrete, falsifiable target for the remaining experiments, instead of running experiments first and assembling a narrative around whatever happened to work.
Log what didn't work, not just what did
Failed configurations, seeds and hyperparameters are usually thrown away once a run fails, but recording them prevents a lab from quietly re-running the same dead end six months later under a different name.
Tools of the trade
PyTorch / JAX
The two dominant deep-learning frameworks; PyTorch leads in most academic and industry research, while JAX is popular for large-scale and Google-affiliated work.
GPU/TPU compute cluster
Shared clusters, scheduled with tools like Slurm, that turn a model design into an actual trained network — often the single biggest constraint on what experiments are possible.
arXiv
The field's de facto publication venue in practice: new results circulate here as preprints, often months before or entirely without formal peer review.
Weights & Biases / TensorBoard
Experiment-tracking dashboards that log loss curves, hyperparameters and outputs across hundreds of runs, without which comparing experiments would be unmanageable.
Jupyter notebooks
Interactive environments for quick data exploration, debugging a model's behavior on specific examples, and prototyping before code moves into a full training pipeline.
How people fail at it
Chasing leaderboard gains without a mechanism
Squeezing out a fractional accuracy improvement through extra hyperparameter search or scale, without a hypothesis for why it works, produces results that are hard for anyone else to build on even when the number itself is real.
Reporting only the best random seed
Running many seeds and publishing only the best one, instead of a mean and a spread, makes ordinary noise look like a genuine effect — a problem flagged repeatedly in reinforcement-learning and NLP reproducibility studies.
Undisclosed data or compute advantages
Test data that leaked into a pretraining corpus, or a compute budget quietly larger than a paper reports, can make a result impossible to reproduce — and once discovered, damages a researcher's credibility more than a modest, honest result would have.
Keep exploring
More in Engineering & Technology
💻
Software Engineer
Writes, tests and maintains the code that runs modern life — and is one of the first professions watching AI automate its own daily work.
AI-resistant 35 🛰️Aerospace Engineer
Designs, analyzes and certifies the aircraft, rockets and spacecraft that leave the ground, working to safety margins that leave no room for guessing.
AI-resistant 74 🔌Semiconductor Engineer
Designs and fabricates the transistors inside every computer, phone and weapon, using machines precise enough that only a few factories on Earth can run them.
AI-resistant 60