Wax On, Weights Off: Knowledge Distillation Explained

What it is, how it works, why it saves money, and why it is rapidly reframing the US-China AI race

No such thing as bad student, only bad teacher.” — Mr. Miyagi, The Karate Kid (1984)

Supervised learning from a more experienced teacher is a credible strategy since the dawn of time. If it were fathers teaching how to hunt, the Platonic Academy teaching mathematics, or Ronald Fisher designing Linear discriminant analysis in 1936.

In the same vein, Daniel LaRusso in the 1984 movie did not rediscover karate from first principles. He painted the fence, waxed the car, and absorbed Miyagi’s decades of hard-won motion over a single intense summer.

The apprentice does not repeat the master’s mistakes. Copy the expert, and you compress a lifetime of trial and error into a few months.

Knowledge distillation is exactly this.

But now you have millions of students harvesting the teacher’s intelligence on an industrial scale. Pretty much all relevant Chinese Frontier Labs have been accused of mind-raping Claude to speed up their training and especially Anthropic stopped having it. In February 2026, Anthropic announced it had caught three Chinese labs, DeepSeek, Moonshot, and MiniMax, running industrial-scale campaigns to extract Claude’s capabilities. As a result, we have seen Anthropic and the US Department of Commerce taking action and restricting (and later re-establishing, probably due to better guardrails and revenue concerns) access to their most powerful models.

In this post, I will be explaining what knowledge distillation is, how it helps you in post-training, and what the implications on the tech stack of frontier labs is.

Table of Contents

1. Where Distillation Comes From

2. Three Types of Distillation

3. Why It Is a Distributed Attack

4. The Architecture and the Data

5. The Economic Equation

6. The Frontier Developer’s Tech Stack

7. Competitive Advantage or Capability Trap?

8. In Closing

Where Distillation Comes From

In the original Hinton (2015) sense, distillation is training a smaller student model to reproduce a larger teacher model’s behavior, instead of learning the underlying task from scratch.

source: https://arxiv.org/abs/1503.02531 - Table 1

Hinton writes:

Starting from the trained baseline full network, the specialists train extremely fast (a few days instead of many weeks for JFT). Also, all the specialists are trained completely independently.

Naturally, in an ultra-competitive world where weeks ahead mean billions of dollars in revenue, speed matters. If a frontier model is reachable through an API, it can be distilled by querying it at scale and training on the transcripts. That single fact turned distillation from a lab technique into a competitive weapon.

This is why DeepSeek-R1 became such a flashpoint in early 2025.

Three Types of Distillation

Before we go further, it helps to know that “distillation” is not one technique and has expanded from Hinton’s initial concept.

Response-based distillation is the simplest and most commonly used distillation method today. The student trains on the teacher’s outputs. In the strong form, it matches the teacher’s full probability distribution over tokens, the soft labels Hinton originally described, which carry far more information than a single right answer because they encode how confident the teacher was and what it almost said instead. In the weak form, when all you have is an API that returns text, the student trains on the generated completions themselves.

Feature-based distillation goes deeper. Instead of matching outputs, the student matches the teacher’s internal representations, the activations in its hidden layers. While this transfers richer structure, it requires access to the teacher’s internals. You cannot do it through an API. And therefore is much more expensive and risky. It is a technique for labs distilling their own models, not for harvesting someone else’s.

Relation-based distillation is subtler still. It teaches the student to reproduce the relationships between the teacher’s representations, how examples sit relative to each other, rather than the representations themselves. It is more of a research direction than a workhorse, but it matters because it points at what distillation is really transferring: not answers, but structure.

The US/China capability war lives almost entirely in the first category. Response-based distillation is the only flavor you can run against a closed model you do not own, which is exactly why it works. And it has worked for China for almost 3 decades. While China established the socialist market economy already in 1978, the true opening to the world only happened in 2001 when China joined the World Trade Organization. If you wanted to invest in the growing Chinese market, foreign firms had to enter joint ventures with Chinese partners, often holding at least 51% to keep control. This facilitated the transfer of technology and know-how to domestic companies, to the effect that many products can not be built in the West anymore.

Of course, Chinese companies, due to their own ingenuity and innovative spirit, have by now surpassed most Western companies in quality and price.

That should not excuse executing a distributed attack to gain an edge in the frontier wars.

Why It Is a Distributed Attack

Dorothy, we are not training narrow logistic regression models for Credit Scoring anymore. Distillation against a closed model takes the shape of a distributed attack and not a single exploit because there is no choke point. Thousands of API accounts, each issuing traffic that looks unremarkable, maybe even successful for the AI provider, on its own, collectively harvest the model’s behavior across the task distribution. This form of IP-Theft is much easier. There is no insider access. No smuggled hard drive. No post-resigning access to company resources. Just API keys and a prompt strategy. The product surface is the attack surface. API abuse is remarkably easy to track and might also suffer from stylistic fingerprinting or watermarking. Therefore, a serious attacker needs to spread data collection across accounts, IPs, and time. Then paraphrases the outputs to defeat. A team running 50,000 API calls to build an eval harness might look identical to a team running 50,000 calls to build enterprise software.

So how did Anthropic detect it then?

To answer this question, we have to go one level deeper in understanding how these attacks work.

But let’s start with what makes them not work.

You do not randomly ask a handful of manually selected questions. You use self-instruct loops or Evol-Instruct-style prompt mutation to sample the teacher broadly, covering the task distribution instead of the easy cases.

When executing the Self-Instruct loop, you seed a small set of instructions, prompt a model to generate new ones in the same style, filter duplicates/low-quality outputs, generate answers for the survivors, fold them back into the seed pool, repeat. Optimizes for diversity of tasks.

seed_pool = load_seed_instructions()  # ~100-200 human-written

for iteration in range(N):
    sampled = sample(seed_pool, k=8)
    new_instructions = model.generate(
        prompt=f"Given these examples: {sampled}, generate 5 new diverse instructions"
    )
    new_instructions = dedupe(new_instructions, seed_pool, threshold=0.7)
    new_instructions = filter_quality(new_instructions)

    for instr in new_instructions:
        output = model.generate(instr)
        seed_pool.append((instr, output))

save(seed_pool)

Then Evol-Instruct mutation is taking an existing instruction and rewrites it via fixed mutation templates. Then you add a constraint, deepen reasoning, concretize, broaden to a new domain, then filter and feed survivors into the next generation. Optimizes for difficulty progression on existing tasks.

mutation_prompts = {
    "deepen": "Add constraints or reasoning steps to make this harder: {instr}",
    "concretize": "Replace general concepts with more specific ones: {instr}",
    "broaden": "Create a new instruction in a different domain, similar difficulty: {instr}",
    "increase_reasoning": "Rewrite to require multi-step reasoning: {instr}",
}

pool = load_seed_instructions()

for generation in range(N):
    next_gen = []
    for instr in pool:
        mutation = random.choice(list(mutation_prompts))
        evolved = model.generate(mutation_prompts[mutation].format(instr=instr))

        if passes_quality_check(evolved) and not is_too_similar(evolved, instr):
            next_gen.append(evolved)
        else:
            next_gen.append(instr)  

    pool = next_gen

save(pool)

Together, this is called Query Diversification.

If you only train on prompt and response pairs, you are teaching the student only what the answer is, not how to arrive at that answer. Therefore, you need to enrich the captured signal with full reasoning traces.

While you need the full reasoning traces, these raw scrapes are extremely noisy. To improve this, you need to apply data engineering methods to filter and deduplicate your data points. Two useful methods are embedding-based clustering for killing near-duplicates and a heuristic approach to strip out the low-value samples. Sometimes LLM-as-a-judge is used, but I am not a fan of this method.

Then you can start training the student model. Here, you perform supervised fine-tuning on the harvested pairs, often followed by preference optimization like DPO or KTO if you also captured comparative judgments. Because the student is fitting an already-compressed function instead of learning from raw reality, this converges far faster than pretraining ever could.

In theory, that’s fine. Let’s build a training pipeline.

The Architecture and the Data

If you draw our pipeline as a diagram, most people put the training run in the center. That is the wrong center.

By 2026, the training run is the easy part. A night on rented H100 GPUs with an off-the-shelf recipe does the trick for most builders. The real system, as most of you have already expected, is a data-engineering beast. Almost every design decision that matters is a decision about data.

Here is how the pieces fit.

At the front is the prompt generation layer. This is the engine that decides what to ask the teacher, and it is where the whole attack lives or dies, because your student can only be as good as the slice of the teacher’s behavior you managed to sample. The naive approach would be to ask whatever questions the AI engineer happens to come up with and end up with a student who is sharp on trivia and useless everywhere else. The better approach is to treat coverage as an optimization target. You seed with a taxonomy of the capabilities you actually want, then run a self-instruct loop (see above) where an LLM generates thousands of task variants from a small set of hand-written exemplars, then layer Evol-Instruct on top to mutate each prompt along two axes, deeper (add constraints, edge cases, multi-step reasoning) and wider (rephrase into new domains). The output is not a prompt list. It is a synthetic distribution engineered to tile the region of capability space you are trying to steal.

Next is the collection layer, and this is where you fight the defenses. As mentioned, you want the richest signal the API is willing to surrender. Full outputs at minimum, chain-of-thought traces if the teacher exposes them, and log-probs or top-k token distributions if you are lucky enough to get them, because soft targets carry orders of magnitude more information per sample than a hard label does. This layer is also where the “distributed” in distributed attack becomes concrete: request volume is spread across many accounts, IP ranges, and time windows to stay under per-key rate limits and below the anomaly-detection threshold, with retry logic, temperature sweeps to capture output diversity, and sometimes a paraphrase pass to launder stylistic fingerprints out of the harvested text before it ever hits storage.

Then the curation layer, which is the part amateurs skip and professionals obsess over. Raw harvest is 60 to 80 percent waste, and training on the waste actively poisons the student. So you build a filtering cascade. Exact and near-duplicate removal through embedding-based clustering, so the model does not overfit to whatever the prompt engine happened to over-sample. Quality scoring, increasingly by an LLM judge rather than heuristics, to drop incoherent, refused, or low-value completions. Correctness filtering wherever you have ground truth, so on math and code, you keep only the traces that actually verify, which is the single highest-leverage move in the entire pipeline because it converts the teacher’s confident-but-wrong answers into discarded rows instead of learned behavior. Format normalization so everything downstream sees one schema. What survives is a fraction of what you collected and is worth far more per token.

Underneath all of it sits the data engineering. Unglamorous and decisive, its a fundamentally important versioned dataset store so you can trace exactly which harvest produced which checkpoint, deduplicated blob storage for the raw transcripts, a feature or example store keyed for reproducible train and eval splits, and a lineage layer that lets you answer “where did this capability come from” when the student does something surprising.

Skip this, and you are not running a research program; you are building a slot machine (and you are not the house).

Only after all, and only after all of that, does the training layer begin.

The rest is mechanical. Supervised fine-tuning on the curated pairs, LoRA or QLoRA to keep the run cheap, optional preference optimization with DPO or KTO if you captured comparative judgments, and a held-out eval harness to measure how much of the teacher actually transferred. Notice the shape of the effort. Four-fifths of the work happened before the first gradient step.

In the end, as for most ML tasks. It is a data pipeline with a training run slapped to the end, not the other way around.

I suppose that also explains why it is comparably cheap.

The Economic Equation

Pretraining a frontier model means paying for discovery. In this context, discovery can mean three things.

1. What ratio of code to math to natural language to multilingual text yields the best reasoning? Nobody will know this in advance. The lab has to find it through ablation runs, each one a costly (time and token) experiment.

2. A priori, the learning rate schedule and the width-versus-depth tradeoff are unknown. Therefore its difficult to assess where to spend the parameter budget. These are settled through scaling-law studies and failed runs that tell you what breaks at 1/10th size before you commit the full compute.

3. We don’t know in advance what good behavior even looks like. Reward modeling, a prerequisite for RL to work, requires RLHF to collect large-scale human preference data to teach the model what a helpful, correct, well-reasoned answer is. That labeling effort is discovery too, and it’s a big share of the cost.

To execute it, efficient data architectures, separate highly-skilled teams, and support functions need to work like a clockwork to not waste trillions of tokens and months of cluster time. Repeated failed runs while the team figures out what actually works is depressing morale.

That’s where Knowledge Distillation is so insidious. It only pays for compression. The target function’s behavior is already known, so the student just has to fit a smaller hypothesis space to match it. The attacker skips any risk on data curation, architecture search, filtering, and most of the alignment tax.

This simplifies the capital expenditures of the distiller compared to the work the model developer needs to invest in.

The Frontier Developer’s Tech Stack

A frontier developer’s stack starts long before any GPU is rented. It also starts with data pipelines. But here these pipelines scrape, license, and deduplicate web-scale corpora, run them through classifiers for quality and toxicity, and mix in code, math, and curated instruction data.

Pretraining then runs on frameworks built for thousands of accelerators talking to each other constantly. Megatron-LM or its successors for tensor and pipeline parallelism. DeepSpeed or PyTorch FSDP for sharding optimizer state. JAX and XLA on TPU pods for the labs that go that route. All coordinated through Slurm or Kubernetes-based orchestration, with Ray increasingly common for the data and RL loops wrapped around it.

Layered on top of raw pretraining is the alignment stack.

1. Supervised fine-tuning on curated instruction data.

2. Reward modeling built from large-scale human preference collection. And especially this data point is extremely expensive to collect.

Each one of these carries its own labeling platforms, annotator pools, and reward-hacking countermeasures.

Running alongside all of that is an evaluation and observability layer. Internal benchmark suites that go well beyond public leaderboards. Experiment tracking through something like Weights & Biases or MLflow. Dedicated red-teaming and safety pipelines that probe for jailbreaks, bio and cyber uplift, and policy violations before anything ships.

The Fable shutdown, incidentally, was about exactly this layer failing to catch something in time.

Also, inference service defensive tooling adds a layer that never shows up in a distiller’s balance sheet at all (unless they start to sell their own frontier model), rate limiting, output watermarking, log-prob suppression, and anomaly detection on usage patterns.

Competitive Advantage or Capability Trap?

Obviously, expenditures in data collection and data engineering are substantially reduced when using the distillation shortcut. It quickly reproduces a similar capability level without ever doing any of the research that produced it.

That is why it is contentious, not merely efficient. And it is why it sits at the center of the capability race.

Distillation lets a fast follower absorb years of another lab’s R&D in weeks. I like to call this advantage “fast-follower economics”. The Fast-follower strategy, in a nutshell, is executed by deliberately letting the competitor be first to market and profiting by moving second, but operating cheaper.

There is no path to genuine frontier leadership this way. Only catch-up will ever be possible. You are, by construction, always one model behind Near-frontier performance. It is a real accelerant for open-source ecosystems and edge deployment. Even though none of the “open-source” labs are truly open-source, merely open-weights.

But there are further downsides to distilling. The distilled student will always inherit the hard capability ceiling bounded by its teacher. It inherits the teacher’s blind spots and errors along with its strengths; besides the real legal and ToS exposure. But then one might argue that LLM outputs are not IP.

For the teacher, the disadvantage is obvious. Distillation commoditizes the expensive part of the moat. It clones frontier capability into cheaper competitors and undercuts the pricing that capability was supposed to re-finance.

The indirect advantage is that it forces API hardening. Rate limiting, output watermarking, ToS enforcement, sometimes withholding log-probs or reasoning traces entirely. Done well, this genuinely strengthens the product surface. But that hardening comes at the cost of friction for legitimate users. And as Fable showed, sometimes the hardening is not even a decision the lab gets to make on its own timeline.

In Closing

We already know that DeepSeek reportedly asked Claude to articulate the step-by-step reasoning behind a completed answer, which is chain-of-thought elicitation, harvesting the trace exactly as I had described above. We also know that they used Claude as a reward model for reinforcement learning through rubric-based grading and generated censorship-safe rewrites of politically sensitive queries to train their own steering behavior.

We also know that Moonshot went after agentic reasoning, tool use, and coding, then pivoted in a later phase to reconstructing Claude’s reasoning traces specifically.

MiniMax, the largest at over 13 million exchanges, was caught mid-campaign, and when Anthropic shipped a new model during that campaign, MiniMax redirected roughly half its traffic to the newer system within 24 hours. The apprentice does not wait for the next lesson. He copies it the day it is taught.

If you are building agents, and I know most of you reading this are, the lesson is not “go distill a frontier model.” The lesson is to know whether you are building your agent on the master or the apprentice. In the latter case, you will always be behind at least a few weeks.

While the apprentice learns fast and gets good cheaply, the apprentice may never surpass the teacher they copied. Unless the Chinese develop a means to surpass the master. And given the state of the automotive and manufacturing industries, this scenario is far from unlikely. European labs won’t win if they don’t play either game.

In my opinion, both are effective strategies to get to an SOTA model. But in my opinion, one way is better than the other. Karate Kid is a great movie. But Miyagi is the one who knows the whole art. Daniel just learned enough of it to win one tournament.

Know which one you are building on.