Design a data labeling platform

The most surprising number in the InstructGPT paper is not a parameter count. Outputs from the 1.3B parameter InstructGPT model were preferred by humans over outputs from the 175B GPT-3, a model more than a hundred times larger, and the thing that closed that gap was a labeling operation, about 40 contractors working through 13k supervised fine-tuning prompts, 33k reward-model prompts, and 31k RL prompts. Meta ran the same play at industrial scale for Llama 2, collecting 27,540 SFT annotations and 1,418,091 binary preference comparisons in weekly batches. Every aligned model you have used sits on top of a system like the one this article designs, a full-stack application that hands tasks to human annotators, collects their judgments, decides which judgments to trust, and exports datasets that SFT and preference-tuning jobs consume directly.

The machine side of this system is deliberately unimpressive. Even a Llama 2 sized effort produces well under one label write per second on average, so there is no sharding story and no fan-out story, and a single Postgres instance with a replica carries the whole thing. What makes the design hard is that the product being manufactured is human judgment, and humans disagree, get tired, drift from the guidelines, and respond to incentives. The platform therefore has two jobs, a small coordination problem, getting the right next task in front of the right annotator without duplicates or stuck work, and a large measurement problem, pricing the reliability of every label it collects so the training pipeline downstream can trust what it eats.

The walkthrough builds the coordination layer with Postgres as the source of truth and Redis for leases and dedup, then spends most of its time where the difficulty actually lives, consensus and gold tasks, agreement metrics like Cohen's kappa and Krippendorff's alpha, model-assisted pre-labeling with its anchoring hazard, and the preference-pair loop that feeds RLHF and DPO.

Scope and requirements

Functionally, ML teams submit batches of tasks, a task being a self-contained unit of work such as a prompt with two model responses to compare, a document to classify, or a prompt for which an annotator writes an ideal response. The platform queues those tasks, routes each one to a qualified annotator, serves an annotation UI fast enough that nobody waits, records every judgment, resolves disagreements, and exports finished datasets. Two task families cover the fine-tuning world. Demonstrations, where the annotator writes the good output, feed supervised fine-tuning, and comparisons, where the annotator picks the better of two model outputs, feed reward models and DPO. The same machinery serves classic classification and span tagging, which is why general tools like Label Studio model all of them as one task format.

Non-functionally, the constraints are unusual for a systems design because the scarce resource is paid human attention. Quality must be measurable, per annotator, per project, and per week, because a batch of quietly bad labels can set a fine-tune back a full training cycle. Task fetch must be fast, under a couple hundred milliseconds, because a thousand annotators each losing two seconds between tasks is real money at the hourly rates involved. Every label must be auditable, meaning traceable to who produced it, when, how long it took, which guidelines version was live, and whether a model pre-label was on screen. And the system must assume annotators disappear mid-task constantly, closed tabs, ended shifts, dropped connections, so work assignment has to heal itself without an operator. Out of scope here are workforce sourcing and payments, audio and video annotation tooling, and 3D tasks, though the task model extends to all of them.

The shape of the real operations is worth internalizing before designing anything. OpenAI ran InstructGPT with a team of about 40 contractors. Meta collected Llama 2 preference data in weekly batches over months, adjusting instructions between batches as the model and the edge cases evolved. A labeling platform is an operations product as much as a software product, and the design below keeps humans, their trust scores, and their disagreements as first-class entities rather than as rows bolted on later.

Sizing the problem

Two real datasets anchor the arithmetic. InstructGPT's entire human-data footprint was 13k SFT prompts, 33k reward-model prompts, and 31k PPO prompts, small enough that about 40 people produced it. Llama 2 sits at the other end, 1,418,091 binary comparisons plus 27,540 SFT annotations. Take the Llama 2 scale as the target and make the assumptions explicit. Assume a binary comparison of two chat responses takes 3 minutes of reading and judging on average, so one annotator produces about 20 comparisons per hour. Then 1.4 million comparisons cost about 70,000 annotator-hours, and a pool of 250 annotators working 30 hours a week clears it in just over nine weeks, consistent with the weekly batches collected over months that the Llama 2 paper describes. At a fully loaded $20 per hour that is $1.4M of labor before any quality control. Layer on the quality machinery designed below, gold tasks at about 10% of everything an annotator sees and a second vote on a 15% audit slice, and the bill rises by about 1.3x, to roughly 90,000 hours and $1.8M.

Now look at what the machines see. Ninety thousand hours of work spread over ten weeks is under one annotation write per second on average, a few per second at shift peaks. The task table holds a couple million rows. A single Postgres instance with one replica is comfortable, Redis barely notices, and nothing in this system ever needs a shard key. The scaling problem lives in the budget, not the database, which inverts the usual systems-design instincts.

The inversion tells you where optimization effort pays. Shaving 10 seconds off the average comparison, through a faster UI, keyboard shortcuts, or a useful pre-label, saves about 3,900 hours across 1.4M tasks, roughly $78k, more than any infrastructure decision in this article is worth. The consequence runs through the whole design, and it is why the annotation serving path and the pre-labeling sections below obsess over seconds while the storage sections stay simple.

The data model in Postgres

Four entities carry the design, a task, an annotator, an annotation, and the project that groups them. The task payload is JSONB because task shapes vary wildly across projects, a comparison carries a prompt and two responses, a classification task carries a document and a label set, and forcing either into fixed columns buys nothing. This mirrors Label Studio's format, where a task is JSON with a data key, completed work lives in an annotations array, and model output lives in a predictions array with a confidence score, and each annotation records lead_time and completed_by. Those last two fields look like trivia and are load-bearing, lead time feeds the cost model and catches annotators who answer faster than the text can be read, and the annotator id is the join key for every trust computation in the system.

The one rule that outranks the rest is that annotations are append-only. An annotation is never updated in place. If an adjudicator overrules a label, that is a new row from a new annotator id, and the task's resolution changes, not the history. Every agreement metric in this article is a computation over the full set of votes, and a platform that overwrites labels has destroyed the raw material for its own quality measurement. The unique constraint on (task_id, annotator_id) enforces one vote per person per task and quietly makes submission idempotent, a retry after a network blip lands on the constraint and becomes a no-op.

Task state is an explicit enum, queued to leased to submitted, then needs_overlap when more votes are required, then resolved or escalated. The priority column is the hook the active-learning section fills in later, and is_gold with a stored gold_answer is the hook for the quality section. Gold tasks live in the same table as real tasks on purpose, because the entire point of a gold task is that nothing downstream of the ingestion job can tell it apart.

CREATE TABLE annotators (
  annotator_id   BIGSERIAL PRIMARY KEY,
  email          TEXT UNIQUE NOT NULL,
  skills         TEXT[] NOT NULL DEFAULT '{}',   -- {'preference','code','medical'}
  trust_score    REAL NOT NULL DEFAULT 0.5,      -- rolling gold-task accuracy
  active         BOOLEAN NOT NULL DEFAULT TRUE
);

CREATE TYPE task_state AS ENUM
  ('queued','leased','submitted','needs_overlap','resolved','escalated');

CREATE TABLE tasks (
  task_id        BIGSERIAL PRIMARY KEY,
  project_id     BIGINT NOT NULL,
  payload        JSONB NOT NULL,        -- prompt, responses A and B, doc refs
  content_hash   BYTEA NOT NULL,        -- SHA-256 of normalized payload
  is_gold        BOOLEAN NOT NULL DEFAULT FALSE,
  gold_answer    JSONB,                 -- set only when is_gold
  required_votes SMALLINT NOT NULL DEFAULT 1,
  state          task_state NOT NULL DEFAULT 'queued',
  priority       REAL NOT NULL DEFAULT 0,   -- active-learning uncertainty
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (project_id, content_hash)     -- dedup backstop, Redis is the fast path
);

-- Append-only. Adjudication adds rows, nothing is ever updated in place.
CREATE TABLE annotations (
  annotation_id  BIGSERIAL PRIMARY KEY,
  task_id        BIGINT NOT NULL REFERENCES tasks,
  annotator_id   BIGINT NOT NULL REFERENCES annotators,
  result         JSONB NOT NULL,        -- {"choice":"B","margin":"better"}
  lead_time_ms   INT NOT NULL,          -- wall-clock time on the task
  prelabel_shown BOOLEAN NOT NULL,      -- for anchoring analysis later
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (task_id, annotator_id)        -- one vote per person, idempotent submit
);

CREATE INDEX ON tasks (project_id, state, priority DESC);

Task ingestion, leasing, and dedup

Ingestion accepts batches over an API and normalizes each task, then computes a SHA-256 over the normalized payload. Dedup runs twice by design. A Redis set membership check rejects duplicates cheaply at ingestion speed, and the UNIQUE (project_id, content_hash) constraint in Postgres is the backstop that holds when Redis has been flushed or the ingestion job raced itself. Duplicate tasks are worth this paranoia because they burn budget twice for one unit of information, and because a duplicated task that lands in both a training export and an evaluation slice leaks between the two. Accepted tasks are inserted as queued and their ids pushed onto a per-project Redis list that serves as the pending queue.

Assignment is the part most first designs get wrong, by writing an assigned_to column and calling it done. Annotators abandon tasks constantly, and a static assignment turns every abandonment into stuck work that an operator has to notice and free. The right primitive is a lease, a time-boxed exclusive claim that expires on its own. Redis is built for this. The API pops a task id off the queue and runs SET lease:{task_id} {token} NX PX 600000, which succeeds only if no lease exists and evaporates after ten minutes, exactly the simple locking pattern the Redis SET documentation describes. The token is a random value, and release runs a small check-and-delete script that only deletes the lease if the token matches, which stops a slow client from releasing a lease that has already expired and been granted to someone else. The UI heartbeats to extend the lease while the annotator is genuinely working, so a long task survives and a closed tab does not.

The failure semantics deserve to be stated plainly, because they are the same at-least-once contract that shows up in every distributed design. A lease can expire while its human is still typing, so two people can occasionally work the same task, and the design accepts this rather than pretending to prevent it. Both submissions land as valid votes, the unique constraint absorbs any retry of either, and the overlap becomes free agreement data. Postgres remains the source of truth throughout. A sweeper job re-queues any task sitting in leased state whose lease key no longer exists, and because queue and leases are both reconstructible from the tasks table, Redis can lose everything and cost the system a rebuild, never a label.

LEASE_TTL_MS = 600_000            # 10 minutes of annotator attention

def lease_next_task(r, project_id):
    while True:
        task_id = r.lpop(f"queue:{project_id}")   # next pending task id
        if task_id is None:
            return None                            # queue drained
        token = secrets.token_hex(16)
        # NX: only if no one holds it. PX: it expires on its own if the
        # annotator vanishes, no operator needed to unstick the task.
        if r.set(f"lease:{task_id}", token, nx=True, px=LEASE_TTL_MS):
            return task_id, token
        # a still-leased id slipped back into the queue, skip it

RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1]
then return redis.call('del', KEYS[1]) else return 0 end
"""

def submit(r, db, task_id, token, annotator_id, result, lead_ms):
    # UNIQUE (task_id, annotator_id) makes this safe to retry, and a
    # submit that arrives after lease expiry is still a valid vote.
    db.insert_annotation(task_id, annotator_id, result, lead_ms)
    r.eval(RELEASE, 1, f"lease:{task_id}", token)  # release only our own lease

Postgres is the source of truth for every state transition. Redis holds only the disposable coordination state, the dedup set, the pending queue, and the per-task leases, so losing Redis costs a rebuild and never a label. The consensus worker reads the votes and writes each task's resolution, and the export job turns resolved tasks into JSONL for the training pipeline.

The annotation serving path

When an annotator finishes a task, the next one must already be waiting, because the sizing section priced every idle second. The fetch path runs in one round trip. The API pops candidate ids from the project queue, filters against two conditions, the annotator's skills array must cover the project's requirement and the annotator must not already have a vote on that task, then takes the lease and returns the full payload, any model pre-label with its confidence score, and the guideline snippet relevant to the task type. The client renders the current task and prefetches the payload of the next leased task in the background, so task-to-task transition is a local render rather than a network wait. Skill routing matters more than it looks, the preference work behind modern assistants leans on annotators with real domain depth, and Surge AI, which supplied RLHF labeling for Anthropic's early assistant work, describes staffing exactly that way, with labelers bringing expertise across law, medicine, business, and STEM.

The UI itself is a throughput instrument. Comparisons render both responses side by side with the choice and a margin selector on single keys, and a submitted task advances immediately with the write happening asynchronously behind the transition. The client stamps lead_time_ms from first render to submit and sets prelabel_shown, and both land on the annotation row. Lead time distributions are the cheapest fraud detector the platform has, a labeler whose median comparison time is 20 seconds on 800-word responses is not reading them, and the downstream analysis of pre-label anchoring is impossible unless the platform recorded what was on screen at judgment time.

One deliberate omission, the serving path never shows an annotator how others voted on the same task. Blind voting is what makes the agreement math in the next sections meaningful, and platforms that surface a running tally converge on the first vote rather than on independent judgment.

Consensus, gold tasks, and annotator trust

For categorical tasks the platform buys redundancy directly. required_votes is set to 3, the consensus worker resolves a task when a majority agrees, and a task with three ways to disagree escalates to an expert adjudication queue, which is a separate project with a smaller, higher-trust pool. But redundancy has a price the preference world cannot pay, triple-labeling Llama 2's 1.4M comparisons would have tripled a seven-figure budget. So at preference scale the bulk of tasks get one vote, and quality control shifts from per-item redundancy to per-annotator measurement, with a 15% audit slice double-labeled to keep an unbiased estimate of pool-level agreement. OpenAI's InstructGPT team ran exactly this kind of measurement and published the numbers, training labelers agreed with each other 72.6% of the time on ranking tasks, and held-out labelers hired for validation agreed 77.3% of the time.

Gold tasks are the per-annotator instrument. A gold task carries a known correct answer, written by the project owner or settled by expert adjudication, and the ingestion job injects gold at about 10% of what every annotator sees, indistinguishable from real work because it lives in the same table, the same queue, and the same UI. Each annotator's rolling accuracy on gold becomes trust_score, and trust gates everything, new annotators work a pure-gold qualification set before touching production tasks, a trust dip below threshold routes the annotator's recent work into review and their queue back to qualification, and payment disputes resolve against gold rather than against opinion. The reason gold exists when consensus already does is subtle and worth stating, consensus measures agreement with the pool, gold measures agreement with the truth, and a pool can drift together, especially when everyone misread the same ambiguous guideline paragraph.

Trust also improves resolution itself. A plain majority weights a three-week veteran at 0.95 gold accuracy the same as a new hire at 0.62, so the consensus worker weights votes by trust score, which converges faster and lets marginal tasks resolve with two votes when both voters are high-trust. The failure mode to guard against is treating trust as permanent, gold accuracy is a rolling window, not a lifetime average, because the useful signal is drift, the moment a previously reliable annotator starts slipping as guidelines evolve under them.

Measuring agreement

Raw percent agreement is the number everyone reaches for and it flatters skewed tasks. Run the standard correction on a concrete case. Two annotators label 100 items for toxicity, they agree on 90, and the naive read is that the task is well-defined. But suppose the matrix is 82 items both called clean, 8 both called toxic, and 10 splits, with annotator A marking 12% toxic overall and annotator B marking 14%. Two annotators guessing independently with those base rates would agree by pure chance p_e = 0.12 × 0.14 + 0.88 × 0.86 = 0.774 of the time. Cohen's kappa asks how much of the observed agreement survives after chance is removed, κ = (p_o − p_e) / (1 − p_e) = (0.90 − 0.774) / (1 − 0.774) ≈ 0.56. The 90% headline was mostly the skew talking, and the task definition has real work left.

Kappa's limitation is structural, it is defined for exactly two fixed annotators with no missing labels, and a production platform has neither, it has a rotating pool where each task is seen by whichever one to three annotators happened to lease it. Krippendorff's alpha is built for that shape, it handles any number of annotators, tolerates missing labels, and generalizes across nominal, ordinal, and interval data, which matters the moment the label is a 1-to-7 rating where disagreeing by one step should cost less than disagreeing by six. The platform computes alpha per project per week from the audit slice and treats it as the primary health metric, and the operational rule is to read changes, an alpha that steps down after a guidelines edit means the edit created ambiguity, and one that decays slowly means the workforce is drifting.

Calibrate expectations against published reality before panicking at the absolute numbers. InstructGPT's 72.6% inter-labeler agreement on rankings is the honest ceiling for subjective preference work, nothing like the high-nineties agreement of clean objective tasks. Tooling agrees on the mechanics here, Label Studio Enterprise computes agreement through consensus and pairwise methodologies built on per-type matching functions, exact match for choices, intersection over union for boxes, span overlap for text, rather than one universal coefficient, and that matching layer is exactly what a platform needs anyway, because structured labels have to be reduced to match scores before kappa or alpha can be computed over them at all.

Pre-labeling and active learning

Once any model exists, ingestion runs it over incoming tasks and stores its output alongside the payload, the pattern Label Studio formalizes as a predictions array shaped like an annotation plus a confidence score. The annotator's job then shifts from creating to verifying, confirm the pre-label or fix it, and the sizing section already priced what that is worth, seconds saved per task compound into tens of thousands of dollars across a Llama 2 sized run. Pre-labeling is the single highest-yield feature in the platform, and it ships with the platform's most insidious hazard.

The hazard is anchoring. Agreeing with a plausible-looking pre-label is less effort than judging from scratch, so a tired annotator's accept rate climbs and the dataset quietly inherits the model's own errors, which is a feedback loop wearing a productivity feature's clothes. Three guards are built in from day one. Per-annotator accept rate is monitored against the pool, an outlier who accepts everything is rubber-stamping. Gold tasks are never pre-labeled, because a pre-labeled gold measures the model-plus-annotator system instead of the annotator, and the prelabel_shown flag on every annotation is what lets an analyst later separate the two populations cleanly. And low-confidence pre-labels are suppressed entirely, showing a coin-flip guess anchors the human without saving any time.

Active learning decides which tasks deserve humans at all, and it is the reason the tasks table carries a priority column. Instead of labeling a random sample of the unlabeled pool, score the pool with the current model and spend annotator-hours where the model is least certain, ranking by prediction entropy or by the margin between the top two classes, written into priority so the queue serves uncertain tasks first. The mirror-image trick runs on the labeled side, tasks where the model confidently disagrees with a single human vote are exactly the tasks most worth a second opinion, so the consensus worker routes model-annotator disagreements into the overlap slice. Both loops spend the budget where a label changes the most, which is the labeling platform's version of the funnel logic that runs through every ML system on this site.

Preference pairs, and how DPO consumes them

The two landmark preference datasets used different protocols, and the difference is a platform design decision. InstructGPT showed labelers K = 4 to 9 outputs per prompt and had them produce a full ranking, which yields K-choose-2 comparisons per screen, so 33k prompts became a much larger comparison set for reward-model training. Llama 2 went binary, annotators wrote a prompt, saw two sampled responses, chose one, and rated the gap on a four-point scale, significantly better, better, slightly better, or negligibly better slash unsure, and Meta collected 1,418,091 of these. Rankings extract more pairs per screen, binary-with-margin is a faster and more consistent judgment per screen, and a platform should support both, which the JSONB result column already does, {"choice":"B","margin":"better"} for one, an ordered list for the other.

The export format is settled convention. Anthropic's hh-rlhf dataset, 161k training rows and 8.55k test rows, is JSONL where every line holds a chosen and a rejected text, and that two-field row is the lingua franca the tooling downstream expects. The export job takes resolved comparisons, applies the quality gates, dropping annotators below a trust floor and comparisons the audit slice flagged, splits by prompt so no prompt leaks across train and test, and writes chosen-rejected rows with margin and annotator metadata carried alongside for weighting experiments.

What happens next is why comparisons are the product's most valuable output. The classic RLHF pipeline, InstructGPT's, fits a reward model on the comparisons and then optimizes the policy against it with PPO. DPO collapsed that pipeline, deriving the optimal policy in closed form so that the standard RLHF objective is solved with a simple classification loss directly on the pairs, no explicit reward model and no sampling from the model during fine-tuning. Zephyr-7B-beta is the concrete existence proof, Mistral-7B-v0.1 fine-tuned on UltraChat and then aligned with DPO via TRL's DPOTrainer on UltraFeedback, a dataset of 64k prompts with completions ranked by GPT-4. Note what that last fact implies for the platform, AI feedback is now a real substitute supplier for some quality tiers, and the honest framing is that your human pairs compete on the judgment quality that GPT-4-as-ranker cannot provide.

The loop structure is the last piece. Preference data goes stale as the policy improves, because pairs sampled from an old checkpoint stop covering the mistakes the new checkpoint actually makes, which is why Llama 2's annotations were collected in weekly batches with fresh samples from the improving model. Each cycle of the diagram below samples prompts, generates two responses from the current checkpoint, runs them through the platform under gold and consensus control, trains a DPO candidate on the accumulated pairs, and promotes it through an eval gate, and then the next batch samples from the promoted checkpoint so the data stays on-distribution.

One batch of the loop. The current checkpoint writes two responses per prompt, annotators pick winners under gold and consensus control, and DPO turns the chosen and rejected rows directly into the next candidate checkpoint, no reward model in between. The eval gate promotes it, and the next batch samples from the promoted model so pairs stay on-distribution, the reason Llama 2 collected weekly.

Retrieval over the guidelines

Labeling guidelines start as two pages and grow into a living document as edge cases accumulate, Llama 2's weekly batch cadence existed partly because instructions evolved between batches. Past a certain size, the guideline document itself becomes a consistency hazard, two annotators resolving the same edge case from memory of different sections. The optional upgrade is retrieval, chunk the guidelines by section, embed the chunks, and at task-serve time retrieve the sections most similar to the current task's payload, rendering the top snippet beside the task with a link into the full document. The same index answers annotator questions in the UI directly, and because the serving path already assembles a task response, this adds one lookup to an existing round trip.

It earns the qualifier optional honestly. For a 10-page guideline a decent full-text search and a disciplined change log deliver most of the value at a fraction of the machinery, and the highest-yield consistency tools remain non-retrieval ones, qualification sets that teach the guidelines, gold tasks that measure adherence, and adjudication feedback that shows an annotator the resolution of tasks they got wrong. Retrieval starts paying when guidelines cross into dozens of pages, when the task mix is heterogeneous enough that the relevant section genuinely varies per task, or when per-batch instruction changes need to reach a thousand annotators the same morning. Version every guideline chunk and stamp the active guidelines version onto each annotation row, because an alpha drop is only diagnosable if you can slice agreement by the guidelines version that was live.

Alternatives that work, and non-starters

Several substitutions are legitimate and worth naming. Buying instead of building is the biggest one, self-hosted Label Studio covers the task model, the UI, and ML-backend pre-labeling for a team that does not need custom preference tooling, and managed vendors like Scale or Surge sell the workforce together with the platform, which is how Anthropic sourced early RLHF labeling from Surge. The build case rests on preference-specific UX, tight active-learning loops against your own models, and owning the trust pipeline. SQS instead of Redis for queue and lease works cleanly, its visibility timeout is the same lease semantic under a different name, a consumed message stays invisible while worked and reappears on timeout, and you trade Redis's latency and its dedup-plus-priority extras for a managed service. Postgres alone also works at small scale, SELECT ... FOR UPDATE SKIP LOCKED over the tasks table is a real queue pattern, right up until lease TTLs and heartbeats start being reimplemented as timestamp columns and cron sweeps, which is Redis's job description. AI feedback instead of human feedback is a genuine alternative at some quality tiers, UltraFeedback's GPT-4 rankings trained Zephyr, and the sober framing is that human pairs must justify their 100x cost on the judgment slices where the AI ranker is weakest, domain expertise, safety, and subtle helpfulness.

The non-starters are the moves that look reasonable and forfeit something structural. A single mutable label column per task destroys the vote history, and with it every agreement metric, every audit, and every re-adjudication, append-only annotations are non-negotiable. Two-vote majority is not a consensus scheme, a 1-1 split resolves nothing and you have paid double for zero decisions, overlap in twos is for measuring agreement, resolving requires an odd panel or an adjudicator. Exactly-once task delivery is the same fiction here as in every distributed system, a lease can always expire under a live annotator, so the design is at-least-once with idempotent submits, and the occasional double-labeled task is free agreement data rather than a bug. Visible gold marking, including any UI difference an annotator community can fingerprint, converts your measurement instrument into a performance, gold only works blind. And grading a model with itself, using the checkpoint being trained as the sole judge of its own outputs, feeds the model's biases back as supervision with no independent signal anywhere in the loop, which is precisely the degenerate case the human platform exists to prevent.

Questions and answers

The core ideas as questions with the answers given outright. Each wrong multiple-choice option is marked with why it is wrong, and the ordering ones show the correct sequence.

References

1. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022), The roughly 40-contractor operation behind the 13k SFT, 33k RM, and 31k PPO prompt datasets, rankings of 4 to 9 outputs per prompt, and the published 72.6% and 77.3% inter-labeler agreement rates.
2. Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models (2023), Preference collection at industrial scale, 27,540 SFT annotations and 1,418,091 binary comparisons with a four-point margin scale, gathered in weekly batches.
3. Rafailov et al., Direct Preference Optimization (2023), The closed-form result that turns RLHF into a classification loss directly on preference pairs, with no explicit reward model and no sampling during fine-tuning.
4. Anthropic, hh-rlhf dataset, The chosen-rejected JSONL convention this platform exports, 161k training and 8.55k test rows of helpfulness and harmlessness comparisons.
5. HuggingFace H4, Zephyr-7B-beta model card, A shipped DPO consumer, Mistral-7B-v0.1 fine-tuned on UltraChat and aligned with TRL's DPOTrainer on UltraFeedback's 64k GPT-4-ranked prompts.
6. Label Studio, task format documentation, The data, annotations, and predictions task structure, with lead_time, completed_by, and prediction confidence scores this article's schema mirrors.
7. Label Studio Enterprise, annotation agreement documentation, Consensus and pairwise agreement methodologies built from per-type matching functions, exact match, IoU, and span overlap.
8. Redis, SET command documentation, The NX-plus-expiry locking pattern with a random token and check-and-delete release that powers task leasing.