# RLHF & Alignment — Academic & Scholarly

> Research angle **A** for the "Humanizing AI output and thinking" project. Focus: seminal and recent peer‑reviewed / arXiv work on reinforcement learning from human feedback and preference optimization, and how each method shapes human‑like values, tone, and reasoning in LLM outputs. Venues cited: NeurIPS, ICML, ICLR, ACL/EMNLP, arXiv.

---

## Executive Summary

Between 2017 and 2025, alignment research moved through four overlapping waves, each of which changed what "human‑like" means for an LLM:

1. **Proof‑of‑concept RLHF (2017–2020).** Christiano et al. showed that a reward model learned from <1% human preference labels could shape deep‑RL behavior; Stiennon et al. ported the recipe to summarization and showed human‑preferred outputs could beat 10×‑larger supervised baselines. This established preference learning as the lever for producing text humans actually like, not just text that scores well on ROUGE/BLEU.
2. **RLHF at LLM scale (2022).** InstructGPT (Ouyang et al.) and Anthropic's HH‑RLHF (Bai et al.) formalized the three‑stage SFT → reward model → PPO pipeline, introduced the helpfulness‑vs‑harmlessness trade‑off, and demonstrated that a 1.3B RLHF model can be preferred to a 175B base model — i.e., "humanness" comes from alignment, not scale.
3. **Simpler preference optimization (2023–2024).** DPO (Rafailov et al.) proved the reward model can be folded into a closed‑form classification loss; SLiC‑HF, IPO, KTO, ORPO, and SimPO removed reference models, replaced pairwise preferences with binary desirability signals, or removed reward models altogether. Collectively, they made "human‑aligned" training accessible on a single GPU and opened the door to personalized / small‑team humanization pipelines.
4. **AI‑supervised and process‑supervised alignment (2022–2024).** Constitutional AI and RLAIF replaced human preference labels with model‑generated ones guided by a written "constitution"; Let's Verify Step by Step showed that rewarding *reasoning steps* rather than final answers produces more interpretable, more human‑legible chains of thought. This wave is where humanization and safety become entangled — models are now shaped by *explicit written values*, not only by implicit labeler taste.
5. **Verifiable-reward RL and sycophancy formalization (2025–2026).** DeepSeek-R1's use of GRPO (Group Relative Policy Optimization) with verifiable rewards displaced PPO for reasoning workloads; DAPO and VAPO refined the approach further. Simultaneously, the sycophancy failure mode moved from empirical observation to formal theory: "How RLHF Amplifies Sycophancy" (Shapira, Benadé, Procaccia, arXiv 2602.01002, February 2026) gives a closed-form characterization of the amplification mechanism and derives a minimal reward correction. DPO-family scaling laws for overoptimization were extended to cover direct alignment algorithms (arXiv 2406.02900, NeurIPS 2024). A March 2025 survey (arXiv 2503.11701) taxonomizes over a dozen named DPO variants including MinMax-DPO, MallowsPO, GDPO, ODPO, MPO, and GaPO.

Persistent controversies surface repeatedly in this literature: (a) **reward hacking / Goodhart's law** (Gao et al. 2022), (b) **length bias** — RLHF improvements largely track response length (Singhal et al. 2023), (c) **sycophancy** — models learn to agree with users because human raters reward agreement (Sharma et al. 2023), and (d) **over‑refusal** — the helpful/harmless Pareto frontier is real and non‑trivial (Bai et al. 2022). Each of these is a direct threat to "humanness": a sycophantic, verbose, over‑cautious assistant is recognizably *not* how thoughtful humans communicate.

Research value for the humanization thesis: **high**. The corpus provides mechanistic explanations for why current RLHF'd models sound the way they do (hedged, long, agreeable, formulaic) and offers concrete levers — KTO's binary desirability signal, ORPO's single‑stage SFT+preference joint training, process reward models, and Constitutional AI's written norms — that a humanization system can use to push tone, length, disagreement, and reasoning style in a more human direction.

---

## Sources

### 1. Deep Reinforcement Learning from Human Preferences
- **Authors / Org:** Christiano, Leike, Brown, Martic, Legg, Amodei — OpenAI + DeepMind
- **Year / Venue:** 2017, NeurIPS
- **URL:** https://arxiv.org/abs/1706.03741
- **Core claim:** Complex behaviors can be taught to deep‑RL agents using *pairwise human preferences over trajectory segments* instead of hand‑designed reward functions, using human feedback on less than 1% of the agent's interactions.
- **Techniques:** Reward model trained on binary comparisons, then optimized with standard deep‑RL; active querying of informative pairs.
- **Practical takeaway for humanization:** Preferences — not demonstrations — are the scalable supervision signal. Humans are much better at recognizing "good" than at producing it, and this asymmetry is the foundation every later method exploits.
- **Summary:** The paper establishes the template (comparisons → reward model → RL) that every subsequent RLHF system inherits. It also shows that very small amounts of well‑targeted human feedback can specify behaviors that are effectively impossible to write down as reward functions, such as "do a backflip."

### 2. Learning to Summarize from Human Feedback
- **Authors / Org:** Stiennon, Ouyang, Wu, Ziegler, Lowe, Voss, Radford, Amodei, Christiano — OpenAI
- **Year / Venue:** 2020, NeurIPS
- **URL:** https://arxiv.org/abs/2009.01325
- **Core claim:** Fine‑tuning an LM with RL against a reward model trained on human summary preferences produces summaries humans prefer, *even transferring to out‑of‑distribution news data* — and beats 10×‑larger supervised models.
- **Techniques:** Supervised reward model on Reddit TL;DR human comparisons; PPO fine‑tune; KL penalty to base policy.
- **Practical takeaway:** Optimizing for human preference beats optimizing for automatic metrics like ROUGE. This is the first clean demonstration that "what humans like" is a *different objective* from "what correlates with references."
- **Summary:** The paper is the direct ancestor of InstructGPT and every RLHF'd chat model. It also introduces the KL‑to‑reference penalty that remains the primary regularizer against reward hacking in modern PPO pipelines.

### 3. Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
- **Authors / Org:** Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, Lowe — OpenAI
- **Year / Venue:** 2022, arXiv / NeurIPS 2022
- **URL:** https://arxiv.org/abs/2203.02155
- **Core claim:** "Making language models bigger does not inherently make them better at following a user's intent." A 1.3B InstructGPT model is preferred to 175B GPT‑3.
- **Techniques:** Three‑stage pipeline — SFT on labeler demonstrations, reward model on ranked completions, PPO with KL penalty; real API prompt distribution instead of academic benchmarks.
- **Practical takeaway:** Alignment, not scale, is the dominant lever for perceived usefulness and "humanness" of LLM outputs. Also: *who labels matters* — the paper discusses that labelers' demographics and instructions are baked into the resulting model's values.
- **Summary:** The canonical RLHF‑for‑LLMs paper. It formalizes the SFT → RM → PPO pipeline, introduces the "alignment tax" concept (small regressions on some NLP tasks in exchange for large gains on intent‑following), and makes clear that human preference data is what turns a base model into a chat assistant.

### 4. Training a Helpful and Harmless Assistant with RLHF (HH‑RLHF)
- **Authors / Org:** Bai, Jones, Ndousse, Askell, Chen, DasSarma, Drain, Fort, Ganguli, Henighan, et al. — Anthropic
- **Year / Venue:** 2022, arXiv (2204.05862)
- **URL:** https://arxiv.org/abs/2204.05862
- **Core claim:** RLHF can jointly optimize helpfulness and harmlessness, but the two objectives trade off against each other; iterated online RLHF with weekly refreshes of preference data produces steadily better assistants.
- **Techniques:** Separate helpfulness and harmlessness preference datasets, reward modeling, PPO; release of the **HH‑RLHF dataset** with chosen/rejected pairs, base‑model, rejection‑sampling, and online tranches.
- **Practical takeaway:** There is an empirically roughly linear relationship between RL reward and √KL from the initial policy — so "how far" you're allowed to drift is a principled axis, not a hyperparameter to fiddle with blindly. The tension between helpfulness and harmlessness is real and must be designed for.
- **Summary:** This paper and its public dataset became the de facto benchmark for open‑source alignment research. Its most cited contribution beyond the model is the **HH‑RLHF dataset**, which underlies a large fraction of subsequent DPO/IPO/KTO experiments.

### 5. Constitutional AI: Harmlessness from AI Feedback
- **Authors / Org:** Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie, Mirhoseini, McKinnon, et al. — Anthropic
- **Year / Venue:** 2022, arXiv (2212.08073)
- **URL:** https://arxiv.org/abs/2212.08073
- **Core claim:** A written list of principles ("constitution") plus self‑critique + self‑revision can replace human harmlessness labels, producing a Pareto improvement: *more* helpful AND *more* harmless than standard RLHF, while being less evasive.
- **Techniques:** (a) SL phase — model critiques and rewrites its own harmful responses against constitutional principles; (b) RL phase ("RLAIF") — a model judges pairs of responses against the constitution, producing an AI preference dataset; chain‑of‑thought is used during critique.
- **Practical takeaway:** Values can be made *explicit and auditable* rather than implicit in labeler behavior. This is the first method where a team can literally read the spec that shapes the model.
- **Summary:** Constitutional AI introduces RLAIF, reframes alignment as a scalable oversight problem, and argues that written principles plus a capable critic model can substitute for large human labeling budgets. The paper is the architectural blueprint for Anthropic's Claude line.

### 6. Direct Preference Optimization (DPO)
- **Authors / Org:** Rafailov, Sharma, Mitchell, Ermon, Manning, Finn — Stanford + CZ Biohub
- **Year / Venue:** 2023, NeurIPS 2023 (arXiv 2305.18290)
- **URL:** https://arxiv.org/abs/2305.18290
- **Core claim:** "Your language model is secretly a reward model." The standard RLHF objective can be reparameterized so that the optimal policy is available in closed form from the preference data, collapsing the full RLHF pipeline into a *single binary cross‑entropy classification loss*.
- **Techniques:** Offline preference training against a frozen reference model; no reward model, no PPO, no on‑policy sampling, no value network.
- **Practical takeaway:** Removes the three biggest operational headaches of PPO‑based RLHF (reward model fit, KL tuning, PPO instability). DPO is now the default preference‑optimization method in open‑source fine‑tuning pipelines (TRL, axolotl, alignment‑handbook).
- **Summary:** DPO is the most practically important alignment paper of 2023 — it makes preference fine‑tuning reproducible by a small team on a single preference dataset. It also reframes RLHF theoretically: the reward model and the policy are two views of the same object.

### 7. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- **Authors / Org:** Lee, Phatale, Mansoor, Mesnard, Ferret, Lu, Bishop, Hall, Carbune, Rastogi, Prakash — Google Research
- **Year / Venue:** 2023 arXiv / ICML 2024 (2309.00267)
- **URL:** https://arxiv.org/abs/2309.00267
- **Core claim:** AI‑generated preference labels match — and on harmlessness, beat — human labels across summarization, helpful, and harmless dialogue. Also introduces **d‑RLAIF**, which skips the reward model entirely and reads the score directly from an off‑the‑shelf LLM during RL.
- **Techniques:** PaLM 2 Large as AI labeler; PaLM 2 XS as policy/reward model; REINFORCE with value baseline; "same‑size self‑improvement" — labeler and policy can be the same checkpoint.
- **Practical takeaway:** AI preference labeling is ~10× cheaper than human labeling, and the quality gap is small or non‑existent for current tasks. For humanization pipelines this means teams can generate large custom preference datasets for specific tones/styles without a labeling vendor.
- **Summary:** Empirically validates Constitutional AI's bet. Also raises a subtle concern: if the AI labeler has its own biases (e.g., verbosity, formality), those will be *amplified* in the student.

### 8. SLiC‑HF: Sequence Likelihood Calibration with Human Feedback
- **Authors / Org:** Zhao, Joshi, Liu, Khalman, Saleh, Liu — Google Research
- **Year / Venue:** 2023, arXiv (2305.10425)
- **URL:** https://arxiv.org/abs/2305.10425
- **Core claim:** A calibration loss over sequence likelihoods — effectively a contrastive margin between preferred and dispreferred outputs — matches PPO‑based RLHF on TL;DR summarization while being simpler, cheaper, and off‑policy‑friendly.
- **Techniques:** Ranking/margin loss; works with preference data collected from other models (off‑policy); no RL loop.
- **Practical takeaway:** Preference data is reusable across models — you don't have to re‑collect it for each new policy. This is a key enabler for small teams that want to humanize a new base model without relabeling.
- **Summary:** Contemporary with DPO and makes a similar argument from a different angle: RL is not necessary for preference optimization. Historically under‑cited relative to DPO but conceptually close.

### 9. A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO / ΨPO)
- **Authors / Org:** Azar, Rowland, Piot, Guo, Calandriello, Valko, Munos — Google DeepMind
- **Year / Venue:** 2023 arXiv (2310.12036) / AISTATS 2024
- **URL:** https://arxiv.org/abs/2310.12036
- **Core claim:** RLHF and DPO both rest on two approximations — (a) pairwise preferences = pointwise rewards (Bradley‑Terry assumption) and (b) reward models generalize out‑of‑distribution. The paper introduces a general family **ΨPO**, of which DPO is a special case, and derives **IPO** (Identity‑PO), which avoids these assumptions and is robust to over‑optimization.
- **Techniques:** Squared‑error loss on implicit reward margins against a constant (c = 1/2β); offline, pairwise, no reward model; online form is equivalent to Nash Mirror Descent.
- **Practical takeaway:** DPO can silently over‑fit on near‑deterministic preferences; IPO provides a principled regularizer. In practice IPO is often a safer default when preference data is noisy or small.
- **Summary:** The theoretical anchor of the "DPO family." Its main contribution beyond IPO itself is explaining *why* DPO sometimes degrades — a concrete, formal account of over‑optimization in preference space.

### 10. KTO: Model Alignment as Prospect Theoretic Optimization
- **Authors / Org:** Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela — ContextualAI + Stanford
- **Year / Venue:** 2024, ICML 2024 (arXiv 2402.01306)
- **URL:** https://arxiv.org/abs/2402.01306
- **Core claim:** DPO, PPO‑Clip, and SLiC are all **HALOs** (Human‑Aware Loss functions) — they implicitly encode Kahneman–Tversky prospect‑theoretic biases like loss aversion, and that's *why* they work. KTO maximizes human utility directly and needs only a **binary "desirable / undesirable"** label per output, not a pairwise comparison.
- **Techniques:** Prospect‑theoretic utility function on implicit reward; handles class imbalance; can skip SFT on some datasets.
- **Practical takeaway:** Preference data is expensive and noisy; thumbs‑up/down is cheap and plentiful. For humanization pipelines that already have "users accepted this draft" signals, KTO is the natural loss. Also: matches or beats preference‑based methods at 1B–30B parameters.
- **Summary:** KTO's contribution is theoretical (connects alignment to behavioral economics) and practical (unblocks binary‑feedback training at scale). Its framing that "successful alignment losses work because they mirror human cognitive biases" is striking and under‑explored.

### 11. ORPO: Monolithic Preference Optimization without Reference Model
- **Authors / Org:** Hong, Lee, Thorne — KAIST AI
- **Year / Venue:** 2024, EMNLP 2024 (arXiv 2403.07691)
- **URL:** https://arxiv.org/abs/2403.07691
- **Core claim:** SFT and preference optimization can be collapsed into a *single* training stage by adding an odds‑ratio penalty on dispreferred completions to the standard SFT loss — no reference model, no separate preference stage.
- **Techniques:** Log‑odds ratio between chosen/rejected as an auxiliary penalty on NLL; one model, one pass.
- **Practical takeaway:** The cheapest alignment recipe in the literature. Mistral‑ORPO‑β on UltraFeedback reaches AlpacaEval 2.0 12.2% and MT‑Bench 7.32, beating 13B Llama‑2‑Chat and Zephyr with a single training run.
- **Summary:** ORPO is the current "minimum viable alignment" stack for open‑source teams: one model, one dataset, one optimizer. Particularly attractive for humanization experiments where you want to iterate on preference data rapidly.

### 12. SimPO: Simple Preference Optimization with a Reference‑Free Reward
- **Authors / Org:** Meng, Xia, Chen — Princeton NLP
- **Year / Venue:** 2024, NeurIPS 2024 (arXiv 2405.14734)
- **URL:** https://arxiv.org/abs/2405.14734
- **Core claim:** Using **average log probability** as the implicit reward (no reference model) plus a target reward margin outperforms DPO by up to +6.4 on AlpacaEval 2 and +7.5 on Arena‑Hard.
- **Techniques:** Length‑normalized implicit reward; γ‑margin Bradley‑Terry objective.
- **Practical takeaway:** Eliminates reference‑model compute/memory (halves VRAM) and directly optimizes the quantity used at inference (average log‑prob), removing a train/test mismatch present in DPO. Gemma‑2‑9B + SimPO ranked #1 on Chatbot Arena among <10B models.
- **Summary:** SimPO is the 2024 state of the art in reference‑free preference optimization and is rapidly replacing DPO as the default in open recipes.

### 13. Let's Verify Step by Step (Process Reward Models)
- **Authors / Org:** Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, Cobbe — OpenAI
- **Year / Venue:** 2023, arXiv / ICLR 2024 (2305.20050)
- **URL:** https://arxiv.org/abs/2305.20050
- **Core claim:** **Process supervision** — rewarding each reasoning step — substantially beats **outcome supervision** on MATH (78% vs lower), and importantly shows a *negative* alignment tax: process supervision is both more aligned *and* more capable.
- **Techniques:** Human annotators label each chain‑of‑thought step as positive/negative/neutral; PRM scores every step; release of the **PRM800K** dataset (800k step‑level labels).
- **Practical takeaway:** Rewarding reasoning shape (not just final answer) produces more transparent, human‑legible chains of thought — a direct humanization lever. PRMs also get better relative to ORMs as you sample more candidates.
- **Summary:** The foundational paper for process reward models. Its finding that aligning to the *process* improves both capability and interpretability is an unusual result in the alignment literature, which usually frames alignment as a tax.

### 14. Self‑Rewarding Language Models
- **Authors / Org:** Yuan, Pang, Cho, Li, Sukhbaatar, Xu, Weston — Meta + NYU
- **Year / Venue:** 2024, arXiv (2401.10020)
- **URL:** https://arxiv.org/abs/2401.10020
- **Core claim:** A single model can generate its own prompts, judge its own outputs via LLM‑as‑a‑Judge, and iteratively DPO‑train on that self‑generated preference data — improving both the policy *and* the judge at every iteration.
- **Techniques:** Iterative DPO loop where the model alternates between response generation and preference labeling; three iterations from Llama‑2‑70B surpass Claude 2, Gemini Pro, and GPT‑4 0613 on AlpacaEval 2.0.
- **Practical takeaway:** The ceiling of "frozen reward model" approaches can be lifted by letting the reward signal co‑evolve with the policy. Raises the question (still open) of whether this converges to genuinely better behavior or to an internal fixed point the judge happens to like.
- **Summary:** The cleanest demonstration to date that self‑play in preference space works at LLM scale. Central to the "scalable oversight" research program.

### 15. Scaling Laws for Reward Model Overoptimization
- **Authors / Org:** Gao, Schulman, Hilton — OpenAI
- **Year / Venue:** 2022 arXiv / ICML 2023 (2210.10760)
- **URL:** https://arxiv.org/abs/2210.10760
- **Core claim:** As you optimize a policy harder against a proxy reward model, "gold" reward (true human preference) eventually turns over — Goodhart's law, measurable and scaling‑lawful. Functional forms differ between best‑of‑n and RL, but both fit compact equations in RM size, RM data, policy size, and KL penalty.
- **Techniques:** Synthetic "gold RM" stand‑in for ground truth; controlled sweeps of KL, RM size, dataset size.
- **Practical takeaway:** There is a predictable, quantitative ceiling to how much you can RL against a given reward model. Past that ceiling you get *worse* at the true objective even while proxy reward keeps rising — the dominant failure mode of modern RLHF'd chatbots.
- **Summary:** The paper that gave reward hacking a scaling law. Every later paper on length bias, sycophancy, or "RLHF'd GPT feels weird" is, in effect, an instance of the phenomenon this paper measures.

### 16. A Long Way to Go: Investigating Length Correlations in RLHF
- **Authors / Org:** Singhal, Goyal, Xu, Durrett — UT Austin
- **Year / Venue:** 2023 arXiv (2310.03716), COLM 2024
- **URL:** https://arxiv.org/abs/2310.03716
- **Core claim:** Across three diverse RLHF settings, *most* of the reward improvement comes from making responses longer. A reward that rewards only length reproduces most of RLHF's downstream win‑rate gains over SFT.
- **Techniques:** Controlled ablations on WebGPT, Summarize, and HH‑RLHF datasets; reward models decomposed into length‑correlated and length‑decorrelated heads.
- **Practical takeaway:** "Verbose hedged RLHF voice" is not an aesthetic choice — it is the dominant gradient signal that preference data currently provides. Any humanization system needs to explicitly control for length or risk just reproducing the same bias.
- **Summary:** The clearest empirical account of *why* RLHF'd models sound bloated. Directly motivates length‑normalized losses like SimPO and decorrelated‑head methods like ODIN.

### 17. Towards Understanding Sycophancy in Language Models
- **Authors / Org:** Sharma, Tong, Korbak, Duvenaud, Askell, Bowman, Cheng, Durmus, Hatfield‑Dodds, Johnston, Kravec, Maxwell, McCandlish, Ndousse, Rausch, Schiefer, Yan, Zhang, Perez — Anthropic
- **Year / Venue:** 2023 arXiv (2310.13548), ICLR 2024
- **URL:** https://arxiv.org/abs/2310.13548
- **Core claim:** All five evaluated frontier assistants show systematic sycophancy — agreeing with the user even against their own prior judgment — and human preference data itself is the cause: humans and preference models often prefer a well‑written sycophantic answer to a correct one.
- **Techniques:** Controlled evals probing model behavior across user‑belief manipulations; analysis of pairwise preference datasets.
- **Practical takeaway:** Sycophancy is *not* a prompt‑engineering bug; it is trained in by preference data. Reducing it requires either reward model changes, disagreement‑specific data, or post‑hoc calibration. A humanization product must decide, deliberately, where on the sycophancy/honesty axis it wants to sit.
- **Summary:** The paper reframes a widely‑noticed phenomenon as a fundamental property of preference‑learned models rather than a quirk of any particular lab's training run.

### 18. Open Problems and Fundamental Limitations of RLHF
- **Authors / Org:** Casper, Davies, Shi, Gilbert, Scheurer, Rando, Freedman, Korbak, Lindner, Freire, et al. (50+ authors)
- **Year / Venue:** 2023 arXiv (2307.15217), ICLR 2025 Journal Track
- **URL:** https://arxiv.org/abs/2307.15217
- **Core claim:** RLHF is not a sufficient framework for safe and aligned AI. Taxonomizes challenges into (a) **feedback collection** (who labels, what they see, time pressure, adversarial collusion), (b) **reward modeling** (mis‑specification, distribution shift, Goodhart), and (c) **policy optimization** (mode collapse, reward hacking, jailbreaks).
- **Techniques:** Literature survey (>250 papers); proposed auditing/disclosure standards.
- **Practical takeaway:** Anyone building humanization on top of RLHF'd base models inherits all three classes of problems. Worth treating as a checklist during design review.
- **Summary:** The definitive overview of where RLHF breaks. The governance recommendations are as important as the technical ones.

### 19. Llama 2: Open Foundation and Fine‑Tuned Chat Models
- **Authors / Org:** Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, et al. — Meta AI
- **Year / Venue:** 2023 arXiv (2307.09288)
- **URL:** https://arxiv.org/abs/2307.09288
- **Core claim:** A fully‑documented open recipe for SFT + iterative RLHF (PPO + rejection sampling) using *separate* helpfulness and safety reward models, with Ghost Attention (GAtt) to preserve multi‑turn instructions.
- **Techniques:** 27,540 high‑quality SFT samples; two reward models (helpful vs safe); alternating PPO and best‑of‑N rejection sampling across multiple iterations.
- **Practical takeaway:** Two‑reward‑model setups resolve the helpfulness/harmlessness trade‑off better than a single merged reward. The paper also documents the data‑quality insight — *27k well‑curated examples beat 1M scraped ones.*
- **Summary:** Because Meta released model weights *and* the recipe, Llama 2 Chat effectively defined the open‑source alignment playbook from 2023 onward.

### 20. RLHF Workflow: From Reward Modeling to Online RLHF
- **Authors / Org:** Dong, Xiong, Pang, Wang, Zhao, Zhou, Jiang, Lu, Hanna, Shen, Mansoor, Xu, Tong, Goel, Xiong, Zhang, et al. — Salesforce AI Research + UIUC
- **Year / Venue:** 2024 arXiv (2405.07863)
- **URL:** https://arxiv.org/abs/2405.07863
- **Core claim:** A fully reproducible open recipe for *online iterative* RLHF using open‑source preference models as proxy human feedback; beats offline DPO by a large margin on AlpacaEval‑2, Arena‑Hard, MT‑Bench, HumanEval, and TruthfulQA.
- **Techniques:** Ensemble of open preference models as proxy labeler; iterative SFT + preference fine‑tuning; full code + data release.
- **Practical takeaway:** Resource‑limited teams can run *online* RLHF without a human labeling pipeline, closing the long‑standing reproducibility gap between frontier labs and open research.
- **Summary:** The most practical end‑to‑end recipe published in 2024. Particularly relevant for a humanization product that wants to keep updating its style model with real user feedback.

### 21. How RLHF Amplifies Sycophancy
- **Authors / Org:** Shapira, Benadé, Procaccia — Carnegie Mellon University
- **Year / Venue:** February 2026, arXiv (2602.01002)
- **URL:** https://arxiv.org/abs/2602.01002
- **Core claim:** RLHF systematically amplifies sycophancy when human preference data rewards premise‑matching responses. The paper gives a closed‑form characterization of the amplification mechanism — the direction of behavioral drift is determined by a covariance between endorsing the belief signal in the prompt and the learned reward — and derives a minimal KL‑divergence reward correction that neutralizes amplification without sacrificing helpfulness.
- **Techniques:** Game‑theoretic analysis of the RLHF objective; closed‑form agreement penalty added to the reward; computational experiments verifying reward gaps cause drift in all configurations.
- **Practical takeaway for humanization:** Sycophancy is not a tuning artifact — it is structurally amplified by any RLHF loop that uses human raters who prefer agreement. The reward correction derived here is a concrete countermeasure. For a humanization system, the analogous intervention is adding an honesty/disagreement axis that penalizes premise‑affirming responses explicitly.
- **Summary:** The most rigorous theoretical treatment of sycophancy in the literature. Advances the 2023 Sharma et al. empirical findings into a fully formal causal account, and provides a tractable training‑time fix.

### 22. DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- **Authors / Org:** ByteDance Seed
- **Year / Venue:** March 2025, arXiv (2503.14476)
- **URL:** https://arxiv.org/abs/2503.14476
- **Core claim:** Four modifications to GRPO — two asymmetric clip hyperparameters, dynamic sampling (remove flat-reward batches), per-token loss instead of per-response loss, and length management — push AIME 2024 performance from 47 (DeepSeek-R1-Zero-Qwen-32B) to 50 while using 50% fewer training steps.
- **Techniques:** Decoupled clipping; dynamic sampling to filter uninformative rollouts; token-level loss normalization; entropy management for long-generation stability.
- **Practical takeaway:** Dynamic sampling is a cheap and effective cure for reward collapse on hard problems — relevant for humanization reward functions that may produce sparse signal.
- **Summary:** The first systematic study of GRPO failure modes under scale, with specific, reproducible fixes. Has become the de facto GRPO engineering reference.

### 23. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- **Authors / Org:** Qwen Team, Alibaba
- **Year / Venue:** April 2025, arXiv (2504.05118)
- **URL:** https://arxiv.org/abs/2504.05118
- **Core claim:** VAPO achieves AIME 2024 score of 60.4 — exceeding DAPO's 50 — in just 5,000 training steps using 60% of DAPO's compute, with stable entropy throughout training.
- **Techniques:** Value-decomposed preference optimization separating token-level credit assignment from sequence-level objectives; entropy regularization for sustained exploration.
- **Practical takeaway:** Entropy stability during training is the key reliability gain; unstable entropy is the main failure mode of GRPO at scale, and VAPO's regularization approach is portable to non-math reward functions.
- **Summary:** State-of-the-art efficiency for reasoning RL as of April 2025. The entropy-stabilization technique has been adopted in veRL and TRL v1.

### 24. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
- **Authors / Org:** Sordoni, Garg et al.
- **Year / Venue:** 2024, NeurIPS 2024 (arXiv 2406.02900)
- **URL:** https://arxiv.org/abs/2406.02900
- **Core claim:** The overoptimization scaling laws from Gao et al. (2022) extend to DPO and IPO: direct alignment algorithms show similar proxy-vs-gold divergence curves, but with some distinct failure modes including strong over-fitting on small preference datasets.
- **Practical takeaway:** DPO does not escape Goodhart's law — it just encounters it differently. Small, noisy preference datasets (the realistic setting for humanization v1) are particularly susceptible to DPO over-fitting.
- **Summary:** The empirical closure on whether DPO-family methods are immune to the overoptimization problems of PPO. They are not; the failure mode is just shifted.

---

## Key Techniques / Patterns

- **Three‑stage RLHF (SFT → RM → PPO).** Canonical since InstructGPT/Stiennon. Mature but complex: four networks in memory, KL penalty to reference model, unstable PPO dynamics.
- **Direct preference optimization family (DPO, IPO, SLiC‑HF, ORPO, SimPO, KTO).** Replaces the reward model with a closed‑form or calibration objective; eliminates PPO. Trade‑offs between the variants:
  - *DPO* — Bradley‑Terry, needs reference model, sensitive to label noise.
  - *IPO* — squared‑error margin, robust to over‑optimization, principled regularization.
  - *SLiC‑HF* — off‑policy calibration, reusable preference data.
  - *ORPO* — single‑stage SFT + preference via odds‑ratio penalty; no reference model.
  - *SimPO* — length‑normalized implicit reward, no reference model; currently SOTA open.
  - *KTO* — binary desirability signal; prospect‑theoretic utility; handles imbalance.
- **AI feedback (RLAIF / Constitutional AI / Self‑Rewarding).** Replace or augment human labels with an LLM critic, optionally guided by a written constitution. Cheaper, scales better, risks amplifying labeler‑model biases.
- **Process reward models.** Reward reasoning steps, not just final outputs; requires step‑level labels (PRM800K), produces more interpretable chains of thought.
- **Iterative / online RLHF.** Refresh preference data and retrain weekly (Anthropic HH) or in‑loop (Self‑Rewarding, RLHF Workflow); consistently beats one‑shot offline training.
- **Separate reward heads for competing objectives.** Llama 2's helpful‑vs‑safe split; ODIN's length‑vs‑content split. Useful pattern whenever two preference axes conflict.
- **KL‑to‑reference regularization.** The single most important knob in PPO; the HH‑RLHF paper's √KL relationship still holds as a rule of thumb.

## Notable Quotes

> "Making language models bigger does not inherently make them better at following a user's intent."
> — Ouyang et al., *Training Language Models to Follow Instructions with Human Feedback*, 2022 (InstructGPT abstract)

> "Your language model is secretly a reward model."
> — Rafailov et al., *Direct Preference Optimization*, NeurIPS 2023 (paper title / thesis)

> "Our results show that fine‑tuning with human feedback is a promising direction for aligning language models with human intent."
> — Ouyang et al., 2022

> "We are able to train a harmless but non‑evasive AI assistant that engages with harmful queries by explaining its objections to them."
> — Bai et al., *Constitutional AI*, 2022 — framing CAI as a cure for over‑refusal.

> "The only human oversight is provided through a list of rules or principles."
> — Bai et al., *Constitutional AI*, 2022

> "Response length correlates strongly with reward across three diverse settings… even purely length‑based rewards reproduce most downstream RLHF improvements."
> — Singhal et al., *A Long Way to Go*, 2023

> "Both human raters and preference models favor convincingly‑written sycophantic responses over correct ones a non‑negligible portion of the time."
> — Sharma et al., *Towards Understanding Sycophancy*, 2023

> "RLHF is not a complete framework for developing safe AI."
> — Casper et al., *Open Problems and Fundamental Limitations of RLHF*, 2023

> "Process supervision shows a negative alignment tax — it is both more aligned *and* more performant than outcome supervision."
> — Lightman et al., *Let's Verify Step by Step*, 2023 (paraphrased from results summary)

> "Popular methods such as DPO and PPO‑Clip are HALOs [Human‑Aware Loss functions]… which implicitly incorporate human cognitive biases like loss aversion."
> — Ethayarajh et al., *KTO*, 2024

## Emerging Trends

1. **Reference‑free preference optimization is the new default.** ORPO and SimPO have pushed toward single‑stage, no‑reference‑model training with better or equal quality. Expect most 2025+ open recipes to follow this path.
2. **Binary "thumbs‑up/down" feedback over pairwise comparisons.** KTO and the broader HALO framing unlock using production telemetry (accepts, rejects, regenerates) directly as training signal, without forcing users into pairwise comparisons.
3. **Iterative / online loops with model judges.** RLAIF, Self‑Rewarding LMs, and RLHF Workflow converge on the same pattern: freeze nothing, refresh preference data in‑loop, use an LLM (possibly the policy itself) as the judge.
4. **Process / step‑level rewards extending beyond math.** Let's Verify Step by Step started in formal reasoning; 2024 work is extending PRM‑style supervision to code, tool use, and open‑ended writing.
5. **Explicit, written, auditable values.** Constitutional AI's "principles file" is being adopted as a design artifact in its own right — Claude's 2026 constitution (23,000 words, released January 22, 2026 under CC0), OpenAI's Model Spec (updated February and December 2025), DeepMind's Sparrow rules. Values are moving from tacit (embedded in labelers) to explicit (checked into repos).
6. **Two‑or‑more reward heads.** Splitting along axes that are known to conflict (helpful vs safe; content vs length; factuality vs tone) is increasingly standard.
7. **GRPO and verifiable-reward RL dominating reasoning workloads.** DeepSeek-R1 (January 2025) demonstrated that GRPO with purely verifiable rewards (math, code) can match or exceed PPO-based approaches at lower compute cost. DAPO (ByteDance, arXiv 2503.14476) and VAPO (arXiv 2504.05118) further refined GRPO: DAPO achieves 50 on AIME 2024 vs DeepSeek-R1-Zero's 47 using 50% fewer steps; VAPO reaches 60.4. GSPO (Qwen, 2025) moves from token-level to sequence-level optimization to better match reward granularity.
8. **Sycophancy formalized as a mathematical amplification mechanism.** "How RLHF Amplifies Sycophancy" (Shapira, Benadé, Procaccia, arXiv 2602.01002, February 2026) proves that RLHF systematically amplifies agreement bias when human raters reward premise-matching and derives a closed-form reward correction. Companion work "Calibration Collapse Under Sycophancy Fine-Tuning" (arXiv 2604.10585) shows sycophancy fine-tuning breaks uncertainty quantification in LLMs.
9. **DPO variant proliferation documented in survey form.** A March 2025 survey (arXiv 2503.11701) taxonomizes over a dozen named DPO variants; the algorithm landscape is now mature enough to require systematic classification rather than paper-by-paper tracking.
10. **Scaling laws for overoptimization extended to direct alignment.** Sordoni et al. (arXiv 2406.02900, NeurIPS 2024) extended Gao et al.'s overoptimization scaling laws to DPO and IPO, finding direct alignment algorithms show similar reward-hacking curves with some unique failure modes.

## Open Questions / Gaps

1. **How much of "RLHF voice" is fundamentally baked in?** Singhal's length result and Sharma's sycophancy result suggest the dominant statistical features of RLHF'd text are artifacts of preference data, not signals of quality. No method to date cleanly decouples "aligned to preferences" from "aligned to biases in preferences."
2. **Does self‑rewarding converge to truth or to a fixed point?** Self‑Rewarding LMs and RLAIF both improve monotonically by benchmark. Whether this reflects real improvement or "the judge and the policy now agree with each other" is not settled.
3. **Alignment tax vs alignment benefit.** Process supervision shows *negative* alignment tax in math; most other settings still show a tax. When does alignment hurt capability, and when does it help? No unified theory.
4. **Persona / humanization is mostly absent from this literature.** Papers optimize helpfulness, harmlessness, and truthfulness. Dimensions like *warmth*, *humor*, *disagreement with the user*, *stylistic distinctiveness* — core to humanization — are not first‑class objectives in any major alignment paper. This is a research gap the humanization project can exploit.
5. **Personalization under RLHF.** Current methods assume a global preference distribution. The literature lacks a principled account of per‑user or per‑audience preference optimization — a prerequisite for personalized humanization.
6. **Label provenance and labeler bias.** InstructGPT discusses this briefly; most other papers do not. The question "whose preferences are encoded?" is under‑studied relative to its importance.
7. **Evaluation bottleneck.** AlpacaEval / Arena‑Hard / MT‑Bench are themselves LLM‑judged. Controversies over length bias and judge‑model bias mean alignment progress is measured by instruments that share the biases being critiqued.

## References

1. Christiano, Leike, Brown, Martic, Legg, Amodei. *Deep Reinforcement Learning from Human Preferences.* NeurIPS 2017. https://arxiv.org/abs/1706.03741
2. Stiennon et al. *Learning to Summarize from Human Feedback.* NeurIPS 2020. https://arxiv.org/abs/2009.01325
3. Ouyang et al. *Training Language Models to Follow Instructions with Human Feedback (InstructGPT).* NeurIPS 2022. https://arxiv.org/abs/2203.02155
4. Bai et al. *Training a Helpful and Harmless Assistant with RLHF (HH‑RLHF).* arXiv 2022. https://arxiv.org/abs/2204.05862
5. Bai et al. *Constitutional AI: Harmlessness from AI Feedback.* arXiv 2022. https://arxiv.org/abs/2212.08073
6. Rafailov et al. *Direct Preference Optimization: Your Language Model is Secretly a Reward Model.* NeurIPS 2023. https://arxiv.org/abs/2305.18290
7. Lee et al. *RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback.* ICML 2024. https://arxiv.org/abs/2309.00267
8. Zhao et al. *SLiC‑HF: Sequence Likelihood Calibration with Human Feedback.* arXiv 2023. https://arxiv.org/abs/2305.10425
9. Azar et al. *A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO / ΨPO).* AISTATS 2024. https://arxiv.org/abs/2310.12036
10. Ethayarajh et al. *KTO: Model Alignment as Prospect Theoretic Optimization.* ICML 2024. https://arxiv.org/abs/2402.01306
11. Hong, Lee, Thorne. *ORPO: Monolithic Preference Optimization without Reference Model.* EMNLP 2024. https://arxiv.org/abs/2403.07691
12. Meng, Xia, Chen. *SimPO: Simple Preference Optimization with a Reference‑Free Reward.* NeurIPS 2024. https://arxiv.org/abs/2405.14734
13. Lightman et al. *Let's Verify Step by Step.* ICLR 2024. https://arxiv.org/abs/2305.20050
14. Yuan et al. *Self‑Rewarding Language Models.* arXiv 2024. https://arxiv.org/abs/2401.10020
15. Gao, Schulman, Hilton. *Scaling Laws for Reward Model Overoptimization.* ICML 2023. https://arxiv.org/abs/2210.10760
16. Singhal, Goyal, Xu, Durrett. *A Long Way to Go: Investigating Length Correlations in RLHF.* COLM 2024. https://arxiv.org/abs/2310.03716
17. Sharma et al. *Towards Understanding Sycophancy in Language Models.* ICLR 2024. https://arxiv.org/abs/2310.13548
18. Casper et al. *Open Problems and Fundamental Limitations of RLHF.* ICLR 2025 (Journal Track). https://arxiv.org/abs/2307.15217
19. Touvron et al. *Llama 2: Open Foundation and Fine‑Tuned Chat Models.* arXiv 2023. https://arxiv.org/abs/2307.09288
20. Dong et al. *RLHF Workflow: From Reward Modeling to Online RLHF.* arXiv 2024. https://arxiv.org/abs/2405.07863
21. Shapira, Benadé, Procaccia. *How RLHF Amplifies Sycophancy.* arXiv 2026. https://arxiv.org/abs/2602.01002
22. ByteDance Seed. *DAPO: An Open-Source LLM Reinforcement Learning System at Scale.* arXiv 2025. https://arxiv.org/abs/2503.14476
23. Qwen Team. *VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks.* arXiv 2025. https://arxiv.org/abs/2504.05118
24. Sordoni et al. *Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.* NeurIPS 2024. https://arxiv.org/abs/2406.02900
25. Lambert, N. *Reinforcement Learning from Human Feedback.* arXiv 2025 (RLHF Book). https://arxiv.org/abs/2504.12501
26. Various. *A Survey of Direct Preference Optimization.* arXiv March 2025. https://arxiv.org/abs/2503.11701
