<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Diy-Llm on 吕炘嵘的博客</title>
    <link>https://blog.lvxinrong.com/tags/diy-llm/</link>
    <description>Recent content in Diy-Llm on 吕炘嵘的博客</description>
    <image>
      <title>吕炘嵘的博客</title>
      <url>https://blog.lvxinrong.com/og-default.jpg</url>
      
      <link>https://blog.lvxinrong.com/tags/diy-llm/</link>
    </image>
    <generator>Hugo</generator>
    <language>zh-CN</language>
    <lastBuildDate>Wed, 28 Oct 2026 12:00:00 +0800</lastBuildDate>
    <atom:link href="https://blog.lvxinrong.com/tags/diy-llm/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>手搓 LLM 笔记 11：PyTorch 与资源核算</title>
      <link>https://blog.lvxinrong.com/llm/llm-11-pytorch-resource-accounting/</link>
      <pubDate>Wed, 28 Oct 2026 12:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-11-pytorch-resource-accounting/</guid>
      <description>训练成本不是黑盒：参数量、显存、FLOPs、时间的估算闭环；混合精度、算术强度、梯度累积与开工踩坑。</description>
      <content:encoded><![CDATA[<p>训练大语言模型时，资源消耗可以直接转换成时间和成本的 所以需要资源核算</p>
<p>FLOPs(Floating Point Operations, 浮点运算次数) 衡量计算复杂度的一个指标，表示一个算法或者模型在执行过程中所需的浮点运算的总次数</p>
<p>经验公式:</p>
<blockquote>
<p>总计算量 ≈  6 ✖️ 参数量 ✖️ Token数量</p>
</blockquote>
<p>6:  fowrad 计算一次约为 2 ✖️ 参数量，backward 约等于前向的2倍，也就是 4 ✖️参数量</p>
<p>时间计算公式:</p>
<blockquote>
<p>时间 = 总工作量 / 总算力</p>
</blockquote>
<p>硬件算力可以查，H100 峰值算力约 1979TFLOPS （FP16/BP16数据类型，且启用结构化稀疏）
训练普通的稠密模型 理论峰值减半 约990 TFLOPs</p>
<p>结构化稀疏: 一种模型压缩的方法，通常是对稠密模型按照50%的稀疏度进行剪纸
<img alt="结构化稀疏：对稠密模型按 50% 稀疏度剪枝" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-11-structured-sparsity.png"></p>
<p>软硬件开销系数 (通常按照 30% ~ 60% 利用率估算更现实)</p>
<p>所以  <strong>在 1024 张 H100 显卡上，训练一个 70B（700亿参数）的模型，数据量是 15T（15万亿 tokens），大概要多久？</strong></p>
$$
\text{时间} = \frac{\text{总工作量}}{\text{总算力}} = \frac{6.3 \times 10^{24}}{5 \times 10^{17}} \approx 1.26 \times 10^7 \text{ 秒} \approx \mathbf{146} \text{ 天}
$$<p><strong>FP16</strong> 和 <strong>BF16</strong> 都是计算机中用于表示小数的<strong>浮点数数据类型</strong>，它们在<strong>深度学习</strong>（AI 模型训练和推理）中被广泛使用。它们的主要目的是<strong>减少内存占用</strong>并<strong>加速计算</strong>（因为占用空间比传统的 FP32 少一半）
FP16 (Half Precision / 半精度浮点数) 精度高 但表示范围小
BF16 (Bfloat16 / 脑浮点数) 精度低 但表示范围极大</p>
<blockquote>
<p>Percy 在课上解释说：“bf16 牺牲了精度来换取范围。对于深度学习来说，范围比精度重要得多，因为数值稳定性的主要威胁是溢出/下溢，而不是尾数精度不够。”</p>
</blockquote>
<p>内存估算
需要同时考虑 参数，梯度，优化器状态(不同优化器 参数量不同)</p>
<h1 id="张量-pytorchtensor">张量 pytorch.tensor</h1>
<p><code>jaxtyping</code> 提供的维度命名（如 “batch seq hidden”）在当前 PyTorch 生态中主要是“文档性质”的，不会在运行时自动强制检查维度是否真的匹配名称，但它能极大地提升代码的清晰度和可靠性。</p>
<pre tabindex="0"><code># 传统方式（容易写错维度顺序）
x = torch.ones(2, 2, 1, 3)  # batch seq heads hidden

# Jaxtyping方式（在类型注解中命名维度）
from jaxtyping import Float
x: Float[torch.Tensor, &#34;batch seq heads hidden&#34;] = torch.ones(2, 2, 1, 3)
</code></pre><p>使用 Einops 库对张量操作进行优化: Einops增加了少量语法开销，但其清晰的维度命名显著降低了调试难度，特别是在复杂的模型架构中
这个后续学习 记个待办</p>
<ul>
<li><strong>FLOPs</strong>（小写s）：执行的浮点运算总数，这是一个衡量计算量的指标，</li>
<li><strong>FLOP/s</strong>（每秒浮点运算）：指硬件每秒能执行的浮点运算次数，也写作 <strong>FLOPS</strong>（大写S）。这是一个衡量“速度”的指标。例如，H100 GPU峰值性能为 990 TFLOP/s。</li>
</ul>
<p>对比矩阵乘法 A (m×k) @ B (k×n) 的 FLOPs = 2 × m × k × n。当 k 较大时（如 k = 1024），矩阵乘法的 FLOPs 是加法的上千倍。对于现代深度学习模型（尤其是 Transformer、MLP 等），90%+ 的 FLOPs 来自矩阵乘法（线性层、QKV 投影、FFN 等）。其他操作（LayerNorm、Softmax、激活函数、加法残差连接等）虽然存在，但 FLOPs 可忽略不计。因此，<strong>在进行粗略但有效的资源估算时，只需计算所有矩阵乘法的 FLOPs 总和即可</strong>。</p>
<p>调用 loss.backward() 后，PyTorch 会从 loss 开始，沿着计算图回溯，自动计算出每个需要梯度的张量的梯度。</p>
<p>一个普适性的结论：在深度学习中，<strong>训练一次（前向+反向）的总 FLOPs 约等于 6 倍的“数据点数量 × 参数数量”</strong></p>
<p>在每次迭代开始前，需要清空上一次计算得到的梯度，否则梯度会累积。优化器在调用 <code>zero_grad</code> 函数清空梯度时，设置 set_to_none=True 是一种更高效的内存管理方式，它会将梯度指针设为 None，而不是将其置零，可以节省内存。</p>
<pre tabindex="0"><code>optimizer.zero_grad(set_to_none=True)
</code></pre><h3 id="检查点checkpointing"> 检查点（Checkpointing）</h3>
<p>大型语言模型的训练耗时极长，过程中几乎不可避免地会发生崩溃或中断，我们需要一种机制来保存进度，以便在中断后能够从中断点恢复，而不是从头开始</p>
<p>检查点需要包含两块，模型参数 和 优化器状态</p>
<blockquote>
<p>为了能够完整地恢复训练，检查点必须包含所有必要的状态信息。主要包括两个核心部分：</p>
<ul>
<li>模型参数：这是模型的核心，包含了所有可学习权重（nn.Parameter）的当前值。通过 model.state_dict() 可以获取一个字典，其中键是参数名称，值是对应的张量数据。
-优化器状态 (optimizer.state_dict())：这是很多人容易忽略但极其关键的部分。优化器（如 Adam、AdaGrad）不仅存储了当前的学习率，还维护着一些内部状态变量。
例如: 如果只保存了模型参数而没有保存优化器状态，那么在恢复训练时，优化器会从零开始，这会导致训练过程不连续，性能下降，甚至可能无法收敛。</li>
</ul>
</blockquote>
<p>checkpoint 保存检查点</p>
<pre tabindex="0"><code># 1. 创建一个字典，包含模型和优化器的状态
checkpoint = {
    &#34;model&#34;: model.state_dict(),
    &#34;optimizer&#34;: optimizer.state_dict(),
}

# 2. 使用 torch.save 将字典序列化并保存到磁盘文件
torch.save(checkpoint, &#34;model_checkpoint.pt&#34;) # 文件名通常使用 .pt 或 .pth 后缀
</code></pre><p>加载也需要两部分同时加载:</p>
<pre tabindex="0"><code># 1. 从磁盘文件加载保存的字典
loaded_checkpoint = torch.load(&#34;model_checkpoint.pt&#34;)

# 2. （后续步骤）将状态加载回模型和优化器
model.load_state_dict(loaded_checkpoint[&#34;model&#34;])
optimizer.load_state_dict(loaded_checkpoint[&#34;optimizer&#34;])
</code></pre><h3 id="混合精度训练-mixed-precision">混合精度训练 (Mixed Precision)</h3>
<p>将低精度(BF16/ FP 16) 用于消耗大但对精度要求不高的部分(激活值)。将高精度用于 对精度敏感的部分(参数和梯度)</p>
<p>PyTorch 的 AMP 库 (Automatic Mixed Precision)：PyTorch 提供了自动混合精度（AMP）库，会自动将安全的操作（如矩阵乘法）转为 bf16，将危险操作（如 exp、softmax）保留为 fp32：</p>
<pre tabindex="0"><code>with torch.amp.autocast(&#34;cuda&#34;, dtype=torch.bfloat16):
    x = torch.zeros(4, 8)  # 自动以 bf16 创建
</code></pre><p>数据搬运也是需要时间，所以总时间取决于 加速器速度FLOP/s 和 内存带宽Bytes/s</p>
<h3 id="算术强度">算术强度</h3>
<p>为了避免每次都要计算两个时间再比较，引入<strong>算术强度（Arithmetic Intensity）</strong>：</p>
<blockquote>
<p>加速器强度 = FLOP/s / Bytes/s   H100 约295FLOP/bytes
算数强度 = FLOPs / Bytes  该操作每搬运1字节能做多少FLOP</p>
</blockquote>
<p>算数强度 &lt; 加速器强度  Memory-bound 瓶颈在数据传输
算数强度 &gt; 加速器强度  Compute-bound 瓶颈在计算</p>
<p>python中 列表的append()和extend 都是原地修改列表。
<code>append(x)</code>：把 <code>x</code> 当作<strong>一个整体元素</strong>加到末尾</p>
<pre tabindex="0"><code>a = [1, 2]
a.append([3, 4])
print(a)  # [1, 2, [3, 4]]
</code></pre><p><code>extend(iterable)</code>：把可迭代对象里的元素<strong>逐个拆开</strong>加到末尾</p>
<pre tabindex="0"><code>a = [1, 2]
a.extend([3, 4])
print(a)  # [1, 2, 3, 4]
</code></pre><p><code>extend</code> 接收的是可迭代对象，比如列表、元组、字符串、集合等</p>
<h2 id="今天的坑m2-开工实录">今天的坑（M2 开工实录）</h2>
<h3 id="梯度累积三次-backward-一次-step">梯度累积：三次 backward 一次 step</h3>
<p>一开始写成了“把所有 loss 攒进 list，最后一次 backward”——这样所有计算图都攒在手里，显存越攒越大，完全背离了梯度累积的初衷。正确姿势是<strong>每个小批立刻 backward（释放计算图），梯度累加在 .grad 里，攒够 N 批再 optimizer.step()</strong>。
归一化也有两种等价写法：<code>sum(loss_i)/N</code> 和每个 <code>loss_i/N</code> 再 backward，因为梯度是线性的，结果一样，但后者省显存。
触发条件用 <code>(i+1) % accum_step == 0</code> 一个表达式通吃，不用给 accum_step=1 单写分支。</p>
<h3 id="zero_grad-的位置决定成败">zero_grad 的位置决定成败</h3>
<p>原来的老代码在循环顶部 zero_grad()，改成累积模式后如果还留着，<strong>前几批攒下的梯度还没 step 就被清掉了</strong>，累积直接白干。口诀：<strong>梯度只在“用掉之后”清零（step 之后），不能在“攒的过程中”清零。</strong></p>
<h3 id="gitignore-的目录匹配会误伤">.gitignore 的目录匹配会误伤</h3>
<p>写 <code>data/</code> 会匹配<strong>所有层级</strong>的 data 目录，把 <code>src/squid/data/</code> 代码包也忽略了（新文件变灰、git status 看不到）。要只排除根目录的语料目录，必须写 <code>/data/</code>（前导斜杠锚定根目录）。</p>
<h3 id="total_loss-是-list-不是张量">total_loss 是 list 不是张量</h3>
<p><code>total_loss / accum_step</code> 会炸，因为 list 不能除以整数；而且把 loss 存进 list 不 backward，等于把计算图全留着。loss 是张量，应该当批归一化后立刻 backward，不是攒起来。</p>
<h2 id="bpb-的含义与下降意味着什么">BPB 的含义与“下降意味着什么”</h2>
<p><strong>BPB（Bits Per Byte）的本质是衡量语言模型把文本压缩了多少：给这段文本编码，平均每个字节要花多少比特。数值越低，说明模型对文本的预测能力越强，压缩效率越高。</strong></p>
<p>两个参照点帮助记忆：</p>
<ul>
<li>BPB 接近 0：模型几乎能完美预测文本，每个字节只需极少比特</li>
<li>BPB = 8：完全不会预测，每个字节都得老老实实存 8 bit，等于没有压缩</li>
</ul>
<p>所以 <strong>BPB 下降 = 模型用更少的比特就能编码同样的文本 = 模型在变好</strong>。它和 val_loss 下降的区别是：val_loss 是“每个 token 的预测难度”，BPB 把它折算成了“每个字节的成本”——这样不同词表、不同模型之间才能公平比较。</p>
<h2 id="容量与深度扫描数据决定上限">容量与深度扫描：数据决定上限</h2>
<p>M2 最小闭环之后，用同一批语料、同一训练量做了容量与深度对照，结论比配置本身值钱。</p>
<h3 id="emb_dim-扫描20-epoch其他锁死">emb_dim 扫描（20 epoch，其他锁死）</h3>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>emb_dim</th>
        <th>最终 val_loss</th>
        <th>BPB</th>
        <th>结论</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>64</td>
        <td>7.02</td>
        <td>3.63</td>
        <td>容量不足</td>
      </tr>
      <tr>
        <td>128</td>
        <td>6.59</td>
        <td>3.42</td>
        <td>仍在受益</td>
      </tr>
      <tr>
        <td><strong>256</strong></td>
        <td><strong>5.99</strong></td>
        <td><strong>3.10</strong></td>
        <td><strong>甜点</strong></td>
      </tr>
      <tr>
        <td>512</td>
        <td>6.26（过拟合）</td>
        <td>3.24</td>
        <td>开始背语料</td>
      </tr>
    </tbody>
  </table>
</div>
<p>512 出现了教科书式的过拟合：<strong>train_loss 一路降到 1.8，val_loss 却从 5.85 回升到 6.29</strong>——模型在背训练集，不是在学规律。所以容量甜点在 256 附近：64 欠拟合、512 过拟合。</p>
<p>验证代码:</p>
<pre tabindex="0"><code>&#34;&#34;&#34;M2 收官：真实语料 + 自训 BPE 的完整预训练。  
  
链路：pack_tokens 打包 → create_stream_loader → 小 GPT 预训练  
      → bpb 曲线 → generate_text 咿呀学语 → save_checkpoint 带 config。  
&#34;&#34;&#34;  
import math  
from pathlib import Path  
  
import torch  
  
from squid.data.token_stream import pack_tokens, create_stream_loader  
from squid.device import get_device  
from squid.generate.text import generate_text  
from squid.model.gpt import GPTModel  
from squid.train.common import calc_loss_loader  
from squid.train.pretrain import pretrain_model  
from squid.tokenizer.my_bpe_tokenizer import MyBPETokenizer  
from squid.checkpoints.store import save_checkpoint  
  
ROOT = Path(__file__).resolve().parents[1]  
DATA = ROOT / &#34;data&#34; / &#34;blog&#34;  
  
  
def test_m2_pretrain():  
    device = get_device()  
    tokenizer = MyBPETokenizer.from_directory(DATA / &#34;m2000&#34;)  
  
    # 1. token 流打包：训练语料与 held-out 分别打包  
    train_text = (DATA / &#34;train.txt&#34;).read_text(encoding=&#34;utf-8&#34;)  
    val_text = &#34;\n&#34;.join(p.read_text(encoding=&#34;utf-8&#34;) for p in DATA.glob(&#34;heldout-*.txt&#34;))  
  
    train_ids = pack_tokens([train_text], tokenizer, tokenizer.eos_id)  
    val_ids = pack_tokens([val_text], tokenizer, tokenizer.eos_id)  
  
    context = 128  
    train_loader = create_stream_loader(train_ids, context, batch_size=8, shuffle=True)  
    val_loader = create_stream_loader(val_ids, context, batch_size=8,  
                                      shuffle=False, drop_last=False)  
  
    # 2. 小模型：emb_dim 64、2 层、context 128  
    cfg = {  
        &#34;vocab_size&#34;: len(tokenizer.vocab),  
        &#34;context_length&#34;: context,  
        &#34;emb_dim&#34;: 256,  
        &#34;n_heads&#34;: 2,  
        &#34;n_layers&#34;: 4,  
        &#34;drop_rate&#34;: 0.0,  
        &#34;qkv_bias&#34;: False,  
    }  
    model = GPTModel(cfg).to(device)  
    optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)  
  
    # 3. 预训练：accum_step=4，bpb 用 held-out 的 bytes/token 折算  
    density = len(val_text.encode(&#34;utf-8&#34;)) / len(tokenizer.encode(val_text))  
    train_losses, val_losses, tokens_seen = pretrain_model(  
        model, train_loader, val_loader, optimizer, device,  
        num_epochs=20, eval_freq=50, eval_iter=5,  
        accum_step=4, bpb_density=density,  
    )  
    assert val_losses[-1] &lt; val_losses[0], &#34;val loss 应该下降&#34;  
  
    # 4. 咿呀学语：给模型一个开头，看它续什么  
    sample = generate_text(model, tokenizer, &#34;大模型&#34;,  
                           max_new_tokens=40, context_size=context,  
                           temperature=0.8, top_k=20, eos_id=tokenizer.eos_id)  
    print(&#34;\n=== 生成样例 ===\n&#34;, sample)  
  
    # 5. 存档：权重 + config + 元信息，完整还原训练现场  
    final_val = calc_loss_loader(val_loader, model, device)  
    save_checkpoint(ROOT / &#34;regression&#34; / &#34;m2-pretrain.pth&#34;,  
                    model, cfg, val_loss=final_val, merges=2000, tokens_seen=tokens_seen[-1])  
    print(f&#34;\n最终 val_loss={final_val:.3f}，checkpoint 已保存&#34;)
</code></pre><h3 id="n_layers-对照emb_dim256">n_layers 对照（emb_dim=256）</h3>
<p>4 层和 2 层基本持平（val_loss 6.03 vs 5.99），深度没有收益，后段 val 还开始上翘。最优配置停在 <strong>emb_dim=256、n_layers=2</strong>。</p>
<p>一句话总结：<strong>小数据面前，模型不是越大越好，是“配得上”才好。</strong> 压缩率有甜点、BPB 有甜点、模型容量也有甜点——全是测出来的，不是猜的。</p>
<h2 id="两个收尾的-bug">两个收尾的 bug</h2>
<p><strong>accum_step 大于每轮批次数</strong>：每 epoch 只有 4 个 batch，accum_step=5 时 <code>(i+1) % 5 == 0</code> 永远不成立，optimizer.step() 一次都不执行，模型完全不学。改成跨 epoch 的 global_step 连续计数才解决。以后看到 loss 完全不动，先查 step 到底执行了没有。</p>
<p><strong>decode 的 UnicodeDecodeError</strong>：字节级 BPE 的 token 可能只是半个汉字，模型自由生成的序列不一定是合法 UTF-8，直接 decode 会炸。改成 <code>decode(&quot;utf-8&quot;, errors=&quot;replace&quot;)</code>，非法片段显示为替换符即可——tiktoken 默认就是这么干的。</p>
]]></content:encoded>
    </item>
    <item>
      <title>手搓 LLM 笔记 10：手写 BPE，从写不出来到甜点区</title>
      <link>https://blog.lvxinrong.com/llm/llm-10-handwritten-bpe/</link>
      <pubDate>Tue, 27 Oct 2026 18:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-10-handwritten-bpe/</guid>
      <description>纯 Python 手写 BPE 训练与编解码，封装成 SquidTokenizer；再用压缩率曲线和小模型 BPB 实验，找到博客语料的甜点区 4000 次合并。</description>
      <content:encoded><![CDATA[<p>上一章遗留的作业：在 <code>squid/tokenizer/bpe.py</code> 里实现三个函数：<code>train_bpe</code>（学合并表）、<code>encode</code>（按表切分）、<code>decode</code>（还原）。先小语料跑通往返，再上博客语料测压缩率。</p>
<p>说实话，开工前我觉得自己写不出来。但拆成小函数之后，一个一个都拿下了。</p>
<h2 id="三个零件">三个零件</h2>
<p><strong>count_pairs</strong>：用滑动窗口把 i 和 i+1 的 id 配成 key，统计每个 key 出现的次数。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">count_pairs</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">],</span> <span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">counts</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">ids</span><span class="p">[</span><span class="mi">1</span><span class="p">:]):</span>
</span></span><span class="line"><span class="cl">        <span class="n">key</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">counts</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">counts</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">counts</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">counts</span>
</span></span></code></pre></div><p><strong>merge</strong>：把 ids 里所有出现过的 pair 替换成新 id。比如 pair=(1,2)、ids=[1,2,1,2,3,4,5]、new_id=9，结果就是 <code>[9, 9, 3, 4, 5]</code>。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">],</span> <span class="n">pair</span><span class="p">:</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">],</span> <span class="n">new_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">new_idx</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="ow">and</span> <span class="p">(</span><span class="n">ids</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">ids</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">])</span> <span class="o">==</span> <span class="n">pair</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">new_idx</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">i</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">2</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">new_idx</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ids</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">            <span class="n">i</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">new_idx</span>
</span></span></code></pre></div><p>这里有个容易写错的点：统计是滑动窗口（允许重叠），替换是贪心扫描（命中消费两格，不匹配消费一格）。所以用 while 手动控制步长，for i in range 里修改 i 下一轮会被覆盖。</p>
<p><strong>train_bpe</strong>：vocab 初始 256 项（所有单字节），每轮取 count_pairs 里 value 最大的 key 合并，新 id 从 256 开始累加。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">train_bpe</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">num_merges</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">vocab</span> <span class="o">=</span> <span class="p">{</span><span class="n">i</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">([</span><span class="n">i</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">256</span><span class="p">)}</span>
</span></span><span class="line"><span class="cl">    <span class="n">merges</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">num_merges</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">counts</span> <span class="o">=</span> <span class="n">count_pairs</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">counts</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">break</span>
</span></span><span class="line"><span class="cl">        <span class="n">best</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">counts</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">counts</span><span class="o">.</span><span class="n">get</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">new_id</span> <span class="o">=</span> <span class="mi">256</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">ids</span> <span class="o">=</span> <span class="n">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">best</span><span class="p">,</span> <span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">merges</span><span class="p">[</span><span class="n">best</span><span class="p">]</span> <span class="o">=</span> <span class="n">new_id</span>
</span></span><span class="line"><span class="cl">        <span class="n">vocab</span><span class="p">[</span><span class="n">new_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">vocab</span><span class="p">[</span><span class="n">best</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">+</span> <span class="n">vocab</span><span class="p">[</span><span class="n">best</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span>
</span></span></code></pre></div><p>注意最后一行：best 里装的是 id 不是字符，要先查 vocab 取出 bytes 再拼接，不能直接 <code>best[0] + best[1]</code>（那是两个整数相加）。</p>
<h2 id="encode--decode-与往返无损">encode / decode 与往返无损</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">],</span> <span class="n">vocab</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="sa">b</span><span class="s2">&#34;&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">vocab</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">ids</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">merges</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">pair</span><span class="p">,</span> <span class="n">new_id</span> <span class="ow">in</span> <span class="n">merges</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="n">ids</span> <span class="o">=</span> <span class="n">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">pair</span><span class="p">,</span> <span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">ids</span>
</span></span></code></pre></div><p>decode 是查词表把 ids 还原成文本，必须先在 bytes 世界拼完整再 decode，不然中文多字节字符会被切碎。encode 是把训练学到的合并规则<strong>按学习顺序</strong>在新文本上重放一遍——merges 存的是规则不是结果，不重放，新 token 不会出现。</p>
<p>往返验收过了：<code>decode(encode(text, merges), vocab) == text</code>。</p>
<p>Python 里 bytes 和 str 的转换我目前还不熟，记入欠账，后面找机会补。</p>
<h2 id="封装成-squidtokenizer">封装成 SquidTokenizer</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyBPETokenizer</span><span class="p">(</span><span class="n">SquidTokenizer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span><span class="p">,</span> <span class="n">eos_id</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span> <span class="o">=</span> <span class="n">vocab</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">merges</span> <span class="o">=</span> <span class="n">merges</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span> <span class="o">=</span> <span class="n">eos_id</span> <span class="k">if</span> <span class="n">eos_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="nb">len</span><span class="p">(</span><span class="n">vocab</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span><span class="p">,</span> <span class="sa">b</span><span class="s2">&#34;&lt;|endoftext|&gt;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">bpe</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">idx</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">bpe</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nd">@classmethod</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">from_directory</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span> <span class="o">=</span> <span class="n">bpe</span><span class="o">.</span><span class="n">load_bpe</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nd">@property</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">eos_id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span>
</span></span></code></pre></div><p>这段学到的 Python 点：</p>
<ol>
<li>三元表达式是 <code>值A if 条件 else 值B</code>，所以 eos_id 那行就是“传了用传入的，没传用词表大小”</li>
<li><code>@classmethod</code> 是挂在类上的方法，类似 Java 静态工厂。<code>cls(vocab, merges)</code> 就是调用 <code>__init__</code>，但比 Java static 多一层：cls 是类本身，子类调用返回子类实例</li>
<li><code>@property</code> 把方法伪装成属性，访问不用加括号</li>
<li><code>self._*</code> 下划线只是约定，不是 private，依然能访问</li>
</ol>
<h2 id="实验一压缩率曲线与甜点区">实验一：压缩率曲线与甜点区</h2>
<p>分别训练了 [64, 128, 256, 512, 1024, 2000, 4000, 8000, 16000] 次的词表，在 held-out 文本上测压缩率（<code>len(text.encode(&quot;utf-8&quot;)) / len(tokenizer.encode(text))</code>，即 bytes/token，越高越省）：</p>
<pre tabindex="0"><code>merges=   64  tokens=10588  bytes/token=1.29
merges=  128  tokens= 9631  bytes/token=1.42
merges=  256  tokens= 8346  bytes/token=1.64
merges=  512  tokens= 7009  bytes/token=1.95
merges= 1024  tokens= 5944  bytes/token=2.30
merges= 2000  tokens= 5043  bytes/token=2.71
merges= 4000  tokens= 4287  bytes/token=3.19
merges= 8000  tokens= 3869  bytes/token=3.54
merges=16000  tokens= 3704  bytes/token=3.69
</code></pre><p>token 数一直在降，bytes/token 一直在涨，但<strong>看每次翻倍的收益</strong>：前期稳定 +17% 左右，4000→8000 掉到 +11%，8000→16000 只剩 +4%。<strong>甜点区在 4000-8000</strong>，再往后基本是在背语料。</p>
<p>直观例子：手机号最好学成 3-4-4 的片段，merges 一直增大就会学成整个 11 位号码——每个号码独一无二，embedding 永远学不到，模型也无法对号码内部做推理。所以并不是 merges 越多越好，每个 token 携带的信息越大，模型训练时也需要更强的能力去掌握它。</p>
<p>另一个好玩的发现：512 次合并时“大模型”还排不上号（语料里只出现 31 次），2000 次之后它终于“拥有姓名”，被切成了一个完整 token。<strong>词表大小决定了多低频的词才配拥有姓名。</strong></p>
<h2 id="实验二bpb把模型拉上一起评">实验二：BPB，把模型拉上一起评</h2>
<p>和 DeepSeek 讨论后确认：评估分词器不能只看 bytes/token，那只是在评分词器本身，还得看“模型+词表”的整体效果。这个指标叫 <strong>BPB（Bits Per Byte）</strong>：</p>
<pre tabindex="0"><code>BPB = (val_loss / ln2) ÷ bytes/token
</code></pre><p>含义：<strong>给这段文本编码，平均每个字节要花多少比特</strong>。val_loss 是模型每预测一个 token 的代价（nats），除以 ln2 换成比特；再除以 bytes/token，就把“每个 token 的成本”折算成“每个字节的成本”。<strong>越低越好</strong>——它同时惩罚两种情况：词表太碎（bytes/token 小）和模型没学好（val_loss 高）。</p>
<h3 id="实验设计">实验设计</h3>
<p>四组：512 / 2000 / 4000 / 8000，同一个微型模型配置（emb_dim=64、n_layers=2、context=128），同一份 train/held-out 文本，只换分词器。</p>
<p><strong>一个关键的控制变量：统一步数，不是统一 epoch。</strong> 因为词表越大，同样文本的 ids 越短，定长窗口切出的窗口数越少，每 epoch 的训练步数也越少：</p>
<pre tabindex="0"><code>窗口数 = (len(ids) - max_length) / stride
每 epoch 步数 = 窗口数 / batch_size
</code></pre><p>如果只统一 epoch，大词表模型等于“饿着肚子考试”。所以给 <code>pretrain_model</code> 加了 <code>max_steps</code> 参数，四组都固定走 500 步。</p>
<p>完整实验代码：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">math</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.data.pretrain</span> <span class="kn">import</span> <span class="n">create_pretrain_loader</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.device</span> <span class="kn">import</span> <span class="n">get_device</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.model.gpt</span> <span class="kn">import</span> <span class="n">GPTModel</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.tokenizer.my_bpe_tokenizer</span> <span class="kn">import</span> <span class="n">MyBPETokenizer</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.train.common</span> <span class="kn">import</span> <span class="n">calc_loss_loader</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.train.pretrain</span> <span class="kn">import</span> <span class="n">pretrain_model</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">ROOT</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="vm">__file__</span><span class="p">)</span><span class="o">.</span><span class="n">resolve</span><span class="p">()</span><span class="o">.</span><span class="n">parents</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">DATA</span> <span class="o">=</span> <span class="n">ROOT</span> <span class="o">/</span> <span class="s2">&#34;data&#34;</span> <span class="o">/</span> <span class="s2">&#34;blog&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">train_model_with_nums_merge</span><span class="p">(</span><span class="n">num_merges</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">vocab_data</span> <span class="o">=</span> <span class="n">ROOT</span> <span class="o">/</span> <span class="s2">&#34;data&#34;</span> <span class="o">/</span> <span class="s2">&#34;blog&#34;</span> <span class="o">/</span> <span class="n">num_merges</span>
</span></span><span class="line"><span class="cl">    <span class="n">tokenizer</span> <span class="o">=</span> <span class="n">MyBPETokenizer</span><span class="o">.</span><span class="n">from_directory</span><span class="p">(</span><span class="n">vocab_data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">cfg</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;vocab_size&#34;</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">vocab</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;context_length&#34;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;emb_dim&#34;</span><span class="p">:</span> <span class="mi">64</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;n_heads&#34;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;n_layers&#34;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;drop_rate&#34;</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;qkv_bias&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="n">device</span> <span class="o">=</span> <span class="n">get_device</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span> <span class="o">=</span> <span class="n">GPTModel</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">train_text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;train.txt&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;heldout-手搓LLM笔记07-指令微调.txt&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_text</span> <span class="o">=</span> <span class="n">val_text</span> <span class="o">+</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span> <span class="o">+</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;heldout-我想把家里的设备组成一个私人AI实验室.txt&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                      <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">train_loader</span> <span class="o">=</span> <span class="n">create_pretrain_loader</span><span class="p">(</span><span class="n">train_text</span><span class="p">,</span> <span class="n">tokenizer</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                          <span class="n">batch_size</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_loader</span> <span class="o">=</span> <span class="n">create_pretrain_loader</span><span class="p">(</span><span class="n">val_text</span><span class="p">,</span> <span class="n">tokenizer</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                        <span class="n">batch_size</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">drop_last</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">AdamW</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_losses</span><span class="p">,</span> <span class="n">val_losses</span><span class="p">,</span> <span class="n">tokens_seen</span> <span class="o">=</span> <span class="n">pretrain_model</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">model</span><span class="o">=</span><span class="n">model</span><span class="p">,</span> <span class="n">train_loader</span><span class="o">=</span><span class="n">train_loader</span><span class="p">,</span> <span class="n">val_loader</span><span class="o">=</span><span class="n">val_loader</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">optimizer</span><span class="o">=</span><span class="n">optimizer</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">num_epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">max_steps</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">eval_freq</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">eval_iter</span><span class="o">=</span><span class="mi">100</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_loss</span> <span class="o">=</span> <span class="n">calc_loss_loader</span><span class="p">(</span><span class="n">val_loader</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">density</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">val_text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">val_text</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">bpb</span> <span class="o">=</span> <span class="n">val_loss</span> <span class="o">/</span> <span class="n">math</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">density</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;val_loss=</span><span class="si">{</span><span class="n">val_loss</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">  bytes/token=</span><span class="si">{</span><span class="n">density</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">  BPB=</span><span class="si">{</span><span class="n">bpb</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m512&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m2000&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m4000&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m8000&#34;</span><span class="p">)</span>
</span></span></code></pre></div><h3 id="结果">结果</h3>
<pre tabindex="0"><code>m512 : val_loss=4.819  bytes/token=1.99  BPB=3.494
m2000: val_loss=6.551  bytes/token=2.80  BPB=3.373
m4000: val_loss=7.584  bytes/token=3.30  BPB=3.315
m8000: val_loss=8.378  bytes/token=3.66  BPB=3.300
</code></pre><p><strong>为什么 val_loss 一路涨？</strong> val_loss 是“每个 token 的预测难度”，词表变大之后，大量 token 在 117KB 语料里只出现几次，embedding 学不充分，预测它们接近靠蒙，困惑度自然上升。这不是词表变差，是“大 token 更难预测”的固有代价。</p>
<p><strong>但 BPB 在降</strong>——因为每个 token 装的字节更多了，折算回每字节成本反而便宜。而且看边际收益：512→2000 降 0.121，2000→4000 降 0.058，4000→8000 只降 0.015。<strong>BPB 的甜点同样在 4000 附近</strong>，和压缩率实验互相印证。</p>
<p>结论：<strong>val_loss 上升不等于词表不好，BPB 才是整体账；但 BPB 的边际收益也会递减，4000 之后再合并基本没红利。</strong></p>
<h2 id="bpe-的三个哲学点">BPE 的三个哲学点</h2>
<p>实验做完再回头看，BPE 的底层逻辑其实就三句话：</p>
<ol>
<li><strong>原子由数据决定。</strong> 什么是“思考的最小单位”，不是人预先规定的，是语料的统计分布决定的。代码语料出 <code>def</code>、<code>();</code>，中文博客语料出“模型”、“训练”——<strong>词表是语料的指纹</strong>，换语料就要重新训词表。</li>
<li><strong>频率决定谁先成为原子。</strong> BPE 每轮只合并最高频的相邻对，它只认频率，不认语义重要性。“大模型”再重要，出现 31 次也得往后排。</li>
<li><strong>预算决定多低频才配拥有姓名。</strong> 词表大小（合并次数）就是预算：高频组合早早拿到独立 token_id，低频组合要等预算够了才有机会。512 次时“大模型”没姓名，2000 次才有了。</li>
</ol>
<h2 id="工程收获">工程收获</h2>
<ul>
<li>训练产物（vocab/merges）用 JSON 存成中间文件（<code>save_bpe</code> / <code>load_bpe</code>），实验和训练可以分开重跑</li>
<li>不同配置分目录存（<code>data/blog/m512/</code>、<code>m2000/</code>……），不然实验产物互相覆盖，回头根本对不上号</li>
<li>路径用 <code>Path(__file__).resolve().parents[N]</code> 锚定，不再受 IDE 工作目录的摆布</li>
</ul>
<h2 id="欠账与下一步">欠账与下一步</h2>
<ul>
<li>Python 的 bytes/str、pathlib 这些基础用法还不熟，找时间补</li>
<li>M2 预训练：token 流打包、梯度累积、bpb 指标，这次 BPB 实验就是预习</li>
</ul>
<p><strong>一句话总结 M1：BPE 的目标不是把文本压到最短，是在压缩率、词表规模、可学习性、组合泛化之间找平衡。</strong></p>
]]></content:encoded>
    </item>
    <item>
      <title>手搓 LLM 笔记 09：diy-llm 分支，从分词器开始</title>
      <link>https://blog.lvxinrong.com/llm/llm-09-diy-llm-tokenizer/</link>
      <pubDate>Tue, 27 Oct 2026 12:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-09-diy-llm-tokenizer/</guid>
      <description>手搓 LLM 开启第二条分支：从零构建大模型（已完结）之外，新增 diy-llm 理论线。M1 自训 BPE，先把分词器这张图看懂。</description>
      <content:encoded><![CDATA[<blockquote>
<p>手搓 LLM 系列从这篇起有两个分支：<strong>从零构建大模型</strong>（00-08，书已读完、代码已重写）和 <strong>diy-llm</strong>（CS336 中文本土化课程，理论线）。diy-llm 与 nanochat 作品线双轨并行，配套关系见 <a href="https://blog.lvxinrong.com/nanochat/lvchat-roadmap/">lvchat 实现导图</a> 和 <a href="https://blog.lvxinrong.com/diy-llm/diy-llm-proposal/">diy-llm 立项书</a>。</p>
</blockquote>
<h2 id="m1-的目标">M1 的目标</h2>
<p>在 squid-llms 里自己训一个 BPE 分词器，不调用 tiktoken。验收两条：</p>
<ol>
<li>编解码往返无损（encode 再 decode，原文一字不差）</li>
<li>压缩率可测（同样的语料，我的词表 vs GPT-2 词表，谁更省 token）</li>
</ol>
<p>配套阅读：diy-llm 第 2 章（分词器）；标准答案：nanochat 的 tokenizer.py。</p>
<h2 id="一张图看懂分词器的位置">一张图看懂分词器的位置</h2>
<p><img alt="分词器全链路：编码、模型、解码、自回归循环" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-tokenizer-loop.png"></p>
<p>这张图值得收藏，因为它把分词器放回了完整链路里，而不是孤立地讲算法：</p>
<ul>
<li><strong>编码 = 切分 + 查表映射</strong>。文本先切成片段，再按词表（Vocabulary）映成 ID 序列。“我喜欢学习AI。” → <code>[15, 290, 3321, 56, 2]</code>。</li>
<li><strong>模型只看得懂数字 ID 序列</strong>。它不认识“我”也不认识“AI”，它看到的永远是 ID。词表就是人和模型之间的翻译官。</li>
<li><strong>模型产出的是下一个 Token ID</strong>，比如 <code>[109]</code>。</li>
<li><strong>解码 = ID 还原 + 文本重组</strong>。109 查表还原成“很”，再拼回文本。</li>
<li><strong>循环：新字加入输入，继续生成</strong>。这就是自回归——每一步都把上一步的输出接回输入。</li>
</ul>
<p>这一过程不仅是简单的编码转换，更隐含了模型对“世界的切分”策略：模型最终“看到”的基本单位，可能是字符、词，或更常见的子词片段。</p>
<p>分词器还是一个<strong>独立优化的模块</strong>：通常先在大规模文本上训练出词表，再固定下来供模型使用。它本身不让模型“理解”语言——理解能力是在后续 Transformer 训练中获得的——但更合理的切分会让输入分布更清晰一致，间接提升训练效率和最终性能。</p>
<h2 id="token-数量注意力的两笔账">token 数量：注意力的两笔账</h2>
<p><img alt="token 数量与注意力计算复杂度、上下文窗口占用的关系" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-token-count-attention.png"></p>
<p>token 划分越碎（同样的文本 token 数越多），代价体现在两处：</p>
<ul>
<li><strong>计算复杂度呈平方增长</strong>：注意力要算 N×N 的分数矩阵，token 数翻倍，注意力计算量变四倍；</li>
<li><strong>上下文窗口被更快吃满</strong>：窗口容量是固定的 token 数，切得碎，同样长度能装下的实际内容就变少。</li>
</ul>
<p>这就是为什么“切得碎和太宽泛都不行”——碎有碎的代价，粗有粗的代价（词表爆炸、长尾稀疏、OOV）。</p>
<h2 id="分词粒度字符字节词bpe">分词粒度：字符、字节、词、BPE</h2>
<p><img alt="4 种分词器对比：粒度、词表大小、OOV、序列长度" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-tokenizer-types.png"></p>
<p>最直观的是<strong>字符分词</strong>：把文本拆成最小字符单位（英文的字母、中文的单字）。</p>
<ul>
<li>优点：词表极小（英文 26 个字母+符号，中文几千常用字）；没有 OOV，任何生僻词都由基础字符组成。</li>
<li>缺点：序列过长，一句话长度翻数倍，吃上下文窗口和算力；单字符通常没有独立语义，模型要花更深的层数把意义拼回来。</li>
</ul>
<p><strong>BPE 的核心思想就是在字符级（太细）和词级（太粗）之间找平衡</strong>：统计语料中相邻字符对的出现频率，迭代地把最频繁的对合并成一个新 token。高频组合自动成词，低频组合保持拆开——数据驱动的压缩。</p>
<h2 id="算法家族速记">算法家族速记</h2>
<ul>
<li><strong>BPE</strong>：统计相邻对频次，每次贪心合并最高频的一对，迭代构建词表。决策只看频率统计。</li>
<li><strong>WordPiece</strong>：看合并对语言模型似然的贡献，选能显著提升语料拟合度的合并。</li>
<li><strong>Unigram</strong>：反过来，从一个过大的种子词表出发做减法，初始化每个 token 的概率再逐步裁剪。</li>
<li><strong>SentencePiece</strong>：不是某个具体算法，而是语言无关的子词分词框架，提供统一的训练与编码流程，BPE 和 Unigram 都能装在里面跑。</li>
</ul>
<h2 id="我想明白的三个点">我想明白的三个点</h2>
<p><strong>一、为什么自己训 BPE，而不是继续用 tiktoken。</strong> GPT-2 的词表是为英文互联网配的，喂中文和代码会把很多字拆成多个 byte token，同样的文本 token 数翻倍，训练和推理都更贵。自己训，词表才为自己的语料服务，词表大小也成为自己的旋钮。</p>
<p><strong>二、训练的产物不是模型，是一张合并表。</strong> BPE 训练就是反复统计“哪对相邻符号最频繁”，合并成新 token，记下轮次。有了合并表（mergeable_ranks），编码就是查表合并。nanochat 的做法印证了这个本质：训练用 rustbpe，推理把合并表交给 tiktoken——训练和编码是两个独立问题。</p>
<p><strong>三、编码最容易写错的地方是“按 rank 顺序应用合并”。</strong> 后学的合并依赖先学的，顺序乱了，编解码往返就无损不了。M1 的坑大概率会踩在这。</p>
<h2 id="第二章读完的四条心得">第二章读完的四条心得</h2>
<ol>
<li><strong>算法和框架是两个层次。</strong> BPE / WordPiece / Unigram 是算法，SentencePiece / tiktoken 是框架。DeepSeek 用的是 BBPE（字节级 BPE），属于 BPE 家族；“用 SentencePiece”推不出“不是 BPE”，因为框架里可以跑 BPE。</li>
<li><strong>BPE 是频次统计，不是概率模型。</strong> 只看“哪对相邻符号最频繁”。引入概率的是 WordPiece（似然贡献）和 Unigram（token 概率）。</li>
<li><strong>分词的好坏是压缩率与一致性的 trade-off。</strong> 过粗：词表爆炸、长尾稀疏、OOV 变多；过细：序列变长、语义被肢解。</li>
<li><strong>分词间接影响最终训练效果，路径有三条</strong>：序列长度（注意力 N²）、词表大小（输出层 emb_dim × vocab_size 的参数与计算）、语义完整性（太细则模型要花更多层把意义拼回来）。</li>
</ol>
<h2 id="接下来的动作">接下来的动作</h2>
<p>在 <code>squid/tokenizer/bpe.py</code> 里实现三个函数：<code>train_bpe</code>（学合并表）、<code>encode</code>（按表切分）、<code>decode</code>（还原）。先小语料跑通往返，再上博客语料测压缩率。顺手做个小实验：同一句“我喜欢学习AI”，两种词表各 encode 一次，对比切出来的片段谁更贴近语义单元。</p>
]]></content:encoded>
    </item>
    <item>
      <title>diy-llm 立项书：给作品线配一条理论线</title>
      <link>https://blog.lvxinrong.com/llm/diy-llm-proposal/</link>
      <pubDate>Sat, 24 Oct 2026 12:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/diy-llm-proposal/</guid>
      <description>diy-llm 与 nanochat 并列：作品线负责做得出，理论线负责懂得深。按里程碑配章节，不顺序通读，选修区按需捡回。</description>
      <content:encoded><![CDATA[<blockquote>
<p>这不是课程笔记的开篇，是立项书。项目名：diy-llm 理论线走读。发起人、执行人：我自己。评委：Kimi。</p>
</blockquote>
<h2 id="为什么并列而不是顺序">为什么并列，而不是顺序</h2>
<p>《从零构建大模型》给了我 GPT-2 时代的零件，nanochat（lvchat）要把零件装成整车。但装车的过程中一定会碰到书里没有的现代地形：RoPE、RMSNorm、KV cache、数据配比、Scaling Laws、DPO/GRPO。</p>
<p><a href="https://github.com/datawhalechina/diy-llm">diy-llm</a>（DataWhale 出品，斯坦福 CS336 中文本土化课程，16 章 + 6 作业）恰好覆盖这些地形。昨晚粗翻了一遍，不少概念已经有了初步认识——说明它不是另起炉灶的新课，而是已有知识的深水区。</p>
<p>所以定位并列：<strong>nanochat 是作品线，diy-llm 是理论线。作品线负责“做得出”，理论线负责“懂得深”。</strong></p>
<h2 id="用法按里程碑配章节不顺序通读">用法：按里程碑配章节，不顺序通读</h2>
<p>每个 lvchat 里程碑开工前，先读对应章节；交付后可选做对应作业当加试题。映射表维护在 <a href="https://blog.lvxinrong.com/nanochat/lvchat-roadmap/">lvchat 实现导图</a>。</p>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>lvchat 里程碑</th>
        <th>diy-llm 配套章节</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>M1 自训 BPE tokenizer</td>
        <td>第 2 章 分词器 + 作业 1</td>
      </tr>
      <tr>
        <td>M2 预训练</td>
        <td>第 3 章 + 第 11 章 + 第 9 章</td>
      </tr>
      <tr>
        <td>M3 KV cache 推理引擎</td>
        <td>第 10 章 推理</td>
      </tr>
      <tr>
        <td>M4 SFT</td>
        <td>第 13 章 训练流程</td>
      </tr>
      <tr>
        <td>M5 评估框架</td>
        <td>第 12 章 评估与基准 + 作业 6</td>
      </tr>
      <tr>
        <td>架构升级（GPT-2 → 现代）</td>
        <td>第 4 章 RoPE/RMSNorm/SwiGLU</td>
      </tr>
    </tbody>
  </table>
</div>
<p>选修区（roadmap 里“明确砍掉”的装备，想捡回来时的教材）：MoE（第 5 章）、GPU/分布式（第 6-8 章）、GRPO/RLVR（第 14 章）、多模态（第 15 章）。</p>
<h2 id="验收">验收</h2>
<p>理论线的验收不是“读完了”，是“用上了”：某个里程碑的实现决策能说出依据，或者某章的作业真的跑通。读不动的时候优先保作品线，理论线可以等。</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
