<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Tokenizer on 吕炘嵘的博客</title>
    <link>https://blog.lvxinrong.com/tags/tokenizer/</link>
    <description>Recent content in Tokenizer on 吕炘嵘的博客</description>
    <image>
      <title>吕炘嵘的博客</title>
      <url>https://blog.lvxinrong.com/og-default.jpg</url>
      
      <link>https://blog.lvxinrong.com/tags/tokenizer/</link>
    </image>
    <generator>Hugo</generator>
    <language>zh-CN</language>
    <lastBuildDate>Tue, 27 Oct 2026 18:00:00 +0800</lastBuildDate>
    <atom:link href="https://blog.lvxinrong.com/tags/tokenizer/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>手搓 LLM 笔记 10：手写 BPE，从写不出来到甜点区</title>
      <link>https://blog.lvxinrong.com/llm/llm-10-handwritten-bpe/</link>
      <pubDate>Tue, 27 Oct 2026 18:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-10-handwritten-bpe/</guid>
      <description>纯 Python 手写 BPE 训练与编解码，封装成 SquidTokenizer；再用压缩率曲线和小模型 BPB 实验，找到博客语料的甜点区 4000 次合并。</description>
      <content:encoded><![CDATA[<p>上一章遗留的作业：在 <code>squid/tokenizer/bpe.py</code> 里实现三个函数：<code>train_bpe</code>（学合并表）、<code>encode</code>（按表切分）、<code>decode</code>（还原）。先小语料跑通往返，再上博客语料测压缩率。</p>
<p>说实话，开工前我觉得自己写不出来。但拆成小函数之后，一个一个都拿下了。</p>
<h2 id="三个零件">三个零件</h2>
<p><strong>count_pairs</strong>：用滑动窗口把 i 和 i+1 的 id 配成 key，统计每个 key 出现的次数。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">count_pairs</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">],</span> <span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">counts</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">ids</span><span class="p">[</span><span class="mi">1</span><span class="p">:]):</span>
</span></span><span class="line"><span class="cl">        <span class="n">key</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">counts</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">counts</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">counts</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">counts</span>
</span></span></code></pre></div><p><strong>merge</strong>：把 ids 里所有出现过的 pair 替换成新 id。比如 pair=(1,2)、ids=[1,2,1,2,3,4,5]、new_id=9，结果就是 <code>[9, 9, 3, 4, 5]</code>。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">],</span> <span class="n">pair</span><span class="p">:</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">],</span> <span class="n">new_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">new_idx</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="ow">and</span> <span class="p">(</span><span class="n">ids</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">ids</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">])</span> <span class="o">==</span> <span class="n">pair</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">new_idx</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">i</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">2</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">new_idx</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ids</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">            <span class="n">i</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">new_idx</span>
</span></span></code></pre></div><p>这里有个容易写错的点：统计是滑动窗口（允许重叠），替换是贪心扫描（命中消费两格，不匹配消费一格）。所以用 while 手动控制步长，for i in range 里修改 i 下一轮会被覆盖。</p>
<p><strong>train_bpe</strong>：vocab 初始 256 项（所有单字节），每轮取 count_pairs 里 value 最大的 key 合并，新 id 从 256 开始累加。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">train_bpe</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">num_merges</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">vocab</span> <span class="o">=</span> <span class="p">{</span><span class="n">i</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">([</span><span class="n">i</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">256</span><span class="p">)}</span>
</span></span><span class="line"><span class="cl">    <span class="n">merges</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">num_merges</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">counts</span> <span class="o">=</span> <span class="n">count_pairs</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">counts</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">break</span>
</span></span><span class="line"><span class="cl">        <span class="n">best</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">counts</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">counts</span><span class="o">.</span><span class="n">get</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">new_id</span> <span class="o">=</span> <span class="mi">256</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">ids</span> <span class="o">=</span> <span class="n">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">best</span><span class="p">,</span> <span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">merges</span><span class="p">[</span><span class="n">best</span><span class="p">]</span> <span class="o">=</span> <span class="n">new_id</span>
</span></span><span class="line"><span class="cl">        <span class="n">vocab</span><span class="p">[</span><span class="n">new_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">vocab</span><span class="p">[</span><span class="n">best</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">+</span> <span class="n">vocab</span><span class="p">[</span><span class="n">best</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span>
</span></span></code></pre></div><p>注意最后一行：best 里装的是 id 不是字符，要先查 vocab 取出 bytes 再拼接，不能直接 <code>best[0] + best[1]</code>（那是两个整数相加）。</p>
<h2 id="encode--decode-与往返无损">encode / decode 与往返无损</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="n">ids</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">],</span> <span class="n">vocab</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="sa">b</span><span class="s2">&#34;&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">vocab</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">ids</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">merges</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">pair</span><span class="p">,</span> <span class="n">new_id</span> <span class="ow">in</span> <span class="n">merges</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="n">ids</span> <span class="o">=</span> <span class="n">merge</span><span class="p">(</span><span class="n">ids</span><span class="p">,</span> <span class="n">pair</span><span class="p">,</span> <span class="n">new_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">ids</span>
</span></span></code></pre></div><p>decode 是查词表把 ids 还原成文本，必须先在 bytes 世界拼完整再 decode，不然中文多字节字符会被切碎。encode 是把训练学到的合并规则<strong>按学习顺序</strong>在新文本上重放一遍——merges 存的是规则不是结果，不重放，新 token 不会出现。</p>
<p>往返验收过了：<code>decode(encode(text, merges), vocab) == text</code>。</p>
<p>Python 里 bytes 和 str 的转换我目前还不熟，记入欠账，后面找机会补。</p>
<h2 id="封装成-squidtokenizer">封装成 SquidTokenizer</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyBPETokenizer</span><span class="p">(</span><span class="n">SquidTokenizer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span><span class="p">,</span> <span class="n">eos_id</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span> <span class="o">=</span> <span class="n">vocab</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">merges</span> <span class="o">=</span> <span class="n">merges</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span> <span class="o">=</span> <span class="n">eos_id</span> <span class="k">if</span> <span class="n">eos_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="nb">len</span><span class="p">(</span><span class="n">vocab</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span><span class="p">,</span> <span class="sa">b</span><span class="s2">&#34;&lt;|endoftext|&gt;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">bpe</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">idx</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">bpe</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">vocab</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nd">@classmethod</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">from_directory</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span> <span class="o">=</span> <span class="n">bpe</span><span class="o">.</span><span class="n">load_bpe</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="n">vocab</span><span class="p">,</span> <span class="n">merges</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nd">@property</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">eos_id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_eos_id</span>
</span></span></code></pre></div><p>这段学到的 Python 点：</p>
<ol>
<li>三元表达式是 <code>值A if 条件 else 值B</code>，所以 eos_id 那行就是“传了用传入的，没传用词表大小”</li>
<li><code>@classmethod</code> 是挂在类上的方法，类似 Java 静态工厂。<code>cls(vocab, merges)</code> 就是调用 <code>__init__</code>，但比 Java static 多一层：cls 是类本身，子类调用返回子类实例</li>
<li><code>@property</code> 把方法伪装成属性，访问不用加括号</li>
<li><code>self._*</code> 下划线只是约定，不是 private，依然能访问</li>
</ol>
<h2 id="实验一压缩率曲线与甜点区">实验一：压缩率曲线与甜点区</h2>
<p>分别训练了 [64, 128, 256, 512, 1024, 2000, 4000, 8000, 16000] 次的词表，在 held-out 文本上测压缩率（<code>len(text.encode(&quot;utf-8&quot;)) / len(tokenizer.encode(text))</code>，即 bytes/token，越高越省）：</p>
<pre tabindex="0"><code>merges=   64  tokens=10588  bytes/token=1.29
merges=  128  tokens= 9631  bytes/token=1.42
merges=  256  tokens= 8346  bytes/token=1.64
merges=  512  tokens= 7009  bytes/token=1.95
merges= 1024  tokens= 5944  bytes/token=2.30
merges= 2000  tokens= 5043  bytes/token=2.71
merges= 4000  tokens= 4287  bytes/token=3.19
merges= 8000  tokens= 3869  bytes/token=3.54
merges=16000  tokens= 3704  bytes/token=3.69
</code></pre><p>token 数一直在降，bytes/token 一直在涨，但<strong>看每次翻倍的收益</strong>：前期稳定 +17% 左右，4000→8000 掉到 +11%，8000→16000 只剩 +4%。<strong>甜点区在 4000-8000</strong>，再往后基本是在背语料。</p>
<p>直观例子：手机号最好学成 3-4-4 的片段，merges 一直增大就会学成整个 11 位号码——每个号码独一无二，embedding 永远学不到，模型也无法对号码内部做推理。所以并不是 merges 越多越好，每个 token 携带的信息越大，模型训练时也需要更强的能力去掌握它。</p>
<p>另一个好玩的发现：512 次合并时“大模型”还排不上号（语料里只出现 31 次），2000 次之后它终于“拥有姓名”，被切成了一个完整 token。<strong>词表大小决定了多低频的词才配拥有姓名。</strong></p>
<h2 id="实验二bpb把模型拉上一起评">实验二：BPB，把模型拉上一起评</h2>
<p>和 DeepSeek 讨论后确认：评估分词器不能只看 bytes/token，那只是在评分词器本身，还得看“模型+词表”的整体效果。这个指标叫 <strong>BPB（Bits Per Byte）</strong>：</p>
<pre tabindex="0"><code>BPB = (val_loss / ln2) ÷ bytes/token
</code></pre><p>含义：<strong>给这段文本编码，平均每个字节要花多少比特</strong>。val_loss 是模型每预测一个 token 的代价（nats），除以 ln2 换成比特；再除以 bytes/token，就把“每个 token 的成本”折算成“每个字节的成本”。<strong>越低越好</strong>——它同时惩罚两种情况：词表太碎（bytes/token 小）和模型没学好（val_loss 高）。</p>
<h3 id="实验设计">实验设计</h3>
<p>四组：512 / 2000 / 4000 / 8000，同一个微型模型配置（emb_dim=64、n_layers=2、context=128），同一份 train/held-out 文本，只换分词器。</p>
<p><strong>一个关键的控制变量：统一步数，不是统一 epoch。</strong> 因为词表越大，同样文本的 ids 越短，定长窗口切出的窗口数越少，每 epoch 的训练步数也越少：</p>
<pre tabindex="0"><code>窗口数 = (len(ids) - max_length) / stride
每 epoch 步数 = 窗口数 / batch_size
</code></pre><p>如果只统一 epoch，大词表模型等于“饿着肚子考试”。所以给 <code>pretrain_model</code> 加了 <code>max_steps</code> 参数，四组都固定走 500 步。</p>
<p>完整实验代码：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">math</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.data.pretrain</span> <span class="kn">import</span> <span class="n">create_pretrain_loader</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.device</span> <span class="kn">import</span> <span class="n">get_device</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.model.gpt</span> <span class="kn">import</span> <span class="n">GPTModel</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.tokenizer.my_bpe_tokenizer</span> <span class="kn">import</span> <span class="n">MyBPETokenizer</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.train.common</span> <span class="kn">import</span> <span class="n">calc_loss_loader</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.train.pretrain</span> <span class="kn">import</span> <span class="n">pretrain_model</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">ROOT</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="vm">__file__</span><span class="p">)</span><span class="o">.</span><span class="n">resolve</span><span class="p">()</span><span class="o">.</span><span class="n">parents</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">DATA</span> <span class="o">=</span> <span class="n">ROOT</span> <span class="o">/</span> <span class="s2">&#34;data&#34;</span> <span class="o">/</span> <span class="s2">&#34;blog&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">train_model_with_nums_merge</span><span class="p">(</span><span class="n">num_merges</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">vocab_data</span> <span class="o">=</span> <span class="n">ROOT</span> <span class="o">/</span> <span class="s2">&#34;data&#34;</span> <span class="o">/</span> <span class="s2">&#34;blog&#34;</span> <span class="o">/</span> <span class="n">num_merges</span>
</span></span><span class="line"><span class="cl">    <span class="n">tokenizer</span> <span class="o">=</span> <span class="n">MyBPETokenizer</span><span class="o">.</span><span class="n">from_directory</span><span class="p">(</span><span class="n">vocab_data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">cfg</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;vocab_size&#34;</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">vocab</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;context_length&#34;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;emb_dim&#34;</span><span class="p">:</span> <span class="mi">64</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;n_heads&#34;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;n_layers&#34;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;drop_rate&#34;</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;qkv_bias&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="n">device</span> <span class="o">=</span> <span class="n">get_device</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span> <span class="o">=</span> <span class="n">GPTModel</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">train_text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;train.txt&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;heldout-手搓LLM笔记07-指令微调.txt&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_text</span> <span class="o">=</span> <span class="n">val_text</span> <span class="o">+</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span> <span class="o">+</span> <span class="nb">open</span><span class="p">(</span><span class="n">DATA</span> <span class="o">/</span> <span class="s2">&#34;heldout-我想把家里的设备组成一个私人AI实验室.txt&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                      <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">train_loader</span> <span class="o">=</span> <span class="n">create_pretrain_loader</span><span class="p">(</span><span class="n">train_text</span><span class="p">,</span> <span class="n">tokenizer</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                          <span class="n">batch_size</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_loader</span> <span class="o">=</span> <span class="n">create_pretrain_loader</span><span class="p">(</span><span class="n">val_text</span><span class="p">,</span> <span class="n">tokenizer</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span> <span class="mi">128</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                                        <span class="n">batch_size</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">drop_last</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">AdamW</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_losses</span><span class="p">,</span> <span class="n">val_losses</span><span class="p">,</span> <span class="n">tokens_seen</span> <span class="o">=</span> <span class="n">pretrain_model</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">model</span><span class="o">=</span><span class="n">model</span><span class="p">,</span> <span class="n">train_loader</span><span class="o">=</span><span class="n">train_loader</span><span class="p">,</span> <span class="n">val_loader</span><span class="o">=</span><span class="n">val_loader</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">optimizer</span><span class="o">=</span><span class="n">optimizer</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">num_epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">max_steps</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">eval_freq</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">eval_iter</span><span class="o">=</span><span class="mi">100</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">val_loss</span> <span class="o">=</span> <span class="n">calc_loss_loader</span><span class="p">(</span><span class="n">val_loader</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">density</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">val_text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">val_text</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">bpb</span> <span class="o">=</span> <span class="n">val_loss</span> <span class="o">/</span> <span class="n">math</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">density</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;val_loss=</span><span class="si">{</span><span class="n">val_loss</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">  bytes/token=</span><span class="si">{</span><span class="n">density</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">  BPB=</span><span class="si">{</span><span class="n">bpb</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m512&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m2000&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m4000&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">train_model_with_nums_merge</span><span class="p">(</span><span class="s2">&#34;m8000&#34;</span><span class="p">)</span>
</span></span></code></pre></div><h3 id="结果">结果</h3>
<pre tabindex="0"><code>m512 : val_loss=4.819  bytes/token=1.99  BPB=3.494
m2000: val_loss=6.551  bytes/token=2.80  BPB=3.373
m4000: val_loss=7.584  bytes/token=3.30  BPB=3.315
m8000: val_loss=8.378  bytes/token=3.66  BPB=3.300
</code></pre><p><strong>为什么 val_loss 一路涨？</strong> val_loss 是“每个 token 的预测难度”，词表变大之后，大量 token 在 117KB 语料里只出现几次，embedding 学不充分，预测它们接近靠蒙，困惑度自然上升。这不是词表变差，是“大 token 更难预测”的固有代价。</p>
<p><strong>但 BPB 在降</strong>——因为每个 token 装的字节更多了，折算回每字节成本反而便宜。而且看边际收益：512→2000 降 0.121，2000→4000 降 0.058，4000→8000 只降 0.015。<strong>BPB 的甜点同样在 4000 附近</strong>，和压缩率实验互相印证。</p>
<p>结论：<strong>val_loss 上升不等于词表不好，BPB 才是整体账；但 BPB 的边际收益也会递减，4000 之后再合并基本没红利。</strong></p>
<h2 id="bpe-的三个哲学点">BPE 的三个哲学点</h2>
<p>实验做完再回头看，BPE 的底层逻辑其实就三句话：</p>
<ol>
<li><strong>原子由数据决定。</strong> 什么是“思考的最小单位”，不是人预先规定的，是语料的统计分布决定的。代码语料出 <code>def</code>、<code>();</code>，中文博客语料出“模型”、“训练”——<strong>词表是语料的指纹</strong>，换语料就要重新训词表。</li>
<li><strong>频率决定谁先成为原子。</strong> BPE 每轮只合并最高频的相邻对，它只认频率，不认语义重要性。“大模型”再重要，出现 31 次也得往后排。</li>
<li><strong>预算决定多低频才配拥有姓名。</strong> 词表大小（合并次数）就是预算：高频组合早早拿到独立 token_id，低频组合要等预算够了才有机会。512 次时“大模型”没姓名，2000 次才有了。</li>
</ol>
<h2 id="工程收获">工程收获</h2>
<ul>
<li>训练产物（vocab/merges）用 JSON 存成中间文件（<code>save_bpe</code> / <code>load_bpe</code>），实验和训练可以分开重跑</li>
<li>不同配置分目录存（<code>data/blog/m512/</code>、<code>m2000/</code>……），不然实验产物互相覆盖，回头根本对不上号</li>
<li>路径用 <code>Path(__file__).resolve().parents[N]</code> 锚定，不再受 IDE 工作目录的摆布</li>
</ul>
<h2 id="欠账与下一步">欠账与下一步</h2>
<ul>
<li>Python 的 bytes/str、pathlib 这些基础用法还不熟，找时间补</li>
<li>M2 预训练：token 流打包、梯度累积、bpb 指标，这次 BPB 实验就是预习</li>
</ul>
<p><strong>一句话总结 M1：BPE 的目标不是把文本压到最短，是在压缩率、词表规模、可学习性、组合泛化之间找平衡。</strong></p>
]]></content:encoded>
    </item>
    <item>
      <title>手搓 LLM 笔记 09：diy-llm 分支，从分词器开始</title>
      <link>https://blog.lvxinrong.com/llm/llm-09-diy-llm-tokenizer/</link>
      <pubDate>Tue, 27 Oct 2026 12:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-09-diy-llm-tokenizer/</guid>
      <description>手搓 LLM 开启第二条分支：从零构建大模型（已完结）之外，新增 diy-llm 理论线。M1 自训 BPE，先把分词器这张图看懂。</description>
      <content:encoded><![CDATA[<blockquote>
<p>手搓 LLM 系列从这篇起有两个分支：<strong>从零构建大模型</strong>（00-08，书已读完、代码已重写）和 <strong>diy-llm</strong>（CS336 中文本土化课程，理论线）。diy-llm 与 nanochat 作品线双轨并行，配套关系见 <a href="https://blog.lvxinrong.com/nanochat/lvchat-roadmap/">lvchat 实现导图</a> 和 <a href="https://blog.lvxinrong.com/diy-llm/diy-llm-proposal/">diy-llm 立项书</a>。</p>
</blockquote>
<h2 id="m1-的目标">M1 的目标</h2>
<p>在 squid-llms 里自己训一个 BPE 分词器，不调用 tiktoken。验收两条：</p>
<ol>
<li>编解码往返无损（encode 再 decode，原文一字不差）</li>
<li>压缩率可测（同样的语料，我的词表 vs GPT-2 词表，谁更省 token）</li>
</ol>
<p>配套阅读：diy-llm 第 2 章（分词器）；标准答案：nanochat 的 tokenizer.py。</p>
<h2 id="一张图看懂分词器的位置">一张图看懂分词器的位置</h2>
<p><img alt="分词器全链路：编码、模型、解码、自回归循环" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-tokenizer-loop.png"></p>
<p>这张图值得收藏，因为它把分词器放回了完整链路里，而不是孤立地讲算法：</p>
<ul>
<li><strong>编码 = 切分 + 查表映射</strong>。文本先切成片段，再按词表（Vocabulary）映成 ID 序列。“我喜欢学习AI。” → <code>[15, 290, 3321, 56, 2]</code>。</li>
<li><strong>模型只看得懂数字 ID 序列</strong>。它不认识“我”也不认识“AI”，它看到的永远是 ID。词表就是人和模型之间的翻译官。</li>
<li><strong>模型产出的是下一个 Token ID</strong>，比如 <code>[109]</code>。</li>
<li><strong>解码 = ID 还原 + 文本重组</strong>。109 查表还原成“很”，再拼回文本。</li>
<li><strong>循环：新字加入输入，继续生成</strong>。这就是自回归——每一步都把上一步的输出接回输入。</li>
</ul>
<p>这一过程不仅是简单的编码转换，更隐含了模型对“世界的切分”策略：模型最终“看到”的基本单位，可能是字符、词，或更常见的子词片段。</p>
<p>分词器还是一个<strong>独立优化的模块</strong>：通常先在大规模文本上训练出词表，再固定下来供模型使用。它本身不让模型“理解”语言——理解能力是在后续 Transformer 训练中获得的——但更合理的切分会让输入分布更清晰一致，间接提升训练效率和最终性能。</p>
<h2 id="token-数量注意力的两笔账">token 数量：注意力的两笔账</h2>
<p><img alt="token 数量与注意力计算复杂度、上下文窗口占用的关系" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-token-count-attention.png"></p>
<p>token 划分越碎（同样的文本 token 数越多），代价体现在两处：</p>
<ul>
<li><strong>计算复杂度呈平方增长</strong>：注意力要算 N×N 的分数矩阵，token 数翻倍，注意力计算量变四倍；</li>
<li><strong>上下文窗口被更快吃满</strong>：窗口容量是固定的 token 数，切得碎，同样长度能装下的实际内容就变少。</li>
</ul>
<p>这就是为什么“切得碎和太宽泛都不行”——碎有碎的代价，粗有粗的代价（词表爆炸、长尾稀疏、OOV）。</p>
<h2 id="分词粒度字符字节词bpe">分词粒度：字符、字节、词、BPE</h2>
<p><img alt="4 种分词器对比：粒度、词表大小、OOV、序列长度" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-09-tokenizer-types.png"></p>
<p>最直观的是<strong>字符分词</strong>：把文本拆成最小字符单位（英文的字母、中文的单字）。</p>
<ul>
<li>优点：词表极小（英文 26 个字母+符号，中文几千常用字）；没有 OOV，任何生僻词都由基础字符组成。</li>
<li>缺点：序列过长，一句话长度翻数倍，吃上下文窗口和算力；单字符通常没有独立语义，模型要花更深的层数把意义拼回来。</li>
</ul>
<p><strong>BPE 的核心思想就是在字符级（太细）和词级（太粗）之间找平衡</strong>：统计语料中相邻字符对的出现频率，迭代地把最频繁的对合并成一个新 token。高频组合自动成词，低频组合保持拆开——数据驱动的压缩。</p>
<h2 id="算法家族速记">算法家族速记</h2>
<ul>
<li><strong>BPE</strong>：统计相邻对频次，每次贪心合并最高频的一对，迭代构建词表。决策只看频率统计。</li>
<li><strong>WordPiece</strong>：看合并对语言模型似然的贡献，选能显著提升语料拟合度的合并。</li>
<li><strong>Unigram</strong>：反过来，从一个过大的种子词表出发做减法，初始化每个 token 的概率再逐步裁剪。</li>
<li><strong>SentencePiece</strong>：不是某个具体算法，而是语言无关的子词分词框架，提供统一的训练与编码流程，BPE 和 Unigram 都能装在里面跑。</li>
</ul>
<h2 id="我想明白的三个点">我想明白的三个点</h2>
<p><strong>一、为什么自己训 BPE，而不是继续用 tiktoken。</strong> GPT-2 的词表是为英文互联网配的，喂中文和代码会把很多字拆成多个 byte token，同样的文本 token 数翻倍，训练和推理都更贵。自己训，词表才为自己的语料服务，词表大小也成为自己的旋钮。</p>
<p><strong>二、训练的产物不是模型，是一张合并表。</strong> BPE 训练就是反复统计“哪对相邻符号最频繁”，合并成新 token，记下轮次。有了合并表（mergeable_ranks），编码就是查表合并。nanochat 的做法印证了这个本质：训练用 rustbpe，推理把合并表交给 tiktoken——训练和编码是两个独立问题。</p>
<p><strong>三、编码最容易写错的地方是“按 rank 顺序应用合并”。</strong> 后学的合并依赖先学的，顺序乱了，编解码往返就无损不了。M1 的坑大概率会踩在这。</p>
<h2 id="第二章读完的四条心得">第二章读完的四条心得</h2>
<ol>
<li><strong>算法和框架是两个层次。</strong> BPE / WordPiece / Unigram 是算法，SentencePiece / tiktoken 是框架。DeepSeek 用的是 BBPE（字节级 BPE），属于 BPE 家族；“用 SentencePiece”推不出“不是 BPE”，因为框架里可以跑 BPE。</li>
<li><strong>BPE 是频次统计，不是概率模型。</strong> 只看“哪对相邻符号最频繁”。引入概率的是 WordPiece（似然贡献）和 Unigram（token 概率）。</li>
<li><strong>分词的好坏是压缩率与一致性的 trade-off。</strong> 过粗：词表爆炸、长尾稀疏、OOV 变多；过细：序列变长、语义被肢解。</li>
<li><strong>分词间接影响最终训练效果，路径有三条</strong>：序列长度（注意力 N²）、词表大小（输出层 emb_dim × vocab_size 的参数与计算）、语义完整性（太细则模型要花更多层把意义拼回来）。</li>
</ol>
<h2 id="接下来的动作">接下来的动作</h2>
<p>在 <code>squid/tokenizer/bpe.py</code> 里实现三个函数：<code>train_bpe</code>（学合并表）、<code>encode</code>（按表切分）、<code>decode</code>（还原）。先小语料跑通往返，再上博客语料测压缩率。顺手做个小实验：同一句“我喜欢学习AI”，两种词表各 encode 一次，对比切出来的片段谁更贴近语义单元。</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
