<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Model_v3 on 吕炘嵘的博客</title>
    <link>https://blog.lvxinrong.com/tags/model_v3/</link>
    <description>Recent content in Model_v3 on 吕炘嵘的博客</description>
    <image>
      <title>吕炘嵘的博客</title>
      <url>https://blog.lvxinrong.com/og-default.jpg</url>
      
      <link>https://blog.lvxinrong.com/tags/model_v3/</link>
    </image>
    <generator>Hugo</generator>
    <language>zh-CN</language>
    <lastBuildDate>Fri, 06 Nov 2026 13:13:15 +0800</lastBuildDate>
    <atom:link href="https://blog.lvxinrong.com/tags/model_v3/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>手搓 LLM 笔记 17：让 model_v3 承接 Qwen2.5</title>
      <link>https://blog.lvxinrong.com/llm/llm-17-model-v3-qwen/</link>
      <pubDate>Thu, 05 Nov 2026 16:34:29 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-17-model-v3-qwen/</guid>
      <description>从 RMSNorm、SwiGLU、GQA、RoPE 与 KV Cache 整合，到 Qwen2.5 Base / Instruct、官方 logits 对齐和 46 项全量回归，完整记录 I0 的落地、排错与收官边界。</description>
      <content:encoded><![CDATA[<blockquote>
<p>今天做的不只是“把 Qwen 跑起来”。更重要的是：每换一个组件，都要知道自己改变了什么；每次模型表现异常，都要先把问题边界缩小，再决定修哪里。</p>
</blockquote>
<h2 id="今天的路线">今天的路线</h2>
<p>第一阶段留下的是 GPT-2 风格的模型。今天新建并完善独立的 <code>model_v3</code>，没有继续在旧模型里反复覆盖实现。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">RMSNorm
</span></span><span class="line"><span class="cl">→ SwiGLU FFN
</span></span><span class="line"><span class="cl">→ GQA + RoPE + KV Cache
</span></span><span class="line"><span class="cl">→ GPTModelV3 整合
</span></span><span class="line"><span class="cl">→ 加载 Qwen2.5 Base
</span></span><span class="line"><span class="cl">→ 接入 Chat Server 做多轮实验
</span></span><span class="line"><span class="cl">→ 加载 Qwen2.5 Instruct
</span></span><span class="line"><span class="cl">→ 使用 Qwen chat template
</span></span><span class="line"><span class="cl">→ 与官方参考实现做控制变量对照
</span></span></code></pre></div><p>一句话概括架构变化：<strong>归一化换 RMSNorm，FFN 换 SwiGLU，注意力换 GQA，位置信息使用 RoPE，生成接入 KV Cache，输入输出做权重绑定。</strong></p>
<h2 id="一rmsnorm不再做均值中心化">一、RMSNorm：不再做均值中心化</h2>
<p>原来的 LayerNorm 会减均值、除以标准差，再经过可学习参数缩放和平移。RMSNorm 不做均值中心化，直接按均方根控制激活值的整体幅度：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">RMSNorm</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">cfg</span> <span class="o">=</span> <span class="n">cfg</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">=</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">eps</span> <span class="o">=</span> <span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;eps&#34;</span><span class="p">,</span> <span class="mf">1e-6</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">scale</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Parameter</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">    LayerNorm 是“减均值、除标准差”，RMSNorm 是“不减均值，直接除以均方根”。
</span></span></span><span class="line"><span class="cl"><span class="s2">    LayerNorm 先把数据中心化，再控制方差；RMSNorm 不管中心点，只控制向量的整体幅度。
</span></span></span><span class="line"><span class="cl"><span class="s2">    两者都在控制激活值的整体幅度，防止经过多次计算后某些特征向量过大或过小，从而让训练更稳定。
</span></span></span><span class="line"><span class="cl"><span class="s2">    RMSNorm 主要通过等比例缩放控制幅度，LayerNorm 还会先做均值中心化。
</span></span></span><span class="line"><span class="cl"><span class="s2">    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">mean_square</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">y</span> <span class="o">=</span> <span class="n">x</span> <span class="o">*</span> <span class="n">torch</span><span class="o">.</span><span class="n">rsqrt</span><span class="p">(</span><span class="n">mean_square</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">eps</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">scale</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">y</span>
</span></span></code></pre></div><p>这里有两个需要说准确的点：</p>
<ul>
<li>RMSNorm 省略的是均值中心化，不是“随便把数值压小”；</li>
<li>输出还受 <code>eps</code> 和可学习的 <code>scale</code> 影响，不能简单理解成永远严格保证方差为 1。</li>
</ul>
<p>它仍然是在控制每个 token 的特征向量幅度，避免经过多层计算后激活值过大或过小，只是计算路径比 LayerNorm 更短。</p>
<h2 id="二swiglu真正的阀门在-silu-之后">二、SwiGLU：真正的阀门在 SiLU 之后</h2>
<p>普通 FFN 是一路升维、激活、再降维。SwiGLU 改成两路：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">SwiGLUFFN</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">ffn_bias</span> <span class="o">=</span> <span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;ffn_bias&#34;</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;ffn_dim&#34;</span><span class="p">,</span> <span class="mi">3</span> <span class="o">*</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">]))</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">gate_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">up_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">down_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">act</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">SiLU</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">    y = down_proj( SiLU(gate_proj(x)) * up_proj(x) )    gate_proj门控信号，SiLU 之后才形成实际的连续开度。
</span></span></span><span class="line"><span class="cl"><span class="s2">    up_proj 升维后的x (B, T, emb_dim) -&gt; (B, T, hidden_dim)  hidden_dim 默认是emb_dim的3倍
</span></span></span><span class="line"><span class="cl"><span class="s2">    down_proj 降维回 (B, T, hidden_dim) -&gt; (B, T, emb_dim)
</span></span></span><span class="line"><span class="cl"><span class="s2">    SwiGLUFFN 相比较之前的 FeedForward 核心是多了一个门阀矩阵。
</span></span></span><span class="line"><span class="cl"><span class="s2">    按照我的理解，这个门阀矩阵是类似一个电阻的装置，允许电流通过大小，也就是hidden_dim每个维度允许通过多少
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2">    但是核心原理和FeedForward不变，依然是在token经过attention 输出后的每个 token 的隐藏表示 做逐位置变换
</span></span></span><span class="line"><span class="cl"><span class="s2">    再更高维度对自身的隐藏表示进行编辑。    阀门也就是在维度升高之后，增加了一个可以由模型自行通过学习修改阀门流量的开关。可以决定这些信息哪些值得保留，放大或者压下去。
</span></span></span><span class="line"><span class="cl"><span class="s2">    SwiGLU FFN 在升维后，是用 gate_proj 单独生成一路门控信号，经 SiLU 变成每个 hidden 维度的连续开度，再与 up_proj 的内容逐元素相乘。这个开度可学习、可大于 1、可接近 0、也可为负，因此模型能逐维度决定信息保留、放大、压制甚至反向。
</span></span></span><span class="line"><span class="cl"><span class="s2">    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">down_proj</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">act</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">gate_proj</span><span class="p">(</span><span class="n">x</span><span class="p">))</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">up_proj</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span></code></pre></div><p>对应：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">gate_proj：生成门控信号
</span></span><span class="line"><span class="cl">up_proj：生成要保留和变换的内容
</span></span><span class="line"><span class="cl">SiLU(gate_proj)：形成连续开度
</span></span><span class="line"><span class="cl">逐元素相乘：决定每个 hidden 维度如何通过
</span></span><span class="line"><span class="cl">down_proj：降维回 emb_dim
</span></span></code></pre></div><p>我一开始把它类比成半导体或电阻，这个直觉有帮助，但必须再精确一步：<strong>真正的阀门不是单独的 <code>gate_proj</code>，而是 <code>SiLU(gate_proj(x))</code>。</strong></p>
<p><code>gate_proj</code> 只是线性变换；经过 SiLU 后，门控信号才具有非线性的连续开度。这个开度可以接近 0，也可以大于 1，因此不只是简单的“开”和“关”。</p>
<p>与第一阶段 FFN 相同的是，SwiGLU 依然对每个 token 的隐藏表示做逐位置变换；不同的是，模型现在可以分别控制高维特征中哪些信息保留、放大或压下去。</p>
<h2 id="三gqa减少-kv-头不是减少-q-头">三、GQA：减少 K/V 头，不是减少 Q 头</h2>
<p>Qwen2.5-0.5B 的配置是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Q 头数：14
</span></span><span class="line"><span class="cl">K/V 头数：2
</span></span><span class="line"><span class="cl">head_dim：64
</span></span><span class="line"><span class="cl">每 7 个 Q 头共享一组 K/V
</span></span></code></pre></div><p>因此形状从 MHA 的：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Q：   (B, 14, T, 64)
</span></span><span class="line"><span class="cl">K/V： (B, 14, T, 64)
</span></span></code></pre></div><p>变成 GQA 的：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Q：   (B, 14, T, 64)
</span></span><span class="line"><span class="cl">K/V： (B,  2, T, 64)
</span></span></code></pre></div><p>为了让 Q 和 K/V 继续做矩阵运算，需要把每个 K/V 头在对应分组内复制 7 份：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">expand_kv</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">x</span>
</span></span><span class="line"><span class="cl">    <span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span>
</span></span><span class="line"><span class="cl">    <span class="n">x</span> <span class="o">=</span> <span class="n">x</span><span class="p">[:,</span> <span class="p">:,</span> <span class="kc">None</span><span class="p">,</span> <span class="p">:,</span> <span class="p">:]</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">x</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span><span class="p">)</span>
</span></span></code></pre></div><p>这里的 <code>expand</code> 不是先合并 K/V，而是沿新增的分组维度扩展，再 reshape 回 14 个头。</p>
<p>GQA 的主要收益在推理：cache 中保存的是未扩展的 2 组 K/V，而不是复制后的 14 组。相对 14 个 K/V 头的 MHA，本模型的 K/V cache 存储约降为原来的 <strong>1/7</strong>。注意力计算时仍然要服务 14 个 Q 头，不能把它理解成训练计算量也同比例下降。</p>
<h2 id="四rope-与-kv-cache位置必须从历史长度开始">四、RoPE 与 KV Cache：位置必须从历史长度开始</h2>
<p>RoPE 在 v2 已经独立实现，今天做的是把它接入 GQA，并与 cache 的位置偏移对齐。</p>
<p>没有 cache 时，token 位置从 0 开始；已有 <code>past_len</code> 个历史 token 时，新 token 的位置必须继续往后排：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">position_ids</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">past_len</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">past_len</span> <span class="o">+</span> <span class="n">num_tokens</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">device</span><span class="o">=</span><span class="n">x</span><span class="o">.</span><span class="n">device</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><p>如果每次生成都从 0 重新计算 RoPE，新 token 就会拿到错误的位置信息。</p>
<p>cache 中保存未扩展的 K/V：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">present_kv</span> <span class="o">=</span> <span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">values</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">keys</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expand_kv</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">values</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expand_kv</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
</span></span></code></pre></div><p>这样既减少存储，又能完成当前注意力计算。</p>
<h3 id="causal-mask-的一个关键切片">causal mask 的一个关键切片</h3>
<p>使用 cache 后，query 长度和 key 长度不再总是相同，不能继续机械地切一个正方形 mask：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">q_len</span> <span class="o">=</span> <span class="n">queries</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">k_len</span> <span class="o">=</span> <span class="n">keys</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="n">q_len</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">mask_bool</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl"><span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">past_len</span> <span class="o">=</span> <span class="n">k_len</span> <span class="o">-</span> <span class="n">q_len</span>
</span></span><span class="line"><span class="cl">    <span class="n">mask_bool</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">mask</span><span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="n">past_len</span><span class="p">:</span><span class="n">past_len</span> <span class="o">+</span> <span class="n">q_len</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">:</span><span class="n">k_len</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">]</span>
</span></span></code></pre></div><p><code>q_len == 1</code> 时，当前 token 可以看到全部历史和自身，没有当前窗口里的未来 token，因此不需要 mask。</p>
<p>这不是单纯为了消除一个 shape 报错，而是在明确三种长度之间的关系：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">k_len = past_len + q_len
</span></span></code></pre></div><h2 id="五gptmodelv3把零件装成整车">五、GPTModelV3：把零件装成整车</h2>
<p>当前 v3 的主链路是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Token Embedding
</span></span><span class="line"><span class="cl">    ↓
</span></span><span class="line"><span class="cl">24 × Transformer Block
</span></span><span class="line"><span class="cl">    ├─ RMSNorm → GQA + RoPE + KV Cache → 残差
</span></span><span class="line"><span class="cl">    └─ RMSNorm → SwiGLU FFN → 残差
</span></span><span class="line"><span class="cl">    ↓
</span></span><span class="line"><span class="cl">Final RMSNorm
</span></span><span class="line"><span class="cl">    ↓
</span></span><span class="line"><span class="cl">与 Token Embedding 共享权重的输出头
</span></span></code></pre></div><p>核心配置与 Qwen2.5-0.5B 对齐：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">vocab_size = 151936
</span></span><span class="line"><span class="cl">emb_dim = 896
</span></span><span class="line"><span class="cl">ffn_dim = 4864
</span></span><span class="line"><span class="cl">n_heads = 14
</span></span><span class="line"><span class="cl">n_kv_heads = 2
</span></span><span class="line"><span class="cl">n_layers = 24
</span></span><span class="line"><span class="cl">rope_theta = 1000000.0
</span></span><span class="line"><span class="cl">eps = 1e-6
</span></span><span class="line"><span class="cl">qkv_bias = True
</span></span><span class="line"><span class="cl">ffn_bias = False
</span></span><span class="line"><span class="cl">tie_word_embeddings = True
</span></span></code></pre></div><p>工程里的 <code>context_length=1024</code> 是当前小规模验证配置，不代表 Qwen 本身的最大上下文能力。</p>
<h3 id="weight-tying-不是修改-shape">Weight Tying 不是修改 shape</h3>
<p>输入 Embedding 和输出头分别声明为 <code>nn.Embedding</code> 与 <code>nn.Linear</code>，但共享同一个 Parameter：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="bp">self</span><span class="o">.</span><span class="n">out_head</span><span class="o">.</span><span class="n">weight</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">tok_emb</span><span class="o">.</span><span class="n">weight</span>
</span></span></code></pre></div><p>两个模块仍然存在，底层权重张量却是同一份。这样既减少参数，也让 token 的输入表示和输出表示使用同一套语义空间。</p>
<h2 id="六加载-qwen25-base权重与-tokenizer-是两件事">六、加载 Qwen2.5 Base：权重与 tokenizer 是两件事</h2>
<p><code>.safetensors</code> 文件只有模型参数，不包含完整 tokenizer。当前工程分别处理：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">权重：data/model_pth/qwen/qwen2_5-0_5b.safetensors
</span></span><span class="line"><span class="cl">分词：AutoTokenizer.from_pretrained(&#34;Qwen/Qwen2.5-0.5B&#34;)
</span></span></code></pre></div><p>权重加载器负责把 Qwen 的命名映射到 v3，例如：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">model.layers.0.self_attn.q_proj.weight
</span></span><span class="line"><span class="cl">→ trf_block.0.att.W_query.weight
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">model.layers.0.input_layernorm.weight
</span></span><span class="line"><span class="cl">→ trf_block.0.norm1.scale
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">model.layers.0.mlp.gate_proj.weight
</span></span><span class="line"><span class="cl">→ trf_block.0.ffn.gate_proj.weight
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">model.norm.weight
</span></span><span class="line"><span class="cl">→ final_norm.scale
</span></span></code></pre></div><p>加载后使用严格的 <code>load_state_dict()</code> 验证键和形状，不是“文件能读出来”就算成功。</p>
<h3 id="squidtokenizer-是-protocol不是门面类">SquidTokenizer 是 Protocol，不是门面类</h3>
<p><code>QwenTokenizerAdapter</code> 对外提供统一的：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">encode()
</span></span><span class="line"><span class="cl">decode()
</span></span><span class="line"><span class="cl">eos_id
</span></span></code></pre></div><p>底层仍然是 Hugging Face tokenizer。<code>SquidTokenizer</code> 则是 <code>Protocol</code>，更接近 Java 的接口，只声明最小契约。</p>
<p>Python Protocol 支持结构化类型，所以 Qwen adapter 最初没有显式继承也能运行。但项目里的 <code>TiktokenTokenizer</code> 和 <code>MyBPETokenizer</code> 都显式继承了，为了清楚表达契约并方便 IDE 检查，最终统一为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">QwenTokenizerAdapter</span><span class="p">(</span><span class="n">SquidTokenizer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span>
</span></span></code></pre></div><p>这件事提醒我：<strong>能运行，不代表接口设计已经表达完整。</strong></p>
<h3 id="pycharm-的-from_pretrained-告警">PyCharm 的 from_pretrained 告警</h3>
<p>PyCharm 一度提示：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Function &#39;from_pretrained&#39; doesn&#39;t return anything
</span></span></code></pre></div><p>实际运行返回的是 <code>Qwen2Tokenizer</code>，并不是 <code>None</code>。当前 Transformers 版本的返回类型包含 <code>SentencePieceBackend</code>，而未安装对应可选依赖时它被设为 <code>None</code>，使 IDE 看到了容易误导的类型。</p>
<p>最后没有为了消除告警安装不需要的依赖，而是显式收窄构造结果：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">tokenizer</span> <span class="o">=</span> <span class="n">AutoTokenizer</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="n">model_name</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="n">tokenizer</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;Tokenizer 加载失败：</span><span class="si">{</span><span class="n">model_name</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="bp">self</span><span class="o">.</span><span class="n">tokenizer</span> <span class="o">=</span> <span class="n">tokenizer</span>
</span></span></code></pre></div><p>IDE 告警值得检查，但不能只看告警文字就改业务逻辑。</p>
<h3 id="base-的实际验收">Base 的实际验收</h3>
<p>目前已经真实验证：</p>
<ul>
<li>权重前向成功，logits 形状为 <code>(1, 7, 151936)</code>；</li>
<li>cache 与完整前向一致，最大差异约 <code>8.58e-06</code>；</li>
<li>英文 prompt <code>The capital of France is</code> 能生成 Paris；</li>
<li>中文 prompt <code>法国的首都是</code> 能生成合理的巴黎相关续写；</li>
<li>选定测试集共 <code>18 passed</code>。</li>
</ul>
<p>这里的 18 项只是选定回归，不写成“全量测试全部通过”。</p>
<h2 id="七base-接入-server多轮输入不等于多轮助手能力">七、Base 接入 Server：多轮输入不等于多轮助手能力</h2>
<p>Base 接入昨天的 Chat Server 后，页面已经能完成三轮中文交互。但第二轮出现了模型自己编造的：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">### Instruction:
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">### Response:
</span></span><span class="line"><span class="cl">...
</span></span></code></pre></div><p>原因是 Base 把 Alpaca 风格模板当成文章结构继续续写，而生成器只按 EOS 停止。第一处修复是在写入 history 前截断：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">responses</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">prompt</span><span class="p">):]</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="n">marker</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;&lt;|endoftext|&gt;&#34;</span><span class="p">,</span> <span class="s2">&#34;&lt;|im_end|&gt;&#34;</span><span class="p">,</span> <span class="s2">&#34;### Instruction:&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">marker</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span></code></pre></div><p>这样编造的轮次不会显示，也不会污染 history。它只是生成后的截断，并不代表生成器已经支持 stop string 提前停止。</p>
<p>Base 还出现了角色错位：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">用户：你还记得我叫什么、喜欢做什么吗？
</span></span><span class="line"><span class="cl">模型：我叫吕炘嵘，喜欢写代码，正在学习大模型。
</span></span></code></pre></div><p>正确的人称应该是“你叫吕炘嵘”。但它能说出姓名和爱好，说明历史确实进入了 prompt。</p>
<p>所以 Base 实验的结论是：</p>
<blockquote>
<p>多轮上下文链路通过，助手式角色对齐不作通过要求。这里的“记得”来自 ChatSession.history，不是模型权重形成了记忆。</p>
</blockquote>
<h2 id="八instruct结构没有变权重和对话格式变了">八、Instruct：结构没有变，权重和对话格式变了</h2>
<p>下载 <code>Qwen/Qwen2.5-0.5B-Instruct</code> 后，先核对结构，再复用原有加载器。验证结果为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">hidden_size = 896
</span></span><span class="line"><span class="cl">intermediate_size = 4864
</span></span><span class="line"><span class="cl">num_hidden_layers = 24
</span></span><span class="line"><span class="cl">num_attention_heads = 14
</span></span><span class="line"><span class="cl">num_key_value_heads = 2
</span></span><span class="line"><span class="cl">rope_theta = 1000000.0
</span></span><span class="line"><span class="cl">rms_norm_eps = 1e-6
</span></span><span class="line"><span class="cl">vocab_size = 151936
</span></span><span class="line"><span class="cl">tie_word_embeddings = True
</span></span></code></pre></div><p>全部一致，Instruct 权重在 MPS 上加载成功。这说明同代同尺寸的 Base 和 Instruct 可以复用当前 v3 结构与权重映射。</p>
<h3 id="transformers-的配置位置变化">Transformers 的配置位置变化</h3>
<p>第一次验收脚本直接读取：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">config</span><span class="o">.</span><span class="n">rope_theta</span>
</span></span></code></pre></div><p>结果得到：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">AttributeError: &#39;Qwen2Config&#39; object has no attribute &#39;rope_theta&#39;
</span></span></code></pre></div><p>这不是权重加载失败，因为程序还没执行到加载器。当前 Transformers 将 RoPE 配置放在：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">config</span><span class="o">.</span><span class="n">rope_parameters</span>
</span></span></code></pre></div><p>实际内容是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;rope_theta&#34;</span><span class="p">:</span> <span class="mf">1000000.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;rope_type&#34;</span><span class="p">:</span> <span class="s2">&#34;default&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>只调整验收脚本的读取位置，自己的 <code>MODEL_V3_DEFAULT_CONFIG</code> 不需要跟着改名。</p>
<h3 id="eos-与-pad-不再相同">EOS 与 PAD 不再相同</h3>
<p>Base 使用：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">EOS = &lt;|endoftext|&gt; = 151643
</span></span><span class="line"><span class="cl">PAD = &lt;|endoftext|&gt; = 151643
</span></span></code></pre></div><p>Instruct 使用：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">EOS = &lt;|im_end|&gt;    = 151645
</span></span><span class="line"><span class="cl">PAD = &lt;|endoftext|&gt; = 151643
</span></span></code></pre></div><p>因此不能只替换权重而继续沿用 Base 的结束符。</p>
<h2 id="九qwen-chat-template显式描述角色和轮次">九、Qwen chat template：显式描述角色和轮次</h2>
<p>Instruct 不再使用 <code>### Instruction / ### Response</code>，而是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">&lt;|im_start|&gt;system
</span></span><span class="line"><span class="cl">你是一个乐于助人的 AI 助手。&lt;|im_end|&gt;
</span></span><span class="line"><span class="cl">&lt;|im_start|&gt;user
</span></span><span class="line"><span class="cl">你好，我叫吕炘嵘。&lt;|im_end|&gt;
</span></span><span class="line"><span class="cl">&lt;|im_start|&gt;assistant
</span></span></code></pre></div><p><code>add_generation_prompt=True</code> 会在末尾补上 assistant 起始标记，模型只需要接着生成回答。</p>
<p>单轮实测输出：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">你好！很高兴见到你。有什么我可以帮助你的吗？
</span></span></code></pre></div><p>回答角色正确，遇到 <code>&lt;|im_end|&gt;</code> 正常停止，也没有编造额外轮次。</p>
<h3 id="让-chatsession-支持替换模板">让 ChatSession 支持替换模板</h3>
<p>没有复制一套 Qwen 专用 Session，而是给现有 <code>ChatSession</code> 注入 <code>prompt_builder</code>：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="bp">self</span><span class="o">.</span><span class="n">prompt_builder</span> <span class="o">=</span> <span class="n">prompt_builder</span> <span class="ow">or</span> <span class="n">build_chat_prompt</span>
</span></span></code></pre></div><p>调用时：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">prompt</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">prompt_builder</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">history</span><span class="p">,</span> <span class="n">user_input</span><span class="p">)</span>
</span></span></code></pre></div><p>不传时使用旧模板；传入 Qwen 构造器时使用新模板。这相当于一个简化的 Strategy 模式，历史保存、生成、清空逻辑仍然只有一份。</p>
<p>Qwen 的 history 则转换为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span><span class="s2">&#34;role&#34;</span><span class="p">:</span> <span class="s2">&#34;system&#34;</span><span class="p">,</span> <span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span><span class="s2">&#34;role&#34;</span><span class="p">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;用户1&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span><span class="s2">&#34;role&#34;</span><span class="p">:</span> <span class="s2">&#34;assistant&#34;</span><span class="p">,</span> <span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;助手1&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span><span class="s2">&#34;role&#34;</span><span class="p">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;当前问题&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span></code></pre></div><p>Server 用 <code>partial</code> 提前绑定 Hugging Face tokenizer，剩下的 <code>history</code> 和 <code>current_input</code> 仍由 Session 提供。</p>
<h2 id="十一个多轮对话异常">十、一个多轮对话异常</h2>
<p>model_v3 接入 Qwen2.5-0.5B-Instruct 后，单轮回答和 <code>&lt;|im_end|&gt;</code> 停止都正常，但三轮对话出现了问题：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">用户：你好，我叫吕炘嵘。
</span></span><span class="line"><span class="cl">用户：我喜欢写代码，也正在学习大模型。
</span></span><span class="line"><span class="cl">用户：你还记得我叫什么、喜欢做什么吗？
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">模型：我叫小助手，主要功能是提供信息和帮助用户解决问题……
</span></span></code></pre></div><p>这个回答没有利用前两轮的信息，但仅凭页面现象，无法判断问题出在哪里：</p>
<ul>
<li>Server 没有把 history 传进去；</li>
<li>Qwen chat template 组装错误；</li>
<li>model_v3 与官方模型实现存在差异；</li>
<li>0.5B Instruct 本身没有可靠完成这次上下文回忆。</li>
</ul>
<p>我先绕过 Server，手工构造包含两轮真实历史的 prompt，并用断言确认姓名和爱好都在输入中。模型仍然给出了同样的错误回答。</p>
<p>这一步只排除了 Server 和 history 丢失，<strong>还不能直接证明是模型能力不足</strong>。</p>
<h2 id="十一引入官方参考实现">十一、引入官方参考实现</h2>
<p>下一步使用 Hugging Face 的 <code>AutoModelForCausalLM</code>，加载同一份 Qwen2.5 Instruct 权重，输入完全相同的 prompt，再比较生成结果：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">同一份权重 + 同一段 prompt + 同一组解码参数
</span></span><span class="line"><span class="cl">                      │
</span></span><span class="line"><span class="cl">          ┌───────────┴───────────┐
</span></span><span class="line"><span class="cl">          ▼                       ▼
</span></span><span class="line"><span class="cl">     model_v3              官方 Qwen 实现
</span></span></code></pre></div><p>这里最重要的是控制变量：</p>
<ol>
<li><strong>权重相同</strong>：都用 Qwen2.5-0.5B-Instruct，不能一边 Base、一边 Instruct；</li>
<li><strong>输入相同</strong>：使用同一个 <code>build_qwen_chat_prompt()</code> 生成的字符串，包括 system、历史轮次和 assistant 起始标记；</li>
<li><strong>解码相同</strong>：都使用 greedy decoding，并统一 <code>max_new_tokens</code>、EOS 和 PAD；</li>
<li><strong>只改变模型实现</strong>：自实现 model_v3 对官方 <code>AutoModelForCausalLM</code>。</li>
</ol>
<p>结果有两种解释方向：</p>
<ul>
<li><strong>官方实现正确，model_v3 错误</strong>：优先怀疑自实现的兼容性。下一步比较最后一个 token 的 logits，再逐层定位，而不是继续调整 prompt；</li>
<li><strong>官方实现也没有通过任务</strong>：Server 和 model_v3 都有了重要的排除证据，接下来才研究提问方式或模型能力边界。</li>
</ul>
<h3 id="参考实现不是标准答案">参考实现不是标准答案</h3>
<p>官方实现也可能给出错误回答。它的价值不是替我们判定“什么回答最好”，而是提供一个<strong>相同条件下的对照组</strong>。</p>
<p>如果两边一个采样、一个 greedy，或者悄悄换了 prompt，即使输出不同也不能说明实现有误。对照实验一旦引入第二个变量，结论就不可靠。</p>
<p>这套顺序以后可以复用到其他自实现模型：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">先检查调用链
</span></span><span class="line"><span class="cl">→ 固定并检查实际输入
</span></span><span class="line"><span class="cl">→ 绕过外围服务复现
</span></span><span class="line"><span class="cl">→ 对齐官方参考实现
</span></span><span class="line"><span class="cl">→ 输出不同再比较 logits 和中间层
</span></span></code></pre></div><p>不要一上来就逐层打印张量，也不要一看到回答错误就归因为模型太小。<strong>先把问题边界缩小，再决定修哪里。</strong></p>
<h2 id="十二本次官方对照的真实结果">十二、本次官方对照的真实结果</h2>
<p>官方实现生成的是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">我是由阿里云开发的超大规模语言模型，我叫通义千问。
</span></span><span class="line"><span class="cl">我擅长回答各种问题，并且可以进行文字生成、翻译等任务……
</span></span></code></pre></div><p>它同样没有回答用户的姓名和爱好。因此，这次“第三轮没有回忆起上下文”的问题，不能再简单归因于 Server 或 model_v3。至少官方参考实现在相同输入下也没有通过这项任务。</p>
<p>但两边生成的文字并不相同：model_v3 自称“小助手”，官方实现自称“通义千问”。所以本次实验支持的是<strong>任务层面的排除结论</strong>，不是“两个实现已经逐 token 等价”。</p>
<p>这里要拆开两个验收口径：</p>
<ul>
<li><strong>任务是否通过</strong>：两边都没有回忆起用户姓名和爱好，下一步应研究提问方式与模型能力；</li>
<li><strong>实现是否完全一致</strong>：两边输出不同，尚未证明。差异可能来自浮点计算、注意力实现或自实现细节，需要比较 logits 才能继续判断，不能仅凭一段生成文本下结论。</li>
</ul>
<p>这次对照最大的价值，是避免在 Server 和 model_v3 里盲目寻找一个可能并不存在的“历史丢失 bug”，同时也没有把“输出不同”草率解释为实现已经出错。</p>
<p>下一步只改变提问方式，保持 history、system prompt 和 greedy decoding 不变：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">请根据上面的对话回答：用户叫什么名字？
</span></span><span class="line"><span class="cl">用户喜欢做什么？正在学习什么？不要介绍你自己。
</span></span></code></pre></div><p>复测后，model_v3 在测试脚本和网页链路中都能正确回答：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">用户叫吕炘嵘，用户喜欢写代码，正在学习大模型。
</span></span></code></pre></div><p>官方参考实现也给出了语义一致的正确回答：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">用户叫吕炘嵘，他喜欢写代码，并正在学习大模型。
</span></span></code></pre></div><p>测试脚本同时报告 <code>1 passed</code>。这次结果说明：在 history、权重和解码方式不变的情况下，只把“用户”这个角色和要回答的任务明确写出来，自实现与官方实现就都能完成这次上下文回忆。</p>
<p>因此，原来的错误更符合“模糊提问触发了小模型的自我介绍”，而不是 Server 丢失 history，也不能据此宣称当前模型完全不具备多轮回忆能力。反向结论同样不能扩大：一次明确提问通过，不代表所有自由提问形式都已稳定，更不代表两个实现已经逐 token 等价。</p>
<h3 id="i0-数值对齐补充2026-11-06">I0 数值对齐补充（2026-11-06）</h3>
<p>昨天只比较了生成文本，今天继续把实验往前推进一步：<strong>不调用 <code>generate()</code>，直接比较相同 input_ids 经过两套模型实现后得到的原始 logits。</strong></p>
<p>测试固定使用 CPU、float32、本地缓存权重和相同 tokenizer，避免网络、设备和数据类型引入额外变量。基础回归断言为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">torch</span><span class="o">.</span><span class="n">testing</span><span class="o">.</span><span class="n">assert_close</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">our_logits</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">official_logits</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">rtol</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">atol</span><span class="o">=</span><span class="mf">1e-4</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><p>这里 <code>rtol=0</code> 表示不额外放大数据量带来的相对容差，只要求所有元素的绝对误差不超过 <code>1e-4</code>。</p>
<h4 id="三组输入的完整前向对照">三组输入的完整前向对照</h4>
<p>在官方默认 attention 路径下，三组输入的 top-1 next token 都一致，绝对误差也都在 <code>1e-4</code> 内：</p>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>输入类型</th>
        <th style="text-align: right">token 数</th>
        <th style="text-align: right">最大绝对误差</th>
        <th style="text-align: right">平均绝对误差</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>中文自我介绍</td>
        <td style="text-align: right">8</td>
        <td style="text-align: right"><code>4.3869e-05</code></td>
        <td style="text-align: right"><code>4.8585e-06</code></td>
      </tr>
      <tr>
        <td>英文常识开头</td>
        <td style="text-align: right">5</td>
        <td style="text-align: right"><code>5.8174e-05</code></td>
        <td style="text-align: right"><code>6.8113e-06</code></td>
      </tr>
      <tr>
        <td>Python 代码片段</td>
        <td style="text-align: right">11</td>
        <td style="text-align: right"><code>7.5340e-05</code></td>
        <td style="text-align: right"><code>6.5186e-06</code></td>
      </tr>
    </tbody>
  </table>
</div>
<p>最大值与平均值相差约一个数量级并不矛盾。平均值来自整段输入的全部 logits，最大值只取其中误差最大的一个元素。它说明大多数元素非常接近，少数元素误差稍大；是否严重仍要看最大误差的绝对量级，而不是只看两者的倍数。</p>
<h4 id="124-token-多轮-prompt-暴露了-sdpa-数值差异">124-token 多轮 prompt 暴露了 SDPA 数值差异</h4>
<p>完整 chat template 包含两轮历史，共 124 个 token。使用官方默认 attention 路径时，测试没有通过 <code>atol=1e-4</code>：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">logits 总数：18,840,064
</span></span><span class="line"><span class="cl">超出阈值的元素：45,350（约 0.2%）
</span></span><span class="line"><span class="cl">最大绝对误差：0.0003409385681152344
</span></span></code></pre></div><p>这次失败不能立刻解释成模型结构错误，也不应该通过随手放宽阈值来消除。下一步只改变一个变量：将官方模型显式指定为 eager attention。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">official_model</span> <span class="o">=</span> <span class="n">AutoModelForCausalLM</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">MODEL_NAME</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">local_files_only</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">attn_implementation</span><span class="o">=</span><span class="s2">&#34;eager&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><p>改为 eager 后，同一组输入得到：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">max abs difference = 0.0
</span></span><span class="line"><span class="cl">mean abs difference = 0.0
</span></span><span class="line"><span class="cl">两边 next token = 20002（用户）
</span></span></code></pre></div><p>这说明此前的微小差异来自官方默认 SDPA 与手写 attention 的数值路径，而不是 tokenizer、权重映射或 model_v3 的结构错误。</p>
<h4 id="sdpa-与-eager-的区别">SDPA 与 eager 的区别</h4>
<p>SDPA 是 <strong>Scaled Dot-Product Attention（缩放点积注意力）</strong>，是 PyTorch 提供的统一、优化过的注意力计算接口。它与手写 attention 使用相同的数学过程：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Q × Kᵀ → 缩放 → causal mask → softmax → × V
</span></span></code></pre></div><p>区别在于具体执行路径：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">eager：逐个执行 matmul、缩放、mask、softmax、matmul
</span></span><span class="line"><span class="cl">SDPA：交给 PyTorch 统一算子，由设备和后端选择优化实现
</span></span></code></pre></div><p>数学公式相同，不代表浮点计算顺序、融合方式和舍入结果一定相同。输入越长，微小差异也可能越明显。SDPA 也不等于 FlashAttention；后者只是某些环境下可能被选择的后端技术之一。</p>
<p>因此，严格数值对齐时固定 <code>attn_implementation=&quot;eager&quot;</code> 不是为了强行让测试变绿，而是为了让官方参考实现与自实现走同一条明确的计算路径。</p>
<h4 id="为什么采样随机性不影响这次零误差">为什么采样随机性不影响这次“零误差”</h4>
<p>当前 logits 测试没有调用 <code>generate()</code>、<code>generate_text()</code>、<code>sample_next()</code> 或 <code>torch.multinomial()</code>，只做：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">固定 input_ids → 模型前向计算 → 比较原始 logits
</span></span></code></pre></div><p>temperature、top-k 和随机采样都发生在 logits 已经算出之后，不会反过来改变这次前向结果。两边又都处于 <code>eval()</code>，Dropout 不生效，因此在输入、权重、CPU、float32 和 eager 计算路径全部一致时，得到 <code>0.0</code> 是合理的。</p>
<p>即使使用 greedy decoding，也只是去掉了随机采样，不能自动消除 SDPA 的数值差异。一旦某一步 argmax 因微小误差选择了不同 token，后续自回归生成还可能继续分叉。</p>
<p>所以这项“零误差”的准确边界是：<strong>固定 124-token 多轮 prompt 下，model_v3 与官方 eager 实现的完整前向 logits 完全一致</strong>。它还不能自动覆盖 KV Cache 增量解码，因此随后单独增加了 cache 对照。</p>
<h4 id="kv-cache-增量解码对照">KV Cache 增量解码对照</h4>
<p>cache 测试先比较 prefill 后第 0 层和第 23 层的 K/V，再连续进行三次单 token decode。两边使用相同权重、CPU、float32 和 eager attention，每一步都从官方 logits 取确定性的 argmax，并把同一个 next token 喂给两个实现。</p>
<p>结果为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">prefill：max abs difference = 0.0
</span></span><span class="line"><span class="cl">cache decode step 1：max abs difference = 0.0，cache length = 8
</span></span><span class="line"><span class="cl">cache decode step 2：max abs difference = 0.0，cache length = 9
</span></span><span class="line"><span class="cl">cache decode step 3：max abs difference = 0.0，cache length = 10
</span></span><span class="line"><span class="cl">每一步两边 next token 均一致
</span></span></code></pre></div><p>这里必须区分直接证据与组合证据：</p>
<ul>
<li>prefill 的 K/V 相同，只直接证明 <code>past_len=0</code> 时的 Value 投影、Key 投影与 Key 的 RoPE 一致，不能单独证明非零 <code>past_len</code>；</li>
<li>cache length 按 8 → 9 → 10 增长，直接证明每轮长度增加 1，不等于每一步的全部拼接内容都被单独比较；</li>
<li>三个不同历史长度下，最终全量 logits 都与官方一致，为 <code>past_len → position_ids → Q/K RoPE → attention → logits</code> 提供强组合证据，但不是对每个内部变量的独立观测；</li>
<li><code>q_len=1</code> 时只有历史和当前 token，没有未来 token，因此 <code>mask_bool=None</code> 在因果定义上成立；三次 decode 与官方一致提供了行为证据，但不是直接比较 attention weights。</li>
</ul>
<p>当前 cache 对照仍不覆盖 <code>q_len&gt;1 且 past_len&gt;0</code> 的分块追加，也不覆盖 temperature、top-k 随机采样或所有可能输入。<strong>组合行为一致与每个组件被独立证明，是两件事。</strong></p>
<h4 id="固定行为回归与全量收官">固定行为回归与全量收官</h4>
<p>随后补了两组真实 model_v3 行为回归：</p>
<ul>
<li>三轮对话后，模型能正确回答“用户叫吕炘嵘，用户喜欢写代码，正在学习大模型”，且不再出现角色错位和模板标记泄漏；</li>
<li>Python 偶数筛选代码的解释完整，包含 <code>find_even_numbers</code>、执行过程与最终输出 <code>偶数列表： [2, 4, 6]</code>；<code>max_new_tokens=512</code> 覆盖了此前的长回答截断场景。</li>
</ul>
<p>最终回归分组结果：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Qwen 专项：7 passed
</span></span><span class="line"><span class="cl">基础与旧模型快速回归：24 passed
</span></span><span class="line"><span class="cl">本地模型评估：7 passed
</span></span><span class="line"><span class="cl">本地慢训练：4 passed
</span></span><span class="line"><span class="cl">DeepSeek 外部评估：4 passed
</span></span><span class="line"><span class="cl">合计：46 passed，0 failed
</span></span></code></pre></div><p>分类微调测试曾出现一次 <code>train_losses[-1] &lt; train_losses[0]</code> 失败。代码审查发现测试没有固定随机种子，并且训练/验证共用带 shuffle 的 loader，每次只抽两个 batch 计算 loss，导致评估值不完全可比。</p>
<p>修复只针对测试的可复现性：增加 <code>torch.manual_seed(123)</code>，并将 <code>eval_iter</code> 改为 <code>None</code> 以覆盖全部玩具样本。训练逻辑与 loss 下降断言均保留，修改后测试通过。没有通过删除断言或随意增加训练轮数来消除红灯。</p>
<p>因此，I0 的“全量回归通过”指当前测试套件中的 46 个固定用例全部通过，不代表任意输入、任意设备和任意生成长度都已经得到数学意义上的完全证明。</p>
<h2 id="十三训练准备只盘点语料尚未开始-midtrain">十三、训练准备：只盘点语料，尚未开始 midtrain</h2>
<p>今天还盘点了后续中文 midtrain 可能使用的语料：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">data/blog/train.txt：约 69,315 个 Qwen tokens
</span></span><span class="line"><span class="cl">私人 AI 实验室 held-out：约 1,564 tokens
</span></span><span class="line"><span class="cl">指令微调 held-out：约 3,894 tokens
</span></span></code></pre></div><p>第一轮只考虑 <code>batch_size=1</code>、<code>max_length=128</code> 的小规模冒烟，目标是验证反向传播、checkpoint 保存恢复和生成链路，不是几十步就学会个人文风。年终总结等个人文本计划后续补充，但还没有合并进训练集。</p>
<p>同时纠正了一个数据切片注释：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">stride 是相邻窗口的起始位置间隔
</span></span><span class="line"><span class="cl">重叠长度 = max_length - stride
</span></span><span class="line"><span class="cl">stride = max_length 时，窗口不重叠
</span></span></code></pre></div><p>midtrain 仍然是 next-token prediction，不需要指令模板，也不需要 SFT 的 <code>-100</code> mask。<strong>今天没有开始实际训练，也没有产生 midtrain loss 或新 checkpoint。</strong></p>
<h2 id="十四工程整理">十四、工程整理</h2>
<p>模型权重目录整理为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">data/model_pth/
</span></span><span class="line"><span class="cl">├── gpt2/
</span></span><span class="line"><span class="cl">├── qwen/
</span></span><span class="line"><span class="cl">└── experiments/
</span></span></code></pre></div><p>Qwen Base 与 Instruct 分成两个文件保存，原始 Base 没有被覆盖。正式代码入口里的旧权重路径也同步更新。</p>
<p>今天新增的主要能力包括：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">src/squid/checkpoints/qwen.py
</span></span><span class="line"><span class="cl">src/squid/tokenizer/qwen_tokenizer.py
</span></span><span class="line"><span class="cl">src/squid/model/v3_modern/
</span></span></code></pre></div><p>新增依赖主要用于读取 safetensors、访问 Hugging Face 仓库和加载 tokenizer。</p>
<h2 id="十五今天的验收边界">十五、今天的验收边界</h2>
<p>已经真实完成：</p>
<ul>
<li>RMSNorm、SwiGLU、GQA、RoPE 与 KV Cache 的 v3 整合；</li>
<li>Qwen2.5 Base 权重加载、前向、cache 一致性和中英文生成；</li>
<li>Base 与 Instruct 的 tokenizer、EOS、PAD 区分；</li>
<li>Instruct 结构核对、权重加载和单轮生成；</li>
<li>Qwen chat template 与多轮 history 组装；</li>
<li>Chat Server 的真实浏览器多轮调用，以及明确提问下的上下文回忆；</li>
<li>中文、英文、代码输入的官方 logits 容差对照，以及 124-token 多轮 prompt 在官方 eager 路径下的全量 logits 零误差对齐；</li>
<li>prefill 与连续三次单 token cache decode 的官方 eager 零误差对照；</li>
<li>固定多轮回忆、代码解释回归和当前测试套件全量 <code>46 passed</code>；</li>
<li>自实现与官方参考实现的同输入生成对照。</li>
</ul>
<p>不能写成已经完成：</p>
<ul>
<li>任意输入与 KV Cache 增量生成全程的逐 token / logits 完全一致；</li>
<li>当前 0.5B Instruct 在任意自然提问下都能稳定完成多轮回忆；</li>
<li>训练吞吐、KV 实际占用与 BPB 对照（已分别移交 I1 / I2 / I3）；</li>
<li>中文 midtrain 已经开始或取得效果；</li>
<li>Qwen3、MoE 或其他尺寸模型的兼容。</li>
</ul>
<h3 id="收官复盘知道怎么做更要知道做得对不对">收官复盘：知道怎么做，更要知道做得对不对</h3>
<p><strong>写出 LLM 很重要，知道怎么测试和评估同样重要。前者让我知道怎么做，后者让我知道做得对不对、为什么对，以及结论的边界在哪里。</strong></p>
<p>模型能运行，不代表实现正确；输出像话，不代表能力可靠；一次测试通过，也不代表任意场景都稳定。今天的测试不是为了得到好看的 <code>46 passed</code>，而是建立一条从数学计算到真实行为的证据链。</p>
<h4 id="1-先比较确定性-logits再进入生成">1. 先比较确定性 logits，再进入生成</h4>
<p>固定 tokenizer、input_ids、权重、CPU、float32 和 <code>eval()</code>，直接比较：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">our_logits</span> <span class="o">=</span> <span class="n">our_model</span><span class="p">(</span><span class="n">input_ids</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">official_logits</span> <span class="o">=</span> <span class="n">official_model</span><span class="p">(</span><span class="n">input_ids</span><span class="p">)</span><span class="o">.</span><span class="n">logits</span>
</span></span></code></pre></div><p>这一步隔离了 temperature、top-k 和随机采样。如果固定输入下的原始 logits 都有明显差异，就先检查模型结构、权重映射和计算顺序，而不是从最终文本里猜测原因。</p>
<p>最小中文输入先建立基线，再扩展英文、代码和完整 chat template，是为了避免“某一个句子刚好通过”的偶然性。每组输入通过，也只代表该组输入得到覆盖，不代表所有文本。</p>
<h4 id="2-官方实现是参考对照组不是标准答案">2. 官方实现是参考对照组，不是标准答案</h4>
<p>官方实现可以帮助我们区分“自实现错误”和“模型能力问题”，但官方模型自己也可能答错。</p>
<p>因此，对照实验必须固定输入、权重、dtype 和解码方式，只改变实现。官方默认 SDPA 与手写 attention 出现微小误差时，没有直接放宽阈值，而是只切换到 eager 路径，最终得到全量 logits 零误差。</p>
<p>这个过程的价值不只是让测试通过，而是找到了差异来自哪条计算路径。</p>
<h4 id="3-kv-cache-要单独覆盖增量路径">3. KV Cache 要单独覆盖增量路径</h4>
<p>完整前向覆盖 <code>q_len&gt;1、past_len=0</code>，真实生成则大量使用 <code>q_len=1、past_len&gt;0</code>。两者的 RoPE 位置、cache 拼接和 mask 逻辑不同，不能只测其中一个。</p>
<p>cache 测试先比较 prefill 后两层的 K/V，再连续三次把相同的 argmax next token 喂给两边，比较每一步的全量 logits。这样既避免生成轨迹分叉，也覆盖了三个不同的历史长度。</p>
<p>同时必须诚实地区分证据：</p>
<ul>
<li>K/V 在 prefill 时相同，不能单独证明非零 <code>past_len</code>；</li>
<li>cache 长度每轮加 1，只直接证明长度变化；</li>
<li>最终 logits 一致，为 <code>past_len → RoPE → cache → attention</code> 提供强组合证据，但不等于每个内部变量都被独立证明；</li>
<li><code>q_len=1</code> 没有未来 token，<code>mask=None</code> 在因果定义上成立，但输出一致不等于直接比较了 attention weights。</li>
</ul>
<p><strong>测试通过和测试覆盖完整，是两件事。</strong></p>
<h4 id="4-数值一致之后还要验证真实行为">4. 数值一致之后，还要验证真实行为</h4>
<p>多轮行为回归经过完整的 <code>ChatSession → history → chat template → model_v3 → KV Cache → 回答清理</code>，检查姓名、爱好、学习内容、角色错位和模板标记泄漏。</p>
<p>断言不要求整段回答逐字相同，而是检查必须包含的信息和已知错误。这样既验证真实目标，也避免把正常的措辞差异当成失败。</p>
<p>代码解释回归则使用 <code>max_new_tokens=512</code>，覆盖此前暴露的长回答截断场景，并要求最终输出 <code>[2, 4, 6]</code> 真正出现。</p>
<p>这些固定场景证明“当前代表性需求通过”，不证明任意自然提问或任意代码都已稳定。</p>
<h4 id="5-每个断言回答的问题不同">5. 每个断言回答的问题不同</h4>
<ul>
<li><code>shape</code>：输出结构是否一致，不检查数值内容；</li>
<li><code>torch.isfinite()</code>：是否出现 NaN 或无穷，不证明结果正确；</li>
<li><code>assert_close()</code>：数值是否在明确容差内；</li>
<li><code>argmax</code> top-1：最终 next-token 选择是否一致；</li>
<li><code>max / mean</code>：分别观察最严重误差与整体误差水平；</li>
<li>行为断言：最终链路是否满足用户真正关心的要求。</li>
</ul>
<p>本次 <code>rtol=0、atol=1e-4</code> 表示只看绝对误差，不因参考 logit 较大而额外放宽标准。top-1 与全量 logits 断言互补，不能互相替代。</p>
<h4 id="6-回归测试还要控制自己的噪声">6. 回归测试还要控制自己的噪声</h4>
<p>分类测试第一次失败时，没有删除 loss 下降断言，也没有盲目增加训练轮数。代码审查发现无随机种子、shuffle loader 和随机两个 batch 评估让测量不完全可比。</p>
<p>增加固定种子并使用全部玩具样本评估后，原断言保留且通过。修复的是实验的可复现性，不是把红灯藏起来。</p>
<p>最终 <code>46 passed</code> 表示当前代码、权重、数据和环境下的 46 个固定用例全部通过，旧能力没有在这些用例上回归。它不表示所有可能的输入、设备、生成长度和模型能力都已被证明。</p>
<p>以后每完成一个里程碑，除了问“功能实现了吗”，还应该继续问：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">我用什么证据证明它正确？
</span></span><span class="line"><span class="cl">这个测试直接观测了什么？
</span></span><span class="line"><span class="cl">哪些结论只是组合行为的间接证据？
</span></span><span class="line"><span class="cl">哪些重要场景没有覆盖？
</span></span><span class="line"><span class="cl">失败后能否通过控制变量缩小问题边界？
</span></span></code></pre></div><p>能写出模型，是把想法变成代码；能测试和评估模型，是把“我觉得它对”变成“我知道它在哪些条件下、以什么证据成立”。后者同样是从零构建大模型不可缺少的一部分。</p>
<p>今天最值得留下的，不只是 Qwen 终于在自己的模型里说话了，而是这条更通用的工程方法：</p>
<blockquote>
<p><strong>先让结果可复现，再让变量可控制，最后才让结论落地。</strong></p>
</blockquote>
<h2 id="十六今天的感悟每一版都重新理解一遍-transformer">十六、今天的感悟：每一版，都重新理解一遍 Transformer</h2>
<p>今天还有一个比跑通 Qwen 更重要的收获：v1、v2、v3 的迭代，不只是工程目录和版本号的变化，更是在一遍遍加深我对原始 Transformer 架构的理解。</p>
<p>每一版，我都会重新亲手拼装 Embedding、归一化、注意力、残差、FFN 和输出头，再让新的组件进入这条主链路。看似是在重复，实际上每次面对的问题都不同：v1 理解基本的数据流，v2 重新思考位置信息，v3 进一步处理归一化、门控、注意力头分组和 KV Cache。</p>
<p>正是在这种“重复但不简单复制”的过程中，我逐渐分清了哪些是不变的骨架，哪些是可以替换的部件。张量形状为什么变化、RoPE 的位置为什么与 <code>past_len</code> 有关、mask 为什么要随 cache 调整，也开始从“记住一张架构图”，变成“能用自己的话解释，并能动手验证”。</p>
<p>对我来说，这是一种很有效的学习方式：<strong>想学，就实现一个；想真正熟悉，就在下一个版本里再亲手实现一次。</strong></p>
<h2 id="十七界面里程碑lvchat-有了自己的实验台">十七、界面里程碑：lvchat 有了自己的实验台</h2>
<p>最后，Chat 页面也完成了一次设计与交互升级。前端由 Gemini 在单文件、无外部依赖的约束下优化，后端接口和会话逻辑保持不变。</p>
<p>这次优化没有把它做成主流 AI 聊天产品的仿制品，而是保留了真实能力和项目身份：<strong>model_v3、Qwen2.5-0.5B Instruct、本地推理、单 Session、非流式纯文本返回。</strong></p>
<p><img alt="lvchat 本地模型实验台首屏：模型信息、单 Session 和真实推理链路" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-17-workbench-welcome.png"></p>
<p>首屏不再只是一个空白的聊天框，而是把这次实验的边界讲清楚了：手搓架构与权重承接、单会话连续上下文，以及完整生成后一次性返回。界面上的说明与后端真实能力一致，没有为了显得完整而添加模型切换、历史会话、流式输出等并不存在的功能。</p>
<p><img alt="lvchat 多轮对话与 Python 代码解释：本地模型完整回答" loading="lazy" src="https://blog.lvxinrong.com/llm/llm-17-workbench-chat.png"></p>
<p>第二张图记录了真实的多轮对话和 Python 代码解释。消息轮次、用户输入、模型回答和长文本阅读层次都得到了清晰呈现，最终输出 <code>偶数列表： [2, 4, 6]</code> 也完整保留在页面中。</p>
<p>对我而言，这个界面里程碑不只是“变好看了”。从第一阶段的 355M 对话页面，到今天由 model_v3 驱动 Qwen2.5 Instruct 的本地实验台，模型、推理、会话和展示终于组成了更完整也更诚实的作品。</p>
<h2 id="附录官方参考实现验证代码">附录：官方参考实现验证代码</h2>
<p>下面这段代码绕过 Chat Server 和 model_v3，使用 Hugging Face 官方模型实现。history 和 prompt 构造器与自实现实验相同，解码统一为 greedy，只生成第三轮回答。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">transformers</span> <span class="kn">import</span> <span class="n">AutoModelForCausalLM</span><span class="p">,</span> <span class="n">AutoTokenizer</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.device</span> <span class="kn">import</span> <span class="n">get_device</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.generate.chat</span> <span class="kn">import</span> <span class="n">build_qwen_chat_prompt</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">model_name</span> <span class="o">=</span> <span class="s2">&#34;Qwen/Qwen2.5-0.5B-Instruct&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">device</span> <span class="o">=</span> <span class="n">get_device</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">tokenizer</span> <span class="o">=</span> <span class="n">AutoTokenizer</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="n">model_name</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">history</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;你好，我叫吕炘嵘。&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;你好！很高兴见到你。有什么我可以帮助你的吗？&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;我喜欢写代码，也正在学习大模型。&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;那太好了！编程和大模型都是非常有趣和有挑战性的领域。&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;你对编程有什么具体的问题或项目吗？&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">current_input</span> <span class="o">=</span> <span class="s2">&#34;你还记得我叫什么、喜欢做什么吗？&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">prompt</span> <span class="o">=</span> <span class="n">build_qwen_chat_prompt</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">tokenizer</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">history</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">current_input</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">input_ids</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">prompt</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">return_tensors</span><span class="o">=</span><span class="s2">&#34;pt&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">model</span> <span class="o">=</span> <span class="n">AutoModelForCausalLM</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">model_name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="n">output_ids</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">generate</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">input_ids</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">max_new_tokens</span><span class="o">=</span><span class="mi">64</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">do_sample</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">eos_token_id</span><span class="o">=</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">eos_token_id</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">pad_token_id</span><span class="o">=</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">pad_token_id</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">output_ids</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="n">input_ids</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]:],</span>
</span></span><span class="line"><span class="cl">    <span class="n">skip_special_tokens</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
</span></span></code></pre></div><p>运行方式：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">uv run python official_qwen_check.py
</span></span></code></pre></div><p>这里只为附录示例约定文件名 <code>official_qwen_check.py</code>；实际也可以通过 <code>uv run python</code> 直接执行这段代码。关键是保持输入、权重和解码参数与 model_v3 对照组一致。</p>
<h2 id="附录2完整的layerspy实现">附录2：完整的layers.py实现</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="n">nn</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.model.v2_rope.rope</span> <span class="kn">import</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">compute_inv_freq</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">compute_cos_sin</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">apply_rope</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">RMSNorm</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">cfg</span> <span class="o">=</span> <span class="n">cfg</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">=</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">eps</span> <span class="o">=</span> <span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;eps&#34;</span><span class="p">,</span> <span class="mf">1e-6</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">scale</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Parameter</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">    LayerNorm 是“减均值、除标准差”，RMSNorm 是“不减均值，直接除以均方根”。
</span></span></span><span class="line"><span class="cl"><span class="s2">    LayerNorm 先把数据中心化，再控制方差；RMSNorm 不管中心点，只控制向量的整体幅度。
</span></span></span><span class="line"><span class="cl"><span class="s2">    两者都在控制激活值的整体幅度，防止经过多次计算后某些特征向量过大或过小，从而让训练更稳定。
</span></span></span><span class="line"><span class="cl"><span class="s2">    RMSNorm 主要通过等比例缩放控制幅度，LayerNorm 还会先做均值中心化。
</span></span></span><span class="line"><span class="cl"><span class="s2">    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">mean_square</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">y</span> <span class="o">=</span> <span class="n">x</span> <span class="o">*</span> <span class="n">torch</span><span class="o">.</span><span class="n">rsqrt</span><span class="p">(</span><span class="n">mean_square</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">eps</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">scale</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">y</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">SwiGLUFFN</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">ffn_bias</span> <span class="o">=</span> <span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;ffn_bias&#34;</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;ffn_dim&#34;</span><span class="p">,</span> <span class="mi">3</span> <span class="o">*</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">]))</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">gate_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">up_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">down_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">ffn_bias</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">act</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">SiLU</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">    y = down_proj( SiLU(gate_proj(x)) * up_proj(x) )    gate_proj门控信号，SiLU 之后才形成实际的连续开度。
</span></span></span><span class="line"><span class="cl"><span class="s2">    up_proj 升维后的x (B, T, emb_dim) -&gt; (B, T, hidden_dim)  hidden_dim 默认是emb_dim的3倍
</span></span></span><span class="line"><span class="cl"><span class="s2">    down_proj 降维回 (B, T, hidden_dim) -&gt; (B, T, emb_dim)
</span></span></span><span class="line"><span class="cl"><span class="s2">    SwiGLUFFN 相比较之前的 FeedForward 核心是多了一个门阀矩阵。
</span></span></span><span class="line"><span class="cl"><span class="s2">    按照我的理解，这个门阀矩阵是类似一个电阻的装置，允许电流通过大小，也就是hidden_dim每个维度允许通过多少
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2">    但是核心原理和FeedForward不变，依然是在token经过attention 输出后的每个 token 的隐藏表示 做逐位置变换
</span></span></span><span class="line"><span class="cl"><span class="s2">    再更高维度对自身的隐藏表示进行编辑。    阀门也就是在维度升高之后，增加了一个可以由模型自行通过学习修改阀门流量的开关。可以决定这些信息哪些值得保留，放大或者压下去。
</span></span></span><span class="line"><span class="cl"><span class="s2">    SwiGLU FFN 在升维后，是用 gate_proj 单独生成一路门控信号，经 SiLU 变成每个 hidden 维度的连续开度，再与 up_proj 的内容逐元素相乘。这个开度可学习、可大于 1、可接近 0、也可为负，因此模型能逐维度决定信息保留、放大、压制甚至反向。
</span></span></span><span class="line"><span class="cl"><span class="s2">    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">down_proj</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">act</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">gate_proj</span><span class="p">(</span><span class="n">x</span><span class="p">))</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">up_proj</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">核心是减少了KV的头，Q的头保持不变
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2">假设Q = 16, K,V = 4
</span></span></span><span class="line"><span class="cl"><span class="s2">所以按照我的理解，GQA是针对推理时候的优化，而不是训练的时候
</span></span></span><span class="line"><span class="cl"><span class="s2">KV的数量后，为了可以和Q进行矩阵运算，是进行的横向复制。那么就可以推出计算量和数据量并没有减少 (1份复制到4份)
</span></span></span><span class="line"><span class="cl"><span class="s2">但是k，v的数据量下降到原本的四分之一了
</span></span></span><span class="line"><span class="cl"><span class="s2">也就是在推理的时候，我们接入了kv cache的空间占用量下降了75%
</span></span></span><span class="line"><span class="cl"><span class="s2">所以kv cache中保留的未扩展的4个头。每个Token的KV存储量大约是原来的1/4
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2">GQA 主要通过减少 K/V 头数来降低推理时的 cache 和带宽压力，而不是直接减少所有训练或推理计算。
</span></span></span><span class="line"><span class="cl"><span class="s2">&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">GroupedQueryAttention</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;n_heads&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;n_kv_heads&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span> <span class="o">==</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">        <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span> <span class="o">==</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span> <span class="o">//</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">kv_dim</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># 每组 K/V 对应多少个 Q 头</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span> <span class="o">//</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">W_query</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;qkv_bias&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">W_key</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">kv_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;qkv_bias&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">W_value</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">kv_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;qkv_bias&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">out_proj</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">emb_dim</span><span class="p">,</span> <span class="n">bias</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;drop_rate&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># 创建一个因果掩码（causal mask）张量，并把它注册成 Module 的 buffer        # register_buffer 是给“不是可学习参数、但属于模型状态、需要跟着模型一起搬设备/转精度/存 state_dict”的张量用的。</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">register_buffer</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;mask&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">torch</span><span class="o">.</span><span class="n">triu</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">torch</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;context_length&#34;</span><span class="p">],</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;context_length&#34;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">bool</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">diagonal</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="n">persistent</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">register_buffer</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;inv_freq&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">compute_inv_freq</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="n">base</span><span class="o">=</span><span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;rope_theta&#34;</span><span class="p">,</span> <span class="mf">10000.0</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="n">persistent</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">expand_kv</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">x</span>
</span></span><span class="line"><span class="cl">        <span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span><span class="p">[:,</span> <span class="p">:,</span> <span class="kc">None</span><span class="p">,</span> <span class="p">:,</span> <span class="p">:]</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">x</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">n_kv_heads</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_queries_per_kv</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">head_dim</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">    (B, T, K/V, D) 横向扩展复制成 (B, T, Q, D)    假设Q = 16， k,v = 4    也就是针对K进行横向复制扩展，每份复制4份，这样后面才能进行 Q @ K.T的运算，不然 Q和KV的维度是对不上的
</span></span></span><span class="line"><span class="cl"><span class="s2">    K0, K1, K2, K3 转换成
</span></span></span><span class="line"><span class="cl"><span class="s2">    (    (K0, K0, K0, K0),    (K1, K1, K1, K1),    ...    )    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">past_kv</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">b</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># (B, T, C) -&gt; (B, T, H, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">queries</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">W_query</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># (B, T, C) -&gt; (B, T, KV, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">keys</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">W_key</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># (B, T, C) -&gt; (B, T, KV,D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">values</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">W_value</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_kv_heads</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1">#(B, T, H, D) -&gt; (B, H, T, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">queries</span> <span class="o">=</span> <span class="n">queries</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1">#(B, T, H, D) -&gt; (B, KV, T, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># (B, T, H, D) -&gt; (B, KV, T, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">values</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">past_len</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">past_kv</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">past_len</span> <span class="o">=</span> <span class="n">past_kv</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">position_ids</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">past_len</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">past_len</span> <span class="o">+</span> <span class="n">num_tokens</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">device</span><span class="o">=</span><span class="n">x</span><span class="o">.</span><span class="n">device</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">cos</span><span class="p">,</span> <span class="n">sin</span> <span class="o">=</span> <span class="n">compute_cos_sin</span><span class="p">(</span><span class="n">position_ids</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">inv_freq</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">queries</span> <span class="o">=</span> <span class="n">apply_rope</span><span class="p">(</span><span class="n">queries</span><span class="p">,</span> <span class="n">cos</span><span class="p">,</span> <span class="n">sin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">keys</span> <span class="o">=</span> <span class="n">apply_rope</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">cos</span><span class="p">,</span> <span class="n">sin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">past_kv</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">past_k</span><span class="p">,</span> <span class="n">past_v</span> <span class="o">=</span> <span class="n">past_kv</span>
</span></span><span class="line"><span class="cl">            <span class="n">keys</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">past_k</span><span class="p">,</span> <span class="n">keys</span><span class="p">],</span> <span class="n">dim</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">values</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">past_v</span><span class="p">,</span> <span class="n">values</span><span class="p">],</span> <span class="n">dim</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">present_kv</span> <span class="o">=</span> <span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">values</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">keys</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expand_kv</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">values</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expand_kv</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">attn_scores</span> <span class="o">=</span> <span class="n">queries</span> <span class="o">@</span> <span class="n">keys</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">q_len</span> <span class="o">=</span> <span class="n">queries</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="n">k_len</span> <span class="o">=</span> <span class="n">keys</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">q_len</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">mask_bool</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">past_len</span> <span class="o">=</span> <span class="n">k_len</span> <span class="o">-</span> <span class="n">q_len</span>
</span></span><span class="line"><span class="cl">            <span class="n">mask_bool</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">mask</span><span class="p">[</span><span class="n">past_len</span><span class="p">:</span><span class="n">past_len</span> <span class="o">+</span> <span class="n">q_len</span><span class="p">,</span> <span class="p">:</span><span class="n">k_len</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">mask_bool</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">attn_scores</span> <span class="o">=</span> <span class="n">attn_scores</span><span class="o">.</span><span class="n">masked_fill</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">mask_bool</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="o">-</span><span class="n">torch</span><span class="o">.</span><span class="n">inf</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">attn_weights</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="n">attn_scores</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span> <span class="o">**</span> <span class="mf">0.5</span><span class="p">,</span> <span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">attn_weights</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span><span class="p">(</span><span class="n">attn_weights</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># (B, H, q_len, k_len) @ (B, H, k_len, D) -&gt; (B, H, q_len, D)</span>
</span></span><span class="line"><span class="cl">        <span class="n">context_vec</span> <span class="o">=</span> <span class="p">(</span><span class="n">attn_weights</span> <span class="o">@</span> <span class="n">values</span><span class="p">)</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">context_vec</span> <span class="o">=</span> <span class="n">context_vec</span><span class="o">.</span><span class="n">contiguous</span><span class="p">()</span><span class="o">.</span><span class="n">view</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">b</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_heads</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">head_dim</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">out_proj</span><span class="p">(</span><span class="n">context_vec</span><span class="p">),</span> <span class="n">present_kv</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">TransformerBlock</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">att</span> <span class="o">=</span> <span class="n">GroupedQueryAttention</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">ffn</span> <span class="o">=</span> <span class="n">SwiGLUFFN</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">norm1</span> <span class="o">=</span> <span class="n">RMSNorm</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">norm2</span> <span class="o">=</span> <span class="n">RMSNorm</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">drop_shortcut</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;drop_rate&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">past_kv</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">shortcut</span> <span class="o">=</span> <span class="n">x</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span><span class="p">,</span> <span class="n">present_kv</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">att</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">past_kv</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">drop_shortcut</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">shortcut</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">shortcut</span> <span class="o">=</span> <span class="n">x</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ffn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">drop_shortcut</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">shortcut</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">x</span><span class="p">,</span> <span class="n">present_kv</span>
</span></span></code></pre></div><h3 id="附录3-完整的gptmodelv3">附录3: 完整的GPTModelV3</h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="n">nn</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">squid.model.v3_modern.layers</span> <span class="kn">import</span> <span class="n">TransformerBlock</span><span class="p">,</span> <span class="n">RMSNorm</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">MODEL_V3_DEFAULT_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;vocab_size&#34;</span><span class="p">:</span> <span class="mi">151936</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;emb_dim&#34;</span><span class="p">:</span> <span class="mi">896</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;ffn_dim&#34;</span><span class="p">:</span> <span class="mi">4864</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;n_heads&#34;</span><span class="p">:</span> <span class="mi">14</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;n_kv_heads&#34;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;n_layers&#34;</span><span class="p">:</span> <span class="mi">24</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;rope_theta&#34;</span><span class="p">:</span> <span class="mf">1000000.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;eps&#34;</span><span class="p">:</span> <span class="mf">1e-6</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;qkv_bias&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;drop_rate&#34;</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;context_length&#34;</span><span class="p">:</span> <span class="mi">1024</span><span class="p">,</span>  <span class="c1"># 先做小规模验证</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;ffn_bias&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;tie_word_embeddings&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">GPTModelV3</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cfg</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">cfg</span> <span class="o">=</span> <span class="p">{</span><span class="o">**</span><span class="n">MODEL_V3_DEFAULT_CONFIG</span><span class="p">,</span> <span class="o">**</span><span class="p">(</span><span class="n">cfg</span> <span class="ow">or</span> <span class="p">{})}</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">tok_emb</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;vocab_size&#34;</span><span class="p">],</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">drop_emb</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;drop_rate&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">trf_block</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="o">*</span><span class="p">[</span><span class="n">TransformerBlock</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;n_layers&#34;</span><span class="p">])],</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">final_norm</span> <span class="o">=</span> <span class="n">RMSNorm</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">out_head</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;emb_dim&#34;</span><span class="p">],</span> <span class="n">cfg</span><span class="p">[</span><span class="s2">&#34;vocab_size&#34;</span><span class="p">],</span> <span class="n">bias</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># 一个 token 被理解成什么样，和它最后被生成出来时对应的向量表示，最好保持一致。</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># 直接减少参数量, 让输入和输出使用同一套语义表示</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">cfg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;tie_word_embeddings&#34;</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># 底层使用的是同一个权重张量</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># tok_emb 和 out_head 不是同一个模块，但它们共享同一个 weight 参数对象。</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">out_head</span><span class="o">.</span><span class="n">weight</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">tok_emb</span><span class="o">.</span><span class="n">weight</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">past_kv</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_cache</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">tok_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">drop_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">present</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="cl">        <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">block</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">trf_block</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="n">past</span> <span class="o">=</span> <span class="n">past_kv</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">past_kv</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">            <span class="n">x</span><span class="p">,</span> <span class="n">present_kv</span> <span class="o">=</span> <span class="n">block</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">past_kv</span><span class="o">=</span><span class="n">past</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">present</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">present_kv</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">final_norm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">logits</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">out_head</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">use_cache</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">logits</span><span class="p">,</span> <span class="n">present</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">logits</span>
</span></span></code></pre></div>]]></content:encoded>
    </item>
  </channel>
</rss>
