<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>多轮对话 on 吕炘嵘的博客</title>
    <link>https://blog.lvxinrong.com/tags/%E5%A4%9A%E8%BD%AE%E5%AF%B9%E8%AF%9D/</link>
    <description>Recent content in 多轮对话 on 吕炘嵘的博客</description>
    <image>
      <title>吕炘嵘的博客</title>
      <url>https://blog.lvxinrong.com/og-default.jpg</url>
      
      <link>https://blog.lvxinrong.com/tags/%E5%A4%9A%E8%BD%AE%E5%AF%B9%E8%AF%9D/</link>
    </image>
    <generator>Hugo</generator>
    <language>zh-CN</language>
    <lastBuildDate>Mon, 16 Nov 2026 09:00:00 +0800</lastBuildDate>
    <atom:link href="https://blog.lvxinrong.com/tags/%E5%A4%9A%E8%BD%AE%E5%AF%B9%E8%AF%9D/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>手搓 LLM 笔记 23：I4 多轮对话 SFT</title>
      <link>https://blog.lvxinrong.com/llm/llm-23-i4-multiturn-sft/</link>
      <pubDate>Mon, 16 Nov 2026 09:00:00 +0800</pubDate>
      <guid>https://blog.lvxinrong.com/llm/llm-23-i4-multiturn-sft/</guid>
      <description>从 Assistant-only loss mask、Qwen 多轮模板和 128 条合成样本开始，完成全参数 SFT、独立行为评测与训练量干预。记录 v1 能力回退、v2 恢复，以及 loss 与行为表现不一致的真实证据。</description>
      <content:encoded><![CDATA[<p>这篇记录的是 <strong>I4-B：多轮对话 SFT 的最小完整闭环</strong>，不是整个 I4 的收官。CoT-SFT、DPO、GRPO、LoRA 与全参对照尚未开始。</p>
<p>这一轮最有价值的地方，不只是最后得到了一个当前最佳 checkpoint，而是第一次完整看见：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">多轮数据如何变成训练目标
</span></span><span class="line"><span class="cl">→ SFT 如何改变模型行为
</span></span><span class="line"><span class="cl">→ 新能力习得为什么会伤害旧能力
</span></span><span class="line"><span class="cl">→ 为什么不能只看训练 loss 判断后训练效果
</span></span></code></pre></div><h2 id="一实验目标与边界">一、实验目标与边界</h2>
<p>本轮使用 <strong>Qwen3-0.6B Instruct</strong>，从我们独立实现的 V4 加载、训练和生成。实验设备为 MacBook M4 Pro，实际运行后端为 MPS，权重和训练使用 FP32。</p>
<p>第一版只研究非 CoT 的多轮行为：</p>
<ul>
<li>名字与偏好记忆；</li>
<li>根据新信息更新旧数值；</li>
<li>简单算术；</li>
<li>按指定格式输出；</li>
<li>自然结束，不复制上一轮回答。</li>
</ul>
<p>这不是开放域对话训练。训练数据是受控合成数据，行为评测也只有 6 条固定基线和 32 条独立验证样本。所有结论都限定在这个范围内。</p>
<h2 id="二先弄清楚多轮样本到底监督什么">二、先弄清楚：多轮样本到底监督什么</h2>
<h3 id="1--100-与--torchinf-不是一回事">1. <code>-100</code> 与 <code>-torch.inf</code> 不是一回事</h3>
<p>M4 的单轮 SFT 已经使用过 <code>ignore_index=-100</code>，但这次重新读代码时，我发现自己仍然容易把它和注意力中的 <code>-torch.inf</code> 混在一起。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">target = -100
</span></span><span class="line"><span class="cl">→ 这个位置的预测不计入 loss
</span></span><span class="line"><span class="cl">→ 输入 token 仍然存在，仍可被后续位置读取
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">attn_score = -torch.inf
</span></span><span class="line"><span class="cl">→ 经 softmax 后注意力概率为 0
</span></span><span class="line"><span class="cl">→ 这条注意力连接被切断
</span></span></code></pre></div><p>所以，User 不参与 loss，不代表模型看不到 User。模型仍然读取完整问题和历史，只是我们不要求它学习生成 User 的提问。</p>
<h3 id="2-为什么-targetsp---1-而不是-targetsp">2. 为什么 <code>targets[:P - 1]</code> 而不是 <code>targets[:P]</code></h3>
<p>假设：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">full:    [p0, p1, p2, r0, r1, EOS]
</span></span><span class="line"><span class="cl">inputs:  [p0, p1, p2, r0, r1]
</span></span><span class="line"><span class="cl">targets: [p1, p2, r0, r1, EOS]
</span></span></code></pre></div><p><code>targets[i] = full[i + 1]</code>。当 prompt 长度 <code>P=3</code> 时，必须保留 <code>targets[2] = r0</code>，否则第一个回答 token 就没有监督。</p>
<p>这也解释了为什么 Full 中的 Assistant 区间不能直接当作 target 下标：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">full[29:35]  → targets[28:34]
</span></span><span class="line"><span class="cl">full[53:59]  → targets[52:58]
</span></span></code></pre></div><h3 id="3-多轮-assistant-都应该监督user-只提供上下文">3. 多轮 Assistant 都应该监督，User 只提供上下文</h3>
<p>本轮使用 Assistant-only SFT：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">System：上下文，不计 loss
</span></span><span class="line"><span class="cl">User：上下文，不计 loss
</span></span><span class="line"><span class="cl">Assistant 角色头：上下文，不计 loss
</span></span><span class="line"><span class="cl">Assistant 正文：计 loss
</span></span><span class="line"><span class="cl">Assistant 的 &lt;|im_end|&gt;：计 loss
</span></span></code></pre></div><p><code>&lt;|im_start|&gt;assistant</code> 是模板提供的答题区，不需要模型自己生成；<code>&lt;|im_end|&gt;</code> 是模型回答结束后需要生成的动作，必须参与训练。</p>
<p>多轮样本里会出现多个 <code>&lt;|im_end|&gt;</code>。System 和 User 的结束标记与 Assistant 的结束标记虽然 token id 相同，但职责不同，不能只保留整个序列中的第一个 EOS。</p>
<h2 id="三把真实-qwen-模板变成-dataset">三、把真实 Qwen 模板变成 Dataset</h2>
<h3 id="1-模板探针">1. 模板探针</h3>
<p>我们首先用两轮对话检查真实渲染结果，而不是凭感觉拼字符串。固定样例渲染后共有 60 个 token，两轮 Assistant 的 Full 区间为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">第一轮：full[29:35]
</span></span><span class="line"><span class="cl">第二轮：full[53:59]
</span></span></code></pre></div><p>第二轮还包含一个空的 <code>&lt;think&gt;...&lt;/think&gt;</code> 区域。当前非 CoT 数据契约中，空 think 区域只作为上下文，不参与 loss；两轮 Assistant 正文及各自的 <code>&lt;|im_end|&gt;</code> 参与 loss。</p>
<h3 id="2-核心掩码构造函数">2. 核心掩码构造函数</h3>
<p><code>src/squid/data/chat_sft.py</code> 中，区间统一使用 Full 下标、右端不取：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">build_assistant_targets</span><span class="p">(</span><span class="n">full_ids</span><span class="p">,</span> <span class="n">assistant_spans</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=-</span><span class="mi">100</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">full_ids</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&#34;full_ids 不能为空&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">targets</span> <span class="o">=</span> <span class="p">[</span><span class="n">ignore_index</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">full_ids</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span> <span class="ow">in</span> <span class="n">assistant_spans</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">start</span> <span class="o">&lt;</span> <span class="mi">1</span> <span class="ow">or</span> <span class="n">end</span> <span class="o">&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">full_ids</span><span class="p">)</span> <span class="ow">or</span> <span class="n">start</span> <span class="o">&gt;=</span> <span class="n">end</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="sa">f</span><span class="s2">&#34;非法 Assistant 区间：[</span><span class="si">{</span><span class="n">start</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">end</span><span class="si">}</span><span class="s2">)&#34;</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">targets</span><span class="p">[</span><span class="n">start</span> <span class="o">-</span> <span class="mi">1</span><span class="p">:</span><span class="n">end</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">full_ids</span><span class="p">[</span><span class="n">start</span><span class="p">:</span><span class="n">end</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">targets</span>
</span></span></code></pre></div><h3 id="3-不写死区间用前缀编码定位">3. 不写死区间，用前缀编码定位</h3>
<p><code>find_assistant_spans()</code> 在渲染文本中顺序定位每轮 Assistant 正文及 <code>&lt;|im_end|&gt;</code>，再分别编码正文前和结束标记后的前缀，得到 token 下标，并断言前缀编码与 <code>full_ids</code> 完全一致。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">start_ids</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">full_text</span><span class="p">[:</span><span class="n">content_start</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="n">add_special_tokens</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">end_ids</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">full_text</span><span class="p">[:</span><span class="n">end_char</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="n">add_special_tokens</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">assert</span> <span class="n">start_ids</span> <span class="o">==</span> <span class="n">full_ids</span><span class="p">[:</span><span class="nb">len</span><span class="p">(</span><span class="n">start_ids</span><span class="p">)]</span>
</span></span><span class="line"><span class="cl"><span class="k">assert</span> <span class="n">end_ids</span> <span class="o">==</span> <span class="n">full_ids</span><span class="p">[:</span><span class="nb">len</span><span class="p">(</span><span class="n">end_ids</span><span class="p">)]</span>
</span></span></code></pre></div><p>这样避免把某组固定样例的下标硬编码进 Dataset，也避免简单搜索 token id 时误切正文。</p>
<p><code>encode_qwen_multiturn()</code> 将真实模板、区间定位和 target 构造接起来：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">encode_qwen_multiturn</span><span class="p">(</span><span class="n">tokenizer</span><span class="p">,</span> <span class="n">messages</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">full_text</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">apply_chat_template</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">messages</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">tokenize</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">add_generation_prompt</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">encoded</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">apply_chat_template</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">messages</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">tokenize</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">add_generation_prompt</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">full_ids</span> <span class="o">=</span> <span class="n">encoded</span><span class="p">[</span><span class="s2">&#34;input_ids&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">full_ids</span> <span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">full_ids</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">list</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">full_ids</span> <span class="o">=</span> <span class="n">full_ids</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">spans</span> <span class="o">=</span> <span class="n">find_assistant_spans</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">tokenizer</span><span class="p">,</span> <span class="n">messages</span><span class="p">,</span> <span class="n">full_text</span><span class="p">,</span> <span class="n">full_ids</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">targets</span> <span class="o">=</span> <span class="n">build_assistant_targets</span><span class="p">(</span><span class="n">full_ids</span><span class="p">,</span> <span class="n">spans</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">full_ids</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">targets</span>
</span></span></code></pre></div><h3 id="4-batch-padding">4. Batch padding</h3>
<p>inputs 使用 EOS/pad id 补齐，targets 使用 <code>-100</code> 补齐，不额外追加 EOS，因为每轮 Assistant 已经包含自己的 <code>&lt;|im_end|&gt;</code>：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">qwen_multiturn_collate</span><span class="p">(</span><span class="n">batch</span><span class="p">,</span> <span class="n">pad_token_id</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=-</span><span class="mi">100</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_len</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span> <span class="k">for</span> <span class="n">inputs</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">batch</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">input_batch</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="cl">    <span class="n">target_batch</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">inputs</span><span class="p">,</span> <span class="n">targets</span> <span class="ow">in</span> <span class="n">batch</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">input_ids</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">long</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">target_ids</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="n">targets</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">long</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">input_batch</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">pad_to_length</span><span class="p">(</span><span class="n">input_ids</span><span class="p">,</span> <span class="n">max_len</span><span class="p">,</span> <span class="n">pad_token_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">target_batch</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">pad_to_length</span><span class="p">(</span><span class="n">target_ids</span><span class="p">,</span> <span class="n">max_len</span><span class="p">,</span> <span class="n">ignore_index</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">torch</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="n">input_batch</span><span class="p">),</span> <span class="n">torch</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="n">target_batch</span><span class="p">)</span>
</span></span></code></pre></div><p>对应测试按阶段实际运行为 <strong>1 passed、2 passed、3 passed</strong>，分别覆盖掩码错位、真实模板 Dataset 和 Batch padding。这些数字不是一次全量回归。</p>
<h2 id="四冻结行为基线也修正评测器自己">四、冻结行为基线，也修正评测器自己</h2>
<h3 id="1-固定基线配置">1. 固定基线配置</h3>
<p>行为基线使用：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">模型：Qwen3-0.6B Instruct
</span></span><span class="line"><span class="cl">device：mps
</span></span><span class="line"><span class="cl">dtype：FP32
</span></span><span class="line"><span class="cl">context_size：8192
</span></span><span class="line"><span class="cl">max_new_tokens：512
</span></span><span class="line"><span class="cl">temperature：0.0
</span></span><span class="line"><span class="cl">enable_thinking：false
</span></span></code></pre></div><p>六条样例覆盖多轮记忆、连续数值更新、算术和格式遵循。</p>
<h3 id="2-expected_contains-为什么不能判断数值答案">2. <code>expected_contains</code> 为什么不能判断数值答案</h3>
<p>第一版规则只检查回答是否包含 <code>912</code>。但：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="s2">&#34;912&#34;</span> <span class="ow">in</span> <span class="s2">&#34;8912&#34;</span>  <span class="c1"># True</span>
</span></span><span class="line"><span class="cl"><span class="s2">&#34;912&#34;</span> <span class="ow">in</span> <span class="s2">&#34;7192&#34;</span>  <span class="c1"># True</span>
</span></span></code></pre></div><p>所以数值题改为精确匹配，并单独拆出两个指标：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">answer_correct：内容或数值是否正确
</span></span><span class="line"><span class="cl">format_compliant：是否遵守输出格式
</span></span></code></pre></div><p>例如：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">24            → 格式正确，答案错误（期望 30）
</span></span><span class="line"><span class="cl">123+789=912   → 格式错误，答案正确
</span></span><span class="line"><span class="cl">912           → 格式和答案都正确
</span></span></code></pre></div><p>评分器新增独立测试，实际运行 <strong>4 passed</strong>；加入复制检测后为 <strong>7 passed</strong>。</p>
<h3 id="3-一个列表共享引用造成的全量误报">3. 一个列表共享引用造成的全量误报</h3>
<p>32 条评估第一次全部显示 <code>copy=True</code>。根因不是模型全都复制，而是：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">case[&#34;history&#34;] 与 session.history 指向同一个列表
</span></span><span class="line"><span class="cl">→ session.ask() 把当前回答追加进 history
</span></span><span class="line"><span class="cl">→ 复制检测读取到刚生成的回答
</span></span><span class="line"><span class="cl">→ 再与 response 比较，永远相等
</span></span></code></pre></div><p>修复为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">session</span><span class="o">.</span><span class="n">history</span> <span class="o">=</span> <span class="n">history</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
</span></span></code></pre></div><p>这次错误提醒我：<strong>训练前先验证评测器，训练后也不要把评测器的异常直接归因于模型。</strong></p>
<h3 id="4-训练前的真实行为">4. 训练前的真实行为</h3>
<p>原始模型的六条基线结果为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">passed = 2/6
</span></span><span class="line"><span class="cl">answer_correct = 4/6
</span></span><span class="line"><span class="cl">format_compliant = 4/6
</span></span><span class="line"><span class="cl">copied_previous_assistant = 1
</span></span><span class="line"><span class="cl">natural_stop = 6/6
</span></span></code></pre></div><p>典型现象包括：</p>
<ul>
<li>苹果已有 24 个，再买 6 个，仍回答 <code>24</code>；</li>
<li><code>25 * 24</code> 错答为 <code>240</code>；</li>
<li><code>123 + 789 = 912</code> 结果正确，但违反“只回答数字”；</li>
<li>颜色题直接复制上一轮 Assistant。</li>
</ul>
<h2 id="五4-step-冒烟先证明链路不证明能力">五、4-step 冒烟：先证明链路，不证明能力</h2>
<p>正式数据之前，用 4 条训练样本、2 条验证样本做了 4 个优化步骤：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">initial train loss：3.390143
</span></span><span class="line"><span class="cl">initial val loss：  2.800545
</span></span><span class="line"><span class="cl">final train loss：  1.958147
</span></span><span class="line"><span class="cl">final val loss：    1.819232
</span></span><span class="line"><span class="cl">optimization steps：4
</span></span><span class="line"><span class="cl">checkpoint reload：passed
</span></span></code></pre></div><p>恢复 checkpoint 后生成两条验证回答：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">期望 28 → 生成 28，正常 stop
</span></span><span class="line"><span class="cl">期望 15 → 生成 6，正常 stop
</span></span></code></pre></div><p>所以冒烟只证明了数据、loss、反向传播、checkpoint 和生成链路可用。第二条答错说明它没有证明模型能力。</p>
<h2 id="六128--32-合成数据与隔离审计">六、128 + 32 合成数据与隔离审计</h2>
<p>正式数据由 GPT 辅助生成，再经过本地审计：</p>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>类别</th>
        <th style="text-align: right">Train</th>
        <th style="text-align: right">Val</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>单轮算术</td>
        <td style="text-align: right">32</td>
        <td style="text-align: right">8</td>
      </tr>
      <tr>
        <td>多轮数值更新</td>
        <td style="text-align: right">32</td>
        <td style="text-align: right">8</td>
      </tr>
      <tr>
        <td>名字记忆</td>
        <td style="text-align: right">32</td>
        <td style="text-align: right">8</td>
      </tr>
      <tr>
        <td>偏好记忆</td>
        <td style="text-align: right">32</td>
        <td style="text-align: right">8</td>
      </tr>
      <tr>
        <td>合计</td>
        <td style="text-align: right"><strong>128</strong></td>
        <td style="text-align: right"><strong>32</strong></td>
      </tr>
    </tbody>
  </table>
</div>
<p>隔离要求包括：</p>
<ul>
<li>train / val 没有完全重复的对话；</li>
<li>同一算术表达式不跨集合；</li>
<li>排除行为基线中的小吕、蓝色、3×8+6、25×24、123+789、17+5；</li>
<li>避免复用已有 smoke 样例；</li>
<li>Assistant 不携带 think 或角色标记；</li>
<li>每条样本都能通过真实模板编码，并且存在有效监督 token。</li>
</ul>
<p>分类审计最初把“叫我”“首选爱好”“昵称”等表达误归为数值任务。修正审计规则后，四类数量恢复平衡，全部模板编码通过，train / val 精确重复为 0，排除项检查通过。</p>
<p>这组数据仍然只是受控任务数据，不代表真实多轮语料分布。</p>
<h2 id="七v164-stepsloss-很漂亮行为却出现回退">七、v1：64 steps，loss 很漂亮，行为却出现回退</h2>
<h3 id="1-训练配置与-loss">1. 训练配置与 loss</h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">全参数 SFT
</span></span><span class="line"><span class="cl">batch_size = 4
</span></span><span class="line"><span class="cl">n_epochs = 2
</span></span><span class="line"><span class="cl">optimization steps = 64
</span></span><span class="line"><span class="cl">peak_lr = 1e-5
</span></span><span class="line"><span class="cl">initial_lr = 1e-6
</span></span><span class="line"><span class="cl">min_lr = 1e-7
</span></span><span class="line"><span class="cl">weight_decay = 0
</span></span><span class="line"><span class="cl">warmup_steps = 6
</span></span><span class="line"><span class="cl">seed = 123
</span></span><span class="line"><span class="cl">context_size = 1024
</span></span></code></pre></div><p>结果：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">initial train loss：3.797954
</span></span><span class="line"><span class="cl">initial val loss：  3.745439
</span></span><span class="line"><span class="cl">final train loss：  0.616244
</span></span><span class="line"><span class="cl">final full val loss：0.859470
</span></span><span class="line"><span class="cl">checkpoint reload：passed
</span></span></code></pre></div><h3 id="2-六条基线从-26-到-46">2. 六条基线：从 2/6 到 4/6</h3>
<p>v1 的六条基线变为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">passed = 4/6
</span></span><span class="line"><span class="cl">answer_correct = 4/6
</span></span><span class="line"><span class="cl">format_compliant = 6/6
</span></span><span class="line"><span class="cl">copied_previous_assistant = 0
</span></span><span class="line"><span class="cl">natural_stop = 6/6
</span></span></code></pre></div><p>改善包括：</p>
<ul>
<li>苹果题从 <code>24</code> 更新为 <code>30</code>；</li>
<li><code>17 + 5</code> 从完整算式变为只输出 <code>22</code>；</li>
<li>名字回答变为“你叫小吕”。</li>
</ul>
<p>但也出现：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">25 * 24：240 → 60，仍然错误
</span></span><span class="line"><span class="cl">123 + 789：结果正确的算式 → 909，数值回退
</span></span></code></pre></div><h3 id="3-32-条独立验证总通过率提升名字却明显退化">3. 32 条独立验证：总通过率提升，名字却明显退化</h3>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>类别</th>
        <th style="text-align: right">原始模型</th>
        <th style="text-align: right">SFT v1</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>名字记忆</td>
        <td style="text-align: right">8/8</td>
        <td style="text-align: right"><strong>3/8</strong></td>
      </tr>
      <tr>
        <td>数值更新</td>
        <td style="text-align: right">2/8</td>
        <td style="text-align: right"><strong>7/8</strong></td>
      </tr>
      <tr>
        <td>偏好记忆</td>
        <td style="text-align: right">7/8</td>
        <td style="text-align: right">7/8</td>
      </tr>
      <tr>
        <td>单轮算术</td>
        <td style="text-align: right">2/8</td>
        <td style="text-align: right"><strong>7/8</strong></td>
      </tr>
      <tr>
        <td>总计</td>
        <td style="text-align: right">19/32</td>
        <td style="text-align: right"><strong>24/32</strong></td>
      </tr>
    </tbody>
  </table>
</div>
<p>v1 的格式合规从 24 提升到 31，但内容正确从 25 降为 24。名字类出现“你”“你。”“你记住了”等无效回答。</p>
<h3 id="4-同一训练集上的对照确认训练引入的回退">4. 同一训练集上的对照，确认训练引入的回退</h3>
<p>为了区分“原本不会”与“训练后改坏”，又比较了原始模型与 v1 在同一 128 条样本上的 <code>passed</code>：</p>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>类别</th>
        <th style="text-align: right">原始模型</th>
        <th style="text-align: right">SFT v1</th>
        <th>净变化</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>名字记忆</td>
        <td style="text-align: right">32/32</td>
        <td style="text-align: right">21/32</td>
        <td><strong>-11</strong></td>
      </tr>
      <tr>
        <td>偏好记忆</td>
        <td style="text-align: right">32/32</td>
        <td style="text-align: right">28/32</td>
        <td><strong>-4</strong></td>
      </tr>
      <tr>
        <td>数值更新</td>
        <td style="text-align: right">12/32</td>
        <td style="text-align: right">27/32</td>
        <td><strong>+15</strong></td>
      </tr>
      <tr>
        <td>单轮算术</td>
        <td style="text-align: right">8/32</td>
        <td style="text-align: right">21/32</td>
        <td><strong>+13</strong></td>
      </tr>
      <tr>
        <td>总体</td>
        <td style="text-align: right">84/128</td>
        <td style="text-align: right">97/128</td>
        <td>+13</td>
      </tr>
    </tbody>
  </table>
</div>
<p>原始模型在名字和偏好样本上全部正确，v1 却在相同输入上退化。这不能简单解释为验证集分布不同，而是全参数训练在获得新行为时干扰了旧行为，符合局部灾难性遗忘的表现。</p>
<p>另一个重要事实是：<strong>v1 的最终 train loss 已降到 0.616，但训练集逐条生成仍只有 97/128 综合通过。</strong> 平均 token loss 与行为成功不是同一个指标。</p>
<h2 id="八v2减少训练量恢复新旧能力的平衡">八、v2：减少训练量，恢复新旧能力的平衡</h2>
<h3 id="1-干预方式与训练结果">1. 干预方式与训练结果</h3>
<p>在数据、Batch、学习率峰值、随机种子等保持不变的前提下，将 <code>n_epochs</code> 从 2 改为 1：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">optimization steps = 32
</span></span><span class="line"><span class="cl">warmup_steps = 3
</span></span><span class="line"><span class="cl">final train loss = 1.104157
</span></span><span class="line"><span class="cl">final full val loss = 1.021104
</span></span><span class="line"><span class="cl">checkpoint reload = passed
</span></span></code></pre></div><p>严格来说，修改 <code>n_epochs</code> 后，总步数、warmup 步数和学习率衰减过程都发生了变化。因此这是一组训练量与调度对照，不能把全部差异只归因于 epoch 数。</p>
<h3 id="2-原始模型v1v2-的独立验证对照">2. 原始模型、v1、v2 的独立验证对照</h3>
<div class="table-scroll" tabindex="0" role="region" aria-label="表格，可横向滚动">
  <table>
    <thead>
      <tr>
        <th>指标</th>
        <th style="text-align: right">原始模型</th>
        <th style="text-align: right">v1：2 epochs</th>
        <th style="text-align: right">v2：1 epoch</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>综合通过</td>
        <td style="text-align: right">19/32</td>
        <td style="text-align: right">24/32</td>
        <td style="text-align: right"><strong>30/32</strong></td>
      </tr>
      <tr>
        <td>内容正确</td>
        <td style="text-align: right">25/32</td>
        <td style="text-align: right">24/32</td>
        <td style="text-align: right"><strong>30/32</strong></td>
      </tr>
      <tr>
        <td>格式合规</td>
        <td style="text-align: right">24/32</td>
        <td style="text-align: right">31/32</td>
        <td style="text-align: right"><strong>31/32</strong></td>
      </tr>
      <tr>
        <td>名字记忆</td>
        <td style="text-align: right">8/8</td>
        <td style="text-align: right">3/8</td>
        <td style="text-align: right"><strong>7/8</strong></td>
      </tr>
      <tr>
        <td>数值更新</td>
        <td style="text-align: right">2/8</td>
        <td style="text-align: right">7/8</td>
        <td style="text-align: right"><strong>7/8</strong></td>
      </tr>
      <tr>
        <td>偏好记忆</td>
        <td style="text-align: right">7/8</td>
        <td style="text-align: right">7/8</td>
        <td style="text-align: right"><strong>8/8</strong></td>
      </tr>
      <tr>
        <td>单轮算术</td>
        <td style="text-align: right">2/8</td>
        <td style="text-align: right">7/8</td>
        <td style="text-align: right"><strong>8/8</strong></td>
      </tr>
    </tbody>
  </table>
</div>
<p>最重要的现象是：<strong>v2 的 val loss 比 v1 更高，行为表现却显著更好。</strong> 这直接说明，不能只凭平均 token loss 判断后训练是否更好。</p>
<h3 id="3-六条冻结基线全部通过">3. 六条冻结基线全部通过</h3>
<p>v2 在最初六条基线上的结果为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">passed = 6/6
</span></span><span class="line"><span class="cl">answer_correct = 6/6
</span></span><span class="line"><span class="cl">format_compliant = 6/6
</span></span><span class="line"><span class="cl">copied_previous_assistant = 0
</span></span><span class="line"><span class="cl">natural_stop = 6/6
</span></span><span class="line"><span class="cl">empty = 0
</span></span></code></pre></div><p>真实回答为：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">名字：小吕。
</span></span><span class="line"><span class="cl">颜色：蓝色。
</span></span><span class="line"><span class="cl">苹果更新：30
</span></span><span class="line"><span class="cl">25 * 24：600
</span></span><span class="line"><span class="cl">123 + 789：912
</span></span><span class="line"><span class="cl">17 + 5：22
</span></span></code></pre></div><p>全部自然结束，没有精确复制旧回答。</p>
<h3 id="4-v2-仍有两个失败样例">4. v2 仍有两个失败样例</h3>
<p>32 条独立验证中仍有两条失败：</p>
<ul>
<li>数值更新第 01 条：期望 <code>7</code>，模型只生成 1 个 token 就 EOS，形成空回答；</li>
<li>名字记忆第 29 条：期望“小棉”，模型回答“你。”，生成 3 个 token 后停止。</li>
</ul>
<p>所以 v2 是当前最佳，不是全样本正确，也不是所有行为问题都已解决。</p>
<h2 id="九这一轮得到的结论">九、这一轮得到的结论</h2>
<h3 id="1-多轮-sft-首先是数据契约问题">1. 多轮 SFT 首先是数据契约问题</h3>
<p>不是把多轮文本拼起来就算完成。必须明确每轮角色、正文、结束标记和错位 target 的关系，否则模型可能在错误的文本上接受训练。</p>
<h3 id="2-格式行为通常比数值能力更容易学到">2. 格式行为通常比数值能力更容易学到</h3>
<p>v1、v2 都很快提升了“只输出数字”的格式遵循，但算术正确性并未同步提升，部分题目还发生回退。loss 下降不能直接解释为推理能力增强。</p>
<h3 id="3-全参数训练会同时产生收益与干扰">3. 全参数训练会同时产生收益与干扰</h3>
<p>v1 在数值更新和格式上明显收益，却损害了原本稳定的名字与偏好回答。后训练需要同时评估新任务收益和旧能力保持。</p>
<h3 id="4-行为评测必须独立于训练-loss">4. 行为评测必须独立于训练 loss</h3>
<p>本轮最关键的证据是：v2 的 val loss 更高，行为却更好。只优化 loss，无法发现这种新旧能力冲突。</p>
<h3 id="5-评测器本身也需要回归测试">5. 评测器本身也需要回归测试</h3>
<p>contains 误判、格式与正确性混淆、history 共享引用，都曾制造假结论。训练实验必须保留原始回答、评分规则和报告版本，不能只保存一个最终分数。</p>
<h2 id="十当前停止边界">十、当前停止边界</h2>
<p>当前只完成 I4-B 的受控多轮 SFT 闭环：</p>
<ul>
<li>v2 是本轮最佳 checkpoint；</li>
<li>不能据此宣称开放域多轮对话能力；</li>
<li>不能据此宣称 1 epoch 普遍优于 2 epochs；</li>
<li>尚未完成真实多轮语料、LoRA / 全参对照、CoT-SFT、DPO 或 GRPO；</li>
<li>名字和偏好类的核心实体包含规则仍是弱语义评测，后续需要更可靠的评估校准。</li>
</ul>
<p>下一步进入 I4-C 时，需要先想清楚：</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">&lt;think&gt; 中的推理过程是否参与 loss？
</span></span><span class="line"><span class="cl">如何分别定位思考区间与正式回答区间？
</span></span><span class="line"><span class="cl">thinking 与正文的 token 预算如何计算？
</span></span><span class="line"><span class="cl">如何判断思考质量，而不只是判断输出格式？
</span></span></code></pre></div><p>这一轮的主干已经搭起来了：<strong>数据契约、训练、checkpoint、独立行为评测、回退分析和干预对照。</strong> 后面新增 CoT、DPO 或其他方法，都应该在这套基线上继续做局部进化。</p>
<h2 id="主要实验产物">主要实验产物</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">src/squid/data/chat_sft.py
</span></span><span class="line"><span class="cl">scripts/i4_qwen_multiturn_template_probe.py
</span></span><span class="line"><span class="cl">scripts/i4_qwen_multiturn_span_probe.py
</span></span><span class="line"><span class="cl">scripts/i4_behavior_baseline.py
</span></span><span class="line"><span class="cl">scripts/i4_synthetic_data_audit.py
</span></span><span class="line"><span class="cl">scripts/i4_multiturn_sft_train.py
</span></span><span class="line"><span class="cl">scripts/i4_synthetic_behavior_eval.py
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">data/reports/i4-qwen3-0_6b-behavior-baseline-v3.json
</span></span><span class="line"><span class="cl">data/reports/i4-multiturn-sft-v1-training.json
</span></span><span class="line"><span class="cl">data/reports/i4-multiturn-sft-v2-1epoch-training.json
</span></span><span class="line"><span class="cl">data/reports/i4-synthetic-val-original.json
</span></span><span class="line"><span class="cl">data/reports/i4-multiturn-sft-v1-synthetic-val.json
</span></span><span class="line"><span class="cl">data/reports/i4-multiturn-sft-v2-1epoch-synthetic-val.json
</span></span><span class="line"><span class="cl">data/reports/i4-multiturn-sft-v2-1epoch-behavior-v3.json
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">data/model_pth/qwen3/i4-multiturn-sft-v1.pth
</span></span><span class="line"><span class="cl">data/model_pth/qwen3/i4-multiturn-sft-v2-1epoch.pth
</span></span></code></pre></div>]]></content:encoded>
    </item>
  </channel>
</rss>
