MetaMath：新數學推理語言模型，訓練大模型的逆向思維

作者：機器之心 2023-10-11 12:32:53

為了提升模型的正向和逆向推理能力，劍橋、港科大、華為的研究者基于兩個常用的數學數據集（GSM8K 和 MATH）提出了 MetaMathQA 數據集：一個覆蓋面廣、質量高的數學推理數據集。MetaMathQA 由 395K 個大語言模型生成的正向逆向數學問答對組成。

復雜數學推理是評價大語言模型推理能力的重要指標，目前常用的數學推理數據集樣本量有限且問題多樣性不足，導致大語言模型存在 [逆轉詛咒] 的現象，即一個訓練于「A 是 B」的語言模型無法推廣到「B 是 A」[1]。此現象在數學推理任務中的具體形式是：即給定一個數學問題，語言模型擅于用正向推理解答問題但缺乏逆向推理解決問題的能力。逆向推理在數學問題中十分常見，如下 2 個例子。

1. 經典問題 - 雞兔同籠

正向推理：籠子里有 23 只雞和 12 只兔，問籠子里有多少個頭和多少只腳？
逆向推理：有若干只雞兔同在一個籠子里，從上面數，有 35 個頭，從下面數，有 94 只腳。問籠中各有多少只雞和兔？

2. GSM8K 問題

正向推理: James buys 5 packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. How much did he pay?
逆向推理: James buys x packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. How much did he pay? If we know the answer to the above question is 110, what is the value of unknown variable x?

為了提升模型的正向和逆向推理能力，劍橋、港科大、華為的研究者基于兩個常用的數學數據集（GSM8K 和 MATH）提出了 MetaMathQA 數據集：一個覆蓋面廣、質量高的數學推理數據集。MetaMathQA 由 395K 個大語言模型生成的正向逆向數學問答對組成。他們在 MetaMathQA 數據集上基于 LLaMA-2 微調得到專注于數學推理 (正向和逆向) 的大語言模型 MetaMath，在數學推理數據集上達到了 SOTA。MetaMathQA 數據集和不同規模的 MetaMath 模型已開源供研究人員使用。

項目地址：https://meta-math.github.io/
論文地址：https://arxiv.org/abs/2309.12284
數據地址：https://huggingface.co/datasets/meta-math/MetaMathQA
模型地址：https://huggingface.co/meta-math
代碼地址：https://github.com/meta-math/MetaMath

本文中構造了一個逆向推理數據集 GSM8K-Backward。實驗結果表明，目前的方法（e.g., SFT、RFT、WizardMath）在逆向推理問題中表現遠差于正向推理。相較而言，MetaMath 模型在正向和反向推理上都取得了優越的性能。

方法

MetaMathQA 包含四種數據增強的方法：

1. Answer Augmentation (答案增強)：給定問題，通過大語言模型生成能得到正確結果的思維鏈作為數據增廣。

Question: James buys 5 packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. How much did he pay?
Answer: James buys 5 packs of beef that are 4 pounds each, so he buys a total of 5 * 4 = 20 pounds of beef. The price of beef is $5.50 per pound, so he pays 20 * $5.50 = $110. The answer is: 110.

2. Rephrasing Question (問題改寫增強)：給定元問題，通過大語言模型重寫問題并生成得到正確結果的思維鏈作為數據增廣。

Question: What is the total amount that James paid when he purchased 5 packs of beef, each weighing 4 pounds, at a price of $5.50 per pound?
Answer: Each pack of beef weighs 4 pounds, so 5 packs weigh 4 * 5 = 20 pounds in total. The price per pound of beef is $5.50, so the total cost for 20 pounds is 20 * $5.50 = $110. Therefore, James paid a total of $110. The answer is: 110.

3. FOBAR Question (FOBAR 逆向問題增強)：給定元問題，通過掩碼條件中的數字為 x，給定原有答案并反推 x 來產生逆向問題，并基于該逆向問題生成正確的思維鏈過程來進行數據增廣（反推樣例：“If we know the answer to the above question is 110, what is the value of unknown variable x?”）。

Question: James buys x packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. How much did he pay? If we know the answer to the above question is 110, what is the value of unknown variable x?
Answer: James buys x packs of beef that are 4 pounds each, so he buys a total of 4x pounds of beef. The price of beef is $5.50 per pound, so the total cost of the beef is 5.50 * 4x = 22x. We are given that the total cost is $110, so we can write: 22x = 110. Dividing both sides by 22, we get: x = 5. The value of x is 5.

4. Self-Verification Question (Self-Verification 逆向問題增強)：在 FOBAR 的基礎上，通過大語言模型改寫逆向問題部分為陳述的語句來進行數據增廣（改寫樣例：“How much did he pay?” (with the answer 110) 被改寫成 “He paid 110”）。

Question: James buys x packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. He paid 110. What is the value of unknown variable x?
Answer: To solve this problem, we need to determine the value of x, which represents the number of packs of beef that James bought. Each pack of beef weighs 4 pounds and costs $5.50 per pound. The total amount James paid is $110. We can set up the equation as follows: Number of packs of beef * Weight per pack * Price per pound = Total amount paid; x * 4 * $5.50 = $110; Let's simplify and solve for x: 4x * $5.50 = $110; 22x = $110; To isolate x, we divide both sides of the equation by 22: 22x / 22 = $110 / 22; x = $5; The value of x is 5.

實驗結果

在兩個常見數學推理數據集（GSM8K 和 MATH）的實驗結果表明，在不借助外部工具（e.g., code interpreter）, MetaMath 在性能上顯著優于已有開源 LLM 模型。其中，我們的 MetaMath-7B 模型在 GSM8K 上達到了 66.5% 的準確率，在 MATH 上達到了 19.8% 的準確率，超過了相同規模的最先進模型分別 11.6% 和 9.1%。特別地，MetaMath-70B 在 GSM8K 上達到了 82.3% 的準確率，超過 GPT-3.5-Turbo。

根據 Superficial Alignment Hypothesis [2]，大語言模型的能力源于預訓練，而來自下游任務的數據則激活了預訓練期間學習到的語言模型的固有能力。由此產生了兩個重要問題：（i）什么類型的數據可以最有效地激活潛在知識，以及（ii）為什么一個數據集在這種激活中比另一個數據集更好？

為什么 MetaMathQA 有用？提高了思維鏈數據的質量 (Perplexity)