中國移動網站控件引發的藍屏問題分析
本周四(2月23日),我接到了我們同事的一個奇怪的藍屏case,據他回憶,他最近沒有安裝任何軟件和驅動,也沒有更改計算機的硬件配置,除了Windows后臺進行的自動更新之外,他實在想不起來到底對計算機有什么額外的改變。可是突然,就從前一天23日周三晚上起,他的計算機就開始藍屏,重啟之后,進系統之前就會藍屏,或者進了系統用不到一會兒也會藍屏。因此,他懷疑是硬件(如內存)故障導致的,或者是 Windows Update 導致的問題。
照說,例如內存條松動的這種突發硬件故障的確有可能引發藍屏,但是由于 Windows Update 推送的補丁缺陷導致的藍屏可實屬少見,在排查藍屏問題時,我們一般應該遵從默認信任微軟自身組件的原則。
據了解,他的藍屏一般有幾個隨機的錯誤代碼,查詢 Debugging Help 之后,得到的解釋如下:
藍屏代碼 (Bug Check) | 關鍵參數 (Parameter) | 說明 |
PFN_LIST_CORRUPT (0x4E) | This is typically caused by a driver passing a bad memory descriptor list. For example, the driver might have called MmUnlockPages twice with the same list. Stack trace examination is needed. | |
MEMORY_MANAGEMENT (0x1A) | P1: 0x41287 | Internal memory management structures are corrupted. To further investigate the cause, a kernel memory dump file is needed. |
NTFS_FILE_SYSTEM (0x24) | One possible cause of this bug check is disk corruption. Corruption in the NTFS file system or bad blocks (sectors) on the hard disk can induce this error. Corrupted SCSI and IDE drivers can also adversely affect the system's ability to read and write to disk, thus causing the error. Another possible cause is depletion of nonpaged pool memory. If the nonpaged pool memory is completely depleted, this error can stop the system. However, during the indexing process, if the amount of available nonpaged pool memory is very low, another kernel-mode driver requiring nonpaged pool memory can also trigger this error. | |
SYSTEM_SERVICE_EXCEPTION (0x3B) | This error has been linked to excessive paged pool usage and may occur due to user-mode graphics drivers crossing over and passing bad data to the kernel code. | |
SYSTEM_THREAD_EXCEPTION _NOT_HANDLED_M (1000007e) |
This indicates that a system thread generated an exception which the error handler did not catch. |
在以上錯誤中,前兩種出現的頻率最高。如果您百度一下以上錯誤,已經有足夠的理由開始拆下內存條,擦拭金手指了。但是我個人認為,這一定不是一個硬件產生的錯誤。在我看來,這樣的錯誤看似隨機,其實應該有一種共性的可能性——系統中存在一個寫的很爛的驅動。為什么這么講呢,我們可以從查到的描述中看見"bad""depletion""nonpaged pool"出現的頻率很高;另外值得注意的是,對于0x24 NTFS文件系統的 bug check,在很多時候容易產生磁盤損壞的誤導,殊不知,它還有一種可能就是非換頁池耗盡,如上表中我加粗了的部分。
對于如此隨機的錯誤,我們往往是無法通過分析棧去找到兇手的。例如,我在這里給出一個棧的示例:
- MEMORY_MANAGEMENT (1a)
- # Any other values for parameter 1 must be individually examined.
- Arguments:
- Arg1: 0000000000041287, The subtype of the bugcheck.
- Arg2: 0000000000000030
- Arg3: 0000000000000000
- Arg4: 0000000000000000
- Debugging Details:
- ------------------
- BUGCHECK_STR: 0x1a_41287
- DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
- PROCESS_NAME: WmiPrvSE.exe
- CURRENT_IRQL: 0
- TRAP_FRAME: fffff88007e6d6e0 -- (.trap 0xfffff88007e6d6e0)
- NOTE: The trap frame does not contain all registers.
- Some register values may be zeroed or incorrect.
- STACK_TEXT:
- fffff880`07e6d578 fffff800`02c62d7e : 00000000`0000001a 00000000`00041287 00000000`00000030 00000000`00000000 : nt!KeBugCheckEx
- fffff880`07e6d580 fffff800`02ccdd6e : 00000000`00000000 00000000`00000030 00000000`00000000 00000000`fffffa80 : nt! ?? ::FNODOBFM::`string'+0x46485
- fffff880`07e6d6e0 fffff800`02dadbc5 : 00000000`000af94a 00000000`00000000 ffffffff`ffffffff 00000000`01464000 : nt!KiPageFault+0x16e
- fffff880`07e6d870 fffff800`02d426b0 : fffffa80`098d5058 fffff6fd`4004c6a8 fffff800`02f055c0 fffff880`07e6db11 : nt!MiResolvePageFileFault+0x1115
- fffff880`07e6d9b0 fffff800`02cdea07 : 00000000`00000000 00000000`01440004 00000000`0240f3c4 fffff800`00000000 : nt! ?? ::FNODOBFM::`string'+0x399d4
- fffff880`07e6dac0 fffff800`02ccdd6e : 00000000`00000001 00000000`01440004 00000000`023ae701 00000000`00000460 : nt!MmAccessFault+0x1e47
- fffff880`07e6dc20 00000000`76b87222 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x16e
- 00000000`0240f394 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x76b87222
可以看見,除了ntkrnlmp.exe里面的函數,最前面發生的0x76b87222根本無法解析出來。dds 命令也是不能夠解析出具體名稱的。
那么,究竟該怎么樣才能找到問題的元兇呢?
其實,將隨機的藍屏錯誤通過啟用特殊池來轉化為明顯的錯誤是比較好的選擇。對于特殊池(special pool)的概念,我并不是第一次介紹了,關于這個神奇的特殊內存區域的調試方法,請參見我早些時候的文章《啟用特殊池解讀 0x000000c5 藍屏》,或者其英文版《Enable "Special Pool" to Interpret 0x000000c5 Blue Screen》。
從安全模式啟動系統,啟動 verifier,配置啟用 special pool. 當然,安全模式下,可能引發問題的驅動也許并未加載,因此,我們最好選擇"從一個列表選擇驅動程序名",然后繼續選擇"將目前沒有加載的驅動程序添加到列表中…",在彈出的選擇文件對話框中,瀏覽 %systemroot%\system32\derivers , 然后增加"版權"和"產品名稱"兩列文件屬性,并按照它們排序。選擇所有不是微軟的程序驅動,或者選擇沒有數字簽名/版權和產品信息不完整的看似不專業的驅動,添加進來選中應用特殊池即可。

這里需要說明一下,其實特殊池的設置保存在注冊表之中,具體是在內存管理器的分支里:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
它靠 DWord 值 VerifyDriverLevel 和 String 值 VerifyDrivers 控制。有興趣大家可以去窺探一下 J
啟用特殊池之后,我們就重啟計算機,正常進入系統嘗試 repro 這個問題。沒一會兒,還沒登錄果然就又藍了。這回直接進入安全模式,獲得內存轉儲文件進行分析:
首先我們可以看見,特殊池生效了,而且成功進行了內存池分配:
- 4: kd> !verifier
- Verify Level 1 ... enabled options are:
- Special pool
- Summary of All Verifier Statistics
- RaiseIrqls 0x0
- AcquireSpinLocks 0x0
- Synch Executions 0x0
- Trims 0x0
- Pool Allocations Attempted 0x2
- Pool Allocations Succeeded 0x2
- Pool Allocations Succeeded SpecialPool 0x2
- Pool Allocations With NO TAG 0x0
- Pool Allocations Failed 0x0
- Resource Allocations Failed Deliberately 0x0
- Current paged pool allocations 0x0 for 00000000 bytes
- Peak paged pool allocations 0x0 for 00000000 bytes
- Current nonpaged pool allocations 0x0 for 00000000 bytes
- Peak nonpaged pool allocations 0x0 for 00000000 bytes
然后,我們可以直接看到問題驅動究竟是誰了:
- DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
- A device driver attempting to corrupt the system has been caught. This is
- because the driver was specified in the registry as being suspect (by the
- administrator) and the kernel has enabled substantial checking of this driver.
- If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
- be among the most commonly seen crashes.
- Arguments:
- Arg1: 00000000000000b2, MmMapLockedPages called on an MDL having incorrect flags.
- For example, calling MmMapLockedPages for an MDL
- that is already mapped to a system address is incorrect.
- Arg2: fffffa800a4e71b0, MDL address.
- Arg3: 0000000000000005, MDL flags.
- Arg4: 0000000000000005, Incorrect MDL flags.
- Debugging Details:
- ------------------
- BUGCHECK_STR: 0xc4_b2
- DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
- PROCESS_NAME: System
- CURRENT_IRQL: 0
- LAST_CONTROL_TRANSFER: from fffff8000311f3dc to fffff80002c95c40
- STACK_TEXT:
- fffff880`033697e8 fffff800`0311f3dc : 00000000`000000c4 00000000`000000b2 fffffa80`0a4e71b0 00000000`00000005 : nt!KeBugCheckEx
- fffff880`033697f0 fffff800`0311ffb3 : fffff880`05926f60 fffff880`05926f60 fffffa80`069ce700 fffff800`0312e09a : nt!VerifierBugCheckIfAppropriate+0x3c
- fffff880`03369830 fffff800`031327bb : fffffa80`0a4e71b0 fffffa80`09b69000 fffffa80`0a4e71b0 fffff880`05926f60 : nt!ViMmMapLockedPagesSanityChecks+0xa3
- fffff880`03369870 fffff880`06220009 : fffffa80`0a4e72c0 ffffffff`8000069c fffffa80`0a4e72c0 00000000`00000000 : nt!VerifierMmMapLockedPages+0x1b
- fffff880`033698b0 fffff880`0624c93a : fffff880`03369970 fffff880`05926f60 fffffa80`00000032 00000000`0000001c : PassGuard_x64!distorm_version+0x6809
- fffff880`033698f0 fffff880`03369970 : fffff880`05926f60 fffffa80`00000032 00000000`0000001c fffffa80`06768f30 : PassGuard_x64!distorm_version+0x3313a
- fffff880`033698f8 fffff880`05926f5f : fffffa80`00000032 00000000`0000001c fffffa80`06768f30 00000000`00000200 : 0xfffff880`03369970
- fffff880`03369900 fffffa80`00000032 : 00000000`0000001c fffffa80`06768f30 00000000`00000200 00000000`00000000 : usbhub!UsbhSyncSendCommand+0x327
- fffff880`03369908 00000000`0000001c : fffffa80`06768f30 00000000`00000200 00000000`00000000 fffff880`06232040 : 0xfffffa80`00000032
- fffff880`03369910 fffffa80`06768f30 : 00000000`00000200 00000000`00000000 fffff880`06232040 00000000`001e001c : 0x1c
- fffff880`03369918 00000000`00000200 : 00000000`00000000 fffff880`06232040 00000000`001e001c fffff880`062563f8 : 0xfffffa80`06768f30
- fffff880`03369920 00000000`00000000 : fffff880`06232040 00000000`001e001c fffff880`062563f8 00000000`00220020 : 0x200
- STACK_COMMAND: kb
- FOLLOWUP_IP:
- PassGuard_x64!distorm_version+6809
- fffff880`06220009 4889442428 mov qword ptr [rsp+28h],rax
- SYMBOL_STACK_INDEX: 4
- SYMBOL_NAME: PassGuard_x64!distorm_version+6809
- FOLLOWUP_NAME: MachineOwner
- MODULE_NAME: PassGuard_x64
- IMAGE_NAME: PassGuard_x64.sys
- DEBUG_FLR_IMAGE_TIMESTAMP: 4e2fb9f4
- FAILURE_BUCKET_ID: X64_0xc4_b2_VRF_PassGuard_x64!distorm_version+6809
- BUCKET_ID: X64_0xc4_b2_VRF_PassGuard_x64!distorm_version+6809
- Followup: MachineOwner
- ---------
- 4: kd> lmvm PassGuard_x64
- start end module name
- fffff880`06218000 fffff880`06261000 PassGuard_x64 (export symbols) PassGuard_x64.sys
- Loaded symbol image file: PassGuard_x64.sys
- Image path: \??\C:\windows\system32\drivers\PassGuard_x64.sys
- Image name: PassGuard_x64.sys
- Timestamp: Wed Jul 27 15:10:44 2011 (4E2FB9F4)
- CheckSum: 0004A5F0
- ImageSize: 00049000
- File version: 1.0.0.6
- Product version: 1.0.0.6
- File flags: 0 (Mask 17)
- File OS: 4 Unknown Win32
- File type: 1.0 App
- File date: 00000000.00000000
- Translations: 0804.04b0
- ProductName: SysEnter Application
- InternalName: SysEnter
- OriginalFilename: SysEnter.exe
- ProductVersion: 1, 0, 0, 6
- FileVersion: 1, 0, 0, 6
- FileDescription: SysEnter Application
- LegalCopyright: Copyright (C) 2011
好了,知道了這個叫 PassGuard_x64.sys 的驅動是罪魁禍首之后,那我們就該移除它的啟動加載了。直接在安全模式打開注冊表編輯器,刪除HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services 下面的 PassGuard 整個鍵,當然,你還需要找到 ControlSet001 / 002 下面的同樣的鍵刪除。這里,我順便把這個 PassGuard 鍵的內容展示出來:
- [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PassGuard]
- "Type"=dword:00000001
- "Start"=dword:00000002
- "ErrorControl"=dword:00000001
- "ImagePath"=hex(2):5c,00,3f,00,3f,00,5c,00,43,00,3a,00,5c,00,77,00,69,00,6e,00,\
- 64,00,6f,00,77,00,73,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,\
- 00,5c,00,64,00,72,00,69,00,76,00,65,00,72,00,73,00,5c,00,50,00,61,00,73,00,\
- 73,00,47,00,75,00,61,00,72,00,64,00,5f,00,78,00,36,00,34,00,2e,00,73,00,79,\
- 00,73,00,00,00
- "DisplayName"="PassGuard"
- "WOW64"=dword:00000001
- [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PassGuard\Enum]
- "0"="Root\\LEGACY_PASSGUARD\\0000"
- "Count"=dword:00000001
- "NextInstance"=dword:00000001
刪除這個鍵,如果沒有其他的多重問題,那么系統就能正常運行了。在重啟進入正常模式之前,記得刪除 verifier 中的特殊池配置。
為了進一步刪除該驅動所關聯的程序或者其他文件(如果有的話),我請這位同事仔細回憶這是什么。于是,我叫他回憶一下任何可能的情況,例如IE插件的變化、某些程序里面的捆綁、惡意程序等等。結果,他想起周三下午有去中國移動的 10086.cn 充值話費,安裝了一個安全控件。
我們打開IE加載項管理器,選擇所有加載項,果然發現了這一個:

還好不是惡意軟件,而且可以通過下面的 remove 按鈕刪除關聯的控件文件。好了,不多說了,看看中國移動為64位操作系統寫的驅動有多爛你就明白了。
從這個案例中,我想我們看到的不僅僅是特殊池的使用方法和排錯思路,而且還應看到現在很多信息服務公司所面臨的一個問題,自己的產品的驅動外包出去,而且承包開發驅動的公司沒有足夠的驅動撰寫經驗和規范,或者沒有經過測試就投入使用,受損的不僅是客戶,更是這個服務公司自身的品牌。這樣的例子太多了,延伸起來不僅有 badly written driver, 還有 badly written software, badly written website… 鐵道部12306網站說多了就沒意思了,說句實話,現在中國聯通在營業廳推行的銀行卡繳費機我真是不敢用。想想要把銀行卡插進聯通外包商開發的機器中去,還要輸入銀行卡密碼我就覺得膽戰心驚.
原文:http://www.cnblogs.com/mvperic/archive/2012/02/26/2369225.html
【編輯推薦】