Rick's Game Stuff

So PIX is broken in Windows 8 for D3D11 (1, 2). I like PIX. I don’t like Visual Studio 2012, and can’t be bothered to install it alongside Visual Studio 2010 just for the graphics debugger.  So I wanted to try and see if I could find out why and possibly fix it.

When running the June 2010 SDK sample SimpleSample11 in PIX, there’s a nice crash in CTexture2D::Map in d3d11.dll.

Here’s the callstack of the sample running with PIX:
[SimpleSample11.exe] wWinMain
[SimpleSample11.exe] DXUTCreateDevice
[SimpleSample11.exe] DXUTChangeDevice
[SimpleSample11.exe] DXUTCreate3DEnvironment11
[SimpleSample11.exe] DXUTUpdateBackBufferDesc
[PIXHelper.dll] CSpyHookedID3D10Texture2D::Map
[PIXHelper.dll] CHookedID3D10Texture3D::Map
[d3d11.dll] CTexture2D::Map (on CLayeredObjectWithCLS<class CTexture2D>::CContainedObject)

This didn’t jump out at me until much further in my research, and I felt pretty stupid when I realized it.

Here’s the callstack of the sample running without PIX:
[SimpleSample11.exe] wWinMain
[SimpleSample11.exe] DXUTCreateDevice
[SimpleSample11.exe] DXUTChangeDevice
[SimpleSample11.exe] DXUTCreate3DEnvironment11
[SimpleSample11.exe] DXUTUpdateBackBufferDesc
[d3d11.dll] CTexture2D::GetDesc (on CLayeredObjectWithCLS<class CTexture2D>::CContainedObject{for `CDeviceChild<struct ID3D11Texture2D>'})

This is where I finally noticed that for some reason PIXHelper.dll is hooking using its D3D10 classes. Which didn’t sound right.

Looking at the MSDN documentation for ID3D10Texture2D and ID3D11Texture2D shows us Map and Unmap got removed, leaving GetDesc in D3D11 where Map would be in D3D10. Clearly the wrong function (and vftable) is being used by PIX when hooking.

I tracked down CHookedID3D10Texture2D being created in CHookMgr::CreateHookedObject, which is called by CHookedIDXGISurface::Init.

When boiled down past specifics, CHookedIDXGISurface::Init basically does this:
if (surface->QueryInterface(IDXGI10Texture2D)) return new CHookedID3D10Texture2D();
if (surface->QueryInterface(IDXGI11Texture2D)) return new CHookedID3D11Texture2D();


Querying a D3D10 interface from a D3D11 implementation works just fine, so it’s passing off an ID3D10Texture2D when things expect ID3D11Texture2D, which expectedly causes everything to explode.

Flipping this logic in PIXHelper.dll and PIXWin.exe appears to fix this crash, though I’d assume D3D10 usage would crash now instead, but that’s OK.

Here are some patches:

PIXWin.exe (32-bit)

original md5: f5aa5e6b49e00a8e66e50984166d0e8c
modified md5: 18fd25c337b3f9f5fb62fa058bd5fc37
offset 0x1131F1: D8 91 -> 88 8A
offset 0x11320D: 2C -> 54
offset 0x113226: 39 -> 69
offset 0x11324E: 88 8A -> D8 91
offset 0x11326A: 54 -> 2C
offset 0x113279: 69 -> 39


PIXHelper.dll (32-bit)

original md5: 01231c7f3860566809304c8c74bc7d00
modified md5: 4dceceafc593092d9dcf450ec0a9fc93
offset 0xF4462: 48 13 -> E8 0B
offset 0xF447E: 2C -> 54
offset 0xF4497: 39 -> 69
offset 0xF44BF: E8 0B -> 48 13
offset 0xF44DB: 54 -> 2C
offset 0xF44EA: 69 -> 39


PIXWin.exe (64-bit)

original md5: 24d27b524d058262c3d32fdac97b563f
modified md5: fc7084fc1b9732192defce9e5d2d5480
offset 0x17AE33: 49 5A -> 39 5C
offset 0x17AE5C: 2C -> 54
offset 0x17AE7B: 39 -> 69
offset 0x17AE89: E3 5B -> F3 59
offset 0x17AEB2: 54 -> 2C
offset 0x17AEC7: 69 -> 39


PIXHelper.dll (64-bit)

original md5: d4bac87c6c9b9a26a022990bd8563000
modified md5: 8ccdf0fccfa3dc13b961be0a24cff8d1
offset 0x152967: 75 06 -> 65 08
offset 0x152990: 2C -> 54
offset 0x1529AF: 39 -> 69
offset 0x1529BD: 0F 08 -> 1F 06
offset 0x1529E6: 54 -> 2C
offset 0x1529FB: 69 -> 39


Tags: Hacking, PIX

This entry was posted on Monday, October 14th, 2013 at 1:22 am and is filed under Direct3D. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.