HLSL 4 – Calls to user defined functions will crash

In one of my current projects I have been confronted with a weird problem of the Effect framework: A syntactically and semantically valid shader would not render anything. But let’s start at the beginning.
The application I have been working on is a DirectX 11 program written in C++ that uses HLSL 4.0 shaders. My testing device is an Intel Onboard GPU. Suppose we have a simple pixel shader myPS() like this:

float4 getColor()
{
return float4(1,1,0,1);
}

float4 myPS(in GSOut i) : SV_TARGET0
{
return getColor();
}

It is obvious that the shader should return a yellow color for all pixels on the first render target. Well, it should. In my case, the shader did not return anything. The pixels were just discarded. Integrating the function call within the pixel shader leads to the desired result of all pixels being yellow:

float4 myPS(in GSOut i) : SV_TARGET0
{
return float4(1,1,0,1);
}

HLSL functions are inline by default, so the compiler itself should integrate function calls. Actually, he did. But not in the way I wanted. Let’s have a look at the compiled shader with the integrated function call:

ps_4_0
dcl_output o0.xyzw
mov o0.xyzw, l(1,1,0,1)
ret

Looks fine. This is, how the shader should look like. The output is declared, filled with the yellow color and returned.
But the compiled shader without the integrated function (first version) looks a bit different:

ps_4_0
dcl_output o0.xyzw
nop
mov o0.xyzw, l(1,1,0,1)
ret

As you see, there is a nop instruction. I don’t know why, but this instruction is probably the one causing the pixels to be discarded. I suppose it is included for compatibility reasons regarding branching.
Now, how can the compiler be forced to remove the nop instuction? The solution is quite simple. I discovered that two compile flags for the shader cause this behaviour: The D3DCOMPILE_DEBUG and D3DCOMPILE_SKIP_OPTIMIZATION. When they are set at the same time, the shader won’t draw anything. So I just disabled one of them and everything was perfect.
If you have any other ideas on this topic feel free to post comments.

, , ,

  1. #1 by Anonymous on November 22, 2012 - 6:25 am

    Thanks for this post, it significantly helped me with the same problem.

  2. #2 by Anonymous on January 15, 2013 - 8:36 am

    I had a different problem, which apparently had the same cause: In my shader, I was testing buffer values with isnan(). If I used a user defined function to retrieve the buffered value, isnan() would always return false on the retrieved value, but if I retrieved the value directly in the shader, I would get the correct result.

  3. #3 by sergioko on June 23, 2014 - 2:21 am

    Thanks! I’m not mad, I’m not mad!

  4. #4 by Huy Dương on October 26, 2015 - 11:22 am

    OMG! Thank for this post 🙂 I have the same problem. And After read this post I can resolve it 🙂 Thanks…

Leave a comment