Currently with DX11 if the vertex-pixel shader input/output signatures don't match exactly, no warnings or errors are given, just garbage output. This patch checks the signatures and generates an exception with a message naming the offending shaders if they aren't compatible.
Because this is done in _render it is only done in OGRE_DEBUG_MODE to avoid impacting performance.
I might be making a huge mistake, and if so please correct me if im doing sth wrong. but this commit produces error on dx11 domain to geometry shader. Stuff that used to work before produces this error. I dont know if this is directly related to ds->gs, but isnt it possible that the ds output has a different signature then the gs input? or for that matter vs->gs?
What does the error message say? My understanding is that the outputs from one shader stage need to match the inputs to the next stage. If the signatures do not match, there will be no warning or error from DX11, the next shader will just silently get garbage inputs.
In this case, it looks like your domain shader is outputting just "SV_Position", while the geometry shader is expecting "SV_Position" plus "SV_PrimitiveID" and "SV_GSInstanceID". I haven't tested this code on anything other than vertex and fragment shaders, so its possible that special cases are needed to handle other kinds of shaders.
Thanks for the fast replay :)
the error i get is from : "Shader " + progA->getName() + " produces not enough output parameters for shader " + progB->getName(), + "D3D11RenderSystem::validateShaderSignatures" ); from my ds to my gs.
As far as I know, both SV_GSInstanceID and SV_PrimitiveID are only used as inputs. they cant be modified. more so SV_GSInstanceID can only be used by the gs.
So it looks like the current shader signature validation isn't robust enough to deal with some of these special SV semantics like SV_PrimitiveID that are passed directly to a stage in the middle of the pipeline. I'll see about fixing the validation to handle these cases. I'll make a pull request when I have something.
Let me know if I can help.
I discovered that a far better way to go is simply to turn on the debug layer, and it will handle the shader signature validation. I'm going to back out this change.