Stencil API is still weird

Issue #1156 open
David Frank
created an issue

2 things I think should be more obvious:

  • calling love.graphics.setStencilTest() currently disables the love.graphics.stencil(), to me a more verbose setStencilShape, setStencilRule, setStencilTest seem more nature and fits the current API design better (the .push, do things, .pop mentality)

  • love.graphics.stencil takes a stencil function, yet it's unclear how one would write a function that accepts additional parameters without resorting to either (a) create a new function on each run, or (b) create some variables outside of stencil function scope so that it can change.

local ox, oy

local function rect()
  love.graphics.rectangle('fill', ox, oy, 200, 200)
end

local function example1(x, y)
  ox = x
  oy = y
  love.graphics.stencil(rect, 'replace', 1)
  -- alternative
  -- love.graphics.stencil(function() love.graphics.rectangle('fill', x, y, 200, 200) end, 'replace', 1)
  love.graphics.setStencilTest('greater', 0)
  -- do things
  love.graphics.setStencilTest()
end

vs

local function example2(x, y)
  love.graphics.enableStencil()
  love.graphics.setStencilShape('rectangle', 'fill', x, y, 200, 200)
  love.graphics.setStencilRule('replace', 1)
  love.graphics.setStencilTest('greater', 0)
  -- do things
  love.graphics.disableStencil()
end

Comments (12)

  1. Gabe Stilez

    If this is doable, I'd only offer one modification: name setStencilShape setStencilFunction instead, with a signature of (function, ...) so it can get any additional parameters as well; otherwise people can't use such special masks like "bubbled spaghetti 543 dot png" instead of simple geometric shapes.

  2. Alex Szpakowski

    love.graphics.stencil causes anything drawn inside the passed function to be drawn to the stencil buffer. love.graphics.setStencilTest() doesn't disable that since there's nothing to disable. setStencilTest causes what's drawn to be affected by the existing contents of the stencil buffer.

    love.graphics.stencil takes a stencil function, yet it's unclear how one would write a function that accepts additional parameters without resorting to either (a) create a new function on each run, or (b) create some variables outside of stencil function scope so that it can change.

    Using an anonymous function is fairly idiomatic Lua: love.graphics.stencil(function() mything:drawStencil() end)

  3. David Frank reporter

    Thx Alex! I did include both examples for current API, using local variable or anonymous function (commented out).

    My problem with using anonymous function: it means function creation on love.draw() loop. It's not much penalty to modern hardware but I think many programmers will have a natural instinct to avoid that.

    I am using the local variable pattern myself with a project on LOVE v0.10.

    As for setStencilTest, I just feel it's weird to think along this line: "if you pass it a nil, then the stencil buffer you previously set won't be used." This may cause further confusion, like "can I reuse the stencil buffer I set last time?" or "do I need to reset stencil with each test?" (I am guessing calling setStencilTest with nil resets the stencil, but I maybe totally wrong.)

    And the disables wording I use is actually from wiki:

    https://love2d.org/wiki/love.graphics.setStencilTest

    And yeah setStencilFunction is more aptly named, cheers Gabe!

  4. Alex Szpakowski

    I am guessing calling setStencilTest with nil resets the stencil, but I maybe totally wrong

    Calling love.graphics.stencil with the keepvalues parameter set to false (the default) clears the stencil buffer before the function parameter is called. The stencil buffer for the active Canvas is also cleared when love.graphics.clear is called.

    And the disables wording I use is actually from wiki

    The stencil test is disabled, but the stencil values associated with each pixel are left untouched. They're just no longer used until setStencilTest is called again.

  5. David Frank reporter

    One last question: is stencil affected by love.graphics.push('all')?

    If so then it's no different from setScissor, and I will rest my case on setStencilTest.

    I do feel setStencilFunction and setStencilRule might be a good convenience.

  6. Alex Szpakowski

    One last question: is stencil affected by love.graphics.push('all')?

    love.graphics.stencil isn't, love.graphics.setStencilTest is. stencil() is in the drawing group of functions and setStencilTest is in the state-changing group.

  7. David Frank reporter

    Ran into another case where setStencilFunction would be desirable, say you are trying to do OOP in Lua, and created a DrawBox class, but you want to have different stencil function for each DrawBox instance, the only solution would be anonymous function approach, eg. function() self:fn() end, because local variable solution doesn't allow self (or at least tricky to do?)

    Again, probably no big deal on modern hardware, but feels weird to do.

  8. Pablo Mayobre

    My issue is that there is no way to get the already set stencil function, so for example if I have many components that uses stencil and have one inside another, I can't get the parent stencil, set the child stencil, and then set back the parent stencil, I'm aware I could keepvalues and use a different value for the child stencil but again how does the parent set back the values? In UI code all this data is unknown so there should be something like a push and pop for the stencil buffer

  9. Pablo Mayobre

    Yes, that's what I mean, there is no way to tell what is in the buffer, or somehow reset it to what it was before. Once overriden there is no going back.

    This is a problem as I said when you have a function that sets the buffer, then calls another function that uses the buffer for other stuff and when the function return and you are back to the first function, that function may be specting the buffer to be set as it was before calling the second function.

  10. Alex Szpakowski

    I think you're misunderstanding what the stencil buffer is – it's basically just another aspect of the screen or Canvas, like the pixel colors and the depth values (although LÖVE doesn't use depth).

    love.graphics.stencil effectively re-routes rendering to affect the stencil values of pixels on the screen, instead of affecting the color values.

    Once you draw to any part of the screen, there's no going back unless you clear and re-render – regardless of whether you're drawing to the stencil buffer or the color buffer.

    That said, since the stencil buffer has 255 possible values, and the stencil test can check if a value is greater than a certain value, you can have each UI layer increment the stencil value if you want. Then you can have nesting via stencils.

  11. Log in to comment