From your link: "Although related (due to being used for the same purposes), transparent colors and alpha channels are techniques which do not involve the image pixel mixage by binary masking."
Games made after the mid-90's typically utilize GPU blending coupled with an alpha channel, rather than bitwise operations. LÖVE's love.graphics.setBlendMode function exposes some of this functionality. Is there a specific effect you're trying to achieve that can't be done with the current blend modes? Maybe a new blend mode could be added, if the effect can be done that way.
On second thought, I don't think any blend mode would accommodate what I'm thinking of. With transparency effect, it's possible to draw an image with transparency at the "top" so you can see things behind you.
I'm imagining this more like a gingerbread cookie mold. The mold would be a black and white image mask. There can be a function where you set the mask on and all subsequent drawing operations would only show up on the white portion of the image mask.
One good example I can think of is using a black and white text as an image mask. You can apply fancy text texture to it by drawing stuff afterwards.
Also, intermediate gray values between 0 and 255 would affect how transparent things can be when applying drawing operations on the mask. There could be a second argument for newImageMask where you specify a specific channel to check for transparency instead of luminance only.