3DFX: 2x Anti-Aliasing modes on VSA-100 cause hard locks with LFB WC [WORKAROUND: set FX_GLIDE_FORCE_OLD_AA=1]

[Frank Sapone]

Setting 2xAA modes via SSTH3_SLI_AA_CONFIGURATION=1 (2xAA with no SLI) or SSTH3_SLI_AA_CONFIGURATION=3 (2xAA with SLI) will cause hard locks if LFB WC is set via a utility like fastvid or mtrrlfbe, or a soft reboot from entering Windows (Windows will set MTRRs).

4xAA modes via SSTH3_SLI_AA_CONFIGURATION=4 work fine, but the card can only really handle 4xAA at 640x480 and maybe 800x600.

This is sort of a minor issue since 2xAA really fudges up the text and makes the UI very blurry, but the 3D game looks nice.

Issue is not unique to SSE, Sage, Mesa, -ffast-math, or GCC3 vs GCC4.

Possible causes: 2xAA becoming broken in the glide3x dev branch, later GCC builds are overzealous with optimizations, or some sort of DJGPP/DOS specific code somewhere fudging something up as 2xAA works OK in Windows with Q2DOS compiled for Windows.

[Frank Sapone]

This issue is indifferent to Mesa or Sage; so it may be a glide3x DJGPP specific code path issue.

[Frank Sapone]

I'll probably have to check the gl debug output log from quake 2 and see if it's always hard locking during a certain gl_enable/gl_disable mode and trace that back if it's always the same. Afaik it always locks up at the same points. Rendering the 2D menu is OK (though fuzzy, because that's how 3dfx is setup).

[Frank Sapone]

Edited title and description now that is isolated to 2xAA with LFB WC, and is not unique to SSE, Sage, Mesa, -ffast-math or GCC3 vs GCC4.

[Frank Sapone]

I also tested 3 separate V5 5500 cards on a P3 800 (overclocked to 900 and stock) and a P3 550 (stock) and happens on all configurations. So we can rule out the possibility of a bad VSA-100 chip that is flaky with 2xAA.

[Frank Sapone]

I tested this a while ago on an AMD64 3200+ with a Voodoo 4 4500 PCI and 2xAA works OK there in single-chip mode. I can't get the card to work on my P3 800 machine (usually green screens on startup).

[Frank Sapone]

http://www.thedodgegarage.com/3dfx/bench_v6k.htm

"The V5-6000 would not run with 2x AA enabled for this test: A well known authority on 3dfx drivers informed me there is a bug in Glide or hardware that causes a hard lock while in 2x AA with certain alpha-blending modes. It is being investigated and if there is a fix I'll return to this benchmark and report the results."

May or may not be part of our issue.

[Frank Sapone]

Workaround: set FX_GLIDE_FORCE_OLD_AA=1

This does ignore changing pixel formats if you follow the code, so the statement from dodgegarage may actually be true. It may even be possible that the glide3x.dll being distributed with 3rd party drivers uses the "old" 2xAA format. This env var is specifically for 2xAA.