BVH Retargeter - LiveLink & FaceCap Eye Rotation Issue

[Joss Kenyon]

Hi Thomas!

I am experiencing an issue with the Diffeomorphic Daz Importer Addon when importing Daz 3D Genesis characters into Blender 4.1. Specifically, the problem occurs with eye rotations after applying facial motion capture data using the Diffeomorphic Daz BVH Retargeter addon.

Details:

• Blender Version: 4.1
• Diffeomorphic Daz Addons Used: import_daz, mhx_rts, retarget_bvh (latest split versions)
• Character Models Attempted: Genesis 9 and Genesis 8
• Motion Capture Tools Attempted: LiveLink and FaceCap

Issue: The eye rotation of the imported character appears to be offset by 90 degrees when motion capture data is applied. Here are the specifics of the misalignment:

• When I look to my left, the character's eyes look up.
• When I look to my right, the character's eyes look down.
• When I look up, the character's eyes look to their right.
• When I look down, the character's eyes look to their left.

This issue occurs consistently across different character models (Gen 8 and Gen 9) and with both LiveLink and FaceCap motion capture data.

Before coming here, I created a post on the Daz Forums asking for advice, a user responded to that post and pointed out it could be a bug with the addon itself:
https://www.daz3d.com/forums/discussion/687391/diffeomorphic-daz-importer-livelink-eye-tracking-issue#latest

I would appreciate any guidance or solutions you can provide to resolve this eye rotation offset - even a quick fix or workaround.

Thank you for your help, and for creating these addons, they have been a major help to me!

I’ve attached a packed .blend file, a video that displays the input footage alongside the character, & the mocap csv.

Kindest regards,
Joss

[Thomas Larsson]

Fixed for Livelink in last commit. If the same problem exists for Facecap, perhaps you could provide me with the same kind of example so I can fix it.

[Alessandro Padovani]

4.1.0.2131

It seems the livelink keywords are not mapped correctly to the daz facs. Here's the correct mapping for G81 G9. Please note that there's a difference between eyes and lids FACS, where lids don't drive the eyes in daz studio whereas they drive the eyes in livelink, so we need to map lids FACS to eyes FACS. Please note that side-side is reversed for left and right.

LIVELINK           G81 G9 FACS

EyeBlinkLeft       Eye Blink Left
EyeLookDownLeft    Eye Look Up-Down Left negative
EyeLookInLeft      Eye Look Side-Side Left positive
EyeLookOutLeft     Eye Look Side-Side Left negative
EyeLookUpLeft      Eye Look Up-Down Left positive
EyeSquintLeft      Eye Squint Left
EyeWideLeft        Eye Wide Left

EyeBlinkRight      Eye Blink Right
EyeLookDownRight   Eye Look Up-Down Right negative
EyeLookInRight     Eye Look Side-Side Right negative
EyeLookOutRight    Eye Look Side-Side Right positive
EyeLookUpRight     Eye Look Up-Down Right positive
EyeSquintRight     Eye Squint Right
EyeWideRight       Eye Wide Right

edit. Oops .. seems you fixed it while I was posting, going to check the new commit ..

[Joss Kenyon]

Thank you Thomas and Alessandro! So I just need to download the updated version of the BVH Retargeter now?
I won’t be able to test it right now as I’m at work, but I’ll do so as soon as I get home.

Once again, THANK YOU for your work and incredible response times! I can’t tell you how grateful I am!

Joss

[Alessandro Padovani]

update. BVH 4.1.0039

Here doesn’t seem to work, as the G81 G9 FACS are not keyframed for the eyes, as shown below. Anyway the eyes seem to move fine enough, though not via FACS as it should be. Let me know.

steps:

1. easy import G81 or G9 with FACS and “transfer to face“
2. import the livelink file
3. look at the FACS, some are keyframed, but side-side is not as well as other eyes FACS

‌

[Alessandro Padovani]

note. about mocap files in general.

I’m noticing that raw mocap files get precision issues causing a sort of “trembling” in the blender animation. Of course it is possible to use the blender graph editor to smooth the curves, but it takes work and time so we could also provide a simple “smooth“ option to import animations if the keyframe values go from 0 to 1. The function below would limit the precision to 1/10 so to limit the trembling, this is a high level of smoothing of course we could use 1/20 or 1/50, but just to get the idea. Let me know.

key = integer(key * 10) / 10

‌

[Thomas Larsson]

The problem was not with morphs but with the eye bone rotations. If you open the csv file in Excel, you will see that the last six columns are named

LeftEyeYaw, LeftEyePitch, LeftEyeRoll, RightEyeYaw, RightEyePitch, RightEyeRoll

The reason that the eyes moved in the wrong direction was that plugin mixed up the yaw, pitch and roll angles. The sliders are zero because the shapekeys are driven by eye bone rotations, not the other way around. If you turn on Show Final Morphs Values, you can see that the final values and the shapekey values are nonzero although the sliders themselves are not.

‌

The plugin puts keys on integer frames only in some versions of Blender, because that is a Blender limitation. But only in some Blender versions. I haven’t checked recently, perhaps we can use non-integer frames again.

[Thomas Larsson]

Joss, yes, the latest version of the BVH retargeter has the fix.

[Alessandro Padovani]

Thomas, of course it is true that the genesis eye bones drive the lids, this is so even in G3 G8 which don’t have FACS. But when we load a FACS animation we are supposed to use FACS keyframes, not bone keyframes. In DAZ studio we can drive the eyes with FACS, and save a FACS animation that will correctly store FACS keyframes when loaded in blender. It is the same when we load facecap or livelink we are supposed to use FACS keyframes as these are FACS animations. This is what FACS is for.

If you use the conversion table I provided above we can load facecap and livelink as FACS keyframes, same as a FACS animation in daz studio. This way FACS drive bones and not the other way around.

steps:

1. easy import G81 with FACS and “transfer to face“
2. import action facs-anim.duf, that’s a facs animation
3. look at the facs correctly keyframed

‌

As for integer frames, no I didn’t mean that and it would have no effect to limit the “trembling“. What I mean is to limit the keyframe value precision, not the keyframe time precision, so close values will be the same and there’s no “trembling”. Admittedly this could cause “jumps” instead of “trembling” especially with a high smoothing factor eg. 1/10 or below, so it is better as an option.

Anyway at the end things work either way, so mine is just to point out a more “correct“ way how it should work. If you like it your way we can close as resolved for me.

[Alessandro Padovani]

FACS animation, FACS drive bones not the other way around

[Joss Kenyon]

Thomas, Alessandro, thank you! I’ve just tested on my side and the eye directions are fixed!

Interesting point you raise about the FACS keyframes Allesandro - I notice some of the movements are being animated via the FACS keyframes.
I used these FACS keyframes in the graph editor to limit the blinking, stopping the eyelashes from overlapping - this might’ve been more difficult without FACS keyframes?

I also noticed the trembling from the raw files in a recent test I did, that smoothing function could be very useful. Would it be best to use the calibrated file (cal) instead of the raw going forward? I personally didn’t notice much of a difference, but I have mainly been using raw. Otherwise I will just smooth in the graph editor.

‌

[Thomas Larsson]

I agree that it is better to do everything with morphs and ignore the eye rotations, so that is done in the last commit. The option to use eye rotations is gone, but head movement still uses bone rotations because that is not part of facs. This is a rather big change in the code so bugs may have sneaked in, but I tested several animations with both G8.1 and G9 and the results look reasonable.

I will look at the keyframe issue later today.

[Alessandro Padovani]

bvh 4.1.0040

Works fine here, thank you for the fix.

note. eyes tongue options. We don’t need to remove the eyes and tongue options, also because they can be useful if one doesn’t want to import the eyes or tongue animation from the mocap. To ignore the eyes or tongue we just need to ignore “eye“ or “tongue“ keywords in livelink. Let me know.

[Thomas Larsson]

Using the eyes animation leads to double counting. First the eye bone is driven by the eyelook-side-side and eyelook-up-down morphs, and on top of that it is rotated further. If we want to use the eye bones directly, the eyelook morphs should be ignored instead. Which is how it was before.

There was also an option to import the animation directly to shapekeys, which has now been hidden. This could work with eye bone animation, e.g. if you want to export the animation to game engines. But for this to work well, we need to extract animations for the jaw and eyelid bones as well. Topic for improvement.

There are a number of things that can be improved. E.g., currently there are tables for the facs morph names for G8.1 and G9. Those tables should be moved to separate json files, so we could add further files to make the plugin work with other names conventions.

[Alessandro Padovani]

No I’m not talking of “using the eye bones directly”, that should not be done for facs. I’m talking of ignoring the “eye“ or “tongue“ keywords in livelink if we don’t want those animations loaded. Unless I misunderstand what you mean.

As for smoothing it can also be done in the graph editor, perhaps something the addon could do as an option when we import the animation. This is instead of the previous smoothing function explained above, using a gaussian smooth instead of stepping values aka limiting the precision. I didn’t find in the graph editor a modifier to “step“ the values.

p.s. disclaimer. I’m not a mocap expert so I don’t know how professional animators deal with this, I just noticed the issue and wondered for a possible fix.

steps:

1. enter the graph editor
2. select all facs
3. key > smooth > gaussian

‌

[Thomas Larsson]

The reason why the option to ignore eyes morphs existed in the first place was that the same information is available in the facecap and livelink files. Otherwise there is nothing special about these channels. You could imagine somebody only wanting to load an animation for the lower or upper half of the face. OK, you need quite a bit of imagination for that, but anyway there is nothing special about the eyes. If we don’t want the animations, we could remove the F-curves in the graph editor afterwards.

[Alessandro Padovani]

Yes I understand your point of view from the code side, mine is from the user side, but I agree we can just delete the unwanted channels afterwards.

[Thomas Larsson]

Motion capture filtering seems to be a well-researched topic. I googled it and a lot of pages came up. I downloaded one that I will read on my way home from work. There might not be a single best answer here. We want to get rid of unwanted trembling, but not suppress fast transitions such as the blink of an eye.

[Alessandro Padovani]

There’s also “simplify f-curves” that seems to do a pretty good job smoothing trembling while conserving spikes. It also has the benefit of reducing the number of keyframes. We need to activate the addon.

p.s. Both gaussian smooth and simplify f-curves are a one step operation, easy to do in the graph editor if the user is aware. So we may not need an option for that in the addon after all.

https://docs.blender.org/manual/en/4.1/addons/add_curve/simplify_curves.html?utm_source=blender-4.1.1

steps:

1. enter the graph editor
2. select all facs
3. F3 > simplify f-curves

‌

[Alessandro Padovani]

update. bvh 4.1.0041

I see the bvh addon also has its own simplify tool, that has been fixed for key f-curves and works pretty good too. Thank you for the fix. Then if there’s nothing to add we can close as resolved.

‌

[Alessandro Padovani]

Issue #2062 was marked as a duplicate of this issue.