G9 extremely slow to import ? (aka we could try with "only vertex groups")

[Alessandro Padovani]

daz studio 4.21.0.5, blender 3.6.1, diffeomorphic 1.7.2.1730

Below there’s the time required on my system to import various figures, fully rigged same as daz studio. We see that G9 takes a lot of time to import, and looking at the console it’s because she has much more morphs to load and transfer.

I’m just wondering if the import times could be optimized in some way. Perhaps we could reconsider the “only vertex groups“ option for transfer that should cut down the times significantly, since the actual method first transfers all then prunes all that’s a two times bogger.

Let us know what you think about this.

fully rigged genesis figure, import time:

• G8F 43s (face units, expressions, visemes, jcms, flexions)
• G81F 62s (facs, facs expressions, face controls, jcms, flexions)
• G9F 212s (facs, facs details, facs expressions, jcms, flexions)

‌

[Alessandro Padovani]

p.s. G9 is very slow to load even in daz studio, compared to other figures, so this is expected. I’m just wondering if we could optimize something.

[Thomas Larsson]

I improved the Print Statistics tool to include also the number of shapekeys and vertex groups. Then I loaded the three figures with easy import, and got the following results.

	Verts	Edges	Faces	Shapekeys	Vgroups	Load time	V*S/T
G8	16556	32882	16368	123	167	38.325	53134
G8.1	16556	32882	16368	203	167	51.575	65164
G9	25182	50338	25156	323	124	141.021	57677

The last column is # vertices times # shapekeys divided by the load time. If we assume that most time is spent making shapekeys, the last column should be roughly independent of the figure, which is reasonably well the case. In fact it is rather G8 that takes disproportionally long to load, but that is probably because it spends more time on drivers.

[Alessandro Padovani]

I'm not arguing that G9 is "disproportionally" slower, I’m rather wondering if we can improve performances in general, that will benefit all the figures.

Essentially, the idea behind “only vertex groups“ is to check the morphs to transfer by bones. That is, if a morph affects b1 b2 b3 bones in the source, then we check the target for b1 b2 b3 vertex groups, if there’s any then we transfer, otherwise we don’t. This doesn’t compute vertex clouds for bounding boxes nor evaluates vertex weights for pruning, just relies on bones that’s much faster.

That is intended as a first step before bounding boxes and pruning. Based on the idea that if a morph drives bones then the shapekeys are corrective shapes for the bones, thus if the target is not affected by the bones the corrective shapes are not needed.

If a morph is shapekey only, without bones, then we proceed with bounding boxes and pruning as it is now.

[Thomas Larsson]

No, I don’t think this is a good idea. Finding the bones that a morph affect also takes significant time, because we have to loop over all armature drivers to make the connection; at least I haven’t found any other way to do it. I am not convinced that this can be done faster than computing the bounding box, where the loop is done in C, or pruning the morph which is done with numpy.

This method may also discard shapekeys incorrectly. Consider the facs_bs_JawOpen morph where the shapekey is as in #1675. The morph affects the jaw bone and the brows don’t have that, but the shapekey should nevertheless be transferred to the brows because it extends up to the temples.

‌

[Alessandro Padovani]

Ok, I understand daz morphs are complex and trying to optimize too much may lead to errors. We’ll take this as a warning that G9 is much more complex than G8 in terms of morphs and drivers, as well as geometry that needs to be HD.

Will leave this open for a while in case someone wants to contribute with a nice idea, then I’ll close as invalid if nothing comes out.

[Alessandro Padovani]

update.

May we use “only vertex groups“ for jcms ? In that case it should work fine and we may get a significant speedup for transferring jcms. That is, jcms don’t make sense if the bone doesn’t affect the mesh. Let me know.

[Thomas Larsson]

I very much doubt that you will gain much for jcms either. Yes, once you know which bone drives a jcm shapekey you could eliminate some transfers, but you much weigh that gain against the cost for finding the bone. That is a quite expensive operation which involves a slow python loop over the entire driver network. At least I haven’t found a way to immediately get to the driver from the shapekey, without looping over all drivers. This is quite awkward, but I suppose that the driver system is optimized to be evaluated quickly, and not for finding relations.

Bounding boxes and pruning after the transfer also involve loops, and much bigger ones, but since those loops are done in C or numpy they are quite fast anyway.

[Alessandro Padovani]

I'm not sure to follow. That is, unless I miss something, jcms are driven by a single bone, so we look at the jcm driver and find the bone directly there's no need to scan anything. For example:

# from jcm to driver to bone
pJCMShinBend_155_L > pJCMShinBend_155_L(fin) > bone lShin
if target has lShin vertex group
    transfer pJCMShinBend_155_L

That said I suck at rigging and understand very little of drivers. Please let me know so I’ll close as invalid if my suggestion doesn’t work.

‌

[Thomas Larsson]

The problem is to get from the shapekey to the bone. Currently the plugin does something like this:

if rig.data.animation_data:
    for fcu in rig.data.animation_data.drivers:
        if fcu.data_path == "pJCMShinBend_155_L(fin)":
            for var in fcu.driver.variables:
                trg = var.targets[0]
                if trg.id == rig:
                    bone = trg.bone_target

which involves a loop over all armature drivers, plus a loop over all driver variables. The latter may be short, but the outer loop is expensive.

Now I check the docs and see that there is a actually a find function that might find the driver directly

fcu = rig.data.animation_data.drivers.find("pJCMShinBend_155_L(fin)")

However, the docs warn that “this function performs a linear scan of all driver F-Curves”, so in principle the issue remains. Plus how do we know that the shapekey is a jcm, except by looking at the name which isn’t foolproof.

[Alessandro Padovani]

Ok, I understand that the blender function find() may be slow, so because of the internal blender data structures there’s no way to get the driver directly. Plus once we load a corrective shapekey we don’t know anymore where it comes from, if it’s a jcm or facs or whatever, so the name is everything we got and it may be deceiving.

Thank you for the explanation.

[Alessandro Padovani]

Closing since it seems we can't do any better.