Codebase Improvements: More Efficient Usage of PropertyGroups

[Midnight Arrow]

Lately I’ve begun scripting the Daz importer, but it’s been difficult to follow what’s going on inside. Thomas previously made comments (“The problem is that the operators which import multiple files use a collection property, and specifying collection properties from Python seems impossible. At least I did not figure out how to do it.”) expressing unfamiliarity with BPY. As somebody who’s been groping blind in the dark learning BPY for a year and a half now, I can certainly sympathize. But at this point, with what I’ve learned, I think the current codebase needs some upkeep.

The most obvious place to start is the dozens of properties which get added to every single object. I don’t see any reason why these need to be individual properties? The same effect can easily be achieved with a PropertyGroup. This is routinely used by many other addons (my own included) to store object-level properties.

Here’s an example:

# Diffeomophic Import Daz (DID) object-level properties
class DID_AssetProperties(bpy.types.PropertyGroup):

  # Used to differentiate between imported assets and those created in Blender
  is_daz_asset: BoolProperty()

  # Pointer to the asset's Blender object
  obj: PointerProperty(type=bpy.types.Object)

  # Identifies the DSF/DUF file this asset was loaded from
  url: StringProperty()

  # Useful properties copied on loading from a DSF/DUF file
  id: StringProperty()
  name: StringProperty()

  # The DSON format type, for automated parsing of assets
  # (http://docs.daz3d.com/doku.php/public/dson_spec/format_description/metadata/content_types/start)
  content_type: StringProperty()

  # Used to determine the datatype stored in "asset_data"
  asset_type: EnumProperty(items=ENUM_ASSET_TYPE)

  # Data specific to each kind of object (armature, mesh, etc.)
  asset_data: PointerProperty(type=bpy.types.DID_AssetData)

This greatly simplifies registering properties:

def register():
  bpy.utils.register_class(DID_AssetProperties)
  bpy.types.Object.did_asset_properties = bpy.props.PointerProperty(type=DID_AssetProperties)

All of the relevant information is consolidated in one place and standardized, which makes it easy to pass the data around as an opaque pointer until some function actually needs to extract data from it.

Furthermore, the current implementation is not user-friendly. It is impolite to dump hundreds of raw sliders in the custom properties panel because that makes it difficult to find properties created by other addons and end-users. Instead of raw sliders, morphs can be stored as nested arrays inside a PropertyGroup, tidying things up and making the UI more convenient for others.

I realize this is a significant undertaking, but I’d be glad to contribute to the design and the coding.

[Alessandro Padovani]

Thomas is back on sunday.

[Midnight Arrow]

I put together a sample script to show how values stored inside a PropertyGroup can be used to drive an object’s properties. Each morph is contained inside a PropertyGroup which has a “slider_value” and a “final_value”. It uses a callback to transfer the slider_value to the final_value, although in theory the callback can be used to handle any kind of processing we want. We could pull data from an arbitrary number of sources using the context, mix them together, and derive whatever value we want. As long as it gets piped into the “final_value” at the end, the driver will read it just fine.

Just copy the following script into Blender’s text editor and hit the run button. Once the cube pops up, a menu named “DazMorphs” will appear. The slider will control the cube’s X position using the DazMorph PropertyGroup and the callback.

import bpy

# ============================================================================ #

# This PropertyGroup stores the morph data. It is nested inside DazProperties.
class DazMorph(bpy.types.PropertyGroup):

    # This callback is triggered every time the float value is updated (i.e. from
    # a UI slider). This sets the slider value to the final value, which is hooked
    # up to the object's driver, but in theory the DazMorph can store a stack of
    # PropertyGroup objects which pull data from other places, look through them,
    # and calculate a final output.
    def _float_update(self, context):

        # "self" in this context is DazMorph
        self.final_value = self.slider_value

        return

    slider_value: bpy.props.FloatProperty(
        update = _float_update,
    )

    final_value: bpy.props.FloatProperty()


# This PropertyGroup is assigned to each object. It holds an array of morphs.
class DazProperties(bpy.types.PropertyGroup):
    morphs: bpy.props.CollectionProperty(type=DazMorph)


# ============================================================================ #

class DazPanel(bpy.types.Panel):
    bl_idname:str = "DAZ_PT_panel"
    bl_label:str = "Daz Morph Panel"
    bl_space_type = 'VIEW_3D'
    bl_region_type = 'UI'
    bl_category = "DazMorph"

    def draw(self:bpy.types.Panel, context:bpy.types.Context) -> None:
        obj = context.active_object
        if obj:
            self.layout.prop(obj.daz_properties.morphs[0], "slider_value", text="Location X")
        return

# ============================================================================ #

def register():

    bpy.utils.register_class(DazMorph)
    bpy.utils.register_class(DazProperties)
    bpy.utils.register_class(DazPanel)

    bpy.types.Object.daz_properties = bpy.props.PointerProperty(type=DazProperties)

    return

def unregister():

    bpy.utils.unregister_class(DazMorph)
    bpy.utils.unregister_class(DazProperties)
    bpy.utils.unregister_class(DazPanel)

    del bpy.types.Object.daz_properties

    return

# ============================================================================ #

if __name__ == "__main__":

    register()

    # Create cube
    bpy.ops.mesh.primitive_cube_add()
    cube = bpy.context.active_object
    cube.daz_properties.morphs.add()

    # Create driver
    driver:bpy.types.Driver = cube.driver_add('location', 0).driver
    driver.expression = "var"

    # Link driver properties
    variable:bpy.types.DriverVariable = driver.variables.new()
    variable.type = 'SINGLE_PROP'
    variable.targets[0].id_type = 'OBJECT'
    variable.targets[0].id = cube
    variable.targets[0].data_path = 'daz_properties.morphs[0].final_value'

‌

[Thomas Larsson]

Well, that is food for thought, but the changes may disrupt lots of things, and I don’t want to try it before the next release. Although combining all predefined object properties into a single property group should not be terribly difficult.

However, what really litters the custom properties are the morph properties: the raw object properties and the final armature properties. There is already a property group for each morph, containing the morph's ID, label, and other info such as the category. E.g.,

‌

This character has 40 expressions, 104 facs and 4 flexion morphs that have raw sliders; the number of final sliders if higher because some morphs are hidden. It might be doable to include the sliders into the property group, but there are questions:

1. Can property group sliders both be driven and drivers?
2. Can these sliders have limits and soft limits?
3. Can the raw sliders be overridden so they work with file linking?

Also, I think you misunderstood how the final sliders work. They are not just a copy of the raw sliders (then they would be unnecessary), but also contain contributions from other morphs. E.g.

Eye blink (fin) = Eye blink
Eye blink left (fin) = Eye blink left + Eye blink (fin)

See On raw and final slider values

[Midnight Arrow]

However, what really litters the custom properties are the morph properties: the raw object properties and the final armature properties. There is already a property group for each morph, containing the morph's ID, label, and other info such as the category.

We could always just code a custom panel in the object’s properties to handle the morphs. Like how Rigify creates custom panels for its armature properties.

def draw(self, context):
  obj = context.active_object
  if obj.did_asset.is_daz_asset:
    for morph in obj.did_asset.morphs:
      create_morph_panel(self.layout, obj, morph)

Each morph gets its own individual UI box, just like how it’s done in Daz Studio. And this panel can contain custom flags, limits, min/max, etc. All these properties can be stored in the DazMorph object. Then, in the Daz Runtime sidebar panel, we expose the slider_value like normal for convenience, while relegating all the heavy-duty options to the object properties menu.

Not sure how performant this would be, but I think we should give it a try.

This character has 40 expressions, 104 facs and 4 flexion morphs that have raw sliders; the number of final sliders if higher because some morphs are hidden. It might be doable to include the sliders into the property group, but there are questions:

1. Can property group sliders both be driven and drivers?
2. Can these sliders have limits and soft limits?
3. Can the raw sliders be overridden so they work with file linking?

Also, I think you misunderstood how the final sliders work. They are not just a copy of the raw sliders (then they would be unnecessary), but also contain contributions from other morphs.

I purposefully created a simple test setup to show the general procedure. As I said above, “in theory the DazMorph can store a stack of PropertyGroup objects which pull data from other places, look through them, and calculate a final output.” The callback is a function. You can do whatever you want inside it, as long as you store the necessary properties inside either the DazMorph or the context.

Inside a DSON file, morph values are calculated based on a stack of operations. So in theory, inside the callback, you can pull an array of operations from the DazMorph object, loop through all of them (using a PointerProperty to access data from other objects), and calculate the final value on the fly with pure Python code.

This is basically what a driver does anyway, except that uses a UI wrapper for convenience.

And if somebody wants to disable the addon, we can have a “Bake to Drivers” operator in the finalize panel.

As for overrides, each property has a “LIBRARY_OVERRIDABLE” flag, so it should work. I haven’t tested it, though.

[Midnight Arrow]

I threw together a quick script showing what a dedicated morph panel would look like (spoiler alert: It looks exactly like the Shape Key panel).

‌

As before, just copy it into the text editor and hit the execute button to see it.

import bpy

# ============================================================================ #

class DID_Morph(bpy.types.PropertyGroup):

    dson_id: bpy.props.StringProperty()

    slider_value: bpy.props.FloatProperty(
        default = 0.0,
        )

    min_limit: bpy.props.FloatProperty(
        default = -10000.0,
        )

    max_limit: bpy.props.FloatProperty(
        default = 10000.0,
        )

    use_limits: bpy.props.BoolProperty(
        default = True,
        )


class DID_Asset(bpy.types.PropertyGroup):

    is_daz_asset: bpy.props.BoolProperty()

    active_morph_index: bpy.props.IntProperty()
    morphs: bpy.props.CollectionProperty(type=DID_Morph)


# ============================================================================ #

class DID_MorphList(bpy.types.UIList):
    bl_idname = "OBJECT_UL_did_morph_list"

    def draw_item(self, context, layout, data, item, icon, active_data, active_property):
        layout.label(text=item.dson_id)
        return 


class DID_MorphPanel(bpy.types.Panel):
    bl_idname = "OBJECT_PT_did_morph_panel"
    bl_label = "Daz Morphs"
    bl_space_type = 'PROPERTIES'
    bl_region_type = 'WINDOW'
    bl_context = "object"

    def draw(self, context:bpy.types.Context) -> None:

        layout = self.layout

        obj = context.active_object
        if not (obj and obj.did_asset.is_daz_asset):
            return

        layout.template_list(
            "OBJECT_UL_did_morph_list", "Morph List",
            obj.did_asset, "morphs",
            obj.did_asset, "active_morph_index",
            )

        all_morphs = obj.did_asset.morphs
        index = obj.did_asset.active_morph_index

        if not (index >= 0 and index < len(all_morphs)):
            return

        active_morph = all_morphs[index]

        layout.prop(active_morph, "slider_value")

        column = layout.column()
        column.label(text="Limits")

        row = column.row()
        row.prop(active_morph, "min_limit")
        row.prop(active_morph, "max_limit")
        row.prop(active_morph, "use_limits")

        return

# ============================================================================ #

def register():

    bpy.utils.register_class(DID_Morph)
    bpy.utils.register_class(DID_Asset)
    bpy.utils.register_class(DID_MorphList)
    bpy.utils.register_class(DID_MorphPanel)

    bpy.types.Object.did_asset = bpy.props.PointerProperty(type=DID_Asset)

    return

def unregister():

    bpy.utils.unregister_class(DID_Morph)
    bpy.utils.unregister_class(DID_Asset)
    bpy.utils.unregister_class(DID_MorphList)
    bpy.utils.unregister_class(DID_MorphPanel)

    del bpy.types.Object.did_asset

    return

# ============================================================================ #

if __name__ == "__main__":
    register()

    bpy.ops.Mesh.primitive_cube_add()
    cube = bpy.context.active_object
    cube.did_asset.is_daz_asset = True

    morph1 = cube.did_asset.morphs.add()
    morph1.dson_id = "Some pJCM"

    morph2 = cube.did_asset.morphs.add()
    morph2.dson_id = "Some eJCM"

    morph3 = cube.did_asset.morphs.add()
    morph3.dson_id = "Some Shape"

‌

[Thomas Larsson]

I will give this a go. However, lots of stuff has to be changed and the code base will be in a flux until it is done, so other bug fixes will have to wait.

[Midnight Arrow]

Why not just create a branch and work in that?

[Thomas Larsson]

Well, I did try to implement this, but it didn’t go so well. There are two new branches..

In Only_attributes, the custom properties that start with “Daz” (either attributes or keys) are replaced by a class with annotations. This should not pose any problems per se, if the code has been designed in that way. But the code is almost ten years old and quite massive, and it is not straightforward to find everything that needs to be changed. I also get the attribute-error “'PropertyDeferred' object has no attribute ‘keys'“, presumably because I’m accessing something that has not been properly initialized.

However, the “Daz” properties are not the big deal, because there are only a dozen or so of them. What’s really annoying are the properties created by importing morphs, because there can easily be many hundreds of them. In the New_attributes branch I attempted to put all morph sliders into a single class (or rather two classes, for raw and final sliders). Issues:

1. Lots of warnings about dependency loops. The loops don’t seem to be real and the morph sliders work as expected, but the warnings fill up the terminal window so nothing interesting can be seen there.
2. Hard and soft limits can be set when the class is defined, but I didn’t figure out how to change them This can probably be fixed though.
3. Custom properties are usually floats, but they can be booleans and integers too. Is there a way to make multi-type properties?

‌

[Midnight Arrow]

I haven’t tested the new code yet, but specifically in regards to # 3:

I was under the impression PointerProperty allows for polymorphism (using an abstract base class), but it turns out I was mistaken. There’s always the brute force option – store an EnumProperty defining the type and implement all the necessary variables for all types. It seems to be the most Blender-ish way, like with obj.type == 'MESH'. The eight channel types are defined here, as subclasses of channel: http://docs.daz3d.com/doku.php/public/dson_spec/object_definitions/channel/start.

“What’s really annoying are the properties created by importing morphs, because there can easily be many hundreds of them.”

True, but that’s no different from shape keys, which is why my mockup looks like the shape key panel. I looked into UIList and there are abstract methods which can be implemented so we can filter them. I’ll look into it some more.

[Thomas Larsson]

On dependency loops. Import the EyesClosedL morph in the master branch. The driver for the final value looks like this and there are no errors in the terminal.

‌

Now do the same thing in the New_Attributes branch. The driver now refers to entries in a collection property, but apart from that everything is the same as before.

‌

A lot of warnings are written in the terminal, but the morph itself seems to work fine.

Dependency cycle detected:
  ARGenesis 8 Female/PARAMETERS_EVAL() depends on
  ARGenesis 8 Female/DRIVER(DazMorphs["ECTRLEyesClosedL(fin)"].float_value) via 'Driver -> Driven Property'
  ARGenesis 8 Female/PARAMETERS_EVAL() via 'RNA Target -> Driver'
 ...
Dependency cycle detected:
  ARGenesis 8 Female/PARAMETERS_EVAL() depends on
  ARGenesis 8 Female/DRIVER(DazMorphs["lEyelidLowerInner:Rot:0:01"].float_value) via 'Driver -> Driven Property'
  ARGenesis 8 Female/PARAMETERS_EVAL() via 'RNA Target -> Driver'
Detected 7 dependency cycles
Info: Import Units finished 

‌

[Thomas Larsson]

Now I remember another problem. The morphs will probably cease to work if the daz importer add-on is disabled. Or at least we must include the definitions of the morph class and collection property. This is a real issue, because if you make an animation with morphs and send it to a render farm, chances are the scene is rendered without the addon. So we need to include a Stripped runtime system, or else we may run into the ugly face problem.

It is in fact already necessary today, but only if you use ERC morphs. See the post on Morphing Armatures.

[Midnight Arrow]

Yeah, I thought of that too.

The way Rigify handles it is to include a reference to the UI script as a PointerProperty inside the object’s PropertyGroup. That ensures it’s always imported along with the figure. And since it’s flagged as ‘REGISTER’, it will run automatically on startup. The user shouldn’t have to do anything (except check the textbox saying it’s okay to autorun scripts).

[Midnight Arrow]

Tested both branches.

The custom properties panel in Only Attributes looks much tidier now. The PropertyDeferred errors is because on line 296 of propgroups.py, you use an equals sign rather than a colon to define morph_urls. Once that’s fixed, it works fine except for the console spam. I get WRONG drivers_disabled DazDriversDisabled <bpy_struct, Object("Angharad 8") at 0x00000278B3F2E408>.

As for New Attributes, when I try and load a figure it fails. These lines in propgroups.py are passing in PHMMouthRealism_HD_div2, even if I’m not loading any morphs, and it causes an exception in setProtected().

        for asset,inst in main.modifiers:
            asset.postbuild(context, inst)

‌

[Thomas Larsson]

The small errors in only_attributes branch have been fixed.

I removed the new_attributes branch and made a fresh start on morphs in the new_morphs branch. Collecting morphs and other attributes are separate issues, and need not be entangled.

The slider and final values are now part of the same class. The definition is made in runtime/morph_armature.py, which is supposed to be registered even if the daz importer is disabled.

The problem with dependency loops persists. Some morphs behave a little strange so I think it is a real issue.

I will be gone until Monday, so no more updates until then.

[Thomas Larsson]

I think what causes the dependency loops. Consider the case that we have two morphs foo and bar, and the corresponding final properties are driven by:

1.  foo(fin) = foo
2.  bar(fin) = bar + foo(fin)

Start from a state where foo = bar = 0, and hence foo(fin) = bar(fin) = 0 too. Now set foo = 1. If the drivers are evaluated first 1 then 2,

foo(fin) = 1
bar(fin) = 0 + 1 = 1

which is correct. However, if the drivers are updated first 2 then 1,

bar(fin) = 0 + 0 = 0
foo(fin) = 1

and the value of bar(fin) is wrong. If the properties are simple properties, Blender is smart enough to always evaluate multiple drivers in the right order, at least after Blender 2.80; there was talk about a new depsgraph at that time. However, if the properties are part of a collection property, it seems like there is no guarantee that the drivers are evaluated in the right order.

This is a showstopper if this analysis is correct. It seems necessary that the final properties are simple properties to make them update correctly. It should still be fine if the raw properies foo and bar are part of a collection, because they are fixed when the depsgraph is evaluated.

[Midnight Arrow]

This is exactly why I said to handle the logic in an update callback (when is triggered when the user changes the slider) rather than drivers. The evaluation order is always known, since it’s controlled by user input. The update callback handles all the logic and dumps the result into the final value at once. The only need for a driver is to extract the final value. There’s a 1:1 relationship: one shape key → one driver → one morph’s final value. Everything else is calculated instantly inside the callback.

[Thomas Larsson]

I’m confused about what you mean. Do you mean to keep the drivers as they are, and only add an update callback for the final value? I don’t understand how that could ever work. More significantly I tested it and it doesn’t work.

‌

Or do you mean to get rid of the drivers altogether and replace them with update functions. That seems like a terribly bad idea. The logic that relates the various morphs cannot go away, so if it isn’t encoded in the drivers it must be encoded in the update functions. Once you have final properties that drive other final properties, you get complicated update functions. Returning to my previous example, the driver

bar(fin) = bar + foo(fin)

corresponds to two update functions:

bar triggers an update of bar(fin)

foo(fin) also triggers an update of bar(fin). So we have cascading updates:

foo updates foo(fin) updates bar(fin).

This sounds very backwards to me. I also wonder if different items in the same collection property can have different update functions, and if they can be changed dynamically when new morphs are added or removed

[Xin]

update functions are also much slower than (non-python) drivers, since update functions are over python, while drivers can be evaluated by the C++ core and have large performance gains for multiple cores. Considering the large amount of interactions between properties that daz figures have, the only sensible option in terms of performance right now in Blender is using non-python drivers.

[Midnight Arrow]

Or do you mean to get rid of the drivers altogether and replace them with update functions. That seems like a terribly bad idea. The logic that relates the various morphs cannot go away, so if it isn’t encoded in the drivers it must be encoded in the update functions.

As I’ve explained, what I mean is that you store the logic inside the morph. This is how DSF/DUF files work already, inside the Formula DSON object. The operations to apply are encoded using enums and dictionaries. Here’s a very simplistic representation:

class DazMorphOperator(PropertyGroup):
  stage: EnumProperty()
  value: FloatProperty()


class DazMorph(PropertyGroup):
  slider_value: FloatProperty(update=_update_float)
  operators: CollectionProperty(type=MorphOperator)
  final_value: FloatProperty()


def _update_float(self, context):

  value:float = self.slider_value

  for operator in operators:
    match(operator.stage):
      case 'SUM':
        value += calculate_sum_operator(operator)
      case 'MULT':
        value *= calculate_mult_operator(operator)

  # Clamp limits, etc.

  self.final_value = value

  return

update functions are also much slower than (non-python) drivers, since update functions are over python, while drivers can be evaluated by the C++ core and have large performance gains for multiple cores. Considering the large amount of interactions between properties that daz figures have, the only sensible option in terms of performance right now in Blender is using non-python drivers.

Daz Studio handles data a different way than Blender does. Therefore, to replicate it, we need the greater flexibility that update callbacks provide. That's why they’re there. Both that, and the hundreds of “final value” properties which need to be removed from the custom properties panel, justifies their usage, in the absence of any concrete benchmarks proving this is an issue.

The update is only triggered once, when the slider is moved. If you dial in “FBMAthletic”, then it’s updated only once. What “interactions” are you referring to that would make this a problem?

[Alessandro Padovani]

@Midnight I’m not sure to understand, actually we have a limit in blender that sliders can’t drive each other as in daz studio, because they can be controlled either by a driver or the user, not both. Does your new logic resolve this issue by using callbacks instead of drivers ?

[Xin]

We actually did some benchmark when we discussed the change to non-python drivers with Thomas. It was a no brainer, the performance gains were very noticeable. In case you are not aware, import_daz used to work similarly to how you describe in the past (i.e. executing formulas over python), but it was slow even while not implementing all formulas/drivers.

Even Blender devs recommend to use non-pytohn drivers in the docs. The driver system is the system that Blender considers the priority for animation, and, as a result, it’s highly optimized relative to the alternatives. The update() functions are mostly for user convenience when you don’t have many properties or high performance demands.

As for that Alessandro, I remember I could mimic that with update() functions in the past and I posted it here, but then again, update() functions are terribly slow and there wasn’t much of a point in mirroring everything daz does (daz itself is not suitable for animation, it’s too slow).

We concluded that it’s fine having the value of sliders in Blender be different from those of daz, it’s not really a problem since Blender shouldn’t be thought as a way to only play daz animations (in essence letting daz control Blender the daz way with poor performance), but to edit/generate new content, giving priority to performance and Blender’s way of doing things. Otherwise, why leave daz?

[Alessandro Padovani]

@Xin I may be wrong but perhaps the new logic by Midnight would simplify drivers a lot. That is, while now drivers consider all the sliders/morphs at once, thus resulting in long computations, the callback logic would compute only the actual slider/morph controlled by the user or the keyframe, thus resulting in much shorter computations. Again I may be wrong I’m not sure I got what Midnight is proposing.

[Xin]

The driver system already implements that internally. That is, it doesn’t execute everything all the time, it detects what changes and performs updates accordingly walking a dependency graph that Blender knows very well since it’s part of the core.

Blender has no concept of dependency graph over update() functions (i.e. Blender doesn’t parse user’s python dependencies through arbitrary code, it would be a very hard task anyway).

That was another huge problem with update() functions we found in the past with Thomas: they don’t execute in an optimal manner since they are out of reach of Blender’s internal graph. In the example given above by Midnight, the callcalculate_sum_operator(operator) depends on a lot of changes from other properties but Blender doesn’t track those dependencies in its dependency graph so execution won’t be good in at least three ways:

• Some updates() won’t be called when they should, effectively breaking the animation system.
• When they do get called, they will take longer than drivers.
• Dependency cycles on drivers are transparent to Blender and Blender will inform you of those. No such thing exists for dependencies over update() functions. A typical problem is an infinite loop where updates() never end.

Blender’s dependency graph is highly optimized for animation, trying to circumvent it is not a good idea. This dependencies' problem is already one of the most challenging for Blender’s devs.

[Alessandro Padovani]

Thank you, I believe your comments will be useful both to Midnight and Thomas to understand and decide what’s better.

[Midnight Arrow]

@Xin Daz Studio has two kinds of morphs: shaping and posing. While sure, you might want to animate posing morphs, I’m not sure under what circumstances you’d need to animate a shaping morph? When would you need to go from 0% FBMAthletic to 100% FBMAthletic in the course of a single scene? So the animation system doesn’t matter. A shaping morph is something you dial in at the start of a scene and never change. Posing morphs then call on shaping morphs to know when to dial in certain other morphs, but the shaping morphs are already computed at that point.