Codebase Refactor

[Midnight Arrow]

Per Thomas’s suggestion (#2209) I’ve begun working on a refactor of the codebase in a separate branch. I’m not ready to submit a pull request yet, but I have cobbled together some snippets into a single file for testing and feedback. One caveat: it only works on the “Preferred” content directory right now, so you may need to change that. Otherwise, you can simply paste it into Blender’s text editor, and it should run.

These changes are mostly back-end stuff to eliminate technical debt and bring it in line with Blender’s coding recommendations. I won’t be touching the UI code, so the user experience shouldn’t be affected (unless Thomas makes changes, obviously). Everything should exist comfortably alongside the current implementation, not interfering with anything else, until it’s stable enough Thomas can begin reliably calling into the API.

The code uses an abstraction layer to make the API as simple as possible. Only four functions are defined as public (right now). parse_dsf_file_for_geometry() and parse_dsf_file_for_uv_set() will open a DSF file (not a DUF file) to extract an asset, either a mesh or a UVSet, from its library. It returns the data in an intermediate format, a DSON_* struct. These structs can be treated as opaque pointers. You call a function to get a reference to a struct, then you pass the reference to another function to do something with it. In this case, create_mesh() and assign_uv_map().

The parsing functions operate on relative, not absolute paths. That’s partly why they only accept DSF files. DSF files are expected to be in a fixed location in the data directory, but DUF files can be moved anywhere.

Most of the code can be safely ignored. Only the functions under the “Public API” header should be called – everything else is for internal usage only. At the very bottom, under the “Sandbox” header, is a demo of how the API works. If you change the hardcoded paths, you should (in theory) be able to load any mesh/UVSet from any DSF file in your preferred content directory (although you’ll need to know its internal ID to fetch it, obviously). These functions by themselves aren’t that useful unless you know what you’re looking for. However, they’re just a very thin wrapper over the functions the DUF parser (still WIP) uses to extract assets (another reason they work on relative paths). So when the DUF parser reads geometry_instance and finds a URL, like /data/DAZ 3D/Genesis 8/Female/Genesis8Female.dsf#geometry, it calls the same underlying function that parse_dsf_file_for_geometry() does, using the same arguments, to produce the same result.

[Thomas Larsson]

I appreciate your effort, and I can see that your code is in many ways superior to mine. However, I don’t see how it would ever be possible to dock your code with the existing codebase. Unless you intend to rewrite the entire plugin, but that would be an enormous task, essentially redoing the work I have done since 2015. For one thing, you cannot use the same coordinate systems as DS does. Not only is the global coordinate system Y up, but daz bones don’t have to be aligned with the local Y axis. If you disable “Z up” and enable “Unflipped Bones”, you can import a character in daz coordinates.

‌

[Midnight Arrow]

However, I don’t see how it would ever be possible to dock your code with the existing codebase.

Right now, the majority of your codebase is like this:

class SomeOperator(DazOperator, Inherit1, Interit2, Inherit3, IsSomeThing, IsSomeOtherThing, DoesThis, DoesThat):

  def check(self, argument):
    if IsSomeThing.function():
      # Dozens of lines of code
    elif IsSomeOtherThing.function():
      # Dozens of lines of code
    return

  def do(self, argument):
    if DoesThis.function():
      # Dozens of lines of code
    elif DoesThat.function():
      # Dozens of lines of code
  return

  def run(self, argument):
    # fifty lines of code with no datatypes or comments
    return

However, as I brought up in #2209, according to the Blender coding standards, your operators should look like this:

class SomeOperator(bpy.types.Operator):

  some_property: BoolProperty()

  def execute(self, context):
    object_to_manipulate = context.object
    library.do_something(object_to_manipulate, some_property)
    return { 'FINISHED' }

Your operators should only be taking things from the current user context and passing it into a library function, not having a rigid block of code locked inside an object which does fifty things. I don’t want to be rude or mean, but it’s not good design. A library function can be called from anywhere in the code. But trapping functionality inside an operator mandates that it be called from Blender’s UI, making it much less useful for both you and anybody else who might want to script it (me). The goal is to make it so you can slim your operators (which currently handle almost all the work) down so they look like the second example, not the first. The current UI (the drawing code, bl_idnames, etc.) doesn’t need to change. What needs to change is the convoluted, operator-based internal hierarchy that makes your codebase extremely inflexible.

Unless you intend to rewrite the entire plugin, but that would be an enormous task, essentially redoing the work I have done since 2015.

In #2209, you said “Starting development from scratch is of course an option, but it is not something that will involve myself.“ Then in the next comment you said, “I suppose it could work if you work on your own fork.“

So yes, I have been rewriting the backend mostly from scratch, but based on your comments it seemed like you said it was okay.

I’ll be honest, I think the way you’ve been building this addon up since 2015 has caused you to overestimate how complicated the task is. Here’s something I came across when I tried to understand your codebase:

def normalizeRef(id):
    ref = quote(id)
    ref = ref.replace("%23","#").replace("%25","%").replace("%2D", "-").replace("%2E", ".").replace("%2F", "/").replace("%3F", "?")
    ref = ref.replace("%5C", "/").replace("%5F", "_").replace("%7C", "|")
    if len(ref) == 0:
        return ""
    elif ref[0] == "/":
        words = ref.rsplit("#", 1)
        if len(words) == 2:
            ref = "%s#%s" % (words[0].lower(), words[1])
        else:
            ref = ref.lower()
    return canonicalPath(ref)

Here’s mine:

from urllib.parse import ParseResult, urlparse, unquote

def parse_url(url:str) -> tuple[str, str]:
    """Takes a DSON-formatted URL and partitions it into a tuple.

    Returns the path (a URL's path) and the ID (a URL's fragment) of
    the desired file.
    """

    # TODO: Expand this to accomodate more aspects of DSON addressing.
    # http://docs.daz3d.com/doku.php/public/dson_spec/format_description/asset_addressing/start
    # May want to return a NamedTuple to avoid tuple indexing errors.

    # Separates the URL into its components. This could also be accomplished
    # with string split("#"), but for the sake of robustness, we use urllib.
    result:ParseResult = urlparse(unquote(url))

    # FIXME: URLs that point to the same file have no path. Should they be
    # None or an empty string? Need to know for comparison operations.
    dson_path:str = result.path
    dson_id:str = result.fragment

    return (dson_path, dson_id)

For one thing, you cannot use the same coordinate systems as DS does. Not only is the global coordinate system Y up, but daz bones don’t have to be aligned with the local Y axis.

I’ve just been keeping things simple and not rotating the meshes/bones.

It’s a modular API. It would be trivial to add a function that rotates everything it’s been loaded and feed it a loop of all imported objects in a high-level abstraction layer. That’s the advantage of having a function-based API, rather than putting all your functionality into an inflexible set of operators that must be called from the UI.

[Midnight Arrow]

I think I’ve got the armature loading code all sorted.

The only thing I’m not sure about is how to flip bones that are oriented on the Y axis in Daz Studio (the legs, mostly). Obviously we can’t rotate bones around the Y axis in Blender, so I just reused the X axis. I think that’s what the current code does.

    RX = Matrix.Rotation(pi/2, 4, 'X')
    FX = Matrix.Rotation(pi, 4, 'X')
    FZ = Matrix.Rotation(pi, 4, 'Z')

But there’s no comments, so I’m not sure.

Anyway, here’s the code I’m using.

  bone:EditBone = armature.edit_bones.new(node.dson_id)

  # For more compact code
  cp:Vector = node.center_point
  ep:Vector = node.end_point

  # Since we are not setting the tail position directly, we need to
  # set the length. It seems like length is applied on the Y axis,
  # so upon creation the bones will have their Y axis pointing to
  # global Y, not global Z (what happens when adding a 'Single Bone'
  # through the UI). This has implications for our matrix math.
  bone.length = (ep - cp).length

  # ================================================================ #
  # In Daz Studio, the first axis in the rotation order defines which
  # global axis the bone points down. However, in Blender, all bones
  # point along their Y axis.
  #
  # To compensate for this, we first need to rotate the Blender bones
  # in increments of ninety degrees, so they match the way they point
  # in Daz Studio's joint editor (while also compensating for Daz's
  # Y-up coordinate system).
  # ================================================================ #

  daz_direction:Matrix = None

  match(node.rotation_order[0]):
      case 'X':
          daz_direction = Matrix.Rotation(radians(-90), 4, 'Z')
      case 'Y':
          # Default Blender axis
          daz_direction = Matrix.Identity(4)
      case 'Z':
          daz_direction = Matrix.Rotation(radians(90), 4, 'X')

  # ================================================================ #
  # 'Orientation' in Daz Studio's joint editor is a rotational offset
  # from the primary axis (the first axis in the rotation order). 
  #
  # https://www.daz3d.com/forums/discussion/151601/a-question-regarding-joint-orientation
  #
  # Due to the way DSON stores angles as Eulers, the 'convenience' 
  # methods in mathutils aren't very convenient (eyelid twisting at
  # 180.0 degrees, etc). So instead, we just brute force the matrix
  # multiplication.
  # ================================================================ #

  daz_orientation:Matrix = None

  x_rot:Matrix = Matrix.Rotation(node.orientation[0], 4, 'X')
  y_rot:Matrix = Matrix.Rotation(node.orientation[1], 4, 'Y')
  z_rot:Matrix = Matrix.Rotation(node.orientation[2], 4, 'Z')

  match(node.rotation_order):
      case 'XYZ':
          daz_orientation = x_rot @ y_rot @ z_rot
      case 'XZY':
          daz_orientation = x_rot @ z_rot @ y_rot
      case 'YXZ':
          daz_orientation = y_rot @ x_rot @ z_rot
      case 'YZX':
          daz_orientation = y_rot @ z_rot @ x_rot
      case 'ZXY':
          daz_orientation = z_rot @ x_rot @ y_rot
      case 'ZYX':
          daz_orientation = z_rot @ y_rot @ x_rot

  # ================================================================ #
  # Bones are flipped according to whether the end point is above or
  # below the center point on the primary axis. They aren't mirrored,
  # just rotated 180 degrees.
  # ================================================================ #

  rotation:Matrix = daz_orientation @ daz_direction

  match(node.rotation_order[0]):
      case 'X':
          if ep.x < cp.x:
              rotation @= Matrix.Rotation(radians(180), 4, 'X')
      case 'Y':
          if ep.y < cp.y:
              rotation @= Matrix.Rotation(radians(180), 4, 'X')
      case 'Z':
          if ep.z < cp.z:
              rotation @= Matrix.Rotation(radians(180), 4, 'Z')

  # The 'center' point of a bone is the position of its head.
  translation:Matrix = Matrix.Translation(node.center_point).to_4x4()

  # Final update of bone
  bone.matrix = translation @ rotation

‌

[Thomas Larsson]

Nope, the end-point doesn’t have anything to do with it. The way a bone transforms a mesh is given by the pivot and the local coordinate system. In Blender, that is encoded in Bone.matrix_local, which is a 4x4 affine matrix which covers both. In DS, the pivot is the center-point, and the local coordinate system is the orientation matrix, described as an XYZ Euler. The end-point is irrelevant, just as the bone length is irrelevant in Blender. The plugin uses the distance between the end and center-points as the length of a bone, but that is just for visual pleasure.

However, the end-point is often located along one of the coordinate axes in DS. For the first few years of the plugin, I used the end-point as the bone tail, but a very smart Japanese guy called Engetuduouiti informed me that this was wrong in general (Unfortuately Engetu’s English left a lot to wish for, so it took a while to understand what he was saying). The plugin now puts the the bone tail along one of the local coordinate axes specified by the orientation entry in the duf file. Finding out the right axis is a major PITA. The way the plugin does it (after half a year of debugging), depends on the Euler order, and still has some special cases. But that is also because MHX (my fault) and Rigify (beyond my control) want the main bend axis to be X.

My previous picture shows how the coordinate system looks in DS. Or rather, it shows the local Y axis, which in Blender is always directed along the bone.

[Midnight Arrow]

In Daz Studio, the visual representation of the bone is defined by the first axis in the rotation order. In this case, the X axis. The bone, with its rotation zeroed, is oriented so it points along the X axis (global right). The length of the bone, as you say, is defined by the distance between the center point and the end point. The length remains the same (albeit rotated) if the orientation is changed.

The primary axis (here, the X axis) defines an infinite plane in global space. If the end point crosses it so that its X position is less than the X position of the center point, the visual bone is (as far as I can tell) rotated by 180 degrees around its local axis.

You are right about the visual representation of the bone being irrelevant in Daz Studio, because Daz Studio uses the part of the mesh the user clicks on to select a bone. But Blender does require the bone length so the user can select it, and for IK bone chains.