1. Thomas Larsson Avatar Thomas Larsson
  2. Diffeomorphic
  3. import_daz
  4. Issues

Blender addon to generate rigged hd meshes, hd shape keys, normals and vector displacement textures from daz HD morph files [part 2]

Issue #814 resolved
Xin created an issue

[ Continued here: https://bitbucket.org/Diffeomorphic/import_daz/issues/1399/blender-addon-to-generate-rigged-hd-meshes ]

The source is available in the following repository: https://gitlab.com/x190/daz-hd-morphs . There you can also find compilation details for the .dll under MinGW on windows x64, and documentation.

If you find this addon useful, consider supporting its development and maintenance here, since my current hardware doesn’t allow me to develop it further or maintain it fast enough: https://xin888.gumroad.com/l/daz_hd_morphs .

Instead of uploading files here, it’s more convenient for me if you downloaded the files directly from the gitlab link above:

The latest version (the main branch) includes documentation in the form of a .pdf in “./blender_addon/documentation/daz_hd_morphs_documentation.pdf”. I included several step by step examples there.

Comments (109)

  1. bouich jules

    Hi Xin,

    the problem came back even with my new fresh install of blender, and when i try uninstall/reinstall the plugin i have this error:

    so might be something related… ?

    Thanks!

    ‌

    ‌

  2. bouich jules

    OK i think im almost sure i’ve figured out this issue, why it doesnt work:

    because of scalling.

    here is the model what you send me compare it to mine. mine are huge:

    ‌

    it’s scalling, my models are way too big…

    my blender unit is: 1

    while unit scale is 0.01 on the plugin

    ‌

    EDIT: changing the unit scale to 1 in the plugin fixed the problem, so it was not related with blender actually!

    So becareful of your mesh scaling and make sure to use the proper unit scale on this amazing plugin!

    ‌

  4. bouich jules

    Hello XIn,

    Any idea how to speed up the viewport after creating the HD mesh ? , i can’t add mannequin ;P

    it’s literally almost impossible to animate with the lag.

    thanks!

  6. Xin reporter

    I would only use the base mesh to create the animation (without subdivision modifiers), since you don’t need to constantly see the HD details to see how the overall animation is going. Then, at certain frames, you can unhide the HD mesh to adjust some HD details. Before rendering, hide the base mesh and unhide the hd mesh.

    Both the base mesh used for animation and the HD mesh could use the same armature. Or, since the armatures have the same hierarchy and bones' names, you can share Actions between them.

  8. Xin reporter

    Are you sure you are using that operator the right way?

    To use it:

    1. Create a copy of the body mesh and geograft mesh just before merge.
    2. Merge one set of body-geograft meshes with import_daz, as usual. Assume the merged mesh is called “merged”.
    3. Generate file with the new Geograft operator: select the body mesh, then the geograft mesh and finally the “merged” mesh, which should be the active one. Click the button. A .json file will be generated in the working directory.
    4. If you ever want to use the “merged” mesh as a “base mesh” with this addon (for example, to create a rigged HD merged mesh from it), you need to also specify the generated file above in this addon’s main settings for the morphs to work properly.

    ‌

  11. Xin reporter

    It needs to be recompiled for 3.1 since they changed some relevant Blender headers again.

    It’s not hard to do, but I currently don’t have Blender installed since my laptop’s fan broke. This laptop is 10 years old now and its problems have started to accumulate. It’s at the point where it can’t be used for anything demanding, so I don’t want to stress it too much since it overheats easily without the fan. I will try to maintain this addon again whenever I get around to buying a new computer (a desktop this time).

    The source code is available in the link in the first post in case anyone wants to compile it or modify it themselves.

    Eventually I would also like to focus on trying to compile a Linux version, and release the “dhdm writer” program as an addon maybe (e.g. it takes a sculpted mesh in Blender and produces a .dhdm file that can be used for hd morphs in daz).

    ‌

  12. bouich jules

    Thank you XIn, i think 3.0 and 3.1 are intercompatible so no hurries about updating your amazing plugin.

    ‌

  13. Alessandro Padovani

    Xin, I was a programmer a long time ago before becoming a IT professional. If you can give me instructions I’ll try to compile it for you. What do I need as a minimum setup for compiling ?

  14. bouich jules

    Actually you can create the HD mesh with 3.0 then open your blend file with 3.1, to be honest i see no need for a rush for it , better to wait and to avoid bugs and problems.

  15. Maneki

    Yes, it’s not a big deal. It works fine with 3.0 and I link characters to projects, which means I need at least 2 instances of Blender anyway. I hope you’ll be able to get a replacement for your laptop soonish.

  16. Aszrael

    Hi Xin,

    thanks for all your effort you put into it. Can’t wait to see your linux compilation as I not managed to due it myself.

    However, I found a bug in your Normals/Displacments baking procedure. For some reason your code cannot load dhdm files with a SPACE in it’s name (error: … cannot find file: … xxx%20yyy.dhdm).

    Seams like you parsing the .dfs and just grap the last part of the hd_url as a string without translating %20 → SPACE?

    example: …/…/SWM%20Face2.dhdm

    should be: …/…/SWM Face2.dhdm

    Either renaming the original file to: SWM%20Face2.dhdm or editing the hd_url by hand in the corresponding .dfs solves it.

    Quick fix for your utils.py would be:

    • additional import of: re → so first line

      import bpy, os, math, time, gzip, json, re

    ‌

    • add: dhdm_fp = re.sub('%20', ' ', dhdm_fp) in “read_dhdm_level(dhdm_fp)” //around line 502
      def read_dhdm_level(dhdm_fp):
      dhdm_fp = re.sub('%20', ' ', dhdm_fp) # strips '%20' from dhdm_fp
      if not os.path.isfile(dhdm_fp):
      raise ValueError("File "{}" not found.".format(dhdm_fp))
      with open(dhdm_fp, "rb") as f:
      h = f.read(8)
      return int.from_bytes(h[4:], byteorder='little', signed=False)

    No clue why I cannot paste python code here - always the format is messed up - sorry. So keep the indent level correct.

  17. Xin reporter

    Thanks for reporting that Aszrael, I think it’s now fixed if you download it from gitlab.

    As for compiling it Alessandro, I can try to explain the process, but it can become quite tedious since you would need to compile the libraries needed too (listed in “README.txt”). I don’t know if that’s worth your time right now. Maybe for future Blender versions (LTS version) it would make more sense.

  18. Alessandro Padovani

    Ok since it works with 3.0 I guess it’s not urgent. I’ll be available if you feel like trying it together and hope you’ll be back soon !

  19. Jochen Sutter

    The “HD expressions” and the “unique Smile HD” morphs are working to generate normal maps, but I tried it with the “Aging Face Details”. It does not work. I get this error

    ‌

  20. Jochen Sutter

    … I figured out. It does not work, if the .dsf and the .dhdm files have spaces in the filenames. I replaced them with anderlines and also corrected the pathnames in the .dsf file and then it works.

  21. Jochen Sutter

    I think the bug in the script is, that the spaces are replaced with “%20” in the filenames.

    The file name is ”AFD Forehead 01.xxx”

  22. Xin reporter

    Yes, that issue was reported above and it was fixed in the gitlab version (see the link in the first post in this thread), but I haven’t yet uploaded a .7z with the fixed version. You can just download the version from gitlab.

    I might also upload an up-to-date .7z soon, since I don’t think I will be getting a new computer soon as I don’t consider current gpu prices reasonable for what I would end up using it (I will be sticking to this old laptop for the time being since it works well enough for non-graphics related programming).

  23. ぬらとん

    Hello, Xin. Thank you for an awesome addon.

    By the way, “HD mesh generation“ and “HD shapekey” return errors.

    How can I solve this?

    ‌

  24. Maneki

    It’s only compatible with 3.0 as of now. If you use 3.1 or 3.2 you’ll get errors. Xin didn’t have time/the possibility to compile a version for the newer Blender applications.

  27. Alessandro Padovani

    That's amazing news thank you Xin for taking the time to do this. Also because your addon is essential to get HD expressions and it’s frequently requested and referenced at the daz forum. Going to test I’ll let you know if there’s anything.

  33. Maneki

    Is there a way to do sculpts with multires on a model and apply them as shapekeys to the true HD mesh? When I try it the transfer works but the mesh is completely messy.

    ‌

  35. Maneki

    Because I want to use HD morphs, even if it’s just subD2 but be able to sculpt with good performance.

  36. Alessandro Padovani

    Then you could keep two copies of the mesh, one with multires for sculpting and one with multires applied to transfer the HD shapekeys to. Doesn’t this work ?

  37. Maneki

    No it doesn’t work, because the multires character apparently gets loaded without vertex groups but when I try to transfer the vertex groups (which works) the weight maps are completely messed up.

  39. Ulrich Owen

    When I try to enable the plugin in Blender 3.2.2 or 3.3.0, I get the following error. Any Idea how to solve that ?

    ‌

  42. bouich jules

    Please Xin, a support for Genesis 9 / Victoria 9 please

    your plugin is just a MUST.

    ‌

    thank you again!!

  43. Maneki

    That’d be really appreciated, yes. Especially since HD morphs are all on one character now it’s more worth than ever.

  44. Maneki

    @bouich jules I got to try Vic9 with the HD addon now. I was able to convert the character into HD and load a nipple morph as test on it. Do you get errors in certain situations?

  45. bouich jules

    @Maneki i didn’t deep try yet, i will try it the weekend hopefully will have support for standard morph by then

    i have no morph to test, but so far yes you can generate the mesh.

    i’m going full in depth test on it the weekend.

    ‌

  46. Maneki

    Does someone here have expierence with baking custom multires HD details with the HD addon? I try to bake selfmade HD details like nipples, feet soles etc. but when I bake all I get is fully grey vector displacement maps.

  48. Maneki

    @Alessandro Padovani I want them as vector displacement maps though and there’s currently no easy way to do that with the Blender tools.

  52. Maneki

    @Alessandro Padovani Normalmaps aren’t Vector Displacement Maps!

    ”A vector displacement map is a displacement map that can displace in multiple directions. The data is encoded similarly to how per-pixel normal data is stored in a normal map, however that is where the similarities stop. A VD map represents depth in complex multi-channel form while a normal map records normal direction in the same RGB space.”

  53. Rincewind

    Maneki is right.

    Vanilla Blender can bake normal and displacement maps but no vector displacement maps.
    This image from the zBrush docu shows the difference between a displacement and a vector displacement map:

    http://docs.pixologic.com/wp-content/uploads/2013/01/VectorDisp02
    Unfortunatelytelly, Blenders displacement mapping baking sucks for displacement maps. I never got there good results.
    Only baking vector displacement via the HD addon gives me good results.

    For easier baking with vanilla Blender, I can recommend this addon: https://blendermarket.com/products/bake-wrangler

    But as already mentioned, displacement maps are troublesome. I always get there seams on the UVs.

    And I can confirm that the HD addon works only well with baking vector displacement maps from DAZ morphs, but not from geometry inside the Blend file.

  54. Alessandro Padovani

    Ok now I see what you mean. Sorry I don’t know if the HD plugin can do it. But vector displacement requires a very dense subdivision to be effective, and scalar displacement is usually enough for daz figures. Then personally I’d go with multires plus normal maps, since everything else is going to be a resource bogger.

  55. Maneki

    @Alessandro Padovani It is true that vector displacement or displacement in general needs more dense mesh but it’s totally useable with the adaptive subdivision feature of Blender.

  56. Rincewind

    I don’t see why vector displacement should require a denser subdivision than regular displacement maps.
    The only difference is that vector displacement has 3 dimensions instead of only 1 dimension.
    And that currently, it’s the only way to get clean displacement maps without seams in Blender ;)

  57. Maneki

    I figured out how it kinda works with the Xin addon to bake custom HD details. It’s not good enough for me personally to use though. I probably will either use it directly with multires modifier or true HD mesh.

  58. bouich jules

    Hello @Xin

    please the only problem that i am having with G9 is i can’t add my favorite morphs without having this error “

    "

    ‌

    ‌

    please can you have a quick look at it? ( i can’t delete them manually cuz they are like thousands of duplicate from favorite morph)

  60. bouich jules

    Another HUGE problem i noticed, the smile and smile open, and many FACS/expressions doesn’t work on the HD mesh.

  63. Maneki

    It does work for me so I assume it’s a mistake in your workflow. Here’s what I do:

    1. Load G9 with the daz importer
    2. Load all morphs on the character AND the face meshes (save as favorite morphs)
    3. Create a true HD mesh for the HD morphs with Xin’s addon
    4. Merge both rigs together (IMPORTANT)
    5. Load the favorite morphs on the true HD mesh
    6. Put the FACS details slider from 0 to max.
    7. It works

    ‌

  64. bouich jules

    hello @maneki

    i tried dozens of time it doesn’t work, you can’t load the favorite morph on the plugin without having the “ duplicate errors” here is exactly the steps that i did.

    ( i want only the standard morph to works on the HD G9)

    ‌

    1. load the base g9 mesh
    2. import all standard morph
    3. save favorite morphs
    4. create an HD mesh with xin plugin in the HD mesh generation
    5. merge the rigs
    6. go to xin plugin again and load favorite morph in the HD shape key

    you get an automatic “duplicate error”

    ‌

  66. Maneki

    At some point I was able to reproduce the error but I’m not exactly sure what caused it. It must be some very specific step in the workflow. With my usual workflow I don’t have any errors and I get “Head”, “FACS Units”, “FACS details” and “FACS Expressions” to work on G9 true HD all at once. “Units”, “Expressions” and “Visemes” are irrelevant because G9 only has FACs.

  67. bouich jules

    i have no idea how you made it work, i tried to uncheck “Units”, “Expressions” and “Visemes” when i import, then save favorite morph, but i still have the same crash that happen when i try load the favorite morph.

    ‌

  68. Maneki

    Try this:

    • Load a normal G9 without anything into Blender
    • With standard morph apply all FACs “morphs” to the character (without transfering it to face meshes, it doesn’t seem to work currently anyway. You can only transfer them manually with the transfer button and I already reported that)
    • Save those morphs as favorite morphs
    • Create a true HD mesh
    • Merge the rigs BEFORE applying any shapekeys (In your video you don’t merge the rigs before applying the shapekeys. If you do merge afterwards you break the driver connection)
    • Import your favorite morphs to the character and select the true HD mesh as target. Don’t change anything. Keep the “both” option for the shapekeys.

    ‌

  70. Maneki

    The addon seems to be unusable with 3.4 currently. I always get immediatly a error when I try to convert a character into HD mesh. I hope you are doing ok Xin, Happy New Year!

  71. bouich jules

    hey @Maneki actually i stopped using the addon, i am using the “ apply subdivision” feature it give good results. ( super slow rendering but it’s okay)

  72. Alessandro Padovani

    Just want to note that “apply subdivision“ doesn’t do HD jcms/morphs, you need Xin’s for that. So, if you need HD morphs better stay with 3.3.2 until @Xin updates.

    note. Personally for production I’m still using 2.93 LTS, since 3.3 has volume bleeding and 3.4 crashes with eevee, and at Blender’s they don’t seem to care fixing those.

  73. bouich jules

    i think Xin left us ;-)

    hopefully thomas can take over the operations of the HD plugin operations.

  75. Maneki

    I was about to ask if maybe Thomas could integrate it into the daz importer and at least keep it functional for further Blender iterations.

  76. Rincewind

    I was about to ask if maybe Thomas could integrate it into the daz importer and at least keep it functional for further Blender iterations.

    +1

  79. bouich jules

    that’s one reason why i always supported and asked devs to have there donation platform.

    so they don’t disappear when you need them cuz at least they will let there donors know :D

    ‌

  80. Thomas Larsson repo owner

    Morgan, I don’t remember having made a file called daz_morphs.py. Is this your own file?

    Anyway, you cannot access the context in the register function, because it is called during the startup process, before a proper context has been established. Code that require a context must be executed later, when Blender has finished initialization.

  81. Xin reporter

    I’m rewriting the dll to remove all dependencies on Blender’s C++ code, and only depend on Blender’s API through python. This is not as straightforward as it might seem since the dll depends on correct face ordering and not just vertex ordering, but it’s doable, and the performance penalty is not as large as I thought it would be (it’s barely noticeable with my old hdd and it will be even better with a ssd).

    The collada exports/imports in Blender are quite fast, and the .obj exports/imports, which used to be very slow on Blender (that was the main reason why I originally made the .dll interact directly with Blender code), are now implemented in C++ by Blender so they are improving quite fast.

    I don’t know when I will finish this, but it shouldn’t take long. Furthermore, I wanted to write documentation for the addon so people don’t need to go looking for instructions all over the place.

    This new version would only target Blender 3.4+. For the current Blender LTS versions, you can use the versions on the gitlab link in the first post here.

  82. Alessandro Padovani

    Xin, welcome back your addon is essential to work with HD figures, so thank you for all your efforts.

    If you have some time, please explain why Mike wasn’t able to compile with code blocks, this may help anyone who wants to contribute your code. See #1369.

  83. bouich jules

    WOW xin is back!

    please also a support for G9 and specially victoria 9 HD it’s not yet supported!

    ‌

    thank you so much ;-)

  84. Xin reporter

    For those interested in trying to compile the .dll, here are the libraries and headers I used, and the compiler settings. If you use another environment other than Code::Blocks, then it’s likely you will need to do some changes.

  85. Alessandro Padovani

    Thank you Xin for sharing the code::blocks settings, that will do great for developers who want to contribute.

  86. Iago Monteiro

    Hello there. Newbie here.

    I downloaded this add-on a long time ago, and I remember there was a walkthrough/guide with it. Downloaded the new version and I can’t find it anymore. Somewhere in the previous thread it was mentioned there was a walkthrough in the notes folder of the .7 release, but it’s gone in the newer ones and I can’t find the old .7 version to download.

    Would anyone made the walkthrough available again? I understand it’s outdated, but it’s still better than nothing and might help me figuring out the issues I’m having and how to properly use the add-on.

  90. Xin reporter

    Hello, I was finally able to remove all dependencies on Blender C++ code and finally release a version for Blender 3.4. Maintenance from now on should be easier. Read the first post in this Issue for additional information.

    Implementing and testing the latest changes caused several crashes on my old laptop, which is not suitable for 3d work anymore (beyond the very slow performance I keep running into driver issues with 3d applications, which crash the entire system). Since my current hardware makes it quite hard and time consuming to continue properly maintaining and extending this addon, a link to a gumroad page is now available. The addon will always be free and open source regardless.

  92. Alessandro Padovani

    Thank you Xin for the nice upgrade, I just got it on gumroad with a little payment and wish you the best of luck with your shop. One advice, if I search for “daz hd morphs“ both on google and gumroad itself there’s no match. If you want people to find your addon and sell it then may be you need a better unique name as “xin hd booster“ for example.

    @Thomas Larsson Not to push in any way if there’s any reason not to. But may be you could place a link to the Xin gitlab and gumroad shop in the diffeomorphic blog, just to help him out.

  93. Maneki

    Me too. I got it on Gumroad and sent you a little bit financial support. Thank you for the update. <3

  95. Maneki

    I still get a error on Blender 3.4 with the “3.4” Gumroad version. Is it supposed to work as of now?

    ‌

  96. Xin reporter

    Thanks for the support.

    Maneki, that problem is a last minute change that I made and I didn’t notice the typo. I have uploaded the fixed version to gitlab and gumroad.

    The documentation was done with the latest version of the addon on Blender 3.4.1, so most things should work fine. If there are any other bugs, let me know.

  97. bouich jules

    Hello Xin,

    i also did a donation on gumroad right now, i would really appreciate if we can get a support for victoria 9 HD.

    i am no longer using victoria 8 hd or genesis 8, i moved to genesis 9

    ‌

    thank you so much

  98. Xin reporter

    bouich jules, are you sure it doesn’t work? I just tried it and it seems fine, although I haven’t tested it much on genesis 9 yet:

    The steps I followed are:

    1. Import the g9 with import_daz

    2. Import brow FACS

    3. Import brow FACS details

    4. Generate Rigged HD mesh (follow steps in the “Generating HD mesh and driven shape keys from current base mesh” section of this addon’s documentation).

    Could you be more specific about your problem? one thing that could definitely be improved is the automatic duplication, parenting and retargeting (to the hd mesh rig) of the additional original meshes (non-HD meshes like eyes, mouth, eyelashes). This isn’t hard to do (it’s just basic python stuff) and I might add it soon. But the HD morphs seem to work fine.

    That aside, another thing I wanted to test later is the speed of the new Blender .obj importer, since right now the addon uses collada files for HD shape keys and I’m not sure if that’s faster than with the new .obj importer. This requires rewriting a little of the dll code since the dll .obj exporter code is quite obsolete.

  99. Maneki

    G9 FACS worked for me in the past and I checked now if it still works for me in the newest version on 3.4 and yes, they work for me. I assume you got an error in your workflow bouich.
    Try to save the FACs and FACs details on G9 base mesh as favorite morphs and load them in after you merged rigs after generating the hd mesh.

    ‌

  100. Maneki

    @Thomas Larsson The issue above seems to be related to easy import. When I import G9 with FACs and FACs details via easy import, the xin addon spits out a error when I save those morphs as favorite morphs and load them on subD3 generated mesh with Xin’s addon.
    When I load in the base mesh with the daz importer WITHOUT FACs and FACs details and add them manually with the daz importer UI, it works.

  101. Xin reporter

    Maneki, that problem was related to this addon, not import_daz. It has now been fixed.

    The new version (updated in gitlab and gumroad) adds the following:

    • Option to copy and properly reparent additional non-HD meshes (like eyes, lashes, etc.) when generating HD meshes.
    • .obj files are now used instead of .dae files for importing back shapes into Blender. These can give some performance improvements.
    • When importing import_daz favorites, the addon now checks the validity of the morph files (based on selected base mesh and optional geograft file), and ignores the ones which are invalid instead of adding everything as the previous versions did.
    • A new button now exists in the morph files panel that lets you clear invalid files (based on selected base mesh and optional geograft file) from the list of morphs. This can be useful when adding a lot of morph files manually.

    Invalid morph files and the causes are reported in the console. Note that, in the context of this addon, an invalid morph file is not just a morph file which doesn’t fit the base mesh, but can also be a morph file which defines no mesh deformations too (for example, a .dsf file which just moves bones, those are automatically transferred with the rig and don’t belong to the morph list).

    Also, I deleted those old notes from the attachments here to not cause confusion. The up to date documentation can be found in the latest version in either gitlab or gumroad.

  104. Xin reporter

    Maneki, can you give more information about it? like the console output? I can’t reproduce that error. I can import morphs either directly while generating the rig or with the favorites file later on, and it works fine in both cases, at least with the morphs I’m using. It also worked while importing all FACS on genesis 9.

    Also, what happens when you press that new button on the list of morphs you are using? what does the console say?

  106. Xin reporter

    Maneki, I still can’t reproduce it. I can import FACS details fine. Can you describe the steps and the figures you are using?

  107. Maneki

    Xin, I think I simply picked the wrong mesh to load the HD shapekeys on. Your addon generates face meshes as HD now too and my muscle memory possibly caused some errors, sorry for that.^^

  110. Log in to comment
Assignee
–
Type
enhancement
Priority
major
Status
resolved
Votes
0
Watchers
12
Jira: the preferred issue tracker for Bitbucket. Join the team!