1. Thomas Larsson Avatar Thomas Larsson
  2. Diffeomorphic
  3. import_daz
  4. Issues

HD export will convert Multires to Subdivision surface modifier when overriding DAZ default subdivision

Issue #1231 resolved
Arrow created an issue

Hello!

First, i’m sorry for my poor grammar, as English isn’t my native language 😅

When exporting from Daz to Blender using Diffeo’s HD Export script (not the Xin’s DLL addon) with DAZ default viewport and render subdivision set to 3 the mesh have a Multi-resolution modifier, which is correct :

‌

But when overriding DAZ subdivision limit and set it to 5 then exporting with the same HD Export script, the mesh have a Subdivision surface modifier (instead of Multi-resolution), which is incorrect :

‌

‌

I’ve already checked “Build HD Meshes” and “Add Multires”, and set the “Max subdivision level” to 5 in the general settings, but it does not seems to work ?

‌

When exporting using Xin’s DLL addon with DAZ subdivision set to 4 (I cannot set it to 5 as indicated in the next paragraph), the mesh have a Multi-resolution modifier which is correct too :

‌

So apparently it’s possible for Diffeo to import subdivision 4 and higher with a Multi-resolution modifier, but it does not seems to work with the native HD export script ?

Also, Xin’s DLL addon cannot export DAZ subdivision 5 and higher (but 4 and less is fine) ? :

‌

I’ve already looked at the documentation and similar issues, but i’m not sure if I did something wrong ?

System informations :

  • OS : Fedora 36 (Linux)
  • Blender : 3.3.1
  • DAZ : 4.21
  • Diffeomorphic : latest master (currently 1.6.2.1253 : 5666333)

‌

‌

Comments (23)

  5. Alessandro Padovani

    daz studio 4.15.0.30, blender 3.3.1, diffeomorphic 1.6.2.1253

    I don’t have the patience to test a 5x G8F as it takes too much time and memory. But it works fine here with 5x primitives and a 3x G8F, without limits. So can’t reproduce the issue. Test scenes included cube-hd and g8f-hd.

    Let us know if my test scenes work fine for you too.

    ‌

  8. Arrow reporter

    Here are the results from your scenes :

    • “cube-hd.duf” at 5x subdivision produces a 5x Multires when imported in Blender
    • “g8f-hd.duf” at 3x subdivision produces a 3x Multires
    • “g8f-hd.duf” at 5x subdivision produces a 5x Subsurf

    Maybe I can send you the DBZ file (700Mo) to save you some time exporting it ? I’m not sure if it’s allowed by DAZ EULA, that’s why i’m asking.

    ‌

    I’ve also tried different objects : DAZ Material Ball, some Bardot default clothes (from Genesis 8 starter package), different props, etc. They all produces 5x Multires, it seems to only affect the characters.

    Surprisingly, Genesis 8 Male at 4x subdivision produces a 4x Multires which is correct, but a 5x Subsurf at 5x subdivision, i’m not sure what’s causing this ?

    ‌

    ‌

  9. Alessandro Padovani

    No my test scenes were only to help Thomas, since a 5x G8 is quite intensive and my hardware can’t take it for example. It may be that multires fails for 5x because there’s not enough memory so the code reverts to subd for some reason. But this is a job for Thomas I can’t help here.

    From your tests I understand it only happens for figures at 5x, while up to 4x it works fine.

  10. Thomas Larsson repo owner

    I recall a previous case where 4 subdivs worked fine but 5 didn’t. That was a limitation of DS; the HD meshes where not written to the dbz file. Please check the file size of the dbz file with 4 and 5 subdivs. The file with more subdivs should be much larger. If it isn’t, the DS API ran into some limits.

  11. Arrow reporter

    It may be that multires fails for 5x because there’s not enough memory so the code reverts to subd for some reason

    I have 65 Go of RAM : during export at 5x, the “DazStudio.exe” process uses 15-20 Go RAM and ~4% CPU, so there’s plenty left 😃

    Moreover I always get the “succeed” message at the end of the 5x export, so the script didn't get killed :

    ‌

    From your tests I understand it only happens for figures at 5x, while up to 4x it works fine.

    Yes, apparently only the figures seems affected. i’ve tested other type of export like props, clothes, etc and 5x seems to work fine with Multires.

    What I dont understand is why the cube (from “cube-hd.duf”) works but apparently not the figures, maybe from a certain number of vertices it fail ?

    Xin’s DLL addon also report this error when trying to export at 5x subdivision, as Thomas said, maybe there is some kind of limitation ?

    ‌

    Please check the file size of the dbz file with 4 and 5 subdivs. The file with more subdivs should be much larger

    The DBZ files are of different sizes :

    • DBZ for 4x is 300 Mo, when extracted is 1.2 Go
    • DBZ for 5x is 670 Mo, when extracted is 2.1 Go

    Here is the log when importing the base “Genesis 8 male” at 5x subdivision DBZ file into Blender (with Diffeo verbosity = 5) in case you need it : https://bitbucket.org/2027890845/test/src/master/diffeo-import5x-log.txt

    I am totally able to fiddle with python scripts and run some tests if it can helps you.

  12. Thomas Larsson repo owner

    Apparently the export to blender script fails silently if it runs out of memory. I’m not really confortable with dazscript - almost all my work is done in python on the blender side - and I don’t know if there is some way the script could detect memory overruns. This api seems to hide this, and the HD mesh simply becomes invisible for the script. In that case the dbz file at subdiv 5 is smaller than at subdiv 4, because the HD info was not written to it. Normally higher subdivs result in much bigger file size. You can check the file size in a file browser.

    This seems to be the case here. The only HD mesh that was detected was the eyelashes (look for lines starting with Highdef).

  13. Alessandro Padovani

    @Thomas May be you can write a flag in the dbz for the HD export. Then the importer can check if all the subd meshes have a HD version, if not, then a warning could be displayed that the HD export failed out of memory. Of course it would be better to fix the daz script but this could be an alternative.

    p.s. Or may be there’s some clue in the script by Xin how to handle the daz script for failure ?

  14. Arrow reporter

    Yesterday evening I did a little rewrite of the DSA export file, instead of writing line by line to DBZ0, i’ve created a massive object with all properties, then save it in one step with JSON.stringify.

    The script is still able to get he 16 millions vertices at 5x so theoretically it should work : you can see “doVertices: 16589418” above. (as you know : subdivision 4 is 4.4M vertices and 5 is 4 more which is 16M)

    2022-11-08 19:31:56.001 [DEBUG] :: ----- start -----
2022-11-08 19:31:56.001 [DEBUG] :: doMesh: DzFigure
2022-11-08 19:31:56.002 [DEBUG] :: doVertices: 16589418
2022-11-08 19:33:48.609 [DEBUG] :: doUVs: 16646064
2022-11-08 19:37:07.725 [DEBUG] :: doFaces: 16584704
2022-11-08 19:50:29.459 [DEBUG] :: doMaterialGroups: 16
2022-11-08 19:50:29.463 [DEBUG] :: doMaterialGroups: 16
2022-11-08 19:50:29.463 [DEBUG] :: doVertices: 16384
2022-11-08 19:50:29.556 [DEBUG] :: doMesh: DzFigure
2022-11-08 19:50:29.557 [DEBUG] :: doVertices: 1620
2022-11-08 19:50:29.565 [DEBUG] :: doUVs: 1674
2022-11-08 19:50:29.571 [DEBUG] :: doFaces: 1408
2022-11-08 19:50:29.585 [DEBUG] :: doMaterialGroups: 2
2022-11-08 19:50:29.586 [DEBUG] :: doMaterialGroups: 2
2022-11-08 19:50:29.586 [DEBUG] :: doVertices: 464
2022-11-08 19:50:29.596 [DEBUG] :: new DzFile: C:/users/fedora/Documents/daz.dbz0
2022-11-08 19:52:28.453 [DEBUG] :: ----- finish -----

    i’m gonna do more debugging and see what happens 😃

  15. Arrow reporter

    So, after many trials I managed a DSA script that correctly export at 5x subdivision with 5 level of Multi-resolution in Blender 😀 .

    ‌

    With so much datas, some DAZ functions just don’t work as expected and fails silently, for example : “String.substr” is fine with 1Go but not 4Go. DzFile.write() / writeLine() somehow overlap or some lines get skipped … The DBZ0 file size is now 4.2Go (instead of 2Go previously) which is the correct size as it’s 4 more than subdivision 4 level.

    The script, instead of calling multiples “writeLine”, constructs a huge object containing all properties in the same format expected by Diffeomorphic. Afterwards the object get “JSON.stringify” into a huge 4Go string, then chunked into 32Ko parts and finally written on disk.

    ‌

    Here is the final modified “export_highdef_to_blender.dsa” script if anyone interested :

    https://bitbucket.org/2027890845/export-highdef-5x/src/master/export_highdef_to_blender_5x.dsa

    Attention: the whole process take about 30-40 minutes, DAZ can eat up 30Go of RAM and Blender 50Go during import. This technique is about 3-4 slower than Diffeo and not optimized 😅

    ‌

  16. Alessandro Padovani

    If I understand correctly.

    Personally I believe it’s not a good thing to use such amount of memory to build a single huge object. If it’s not possible to use some functions with more than 1GB, the way to go would be the opposite and write multiple lines to file. At least in #191 we found that writing multiple lines to file is not too slow compared to a single write because the file system caches it. See the start of the discussion.

  17. Arrow reporter

    @Alessandro Padovani Thanks for your reply, I did this single huge object because apparently in the original Diffeo’s DSA script, some calls to “writeLine” get skipped (not sure if it’s really the case, but the file size is half as much).

    Afterwards, the string get chunked into multiple smaller parts which are appended to the file. it’s not a single write.

    I’m not an expert in optimisation, and i’m not sure if there is an other solution as Diffeo requires a single JSON file.

  19. Alessandro Padovani

    @Thomas, the c++ version by Donald is certainly an option, though it doesn’t work with shells and geografts. But unless the official script is fixed it should be noted in the docs that it can’t export huge meshes because of limits in the daz api. Then the fact that it fails silently and can’t even warn the user is another more issue.

  21. Arrow reporter

    Yes, I tried Donald’s export plugin, unfortunately it didn't work because of the “Export not successful” error. Is the source code of Donald Dade’s export script publicly available ? Does the licence permit modifications and re-compilations ? i’d like to take a look if possible to know where this error come from. On this issue : possible-speedup-for-the-hd-exporter he says it handle over Thomas, but that’s it.

    Note that i’m on Linux, I tried cross-compiling the “sample plugin” from DAZ SDK with Mingw and Wine, but I wasn’t very lucky with these two methods… so I could not do much except looking at the code 😥

  22. Alessandro Padovani

    I added the files in #191 since the link provided by Donald is no more available. That includes the source code. I know that the 32 bit version of daz studio can run in Wine, but not the 64 bit version. Though it is easier if you get access to windows.

    The HD exporter by Donald has issues with geografts and shells, and with the geometry editor in daz studio.

  24. Log in to comment
Assignee
–
Type
bug
Priority
trivial
Status
resolved
Votes
0
Watchers
2
Jira: the preferred issue tracker for Bitbucket. Join the team!