Running out of memory constantly when working on NEF files

[Olivier Croquette]

When working on NEF files from a NIkon D800E, ART uses 2GB more memory every time a parameter is changed, for instance the color temperature. Once the system memory is used completely, the ART process is killed by the system. This happens as well when going through different pictures.

This does not happen with other file formats. With the later, the available memory goes done, but the free memory stabilizes at around 2GB, even when more changes are done.

Let me know where I can send the NEF file privately if it helps.

Reproducible with 1.15 and 1.17.2

[agriggio]

Hi, are you building art yourself? If so, which platform and compiler? Can you attach your AboutThisBuild.txt file?

And yes, a raw file would be useful. You can upload to Dropbox or similar. Thanks!

[Olivier Croquette]

Yes, I build ART myself, because otherwise it does not even start on my system, due to:

‌

(ART.bin:71335): GLib-GIO-ERROR **: Settings schema
'org.gnome.settings-daemon.plugins.xsettings' does not contain a key
named 'antialiasing'

Here is the build information:

Version: 1.17.2
Branch: 1.17.2
Commit: d8be8afed
Commit date: 2022-11-30
Compiler: cc 11.3.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: 3.24.5
Lensfun: 0.3.2.0
Exiv2: 0.27.5
LCMS2: 2.12
LibRaw: N/A
OpenColorIO: N/A
Build type: Release
Build flags:  -std=c++11 -ffp-contract=off -march=native -Werror=unused-label -fno-math-errno -Wall -Wuninitialized -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags:  -march=native
OpenMP support: ON
Mi-malloc: N/A
Build OS: Linux
Build date: 2022-12-17T06:58:00Z

‌

[agriggio]

thanks for the info. Please try rebuilding with mi-malloc enabled, there's more than one chance that it will solve the issue (it is on by default now, and I'm considering making it mandatory). I would also strongly recommend to enable libraw, but this is probably not related to your issue here

[Olivier Croquette]

Enabling mi-malloc solves the issue. It was not so easy to build with it on Ubuntu 22.04 because the pkg libmimalloc-dev 2.0.5+ds-2 does not provide CMake integration. I applied to following hack for that:

commit 1fac608e2372d3c4cb0485c4c1db5372b718478d (HEAD)
Author: Olivier Croquette <[e-mail]>
Date:   Sat Dec 17 10:39:41 2022 +0100

    Patch CMakeLists.txt for mimalloc

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5cc6b4e2b..f1a1e42ea 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -569,7 +569,9 @@ int main()
 set(HAS_MIMALLOC)
 set(ART_MIMALLOC_VERSION_INFO "N/A")
 if(ENABLE_MIMALLOC)
-    find_package(mimalloc)
+    # find_package(mimalloc)
+    set(mimalloc_VERSION 2.0.5)
+    set(mimalloc_FOUND TRUE)
     if(mimalloc_FOUND)
         if ("${mimalloc_VERSION}" VERSION_LESS 1.4)
             message(STATUS "found mimalloc ${mimalloc_VERSION}, but at least 1.4 is required")

‌

[agriggio]

thanks for testing, I'll try to take care of the cmake integration on my side. And I'll make mi-malloc mandatory from now on

[Olivier Croquette]

Thanks for the quick support. It would be also great if you could provide exact instructions how releases are build (which system, what packages are installed, what command line is executed) so that users can reproduce them more easily. That would also reduce the number of variants in the field and the support effort.

[agriggio]

I'll see what I can do. AboutThisBuild.txt does provide some of the info though

[Danny Heijl]

I have detailed build instructions for Ubuntu 22.04 where all relevant packages (libraw, exiv2, lensfun, ocio, mimalloc, …) are built from source. I wrote them down for my own use, any interest?

[Danny Heijl]

You can find my build recipe here

[agriggio]

Thanks! I’ll link it from the wiki

[Olivier Croquette]

Thank you Danny, I will try it soon.

[Danny Heijl]

@agriggio I see that “tools/build-art” in master still uses tcmalloc and does not use libraw and ocio, is that intentional? Or is it simply because you do not use it yourself to build ART?

[agriggio]

Indeed I don’t use that. But I’ll update it, thanks for reminding me!

[agriggio]

made mimalloc a hard requirement