A surge in JVM process memory while exporting to PNG/PPM format

[mohammed faizaan]

USE CASE: WHAT DO YOU WANT TO DO?

Export a file to PNG or PPM format

STEPS TO REPRODUCE AN ISSUE

1. Start TreeView3 with program memory 8GB (set VM arguments -Xmx8200m)
2. Open large_6kx6k.txt
3. After importing, check the JVM size of TreeView3 using Task Manager. usually is around 3.8GB and is named as javaw.exe process
4. Export the whole matrix to PNG format. Keep monitoring treeview3 JVM size. When the export is going on, it suddenly increases to 6.7GB
5. After the export is done, TreeView opens the file. Close the file. Now check the JVM size. It is still 6.7GB even after closing the Export dialog box. The JVM size remains the same post this issue.
6. This issue does not occur in PDF/SVG/PS format.

CURRENT BEHAVIOR

Sudden increase in JVM size.

EXPECTED BEHAVIOR

JVM should not be taking so much space, otherwise TreeView will run into OutofMemory Exception

DEVELOPERS ONLY SECTION SUGGESTED CHANGE (Pseudocode optional)

unknown

FILES AFFECTED (where the changes will be implemented) - developers only

Might be in ArrayDrawer.java. (while painting into the graphics object)

LEVEL OF EFFORT - developers only

medium

COMMENTS

When you follow the same steps but export to PDF/SVG/PS format, the JVM size remains same i.e. 3.8GB during the whole export as well as post export. PNG and PPM follow the same code path. So, resolving one might resolve the other.

I have used the latest code from the master branch to reproduce this issue I am attaching a video - https://bitbucket.org/smd_faizan/treeview3/downloads/ExportHeapSpaceIssue.mp4

[Robert Leach]

If you export a second time (same data) or a second time (after opening a new file), does the size grow more?

[Srikanth Bezawada]

I think one way for resolving this issue would be to use eclipse memory analyzer and examine the variables and code in parallel by looking at the memory snapshot(heap dump file) provided by analyzer. I shall give this a try this weekend.

[mohammed faizaan]

No, the size does not grow more after subsequent exports. It remains 6.7GB post export process is done.

[Christopher Keil]

This could be a hint. https://stackoverflow.com/questions/18712982/java-imageio-memory-leak

Looks like it is about ImageIO in the ExportHandler.exportImage() function.

If you use a tool like jVisualVM that allows you to manually run garbage collection, is this memory being freed?

[Srikanth Bezawada]

@TreeView3Dev Not sure about jVisualVM.

Many links in stackoverflow recommend that we need to focus on alternatives instead of ImageIO which use less RAM. I ran heap dump analysis using eclipse memory analyzer and this is the only clue I got.

[mohammed faizaan]

I would suggest putting this issue on hold.

I found out that this is neither the stack space nor the heap space that is increasing. It's File/kernel or user space of the program that has increased (Something which BufferedImage uses). HeapSpace and stackspace remain the same before export and after export. Saw this by taking heap dump and using max stack space to 1MB (which is default).

As Srikanth said, I too saw few links which say we shouldn't be worried about this memory. Rather, we should be focus on decresing the time taken or may be use a buffer while exporting (which again is low priority as seen in some of the comments)

[Robert Leach]

Removing milestone: Import/export data (automated comment)