Odin 3.0.1 Validation performance drops significantly in comparison to 2.1.13

[Jeffrey Campbell]

Discussed this with Tor on Discord, performance for validation runs from 2.1.13 to 3.0.1 took a huge hit with the integration/replacement of the validation backend to use the inspector/property drawing system. A validation run to scan the entire project went from less than 30 seconds to around 8:30-9 minutes average. This seems to be caused by a few things, but the principal issue is that some of the types used for inspector usage are not as optimized for validator usage as they could be, particularly for memory usage and creation of Unity types that incur a lot of garbage collection (scanning the entire project took up the majority of my free RAM out of 16 GB total). An example symptom of this is that AddComponent is called many times to add Unity components which I did not previously see when validating in 2.1.13.

Tor has an example profiler binary file to examine based on a capture of a smaller example; I’ve not included it here as an attachment since it’s near 2 GB in size, but it’s available for download from Google Drive here (https://drive.google.com/file/d/1FOHhfd7Ba4U18INwb3EnhHUGO8acsxh6/view?usp=sharing).

Profile Comparisons (non-deep profiles) for the same run

Odin 2.1.13

Odin 3.0.1

Deep Profile Example (using a smaller example sample, not scanning the entire project).

‌

[Tor Esa Vestergaard]

It seems like there are several different hotspots, but the largest guilty party is clearly how we detect which properties we should make for Unity types. I can think of several ways to speed this up, primarily by caching the results by type in a far better way than we do at present. If I’m correct, that should almost completely eliminate this part of the code as a performance issue, which will hopefully help a great deal.

I’ll be sending you builds on Discord whenever I feel like I’ve made significant progress, to confirm that my optimizations work as intended in your project Ideally I should be able to get it close to what it was before 3.0, but keep all the nice new features that the transition to the property system as validation backend have given us.

[Jeffrey Campbell]

Sounds good; I have a side branch that has the example scene to test scanning that I can take deep profile captures of and make available as you send me builds to test so that you can see the results, happy to help so that we can take advantage of 3.x!

[Jeffrey Campbell]

Happy new year! Just checking in to see if there is anything I can help test or check to help progress this issue

[Jeffrey Campbell]

Uploaded screenshots of various validation profile tests and profiler frame capture for the 3.0.31 Performance Fix Test version sent on 1/4/2021, they are available at this Google Drive link here. These should be comparable to the original ones on the ticket 1:1 in terms of comparing performance for specific areas and to be able to iterate on future fixes if desired.

The outcome of this fix from my local testing is that the performance for time was much better; overall times dropped from 8-9 minutes to about 2-3 minutes which is more inline with our 2.X Odin Validator usage. Garbage collected seems to be up on scanning the entire project from the original 3.0 test from 0.6 GB to 1.5 GB, but on the smaller test is down from 60 MB to 20 MB.

If there is any other fixes or tests you’d like me to run please let me know on Discord.

‌

[Jeffrey Campbell]

Just tested those changes out and am uploading new screenshots for the results. I didn't see much in meaningful changes for the smaller single scene test, but for scanning the entire project GC overall shot up from 1.5 GB to just under 3 GB and time to validate also increased from 120 seconds to 190 seconds.

[Tor Esa Vestergaard]

Should be resolved in 3.0.4 - if anyone still sees significantly slower validation times, let us know and send us some deep profiler data files of a validation run so we can see where the performance issue is.

[Ben Ward]

Hey Tor, I’m seeing super slow validation times after upgrading to v3 too. We previously validated in about 4 minutes, but now it takes several hours and then crashes (tested on four machines so far, with the same result). After profiling in the editor I’m seeing gigabytes of garbage being generated... about 25 million calls to GC.Alloc in one frame. :-/ Seems like this particular aspect of Odin has gone backwards quite significantly for us, which is unfortunate as we were forced to upgrade due to the new licensing requirements. As things stand we can’t use the Validator at all at the moment. Anything we can try? Thanks for your help.