Concurrent module validation
When I open my IC2 project, I see a message “Validating modules for project XXX” at the bottom of the IDE, along with a progress bar.
We chose to use a git monorepo for our org, and for reasons I no longer remember, we structure the repo as follows:
- The project root is not a DX project (i.e., no
sfdx-project.json
file) - MOst (but not all) top-level folders are DX projects (i.e., with a
sfdx-project.json
file) that contain a single DX package. We have 29 of these folders so far, and they all exist as IC2 modules within a single IDEA project.
Watching the progress bar, it appears that module validation is single-threaded. I don’t know whether this is the IDE doing this or your code. If it’s your code and it’s possible to speed up by using multiple threads, that would be appreciated as it now takes about a minute to validate all modules.
Comments (13)
-
repo owner -
repo owner Mark, before I optimize something that’s not actually where the time is spent, can you please capture a debug log of the overall validation for this project? You can do so by adding the following to Help>Diagnostic Tools>Debug Log Settings:
#com.illuminatedcloud.intellij.moduletype.IlluminatedCloudModuleValidator
and then opening the project. Once validation completes, please grab your
idea.log
and either attach or email it for review. Note that because of the large number of modules, you may just want to grab allidea*.log
files that have last modified timestamps from when the project was opened. -
repo owner Hi, Mark. Did you ever get a chance to capture the requested debug log so I can see where the time is being spent during module validation? Thanks!
-
reporter - attached idea.log.zip
<div class="preview-container wiki-content"><!-- loaded via ajax --></div> <div class="mask"></div> </div>
</div> </form>
-
repo owner Thanks, Mark. There’s not as much detail as I’d hoped for in there, but obviously I can see that module validation is taking ~2 seconds on average per-module which certainly adds up with as many modules as this project has.
Before I ask for CPU profiling info and/or instrument this for more detailed debug logging, I have a hunch. Please disable the Salesforce DX option Automatically set default username:
and then close/reopen the project so that it validates all modules again. Does it still take a long time, or does that seem to address the problem?
-
reporter Hi Scott! Clearing that checkbox did the trick.
However, I do like the idea of ensuring that the correct default user is set across all of the modules. My first thought was that maybe you could invoke
sfdx force:config:get defaultusername
in each module and compare that result with what it “should” be. But then I ran this:$ time sfdx force:config:get defaultusername === Get Config Name Value Success ─────────────── ─────────── ─────── defaultusername mmscratch02 true sfdx force:config:get defaultusername 1.02s user 0.20s system 101% cpu 1.189 total
I’m not sure what happens within that command that makes it take over 1s! The default username value (if set) is stored in
.sfdx/sfdx-config.json
, but the command might be making a network call to validate that a scratch org with that name/alias exists? Or perhaps the startup overhead of thesfdx
tool is ~1s for any command?So here’s a proposed optimization that would allow keeping the “Automatically set default username” checkbox set, but with negligible overhead when things are already set correctly:
- When the checkbox is set, instead of invoking
sfdx force:config:get defaultusername
to get the value, just open.sfdx/config.json
and read the value directly. - If the value already corresponds to the current Connection alias, do nothing.
- If the value is missing or incorrect, execute
sfdx force:config:set defaultusername=<whatever>
- When the checkbox is set, instead of invoking
-
repo owner Okay, that confirmed where the time is going. And yes, I discussed bypassing the CLI and going straight to the file with one of the members of the Salesforce CLI team yesterday. As you can hopefully appreciate, IC2 tries to treat those files as opaque and the CLI as the sanctioned interface to their contents. However, it looks like it should be fine to read directly from the file for this purpose since it’s read-only and not 100% critical. I’m planning to include that optimization in the next build. I’m not sure if that will be next week or after the holidays, but in the interim you can disable that option.
Oh, and are you on Windows? If so, that’s the reason that the command takes >1 sec. There’s a pretty fixed overhead to running even the simplest of CLI commands on Windows. Well, to be clear, there’s an overhead on Mac and Linux as well, but it’s a much lower overhead. Still, reading from the file directly for this purpose should make that go to nearly 0.
-
reporter Regarding this:
I’m not sure what happens within that command that makes it take over 1s! The default username value (if set) is stored in
.sfdx/sfdx-config.json
, but the command might be making a network call to validate that a scratch org with that name/alias exists? Or perhaps the startup overhead of thesfdx
tool is ~1s for any command?
There’s definitely nearly 1s of overhead with the
sfdx
tool! I just timed it with the-h
flag for a command (definitely no network call happening for that), and that took nearly 1s to complete:$ time sfdx force:config:get -h get config var values for given names USAGE $ sfdx force config get [--verbose] [--json] [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL] FLAGS --json format output as json --loglevel=(trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL) [default: warn] logging level for this command invocation --verbose emit additional command output to stdout sfdx force:config:get -h 0.92s user 0.20s system 87% cpu 1.281 total ^^^^^
-
reporter Oh, and are you on Windows? If so, that’s the reason that the command takes >1 sec. There’s a pretty fixed overhead to running even the simplest of CLI commands on Windows. Well, to be clear, there’s an overhead on Mac and Linux as well, but it’s a much lower overhead. Still reading from the file directly for this purpose should make that go to nearly 0.
No, I’m on a Mac (and a super fast one at that: 2021 16” M1 Pro CPU with 32GB RAM).
-
repo owner Gotcha. I’ve passed that on to the person on the CLI team as feedback.
-
repo owner - attached IlluminatedCloud2.zip
Mark, here's a build of IC2 based on today's official release (2.2.5.1) but with the optimization to load directly from the
sfdx-config.json
file if possible before degrading gracefully to the CLI.You can install it by download (but not extracting) the attached archive and using Settings / Preferences > Plugins > Install plugin from disk (under the gear drop-down menu), then allowing the IDE to restart.
If you don't mind, please install it and let me know if it doesn't resolve the issue for you while still validating the default username setting. You'll need to re-enable that other config option as well, of course.
As stated previously, the plan is to have this in the next official build, either mid-next week or, if that doesn't happen, after the holidays.
-
reporter Hi Scott, just gave that a try and it works great! Thanks so much for working on this. I think now that we’re working through several of these issues, we can make a better case for why the IJ/IC2 experience is superior to VSCode.
-
repo owner - changed status to resolved
Delivered in 2.2.5.2.
- Log in to comment
Yes, module validation is single-threaded. I’m not sure if changing it to a multi-threaded model would be very useful as the bulk of the time is spent scanning files in the module, and trying to spread that out across multiple threads is likely just going to thrash the filesystem. My guess is that gains would be minimal. Having said that, I have a couple of thoughts of how I could perhaps make this much more efficient in such a project. I’ll take a look and see what I can do.