Unable to fetch additional LWC JS files

[Dmytro Kurylo]

I don’t know when this started to happen.

The problem is that when you have an LWC component with additional js files, they are not retrieved (or if retrieved, they are not created as actual files on the component folder), even if I’m sure they exist on that component.

This is easily repeatable on my set up:

1. On screenshot 1, here’s the older component that was created a while ago and does have an additional js file (labels.js).

2. On screenshot 2, I delete the component folder physically (assuming that I’m the new dev has not seen it yet)

3. On screenshot 3, I retrieve the whole org. Component I’ve deleted re-appears, but it lacks the js file (in my case it’s labels.js).

The component still functions correctly, as if the file was there, so I’m assuming that something is wrong on the sync/retrieval side.

UPDATE: if I use ANT, it retrieves the additional js files nicely together with the component.

IC 2.1.8.5,

IntelliJ IDEA 2021.1.3 (Community Edition)
Build #IC-211.7628.21, built on June 29, 2021
Runtime version: 11.0.11+9-b1341.60 amd64

[Vitalii Andrusha]

I have the same problem

[Scott Wells]

I wonder if it's perhaps matching the extra file against another same-named file in the filesystem. Can you please enable debug logging for metadata retrieval, reproduce the behavior, and either attach or email ([e-mail]) the resulting idea.log file(s) for review? That should show me a) whether those missing files are being retrieved at all; b) if so, where they're ending up.

[Dmytro Kurylo]

Here’s the example of reviewsRelatedList component deleted and whole lwc folder retrieved.

These are lines 1050…1100?

OMG! Am I correct that not only is does not add the file to the correct place but it also spoils a different component? Let’s hope nobody presses ctrl+s on that component, as it can be rendered unusable…

https://www.dropbox.com/s/kqwdk5j7i16vsa1/idea.log?dl=0

[Scott Wells]

Thanks. No, your assessment is correct. Just so I understand, you're retrieving the reviewsRelatedList LWC component for the very first time, correct? In other words, when this retrieval occurs, there's not a local copy of that component yet. I ask because I see all of the candidates for labels.js, and none of them are under that component's bundle directory. Had that bundle existed locally, there would have been a 100% match against that version of labels.js. I think that's why others wouldn't have run into this in the past (and this matching heuristic has been in place for a long while).

So having said that, there's certainly a problem here that needs to be addressed. I can update the heuristic to be bundle-aware and verify that the bundle name matches in addition to the current matching logic when it's certain that it's dealing with an Aura, LWC, static resource, experience, etc., bundle. I'll plan to do that for the next release.

[Dmytro Kurylo]

Yes, you are correct. I already had that component locally but for the sake of testing I deleted the whole component folder and retrieved the whole lwc folder assuming that the new component would reappear.

Yes, when I retrieve the component, it does not exist locally on my PC yet.

This simulates the case where you have an older copy of the org, and someone else has created a new component at some point (so I don’t have the component locally yet). Then I when a new piece of work is assigned (to me), I retrieve the whole org to sync the local codebase with it and continue working from that spot. In this case it was only the lwc folder but it does not matter I think.

I saw a similar behavior previously 1 or 2 times with additional js files being places incorrectly (repo showed a diff for files which definitely were not changed) but I thought that was my mistake (e.g. that I’ve placed the code in a wrong place by mistake), but now I’ll probably stop blaming myself in such cases :)

[Scott Wells]

Nope, that was IC2 most likely. Kind of a corner case, but obviously can lead to less-than-desirable results when that corner case is encountered. For what it's worth, I personally use Retrieve for Merge almost exclusively, and I recommend most folks do the same. That way you get a chance to audit the retrieved metadata before it gets placed in the local project directory. There are still good use cases for simple Retrieve directly into the local project, but those are mostly limited to retrieving more simple metadata--not bundles or decomposed metadata--when you know 100% that you want the latest server copy.

Having said that, I'll definitely treat this with the appropriate level of concern and take care of it shortly.

[Scott Wells]

Yeah, and the heuristic had to get more complex as some bundles--more static resource bundles than Aura/LWC bundles--have files with the exact same name at different paths in the same bundle. The latest heuristic is actually quite solid, though what you've identified here is a pretty significant (albeit specific) gap in it. I'll address that by making it aware of bundle names in relative paths and hopefully that will seal it up more tightly.

[Dmytro Kurylo]

So for now in order to sync with an org, the most solid way is either use Retrieve for Merge, or (which is simpler) killing the whole src/ contents and retrieving everything from scratch (in this case it won’t find another “copy” in a different place)?

[Scott Wells]

I suppose it really comes down to preference and familiarity, but I can't imagine removing the entire source tree and then retrieving being the most efficient route. I use Retrieve > Retrieve for Merge filtered by the metadata subscription and have registered Beyond Compare as an external diff tool for IntelliJ IDEA as it does a much better job of showing directory/file diffs (IMO). As soon as the retrieve completes, Beyond Compare is opened with the retrieved metadata on the left and the project metadata on the right, and all files that are identical between the two (taking into account unimportant diffs such as whitespace, line endings, etc.) are immediately filtered out. I'm left with things that are different, added, and removed, which is generally a very small list. If something wasn't aligned properly by IC2 during the retrieval (which you're seeing but I haven't in a very long time), the diff would show that clearly and I could adjust for it pretty easily. Otherwise it's quick work to bring the local project up-to-date with the org changes.

My $0.02 based on the process that I follow.

[Dmytro Kurylo]

Yes, but the problem with this use case is that you don’t know whether it were the files themselves that were placed incorrectly, or it was indeed some valuable ongoing change placed there by some other dev. But thanks for sharing, this is definitely something worth thinking about.

[Dmytro Kurylo]

Hello! Are there any updates on the issue?

[Scott Wells]

Nothing new to report at present. I'm traveling this week but can put this on my work queue for either next week or the following week.

[Scott Wells]

Hi. I tried to reproduce this behavior this morning unsuccessfully. I'll explain what I've done, and hopefully either we'll be able to determine what else I need to do to reproduce the behavior properly, or I'll need to get a debug log from you of the issue occurring that I can use to see it that way.

Here's what I've done:

1. Created a new empty project.
2. Created a new LWC component, component1.
3. Added a new JavaScript file to component1 named labels.js that includes a comment saying that it's "Component 1 labels".
4. Created another new LWC component, component2.
5. Added a new JavaScript file to component2 also named labels.js that includes a comment saying that it's "Component 2 labels".
6. Deployed both LWC components to the org.
7. Removed component2 from the project.
8. Started Retrieve Metadata and unchecked Local + Server so that only component2 is displayed for retrieval.
9. Clicked Retrieve to retrieve component2 directly into the project source directories.
10. The end result is that component2 was retrieved correctly with its own labels.js file, and component1 was not affected.

Here are the relevant debug log entries:

2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever - Processing C:/Users/Scott/AppData/Local/JetBrains/IntelliJIdea2021.2/plugins-sandbox/system/tmp/issue_1949-issue_1949-retrieve1/tmp/force-app/main/default/lwc/component2/labels.js. 
2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever -   Using filename labels.js. 
2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever -   Found two files with matching file names but different metadata types and/or metadata names:
  retrieved file = 'C:/Users/Scott/AppData/Local/JetBrains/IntelliJIdea2021.2/plugins-sandbox/system/tmp/issue_1949-issue_1949-retrieve1/tmp/force-app/main/default/lwc/component2/labels.js', metadata type = LightningComponentBundle, metadata name = 'component2'
  existing file  = 'C:/Users/Scott/dev/projects/IlluminatedCloudTestProjects/issue_1949/force-app/main/default/lwc/component1/labels.js', metadata type = LightningComponentBundle, metadata name = 'component1'
 2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever -   No matching existing file found. 
2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever -   Placing in the default source root. 
2021-11-29 11:15:32,071 [4602626]  DEBUG - .ForceComSfdxMetadataRetriever -   Moving/copying C:\Users\Scott\AppData\Local\JetBrains\IntelliJIdea2021.2\plugins-sandbox\system\tmp\issue_1949-issue_1949-retrieve1\tmp\force-app\main\default\lwc\component2\labels.js to C:\Users\Scott\AppData\Local\JetBrains\IntelliJIdea2021.2\plugins-sandbox\system\tmp\issue_1949-issue_1949-retrieve1\diff\force-app\main\default\lwc\component2 as labels.js. 

As you can see, it's already looking to see whether the metadata object names--in this case the LWC component names--are the same before deciding that the file also matches. This is what doesn't seem to be happening properly in your situation.

If what I've done above pretty much matches what you're doing, please add the following to Help>Diagnostic Tools>Debug Log Settings, reproduce the issue, and attach or email your idea.log* file(s) from the timeframe of the reproduction for review:

#com.illuminatedcloud.intellij.builder.RefreshAction
#com.illuminatedcloud.intellij.builder.RetrieveAction
#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComBuilderUtil
#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever
#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever!trace
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataRetriever
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataRetriever!trace
#com.illuminatedcloud.intellij.builder.ForceComSfdxRetriever
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataUtil
#com.illuminatedcloud.intellij.util.IlluminatedCloudDiffUtil
#com.illuminatedcloud.util.VariableLengthPollingInterval

Hopefully that will show me what's going on for you that's not happening for me so that I can reproduce/fix it.

[Dmytro Kurylo]

@Scott Wells Yes, all steps were correct except one - you don’t need to delete component2, you need to delete only the labels.js file from compoent2. So we’re assuming that the last time you saw that org, component2 did exist, but labels.js file for that component was created later. So to you only component2 → labels.js is new, not the whole component2.

P.S. I tried to reproduce the issue on an isolated environment but without success: all files were placed correctly with the use case above.

[Scott Wells]

Hi. I'd need to see a debug log from the reproduction as that would show me why it failed to match properly. It should already be requiring a match on metadata type as well as metadata name and relative path, so I'd be curious to see why/how it's still matching incorrectly.

[Dmytro Kurylo]

@Scott Wells

@Scott Wells

I faced that issue again on one of my dev orgs and I have a live example of it right now. Here’s the video with a clock, and logs stored for the entire run of the video:

https://www.dropbox.com/s/qr14b4m0ut23kkp/Recording.zip?dl=0

Take closer look at iaViewAll and newIAModal components. I see that it’s not the original use case, but still. In this particular example such things happened:

• newIAModal is a component that exists on the org, but is not part of the repo, so after cloning the repo it was not present in the filesystem. It also has labels.js file (the smaller one).
• iaViewAll component does exist on the repo thus was cloned and is present in the filesystem. It also contains labels.js file (the larger one).
• After retrieval, IC sees a new component (newIAModal) that was not there before and adds it to the source tree. But its labels.ls (smaller one) file, instead of being saved to the newIAModal’s folder, travels to the iaViewAll folder instead and was stored there replacing its original content (larger one).
• After that, I’ve created a net new labels.js file in the newIAModal folder to fool IC’s matching algo so that it sees that file. Next retrieval matches these files correctly, and both labels.js files are now correctly overwritten with their original content.

P. S. Experienced that on IC 2.2.0.0.

P. P. S. Please tell me when you’ve downloaded the file as I don’t want it to be public for longer than needed.

[Scott Wells]

I'm glad you have a solid reproduction. Note that I won't get a chance to take a look until next week due to the holidays, though. If you want to take down or restrict access to the video until Monday, I'll let you know when I'm ready to take a look at it.

[Dmytro Kurylo]

@Scott Wells Hello, were you able to download the file?

[Scott Wells]

Hi. Yes, I've downloaded the files now and will take a look at them this week. Feel free to pull the files from DropBox if you'd like.

[Scott Wells]

Okay, that log showed me very clearly what's happening...and why I didn't see the issue previously. I've attached a build that's effectively the same as the current released build but with a prospective fix for this. Please download the attached archive (don't extract it), and install it using Settings / Preferences > Plugins > Install plugin from disk (under the gear drop-down menu). Allow the IDE to restart and then please let me know if the issue isn't resolved.

[Scott Wells]

Delivered in 2.2.0.2.

[Dmytro Kurylo]

@Scott Wells Thanks, seems to work fine now.

[Scott Wells]

Glad to hear! Thanks for helping to confirm the fix, and of course for providing the detail required to corner it.