Custom Metadata Retrieve not working properly

[[e-mail]]

Whenever I am trying to retrieve Custom Metadata, the retrieve window is not presenting with the list of all the Custom Metadata records in the org. In the screenshot below, it's just showing one metadata, though there are more in the org that are not in the local

In the screenshot below, here also it's just showing one metadata, though there are more in the local and org as well.

[Luke Kozakewycz]

Hi,

This might be a SFDX issue. There is a known issue around custom metadata records. You will need to go into the UI, click Edit and Save on the record and then you should be able to pull that in. If that still doesn't work - then yes, it could be the IDE.

[Scott Wells]

Well, it's actually a Salesforce API issue. It's on my short list to be worked (likely the next-next build). Basically when you ask the Salesforce metadata API to list metadata for the CustomMetadata type, it's not returning everything that's in the server. As a result, the listing in IC ends up being incorrect. When I do work this issue, it's very possible it will just result in a bug against Salesforce. If there's a workaround, I'll implement it, but if there's no viable workaround then we'll all just need to weigh in on the bug I open (or perhaps tie into an existing one) to try to get Salesforce to address it.

[Henry Zhao]

Just tested custom metadata retrieval using MavensMate, and it seems to be pulling all the custom metadata correctly, as seen in the screen shot below:

I believe Mavensmate also uses the metadata API behind the scenes.

[Scott Wells]

That's actually great to hear, Henry. That puts it completely under my control. This is teed up for the next batch of work. Hopefully it'll be something really simple. Thanks for the update!

[Rohan Gore]

I can double confirm that. I used WSC Connector Library to list CustomMetadata and was able to retrieve all the CustomMetadata defined in the org in a Java program. Also was able to retrieve them with ant migration toolkit and for the time being relying on it on my day to day commits to Version Control and I believe ant uses the retrieve function behind the scenes unlike the listMetadata function. So basically both the ways of retrieval in my knowledge are working. Not tested with Tooling API though and not sure even if it possible with it.

Seems to be 100% under your control Scott!

Hoping this could be fixed in next release. Around 25+ devs here, including Henry are relying on Illuminated Cloud on a day to day basis. I am hoping this number can even go up in the coming days ;)

[Scott Wells]

Hmmm....I'm still unable to reproduce this. So here's what I've tried in a project that has a large number of custom metadata objects:

1. Removed all of the *.md files locally so that I can simulate retrieving them from the org for the first time.
2. Verified that I can see the custom metadata objects in the subscription editor using Illuminated Cloud>Configure Module then switching to Selected and clicking the Refresh button to get a list of the latest org metadata. I see all the rows under CustomMetadata, but in this case I didn't select any of them because I wanted to try to reproduce what you're showing as closely as possible.
3. Started a Retrieve Metadata operation, switched Contents to Custom, unchecked everything but Server Only, and clicked the Refresh button.
4. Verified that I can see the custom metadata objects in the selection tree under the customMetadata folder.
5. Selected those custom metadata objects and clicked Retrieve for Merge to verify that the actual custom metadata files were retrieved from the server.

Is there something that you're doing differently? Do you see any errors in the log when you try to list/select the custom metadata objects and they're not displayed? Can you add the following to Help>Debug Log Settings and reproduce the issue?

#com.illuminatedcloud.intellij.builder.ForceComBuilderUtil

then send me the resulting idea.log so I can see what else might be going on? Hopefully that will provide some additional info to help me reproduce it. Thanks!

[Henry Zhao]

Hey @RoseSilverSoftware sorry this took a while. I have attached my idea.log. We are still facing this issue.

[Scott Wells]

Thanks, Henry. To everyone having this issue, I've as yet been unable to reproduce it. I obviously trust that all of you are, but without being able to reproduce it, it's very difficult to debug and fix it. Is there any chance that one of you could make an environment available where the problem occurs? I'm not asking for access to your sensitive project metadata. In an ideal world perhaps you could reproduce the issue in another org with just some custom metadata deployed there and provide a login for me. If that's not possible, I'm happy to do some kind of NDA for access to a populated org where this is happening long enough for me to reproduce, debug, and fix it. I'm just not sure how I'm going to get to the bottom of it very quickly otherwise. The alternative would be to post builds here with more and more diagnostic logging and try to corner it that way, so if I'm unable to get a live environment, I'll go that route.

Sorry to ask...it's always frustrating when I can't just reproduce an issue on my own, especially when it's something that's causing user their own frustration.

[Scott Wells]

I just exchanged notes on this topic with Henry in another issue and figured I'd bring it back here. Above I listed the steps I've taken to try to reproduce this issue, so far unsuccessfully. Could one of you who is having the issue confirm that those steps DO or DO NOT reproduce the issue for you. If they DO NOT reproduce the issue, could you provide the steps you do take to reproduce it?

Then either way, could you make sure the following are under Help>Debug Log Settings:

#com.illuminatedcloud.intellij.builder.RefreshAction
#com.illuminatedcloud.intellij.builder.RetrieveAction
#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComBuilderUtil
#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever
#com.illuminatedcloud.intellij.builder.ForceComSfdxRetriever
#com.illuminatedcloud.intellij.util.IlluminatedCloudDiffUtil
#com.illuminatedcloud.util.VariableLengthPollingInterval

reproduce the issue, and send me your idea.log from that metadata retrieval? Thanks!

[Scott Wells]

Okay, I still haven't gotten to the bottom of this, so I've instrumented the key areas with WAY more diagnostic/debug logging. Here's a build with that additional logging. Download the archive and install it using Settings/Preferences > Plugins > Install plugin from disk.

In addition to the entries listed above, please also add the following entry under Help>Debug Log Settings:

#com.illuminatedcloud.intellij.builder.BuildOptionsDialog

Then open the retrieve metadata dialog, click the refresh toolbar button, and send me all idea.log files that were a result of that along with the name(s) of the metadata that isn't showing up for retrieval properly (a representative name will be sufficient). I'm hoping that will show me why these items are included in the subscription editor but not in the metadata deployment/retrieval dialog.

[Henry Zhao]

Hey @RoseSilverSoftware, installed your custom IC2 plugin and added the debug line. Let me know if this helps.

An example custom metadata that isn't showing up is:

CC_Link.Prospects

[Scott Wells]

Thanks, Henry! I'll review the logs tonight and hopefully they'll show me what's going on...or at least move the investigation forward a bit so I know where to look next. Fingers crossed!

[Scott Wells]

Due to a pesky migraine last night, I wasn't able to review Henry's logs. I'll try to do so today and provide feedback on what I see. In the interim, this enhanced debug logging is included in the latest official build, 2.0.3.6, so feel free to move back to the official build without fear of losing the additional diagnostic info.

[Scott Wells]

Okay, so there is something quite telling in the logs provided by @henrymzhaoGSCS. When I look at the enumerated list of custom metadata from the server, it's missing just about everything:

  "metadataTypes": [
    {
      "name": "CustomMetadata",
      "members": [
        "CC_Link.CC_Payday"
      ]
    },

IC reconciles what it sees locally with what it knows about from the server and shows that set of metadata in the deployment/retrieval dialog. I can see where IC knows about this metadata based on local files:

  CC_LINK.PROSPECTS.MD: {
    customMetadata/CC_Link.Prospects.md
  }

but I'm concerned about the fact that the server isn't reporting everything properly. I'm putting together another test build with even more debug logging right now so I can see exactly what is being loaded from the server in terms of enumerated metadata. That should make it clear whether the server is at fault here or whether IC has a bug that's causing it to lose this metadata as it maps things around. I'll post that build shortly and would appreciate it if @henrymzhaoGSCS could repeat this process so I have a little more info from which to work.

[Scott Wells]

I think I may have found it! More info coming very shortly!

[Scott Wells]

Okay, can you guys try the attached build? As always, install it using Settings/Preferences>Plugins>Install plugin from disk. After the IDE restarts, let me know if you don't see all of your custom metadata (or whatever metadata type was experiencing this issue) enumerated properly in the deploy/retrieve dialog. Fingers crossed!

[Rohan Gore]

Wooh, got all custom metadata!! Let me also confirm this with @henrymzhaoGSCS. Must be juggling between his school and work..! Thanks a lot.

[Scott Wells]

Ah, that's great to see! Yeah, once it's confirmed, I'll make sure it goes into the official build (likely next Monday). Sorry this one took so long, but thanks SO MUCH to everyone who helped me to corner it!

[Henry Zhao]

Can confirm it works beautifully. Thanks a ton @RoseSilverSoftware , this was a huge help!

[Scott Wells]

That's how I like to see a Friday wrap up! Thanks for confirming, guys. Like I said, this will be in the next official build. Just sit tight on this one until I push it out and you should be good. Thanks again for all the help (and patience!), and have a great weekend!

[Scott Wells]

FYI, I will be releasing a new official build with this tomorrow barring unforeseen events. I was waiting for one other user to verify that this also fixed his problem with retrieval of web link metadata. I just received that confirmation, so it should be good to go. Thanks to everyone who helped me work through this one!

[Scott Wells]

Delivered in 2.0.3.7.