Selected/Custom metadata retrieval doesn't retrieve metadata that's not yet in the local project

[Zoran Žunko]

once new classes are added to the org from other people retrieve metadata only enables you to select from the classes currently present on the local file system.

[Scott Wells]

Project, Module + Dependencies, and Module should still fetch new metadata (let me know if it's not working that way), but Selected and Custom work more like Refresh in other IDEs where only the selected files are retrieved. I'm not sure that's right, though. I'll take a look at this for the next release that includes deploy-to-another-connection and see if I can make it treat branch node selection as a wildcard for metadata types that support wildcarding.

[Scott Wells]

Okay, I just verified that I see missing metadata on the left-hand side when you use Project/Module+Dependencies/Module, just not when you use Selected/Custom. Given that there is a way to get that metadata, I'll get out the next release with deploy-to-another-connection and then take a look at making the more efficient retrieval contexts support pulling down missing metadata.

[Zoran Žunko]

After i updated the plugin it worked. (btw i tried all the choices in the select list before).

Thanks once again.

[Scott Wells]

Okay, glad you're able to get missing metadata again, but I'm going to leave this issue open to allow efficient retrieval of a subset of metadata including things that aren't on the local machine yet.

[Zoran Žunko]

nice wording. Sorry for me not being to verbose but i'm fighting some other battles while trying to contribute here as much as i can.

[Scott Wells]

Your wording was totally fine. You're the one who brought this to my attention for which I'm very grateful! Just making it so that in the summary list of issues, I can remember what this one is at a glance.

[Vivek M. Chawla]

It seems like this is the right place to describe what I feel I'm missing out on while fetching new metadata with IC.

I often work in shared environments with multiple developers (we each have our own logins). I'd like to be able to right-click on a metadata folder, like "Classes" or "Objects", then click on "Illuminated Cloud >> Retrieve Metadata", and have ALL metadata of that folder's type (ie. all classes) downloaded to my local project.

Because I initiated the "retrieve metadata" command via a context menu on a specific folder, I would like IC to understand the narrow scope of my request and NOT push me through the "Select Retrieval Scope" dialog. And, just to be clear, this action will pull ALL of that particular metadata type (ie. classes) from the org, even if I don't have those files in my local project.

In other words, I'd really like IC to match this feature from Eclipse:

This kind of simplified metadata retrieval seems way more complicated than it should be in IC vs. other Force.com IDEs.

[Scott Wells]

Vivek, I agree totally that a distinct separate notion of Refresh vs. Retrieve is needed where the former is analogous to Refresh from Server and the latter is (more) analogous to Synchronize with Server. I'll sort through the various issues and enhancements around metadata retrieval and figure out which one would best represent this work.

[Scott Wells]

This issue is now being used to track inclusion of metadata files that are not yet in the local project when retrieving. As of 1.6.7.3, that will be the case for all retrieval scopes (does not currently work that way for Selected or Custom) as long as the directory node is selected.

[Scott Wells]

Delivered in 1.6.7.3.

[Vivek M. Chawla]

@RoseSilverSoftware Unfortunately, I don't think this has been fixed.

I'm using 1.6.7.3 and have a project with two Illuminated Cloud modules, each one pointing to a different Developer Edition org.

After setting up the IC module in IDEA, I performed an initial metadata retrieval which did not include any Visualforce pages. VF pages were not included because the org was brand new and I had not created any VF pages yet.

I then created a Visualforce page called "TestPage" using the Developer Console. I wanted to test that IC would properly retrieve this missing metadata using a Selected or Custom retrieval option.

I right-clicked on the module and navigated to the "Retrieve Metadata..." menu option. The screenshot below shows what I saw:

Unfortunately, this retrieval operation did not pull down the newly added Visualforce page.

I wanted to see if I could get the new metadata using the Module scope, because I had two modules and did not want to have to resort to a full-project pull (which I'm assuming would pull metadata from BOTH orgs, one for each module).

Here is the screen shot from my second pull attempt (using the "Module" option)

Same problem. No VF pages came down.

I finally tried doing a full "Project" pull. This finally worked.

I did not try using Custom, but I'm assuming it's broken as well.

Thanks, Vivek

[Scott Wells]

I'll take a look shortly, Vivek. Thanks for letting me know!

[Scott Wells]

Okay, here's what I just did:

1. Logged into my org and created a new page called MissingPage.
2. In Illuminated Cloud, right-clicked on the pages folder in the project view and clicked Illuminated Cloud>Retrieve Metadata....
3. When the retrieve dialog was displayed, verified that the scope was Selected, the pages node was checked, and Include Missing Metadata was checked.
4. Clicked OK. When the directory diff was displayed, the files MissingPage.page and MissingPage.page-meta.xml were included on the left-hand side and could be copied over to the local source tree.

I verified this behavior with the ApexPage metadata type selected and deselected for the module. So that's the expected behavior, but it's obviously not what you're seeing.

If you want to run one of these retrieves that's not working properly again with debug logging enabled, I should be able to see what's going on. Just add the following under Help>Configure Debug Log Settings and run the retrieve:

#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever
#com.illuminatedcloud.intellij.builder.RetrieveAction

Then just send me the relevant extract from idea.log.

[Stefan Abramiuk]

@RoseSilverSoftware Please check situation when Pages were not included in project from the begging. Even if I add this section in package.xml there is no way to update project with new metadata type.

[Vivek M. Chawla]

@RoseSilverSoftware Your steps to reproduce did not match what I described.

The problem was not with pulling in additional new Visualforce pages after a previously successful VF page retrieve. The problem is that the initial retrieve of new VF pages when there previously were zero VF pages in the local project (because zero VF pages existed in the org when the initial "full retrieve" was performed.

I'll be OOO this morning, but will run the debug this afternoon in case you can't reproduce by following the steps in a new org with no existing VF pages.

Thanks, Vivek

[Scott Wells]

Thanks for clarifying, Vivek. I'll try to reproduce according to those steps in a bit and let you know what I find.

[Vivek M. Chawla]

Well, here is something odd. Now I can't even get the Project-scoped retrieve to pull down the VF pages. This is strange, since at least I was seeing the remote files in the "Pages" folder when I did a project retrieve last night.

I tried "invalidate cache and restart", but still no luck. At this point there seems to be nothing I can do to retrieve Visualforce pages when I don't already have a "/pages" directory in my local project.

I'll email you the idea.log files. If you can think of anything that can help pull down VF pages, please let me know. This is now a blocker for my work.

[Scott Wells]

Let me see what I can find, Vivek. I'll let you know something ASAP.

[Scott Wells]

Vivek, I just want to make sure that I'm testing the right thing:

1. I just created a new IC project against an org with no pages in it.
2. I set up the selected metadata to include all of ApexClass and ApexPage (empty branch nodes).
3. I added a page to the org via the Web UI.
4. I performed a retrieve from within IC. Because I don't have any metadata locally right now, the retrieve selection tree shows just the empty module node which is selected. This is true regardless of the scope.
5. The directory diff window shown after the retrieve includes the new page on the left-hand side.
6. I copy the page to the right-hand side and close the directory diff window.
7. I am able to open and edit the new page in IC.

Please let me know if these steps don't model your steps properly. I apologize if I'm misunderstanding the use case.

[Scott Wells]

Hold tight. I may have an idea about what's going on...

[Vivek M. Chawla]

One difference: When I set up my project, I did not specify specific metadata. I set up the module to use the "All / Package" option for "Contents".

Because of this, there was never an option to retrieve pages, or the pages folder.

I was able to find a work-around, though. I created a Visualforce page from IC, and then deployed it. From that point on, I was able to do a right-click on the "pages" folder, and non-local VF pages were finally able to be retrieved.

So, no longer a blocker, but not an ideal situation.

[Scott Wells]

Yeah, I think I see what's going on, but it doesn't explain the omission of pages when using Project or Module retrieval scope. Let me explain, though, and we'll see where that takes us.

You're correct that the absence of the pages directory is keeping you from pulling down any Visualforce pages when using the Selected and Custom scopes. I can see why that's happening and it should be easy to fix. An ugly workaround is be to create a pages directory under src with an empty Foo.page under it before the retrieve. Once you pull down real pages, you can delete Foo.page. Like I said, ugly, but you mentioned that you're currently blocked so I'm trying to offer up all options.

Even without this workaround, I am able to pull VF pages when using Project or Module as long as I've selected ApexPage as a metadata type for the module. You said that wasn't working for you. Can you verify that your module selection includes ApexPage?

Hopefully some part of this gets you unblocked. If not, let me know. I won't be able to do much this evening as I have an outage starting in about 30 mins, but I can take another look tomorrow if necessary. If this does get you unblocked, I'll start on a real fix for this where all top-level metadata types can be retrieved using Selected or Custom even when the corresponding directories don't exist locally. I'm a little bummed that I let that one through!

[Scott Wells]

Hah! Sounds like you found the same workaround. Okay, glad you're unblocked. Look for a real fix soon. I'll move this one back out of resolved and into my work queue.

[Scott Wells]

1.6.7.7 includes better much support for this feature including the ability to retrieve metadata for types that have never been retrieved/created locally and the ability to manage the module metadata selection directly from the deploy/retrieve dialog for convenience.

[Vivek M. Chawla]

Hi Scott. HUGE thanks for getting this out. I'll do some testing over the next couple of days and let you know if I find any issues. Thanks!!

[Scott Wells]

Moving back to Open as there is still one use case (using All/Package for the metadata selection) that's not working properly. I'll see if I can address it in very short order.

[Scott Wells]

1.6.7.8 should resolve the known remaining issues with All/Package and Package.xml metadata selections and foldered metadata.