can't selectively retrieve data when only package.xml exists in Idea's project

[Petr Švestka]

When I try to selectively retrieve items, I always get everything instead of, say, 2 classes, I selected.

This is required for me to be able to submit my changed code to a repository.

Maybe the problem is that I'm unable to set correctly the 2 windows: Project Structure | Facets | Selected (I check off the items that I want) and Build | Illuminated Cloud | Retrieve Metadata... (anything that I pick here for "Contents", it either stops immediately and doesn't fetch anything, or it fetches everything:-)

Logs attached.

Can you have a look pls if I do something wrong or there's a glitch somewhere?

Thank you.

[Scott Wells]

Let me take a look in a few and I'll let you know. But first let me repeat back to you what's happening so I'm trying to reproduce the right thing. In your local source project source, you only have a package.xml. No matter what you select for the module metadata selection (under the facets area), when you try to retrieve specific metadata in the retrieve dialog, e.g., two specific classes, either nothing gets retrieved or everything gets retrieved. Is that correct?

[Petr Švestka]

Aaah, it's formatted not nicely above...

Basicaly, I have two classes selected and I want them retrieve. The project is empty with a simple package.xml stating only a version. I hit retrieve and DEPENDING on what pick-list option I select, it either retrieves everything or nothing.

What should be the correct way how to retrieve only, say, two classes?

BTW I'm able to create a new project with the two classes as per a fix to a previous issue. Maybe this helps.

[Scott Wells]

Okay. I've reproduced it. It's a side-effect of the recent changes to allow you to retrieve missing metadata. Right now it's quite binary in that you're either going to get everything from the metadata types you've selected or you're only going to get the items that you already have locally. What's missing is an option to only retrieve the metadata you have selected even if you don't have it locally. I'll keep this issue open an use it to track adding that option.

[Petr Švestka]

Hi Scott,

Any chance a fix can be implemented soon? This is quite an obstacle for our team.

The reason is that we commit selected sources + package.xml to SVN when we migrate between sandboxes. This bug currently means that we need to create a new project everytime we migrate:-) Note that SVN is used only as a integration means, so it's always empty except for a basic package.xml.

[Scott Wells]

Petr, let me take a look. Shouldn't take long to address. I'll post an update here once I have a better idea of when I'll deliver it.

[Scott Wells]

Okay, I'm back from a week-long business trip but not quite at 100% yet (got stuck overnight in Charlotte due to Snowmageddon and currently operating on ~3 hours of sleep). I'm going to organize this work alongside a few other commitments and see if I can get them resolved soon. Give me a day or two to get things ordered properly and I'll give a revised estimate on delivery of the fix.

[Petr Švestka]

Hi Scott,

We appreciate your effort on this very much. Do you know already some estimation when this could be fixed?

Thank you.

Best, Petr

[Scott Wells]

Petr, I'll shoot for having this addressed next week. I'll let you know if anything gets in the way of that, but right now that looks very doable. I apologize for the delays on this, but last week and this week have been very busy for me.

[Scott Wells]

FYI, I've gotten started on this in earnest and am making solid progress. I've decided to use this as an opportunity to address a number of usability issues with the deployment/retrieval dialogs, so it will likely take me a few days to get this just right. The resulting work will also lay the foundation for some of the other work being requested in #152, #155, #232, and perhaps a few others.

I'll update this issue with more details soon. Just wanted to let you guys know that I'm making good progress on it.

[Scott Wells]

Just wanted to provide a quick update on this. I'm still making solid progress, but as I said earlier, I'm using this as an opportunity to address much of the UX feedback I've gotten on this aspect of IC and to set the foundation for some other upcoming enhancements such as #155 (ability to perform a quick diff of local vs. server). Here's a screenshot of the work-in-progress:

On retrieval, the dialog now shows the full set of metadata in the org. On deployment, only files in the local file system are displayed. In either case, the status of each file is clearly indicated as to whether it's local-only (green), server-only (red), or local+server (blue), and whether it's part of the module content selection (bold) or not (normal).

I appreciate your patience as this is taking long than originally expected, but the goal is to take a step back and try to address the broader set of problems and confusion around this part of IC rather than a tactical fix.

[Vivek M. Chawla]

Scott,

I'm thrilled that you're taking a strategic approach to this. One of my biggest worries about adoption of Illuminated Cloud is that developers trying it out would get hung up on deployment / retrieval, and just give up trying to make IC work for them before realizing how awesome IC is!

I like the direction you're going with this! The different colors are a nice way of providing some context about each metadata component.

As you go through this process, there are a couple of things that I've been thinking about regarding this UI, but haven't had time to share yet. This seems like the place to do so.

Thought 1: Do not allow selections (check/uncheck) in the Retrieval Scope treeview unless the developer explicitly chooses "Custom" in the Contents drop-down

All of the pre-defined metadata retrieval scopes (Project, Module, and even Selected) are...well...pre-defined. If I want to specify a retrieval scope that is not pre-defined, then I should be forced to select "Custom" in the Contents drop-down first.

The way things are set up now, the Contents drop down never quite "feels" like it has full authority over what happens in this dialog because at any time I can just start checking / unchecking components in the treeview. This always seems to make me feel uncomfortable using this dialog (I know it sounds strange, but it's true).

Also, any time the UI shows boxes that can be checked / unchecked, you are telling my brain that there are LOTS of choices right in front of me. If I do not choose the "Custom" scope, then it means that I have already made my choice to fetch the Project, a specific module, or the scope that I right-clicked on in the IDE's project folder view.

By confronting my brain with additional choices that I don't need to consider, then all you're doing is increasing my cognitive load without providing any additional value. I think that's maybe a smarter-sounding way of saying "I feel icky when I use this dialog". ;-)

Here's how I wish it worked.

If I right-click on "classes" in the IDE's Project view, then the Retrieval Scope dialog should only show me the "classes" folder, and then list all of the classes. If the "Include Missing Metadata" box is checked, then it would show all the classes in the org, and not just the ones that are in my local project. No other metadata folders would be included.

On the other hand, if I right-click on the "src" folder, then ALL of the metadata would be in scope, so I'd see everything (again, depending on the "include missing metadata" checkbox)

Thought 2: Show the "Manage..." button when (and where!) it is contextually appropriate

I'm not quite clear about when I should click the "Manage..." button in this dialog. It seems like I can change the connection that is associated with any given module (or change the package in a specific connection, or the scope of the metadata retrieved by that connection, etc.)

I'm not sure about when this is actually relevant. For example, if I've entered this dialog via a right-click on a folder or component in the IDE's Project tree, then changing connection details doesn't make sense. Maybe this button should disappear in that context?

Also, could this button be renamed? Is it fair to name it "Manage Connections..." instead of just "Manage.."? Again, the goal is to reduce cognitive load when viewing this page. "Manage..." is so general that I can't instantly dismiss it. Plus, since "Manage..." is right next to the "Include Missing Metadata" check box, it sounds like somehow I'm managing missing metadata, and not the connection itself.

Here is how it could work...

Perhaps the "Manage..." (or "Manage Connections...") button could be placed in the same row as the "Contents" drop down? That way the button is grouped with the thing that it is more relevant to.

Thought 3: Provide a legend for the color coding system you're implementing

I am very excited about your color-coding idea for the file names. Would it be possible to have a legend below the treeview that explains what each color means?

Conclusion

Again, thank you very much for paying attention to this aspect of Illuminated Cloud. I'm super excited to see the changes you're working on, and I hope that nailing down this aspect of the IC experience will lead to a much better User Experience (and more sales!)

Thanks!

[Scott Wells]

Thanks for taking the time to provide your thoughts, Vivek! A few comments:

1. Thought 1 - Let me chew on this one. The automatic switch from a pre-defined scopes to Custom when you change the selected metadata in one of those pre-defined scopes was actually requested explicitly by some users when they wanted to start with one selection and add to it (contextually-derived) or remove from it (project/module). Perhaps a prompt with an option to cancel might help.
2. Thought 2 - As for the Manage... button in the deploy/retrieve dialog, that's less about managing connections and more about managing the configured "working set" of metadata. Your point is correct that the generic term "Manage" doesn't convey much, though. I originally had the label as "Manage Content Selection", but it got pretty crowded. As part of this rework of the dialog that area is going to be changing by necessity, so I'll revisit that button while I'm at it.
3. Thought 3 - Definitely already adding a legend. It wasn't in the screenshot as it's a work-in-progress, but that's a major aspect of this to make it obvious what the user is seeing and what it means.

Thanks again!

[Vivek M. Chawla]

Regarding Thought 1: The primary need of the other users (the ones who want to start with a selection and add to it) could still be delivered even if the treeview starts off as unselectable during a right-click driven retrieve.

If the developer enters the Scope Retrieval dialog via a contextual selection, and then explicitly changes the "Content" drop-down to "custom", the "normal" (ie. fully editable) treeview would appear and be auto-populated based on whatever their contextual selection was.

Basically, the UI would work exactly as it does now, but only if the developer is explicit about wanting to choose a "Custom" metadata set. Then, whatever selections / deselections the developer wants to make can be made.

I think that doing it this way incurs a lower cognitive cost than the current implementation. Here's why.

If you enter the Retrieval Scope dialog box with the intent to modify your selection, then you are already prepared for the additional cognitive cost of being faced with more choices. You're entering the situation expecting to "pay" more, so you don't really "feel" the extra cost.

Unfortunately, for anyone who enters that dialog feeling like they have already made a choice (eg. "I want everything from the "pages" folder), or for anyone who makes a choice in that dialog by selecting "Module" or "Project", having all those checkable / uncheckable boxes is all "cost" and no "value".

One counter to my argument is that the developers who want to further customize their selection will have to engage in two extra clicks: One to open the "Contents" drop down, and another to select the "Custom" menu item.
I would argue that those clicks quickly become muscle memory, requiring little (if any) thought. This means that you've solved a "cheap" problem (2 physical clicks) by creating an "expensive" problem (additional choice-driven cognitive load).

I think the issue of "choice-driven cognitive load" is really important to keep in mind, especially given the potential market for Illuminated Cloud (ie. people who are net-new to the IntelliJ IDE).

IntelliJ / IDEA has an overwhelming number of customization options, which is can be a double-edged sword. Great because it's so flexible. Not so great because it hard on the brain (especially for people who have not invested multiple years into using it).

When I think of how people who are coming from the Force.com IDE (or Maven's Mate, or Cloud9, etc) to evaluate IC, I worry that the extra cognitive load of IntelliJ IDEA / Illuminated Cloud will hurt conversions. That would suck, because I want IC to be a commercial success. Basically, I want enough people to buy licenses so that you can keep working on IC...which means that I will get to keep using an ever-improving IC! Win-win, right? :-)

[Scott Wells]

I'm down to just a few items remaining before I'll run this through a solid test cycle. I'm shooting for a release on Monday or Tuesday with it barring unexpected findings during testing.

[Scott Wells]

Quick update on this. I've reached a feature-complete state and plan to spend a little more time really running it through the paces. Right now I'm thinking I'll be releasing it either Tuesday or Wednesday morning. Here's an updated screenshot to give a preview of what I've done (and yes, it does address the original stated issue from this ticket!):

Here's what it looks like when initiated using the context menu with only the contextual selection displayed to the user:

Similarly, the retrieve and deploy modes of the dialog are pre-configured to show a more focused view based on the operation. These can be configured by the user as needed, but the defaults should be sufficient for most purposes. Oh, and it now supports retrieval from alternative connections as well. I need to update that enhancement request to let folks know that's coming.

More to come in the next couple of days...

[Vivek M. Chawla]

Wow!

I like the tabs at the top that narrow the scope of what's shown. That is AWESOME! Using this, I could quickly see everything that's on the server and not local (or vice versa). For some reason, I wasn't expecting you to have this feature, but now that you do I don't know how I could have lived without it! Very cool!

Two small suggestions:

Suggestion One: Optimize placement of the Refresh Button

You may want to reconsider the placement of the "Refresh" button. It feels like the "scope narrowing" buttons shouldn't be so close to the "Refresh" button.

By using the same separator (the | glyph) between "Refresh" and "Local Only" as you do between all of the other scope-limiters, it feels like there is an implicit grouping of all five.

Maybe if "Refresh" was aligned with the right side of the treeview while the scope-limiters remained aligned-left? That way the physical separation implies they are not grouped together.

Suggestion Two: Simplify the Legend

The legend is great! Thank you for adding this! I think you might be able to simplify it into one line, though.

Picture using the same font size and coloring as you currently have, but cut the "Not Subscribed" items and then modify the layout in the following way:

--Legend---------------------------------------------------------------------------------------------
Local Only (?)     Local + Server (?)      Server Only (?)     Subscribed Components use Boldface (?)
-----------------------------------------------------------------------------------------------------

                                                                                  | OK |    | Cancel |

Here's what I'm trying to show...

1. All text moves to one line inside of the "Legend" block
2. The order of the text specifically matches the order of the "scope-limiter" buttons above the treeview, which provides a more visually consistent presentation.
3. The difference of "Subscribed" vs. "Not Subscribed" is indicated implicitly by explaining that only "Subscribed Components use Boldface". Note: The text "Subscribed Components use Boldface" would be colored black, and would itself use boldface as an example.
4. The question marks (?) indicate places where you could add a hover-tip help icons. That way, if new users are unsure of what these selection options mean, they can do a quick hover and get a short explanation.

Again, awesome work on this, Scott! Outside of these two suggestions, it feels like this dialog is just about perfect! I'm very excited to see how well this part of IC is maturing! Thanks for your hard work!

[Scott Wells]

Delivered with 1.6.8.5.

[June Bischoff]

I actually strongly preferred it when the scope automatically changed to Custom when you added or subtracted components from the default selections for Context or Module/Project. Showing me checkboxes that I can't check or uncheck is frustrating, at least when I'm accustomed to being able to modify em on the fly, and as an experienced user, I find the extra clicks to be more expensive than recognizing that I've changed the Content selection type by checking or unchecking.

Additionally, for a hypothetical new user who isn't sure what all the Content selections mean, I think it is WAY more intuitive to have the checkboxes behave consistently and the Content selection change appropriately/automatically. (On the fly learning for the user). I could see it being intimidating and requiring more brainpower to try all the Content selections to figure out when you can select/deselect and when you can't.

However, it is usable as is and different users are going to have different preferences. I'm also thrilled to see the new functionality in this release!

[Scott Wells]

I appreciate the feedback, June. I went back and forth on this, not just based on Vivek's thoughts, but also after "interviewing" several other users (and even some folks who aren't users) about the options. In the end I received enough feedback around confusion with the automatic switching of scopes when clicked that I decided to move away from it. Perhaps more importantly, I knew that line-oriented actions such as the new Compare With Server action that I'm about to release would require users to click on rows, potentially changing the selection/scope inadvertently. Of course, now that I've implemented that, it's pretty simple to avoid clicking on the checkboxes themselves while still selecting the desired row for the action.

I may revisit this decision soon now that I have the foundation in place. That's the beauty of this all being fluid!

[Petr Švestka]

Hi Scott,

I don't think it has been fully resolved.

When I have an empty package.xml and I want to fetch components from an org, they aren't retrieved.

Pls see the following screenshots documenting the flow (attached to this defect) 217-1.png, 217-2.png, 217-3.png

I tried different retrieval scope but unfortunately to no avail.

[Scott Wells]

Hmmmm...let me take a look, Petr. Sorry for the inconvenience. Hopefully at this point it should be very quick to address for you.

[Scott Wells]

Petr, I'm trying to reproduce this but haven't successfully yet, at least by following the flow that I'm understanding from the provided screen shots. Let me tell you what I've done and the results, and you can tell me what we're doing differently.

1. Create a shell directory structure in the file system that only includes src/package.xml with an empty package.xml (looks like your first screenshot except I fixed the leading space or underscore in xmlns).
2. Create a new IC project against that directory structure with a connection to one of my orgs and a subscription of Selected with only ApexClass selected. I did not retrieve metadata or generate the OST during the new project wizard.
3. Now I have a .idea folder and a .iml file sitting next to the empty src/package.xml.
4. Started a retrieve operation. The default filter is Local + Server, Server Only, and Subscribed Only. I see all of the Apex classes in my org listed in red/bold meaning Server Only + Subscribed.
5. Clicked OK to allow the retrieve to proceed.
6. All Apex classes in the org were retrieved along with a package.xml specifying those classes. I copied these into the local file system.

I also tried a variation where I started with a populated package.xml (in this case, all Apex classes) and specified that file as my subscription in the new project wizard. That also worked as expected when I did my first retrieve.

Please let me know if I'm not following the same steps as you and where/how I might change my steps to reproduce the remaining issues.

[Petr Švestka]

Ok, I this works. Although my IntelliJ 14.1.4 C doesn't offer me a wizard with these exact steps and I had trouble setting Project SDK. Should I upgrade IntelliJ 15 to be more aligned with you? Also had to generate the OST in the process. But managed to get this work.

This got me thinking that IC abilities rely on something – it just behaves differently based on some steps. Does the OST generation play a significant role in retrieval, or being aware of the sources already in the project?

Let me explain more in detail what we need and whether it should be possible with current IC: * fetch sources from repo * retrieve sources from an org and merge them into the sources from repo * ideally without having to create a new project each time

What I was doing: 1. new project 2. don't retrieve 3. enable VCS and fetch src into the src folder of the project 4. delete all sources and make package.xml empty 5. commit so as to clean the repo 6. select and retrieve components 7. commit components (this didn't cover the scenario of merging retrieved code into repo code and I had to create the project anew)

Can you help me understand please what're the correct steps (if at all possible)?

I think it beneficial if there were supported scenarios around retrieval/deployment described on your web. I trust that different folks use IC for different purposes.

[Scott Wells]

Let me look at IntelliJ IDEA 14 and see the differences. I support IDEA 12/13/14/15, and the UIs in these areas varies quite a bit across versions. I don't think you need to upgrade to 15 unless you want to do so. It's my responsibility to make sure that 14 works as intended.

As for whether IC relies on source already being present locally, the answer is "sometimes". In the case of retrieval, it should not anymore. That was one of the main things I was trying to address (aside from some of the UX shortcomings) in this issue. There are other features that do depend on it, though, e.g., Apex unit test run configurations which only list the test classes that are present in the local project. But that shouldn't come into play here.

Let me put together the explicit steps to get an existing project going with VCS and Force.com integration. I agree that documenting this would be beneficial, though I'm concerned about being able to do so in a way that is accurate across all versions given the significant differences. I'll chew on that as well.

[Scott Wells]

I ran through this in IDEA 14 and it was pretty much the same process for me. As for your described process, I'm not 100% sure I understand a few steps:

1. New project
2. Don't retrieve - is there any reason you're not retrieving if you know the code base against which you're going to be working?
3. Enable VCS and fetch ... - At least with Git in IDEA, this step is problematic for me as the two ways to retrieve metadata won't allow you to do it into an existing directory structure. Instead I used the CLI to retrieve into the project created in step 1 above.
4. Delete all sources and make package.xml empty - What exactly is the motivation for this step when you're working against a known VCS repo and a known org?
5. Commit so as to clean the repo - Same question as above.
6. Select and retrieve components
7. Commit components (didn't cover merging retrieved code) - Same question as 4 and 5 above. If you didn't start clean, you would have been able to merge/reconcile to two sets of metadata.

Not trying to question your process at all...just trying to understand it. Thanks!

[Petr Švestka]

2 – this is just a way I selected once realizing I need to create a new project for this scenario. I guess I could take the other approach – create an Idea project from sources 3 – for SVN I'm able to select the src folder from the repo and fetch it to the src folder of the project 4 – this is crucial. We use the repo only as an integration means, so it never has all sources in the latest version. Once the sources are used for a deploy, we delete them and commit only an empty package.xml so that another developer can easily use it to commit his sources.

We use this scenario several times a day (each of developers). Several IC versions ago this started NOT to work in all cases and the only 100% way for us is to create a new project each time:-)

Currently, I'm facing a following exact issue (will keep the project open so I can set up logging if you instruct me to do so): * empty package.xml – as fetched from SVN * select a couple of list view components for retrieval * the list views are never retrieved in this project * however, if I select also an Apex Class (or a whole object) for retrieval, the class/object is retrieved, but not the list views

[Scott Wells]

Petr, I just worked with another user to reproduce what is likely a related (perhaps the same) issue where the deploy/retrieve dialog is showing the wrong metadata (sometimes completely empty) when going against an empty org or an org without any instances of the desired metadata types. I have solid steps to reproduce this behavior now and plan to issue a fix ASAP. Because I'm pretty confident that the issue you're reporting and the issue I just observed are closely related or identical, let me get a fix for it out and let's see if that resolves your issue as well. I'm hoping to turn something around in the next day or two, so it should happen quite quickly. Thanks much for your patience as I finally get to the bottom of this and make your team's workflow less onerous!

[Petr Švestka]

Thank you Scott, You're doing a superb job!

[Scott Wells]

Petr, I was able to reproduce the related issue and have created a fix for it this evening. I want to test it thoroughly before releasing but plan to do so in the next couple of days. Hopefully it will address the remaining issues you're seeing in this area as well.

[Scott Wells]

Okay, here's a build with the prospective fix for the remaining issues. I'm going to continue testing on my side, but if you guys want to verify that this addresses it for you, go for it! You can install the test build by downloading the archive and using Settings>Plugins>Install plugin from disk.

[Petr Švestka]

Thanks Scott, I just tried on the same project as yesterday – no change.

The List Views are still not fetched.

I was browsing in the logs and found some interesting lines there:

2016-02-23 16:54:55,185 [ 168756]  DEBUG - tellij.builder.ForceComBuilder -   Processing entry Account.All_Active_Agric_CommB_Clients 
2016-02-23 16:54:55,185 [ 168756]  DEBUG - tellij.builder.ForceComBuilder -   Adding entry explicitly. 
...
2016-02-23 16:54:57,124 [ 170695]   INFO - ntellij.builder.RetrieveAction - No files selected for retrieval. 

All_Active_Agric_CommB_Clients is one of the List Views I'm trying to retrieve. And yes it's well selected in the project module:-)

Looks like it's losing track of the selected components.

List Views are kept on the object level right? Because when I select a field from the same object as one of my List Views and these List Views, it won't retrieve anything. But (!) when I select a VF page or Apex class along with my List Views, that page/class is retrieved (not List Views). So it got me thinking it may be important – pages/classes reside in separate files from objects, whereas List Views will be only part of the object file.

[Scott Wells]

Thanks for checking, Petr. Yeah, you raise a good point that leads to another question. Can you verify that you have the CustomObject entry in your subscription for the ListView (or CustomField, ValidationRule, etc.) that you're wanting? Let me do a little testing on this as well and see if I can reproduce it for a child metadata type like that.

[Petr Švestka]

The objects (Account, Contact, Opportunity) aren't selected. But I tried that and it leads to the object file being retrieved including my List Views but everything else too.

[Scott Wells]

Yeah, I'm seeing the same thing. That definitely gives me what I need to address this, at least in part. I can definitely ensure that child metadata types are included in the package.xml even when the parent metadata type isn't selected. However, as you've noticed, if you select the parent metadata type but not the child metadata types, that expresses a type of recursive select all on the child types as well. I'll get this addressed as well before releasing this new build. Thanks for helping me characterize it!

[Scott Wells]

Okay, I'm making solid progress on this. I probably won't have it out tomorrow but perhaps Thursday or Friday morning.

[Scott Wells]

I'm hoping this is finally addressed in 1.6.9.0. Let me know if not.