Proposal: Modify the managment of static resource bundles so that it plays nicely with version control systems

[Tom Fuda]

The support for managing static resource bundles in IC doesn't currently play so nicely with version control systems. We've been using the Force.com IDE with Git for version control for several years in our shop. Right now, we keep the source files (.css, .js, etc) that comprise our static resource bundles in a separate folder, outside of the project's "src" folder. We use an Ant task to zip the files into a bundle and then move zipped archive into src/staticresource/xxx.resource for deployment to the Saleforce org. The nice thing about this approach is that it allows us to have full version control over the individual source files that go into the bundle. I propose that IC's static resource bundle management adopt a similar approach. When IC is told to explode an existing bundle it should explode the files into a folder outside of the project's "src" folder so that the contents of that folder can be checked into VCS. You could still support the automatic bundling and deployment of the resource when contents change by observing when the exploded files are modified and firing off a zip and deploy operation.

[Scott Wells]

Another user asked for this as well a bit back, but I'm currently unable to find the original enhancement request. I agree with the proposal and will implement it at some point. Because IntelliJ IDEA really does depend on working source files being under a source root, what I will likely do is create (and/or allow selection of) a second source root outside of the Salesforce src dir and extract into that, but otherwise the same idea holds.

Thanks for the proposal!

[David Esposito]

MavensMate creates a /resource-bundles directory off the root of the project and unpacks the zip file there. The tree looks like

\resource-bundles
  \BOE.resource
    \css
      styles.css
    \js
      whatever.js
\src
  \staticresources
    BOE.resource
    BOE.resource-meta.xml
  package.xml

It would be best to use the same hierarchy so that users of MavensMate can play in the same playground as IC users

[Scott Wells]

Thanks for bringing that up. Yeah, the plan would be to do it with a level of flexibility that would allow easy integration with other tools that provide the same/similar functionality if desired, but also the ability to place the exploded resources elsewhere if wanted.

[Tom Fuda]

Just a suggestion here, but Maven's Mate does something similar to what I propose in that it keeps the exploded files for each resource in a folder called "resource-bundles", located at the same level as the "src" folder. Within "resource-bundles", the content of each static resource is expanded into a folder with the same name as the bundle. This would make it easier for users of MM to migrate to IC, while still maintaining the version history of bundled files. ;-)

[Tom Fuda]

In case you haven't guessed, Dave and I work together... sorry for the dupe comment.

[Royce Nobles]

Great idea! This would help adoption at our company immensely.

[John Stamets]

Glad to see this is on the roadmap. Do you have any idea when we might expect this functionality? A lot of our developers are holding off on IC until this goes in... If it will be a while, we may need to look at alternatives for handling this functionality. Thanks!

[Scott Wells]

John (and others waiting on this), I'm hoping to get started on it next week. This week I'm trying to get out some changes to maximize IC's use of the Tooling API during deployment (including static resources and Lightning), especially given some of the slow Metadata API deployment times folks have been seeing for the past week or two. Once that's wrapped up, this is next on my list.

[Kenneth Yeung]

So excited to hear this!!! Looking forward to it! Thanks so much!

[John Stamets]

That's awesome news. Thanks!

[Scott Wells]

Quick update...I haven't started on this yet but hope to this week. With some pods experiencing very slow metadata API deployment times, I've been working hard on maximizing IC's usage of the tooling API for supported metadata types. I think I've finally wrapped up that work, so next I'll turn my attention back to this. Hopefully I'll have something ready in a week or two.

[John Stamets]

Thanks! Appreciate the update. Still really looking forward to it.

[Scott Wells]

For all those who are following along at home, I'm finally getting started on this in earnest today. I'm initially figuring out all of the changes that are going to be required so that I can support both modes (current in-place and requested interop with MavensMate), and I think I have my head wrapped around it pretty well. Let me share my findings/thoughts so far here so I can make sure that what I deliver matches the needs/expectations of those requesting it. I'm not planning to make this a design-by-committee session, but I do want to make sure that I'm not missing anything critical. Here's how I see it working:

• Add a new option in Validation and Deployment to specify whether static resource bundles should be extracted into the main source root (how it works today) or a separate source root.
  • The former is the default to ensure no unexpected change to existing users and the overall behavior is 100% unchanged from how it works now.
  • When the latter is selected, a new source root called resource-bundles is created if necessary as a direct peer to src and added to the module as an actual source root directory. Directories for extracted static resource include the .resource extension unlike when the current behavior is used. I considered making all of this configurable and may still if there's a need, but to start I wanted to focus on the primary driving requirement which is interop with an existing model.
  • When this option is switched, all existing static resource bundles will be converted back into static resource files using the previous setting and then converted back into static resource bundles using the new setting.
• Any deployment of a static resource bundle in a separate source root will first reconstitute the corresponding static resource file, then will deploy that file.
• The existing Deploy static resource bundle on contents change option (and deployment overall) will work the same regardless of where resource bundles are stored.
  • When this is enabled, any file create/update/delete in a static resource bundle directory will trigger a deploy-on-save of the static resource bundle.
  • When disabled, explicit deployment operations including Force Save and Deploy Modified/All metadata can still be used to deploy static resource bundles as desired.
• Update metadata retrieval to be aware of static resource bundles in a separate source root and extract retrieved static resources into a parallel directory structure.
  • If static resource bundle contents were updated as a result of retrieval, the corresponding static resource file should be updated as well based on the latest bundle contents.
  • This will require the directory diff tool to be opened one level higher so that both src and resource-bundles can be included in the comparison. An unfortunate side-effect of this is that any other peer directories/files will be visible in the comparison as well because there's really no good/consistent way to filter out that noise, especially if an external diff tool is configured in IntelliJ IDEA.
• Update metadata refresh to be aware of static resource bundles in a separate source root and refresh both the static resource file and the bundle contents directory (likely a delete and re-convert like it is today).

There are other internal implementation details that I'll have to address, but from an end user-facing perspective, I think that covers it. If I've missed something critical, please let me know. Otherwise I'm hoping this will only take a week or two to implement. I'll keep you all posted on my progress here.

UPDATE: Upon further consideration, removed the option to decide whether the .resource file should be updated when bundle contents are updated. I think I'm just going to do this automatically on deployment, whether deploy-on-save or explicit deployment, and just deploy the .resource file itself as if it weren't managed as a bundle directory.

[Scott Wells]

FYI, this is coming along much faster than expected. I imagine it'll be one of those "the last 20% is 80% of the work" things, but I've already implemented just about everything but the metadata retrieval changes. I'll keep you posted, but so far, so good...

[Brian Triplett]

That is an excellent update! Glad to hear it's moving forward well. Like many things software I'll probably have to fiddle with it to give meaningful feedback but the approach above sounds solid.

[Tom Fuda]

Thanks for the update Scott. This sounds reasonable. Glad to see it moving forward.

[Scott Wells]

Okay, if you're in IDEA 14 or higher and want to live on the edge a bit, here's a pre-release build with this feature. The pre-release build also includes everything in 1.7.4.1 which I released this morning.

To use it, download the archive and use Settings>Plugins>Install plugin from disk, allow the IDE to restart, and then activate the feature using Settings>Illuminated Cloud>Validation and Deployment>Use separate source root for static resource bundles. Note that you can always switch back to the released build by downloading it from the main plugin download page.

Now when you right-click on an archive-type static resource file and click Illuminated Cloud>Convert to static resource bundle, the file should be extracted into a resource-bundles directory as an immediate peer of your existing src directory, and the static resource file should be left alone in its original location.

Here are some of the implemented features:

• Navigate to the contents of the extracted bundle using Go To File.
• Edit the contents in-place (create/update/delete) and see the results pushed to the org as you save or deploy, etc. Note that there's no explicit Convert bundle to resource action because that happens implicitly on deployment.
• If you attempt to convert a static resource file that's already managed as a bundle, you'll be prompted to recreate the bundle or cancel the operation. I'm planning to add a merge option as well to allow you to merge/integrate changes to the static resource file from source control or the org with any uncommitted changes you might have locally in the resource bundle directory.
• If you delete a static resource file that's also managed as a resource bundle, you'll be prompted to delete the existing resource bundle directory. You can also delete the resource bundle itself by using IntelliJ IDEA's standard file/directory delete functionality.
• When you retrieve metadata, if the retrieval includes static resources managed as resource bundles, the resulting directory diff will occur a level higher so that both src and resource-bundles are included in the comparison.
• When you refresh one or more static resource files from the org, the files are refreshed in-place and, if they are also managed as resource bundles, you're prompted to recreate the bundles. As with the third item above, I plan to make this support a merge/integrate option as well as an overwrite/recreate option.

It doesn't currently migrate existing resource bundles back-and-forth as you switch models (in-place vs. separate source root). I am planning to add that before releasing the feature.

There are almost certainly some bugs here, but hopefully all minor. I'll be testing it thoroughly over the next few days, but I'd also love early feedback from those of you who plan to use it as part of your daily workflow, both on bugs you might encounter and also whether it's missing anything critical to help support your desired workflow around static resources.

[Tom Fuda]

I probably won't be able to try this for a few days, at least.

[Mike Bernard]

This is awesome! Nicely done @RoseSilverSoftware

I am using IntelliJ on a Mac and it works VERY well! The only bug I have found so far is that when you right click on the static resource and select Convert To Static Resource Bundle, it adds a _MACOSX folder to the resource bundle. I have run into this before and if you zip with -X as a parameter, it will exclude those files.

This post speaks on that a little more http://stackoverflow.com/questions/10924236/mac-zip-compress-without-macosx-folder

So far, very nice work. I have been waiting for this for a while!

[Scott Wells]

Glad to hear that it's working well for you. I'll look at the _MACOSX folder and make sure it's handled properly. Let me know any additional thoughts you have as you play around with it!

[Mike Bernard]

@RoseSilverSoftware

When I create an IntelliJ project from source, although the resource bundles are there, I can't deploy the files in the resource bundle until I go to the static resource and right click and choose Convert to Resource Bundle. This isn't a big deal at all. I'm just wondering if it could pick the resource bundles up automatically on project creation.

Again, not a big deal.

Thanks!!

[Scott Wells]

Ah, so when you create a new IC project against a code base that already has resource bundles, it's not adding that directory as a source root immediately, correct? You can also resolve that by going into the module settings and selecting that as a source root, but I'll see if I can do it automatically if things are configured for external resource bundles. Thanks!

[Mike Bernard]

Oh nice! I didn't realize that's what it was doing. Yeah, adding as a source root in the module settings works.

[Scott Wells]

Attaching a refreshed build based on 1.7.4.3. There are no updates to the resource bundles feature in this build, but it does include all of the fixes in 1.7.4.2 and 1.7.4.3 in addition to the same state of the resource bundles feature from the previous pre-release build. Please update to this build if you're playing with it. I'm hoping to release the full feature at the very beginning of next week.

[Scott Wells]

This is now functionally complete. I'm running through functional and regression tests and plan to release it on Monday.

[Scott Wells]

Delivered in 1.7.4.4. See this video for an overview of the changes.

[David Esposito]

Chiming in with a 'me too' response: This is working as desired and is even better than MavensMate (which requires you to use a hotkey to compile/deploy the updated static resource after saving). And it's lightning fast compared with deploying the Static Resource independently (2s vs 10s+).

Nice work, thanks.

[Scott Wells]

Thanks for the feedback, David. Glad to hear that it's meeting/exceeding expectations!

[Mike Bernard]

@RoseSilverSoftware

I will second David's response on behalf of the 20 or so developers at my company currently using this feature. I have talked with many of them and also ensured that people are using it and I have had nothing but positive feedback. This definitely beats mavensmate just by compile time alone! Very nice work. We really appreciate it.

[Scott Wells]

Thanks for passing on the good news, Mike!

[Tom Fuda]

@RoseSilverSoftware - I finally got around to trying this. I might have found a minor issue. I'm developing a resource-bundle called "jquery.resource" in one of my Developer Edition orgs that also happens to have a couple of managed packages installed that also happen to have a Static Resource called "jquery", but obviously with different values of Namespace Prefix. When I make a change to one of the source files in my jquery.resource bundle Illuminated Cloud tries to deploy it, but I get the following error: http://screencast.com/t/Kx6foPWD6. When I query my org for Static Resources with Name = 'jquery', I get the following: http://screencast.com/t/LWD1uDGr. I think perhaps IC is having trouble resolving the Static Resource name properly in this case. Should I open a separate issue for this?

[Scott Wells]

Yeah, let's manage that as a separate issue if you don't mind. Please convey these same details into that issue, and if you could provide a relatively simple step-by-step example of how to reproduce it, that would be greatly appreciated. Given that I've already taken namespaces into account pretty pervasively, my guess is that once I'm able to reproduce it, a resolution will come quite quickly.

[Tom Fuda]

See #417