Static resources created using grunt-zip not deployed correctly to SalesForce

[Marko Cekol]

I have a grunt task which takes a certain folder structure and compresses it to a static resource file, creating xxx.resource file along with the xxx.resource-meta.xml file for that resource. The issue I have is, when using deploy or force save the resource gets deployed to SalesForce but it is approximately 150 kb larger in size. When downloading the static resource uploaded this way I am unable to extract it, all of the zip utilities say that it is not a valid zip file.

What are my options here?

[Marko Cekol]

Oh, forgot to mention, manual upload of static resources created this way works without issue.

[Scott Wells]

Hi, Marko. Sorry for the issue. Just so I understand the full scenario, can you enable debug logging for metadata deployment as documented here:

http://www.illuminatedcloud.com/support/debuglogging

then reproduce the issue and send over the resulting idea.log?

Also, are you using any deployment substitution rules or anything? And are you using the IDE's grunt integration to build the resource file? If so, can you describe the timing of when the resource bundle is built vs. when you deploy? My guess is that it's all just fine, but I'd like to understand that just so I know everyone that might be manipulating that file including IC. Oh, and what value is being used for contentType in the resource's meta.xml file?

[Marko Cekol]

I'm not using any deployment substitution rules. I'm running the grunt task manually through the terminal app - I didn't think about integration with WebStorm yet - I just want to get it to work initially. After building the resource I right click on the static resource and choose Illuminated Cloud -> Force Save. The contentType is application/octet-stream, but the same content type is visible in SalesForce if I upload it manually through the browser.

[Scott Wells]

Thanks. Do you mind trying a couple of things? First, disable Tooling API-based deployment of static resources under Illuminated Cloud>Configure Application>Validation & Deployment, try saving again, and see if the same problem occurs. I'm curious as to whether this is a Tooling API issue. Second, do you mind changing the content type to application/zip and see if the problem occurs? Just trying to pin it down. If it still fails under all of those situations, can you create an archive-type static resource that reproduces the issue that you can share with me so I can try to reproduce it/debug it locally?

Thanks!

[Marko Cekol]

Using Tooling API with the application/zip Mime Type yields the following issue: "Deployment Failed: REQUIRED_FIELD_MISSING - You must provide a valid Metadata field for StaticResource".

Unchecking the checkbox (and using Metadata API?) with the application/zip Mime Type sometimes I get the following: "ERROR deploying StaticResource staticresources/clmapp.resource: The specified Static Resource is not a valid zip file", while sometimes it uploads the file without issue, but the file has the same problem as reported initially.

[Marko Cekol]

I'll try to create an archive file which does not have any sensitive data in it and upload it here.

[Marko Cekol]

The sample resource file which exhibits the same behavior, along with the *-meta.xml file.

[Scott Wells]

Perfect! I'll use that to debug the issue. I'm traveling for business today but will try to take a look over the next day or two. I'll let you know what I find. I appreciate all the help trying to corner this one!

[Marko Cekol]

Thank you Scott! If there is any way I can help/contribute just let me know!

[Marko Cekol]

Hi Scott! Any update on the issue? Any way I can help?

[Scott Wells]

I've been massively sidetracked. I'll be catching up over the next few days. I'll let you know if I need any additional info, though. Hopefully I'll have thoughts and/or a resolution shortly.

[Scott Wells]

Marko, I started working this issue today and have not been able to reproduce it. I took the attached static resource and added it to the staticresources directory of an existing project and to the metadata subscription. I then deployed it into the org. Finally I retrieved it from the org and the retrieved file is identical.

Is there some other action I need to take? Do you happen to have any deployment or retrieval substitution rules configured? I'm wondering if perhaps they're operating on this file when they shouldn't be. How about namespace translation (if you don't know what I mean you probably aren't using it)?

Please let me know what else I might need to set up to reproduce it and I'll dig back in.

[Marko Cekol]

Hi Scott, I've created a short recording which explains what is happening much better than I can express using words. I'm not using substitution rules, nor am I using namespace translation. Hopefully it helps with debugging.

[Scott Wells]

Thanks for the video, Marko. I just went through the exact steps you did and wasn't able to reproduce the issue. Don't get me wrong...I 100% believe you that it's happening! That video shows it VERY clearly. I'm just a bit perplexed about why we're getting such different results.

The only thing that comes to mind is the character encoding. In general IC deals pretty much exclusively in UTF-8. However, after reviewing the underlying implementation VERY closely, it appears that the mechanism currently used to load the file contents in the scenario you showed doesn't have the ability to specify an explicit character encoding. Instead, the plugin SDK is trying to infer one. I'm concerned that perhaps it's inferring the wrong one on your machine.

I'll post a test build here shortly that uses raw file I/O to load the file bytes for deployment. Let's see if that resolves the issue or not.

[Scott Wells]

Okay, here's a test build with the change to how file contents are loaded to avoid the potential issue with incorrect character encoding inference. It also has quite a few other changes for the next official build (which I hope to release tomorrow).

You can install it using Settings>Plugins>Install plugin from disk. Please do so and repeat the same process from the video that you created to see if this resolves the issue, then please let me know either way whether it took care of it.

[Marko Cekol]

I installed the attached file as a plugin and restarted WebStorm afterwards. The same issue is still happening for me. :(

[Scott Wells]

Well that's a real bummer. I was hoping that would be it.

And so just to clarify one more time so that I know exactly where to focus my efforts when adding new debug logging, the process you've observed is this:

1. Add the static resource in Salesforce via the Salesforce Web UI - GOOD
2. Retrieve the static resource from Salesforce into the local filesystem via IC - GOOD
3. Deploy the static resource to Salesforce from the local filesystem via IC - BAD, i.e., the byte count on the static resource in the Salesforce UI is already incorrect at this point
4. Retrieve the static resource from Salesforce into the local filesystem via IC - BAD, i.e., the same corrupted version of the static resource is now present in the local filesystem

Is that correct? It's not until IC saves it the first time that it gets corrupted? If not, please let me know the exact point in the process you demonstrated on that video that the static resource is in a known bad state.

If so, though, can you enable debug logging for metadata deployment and verify one more time that the deployment of the static resource via both the Tooling API and the Metadata API produces the same errant result for you? Please attach the latest idea.log showing the deployment both ways resulting in the errant behavior. Sorry for having you jump through so many hoops, but at this point I'm doing differential diagnosis and just trying to corner the issue so that I can hopefully add some debug logging to see if the file is being loaded improperly by IC, whether it's an API issue, or what.

Regards, Scott

[Marko Cekol]

Hi Scott,

I've attached both logs. The end result is the same in both instances. Only thing I've noticed, when saving using the Metadata API, it is visible in the log that the file type cannot be determined for the static resource. I haven't seen anything like this when using the Tooling API (but the issue persists). Let me know if there is anything else I can do.

[Marko Cekol]

Oh, and yes, you are correct, the file gets corrupted when IC saves it. I've tried to save the file through workbench.developerforce and the same corrupted file gets saved to salesforce. The only thing that works is saving through the web ui.

[Marko Cekol]

It seems I have been doing something wrong when trying to save the static resource through the workbench - initially. I've managed to save it correctly this time, I'll try to dig around some more and report my findings.

[Marko Cekol]

Okay, here is something else I've noticed. When querying the static resource record from Salesforce through workbench the result is the following:

UEsDBAoAAAAAAO1MP0wr59PmnpUCAJ ... AcAAAA=

Making a request through workbench using this base64 string - everything works as expected.

While the static resource body - from the log - when using tooling api looks like this:

UEsDBAoAAAAAAO+/vUw/TCvvv73vv73 ... AACe+/vQcAAAA\u003d

Making a request through workbench using this base64 string - the file is corrupted.

Seems like some kind of encoding(?) issue?

[Scott Wells]

Hi, Marko. Yeah, I haven't found anything productive on this myself. Regarding your findings, are you saying that the seemingly corrupted base64 encoding for the file is being displayed in idea.log, or are you bypassing IC and seeing whether perhaps the underlying API is causing the issue? Sorry...just not sure I follow your explanation, though I agree that it looks like a solid lead based on the differences.

[Marko Cekol]

Sorry for the number of messages, I was trying out various things and just started typing (and venting) at the same time.

What I'm trying to say is the following: Base64 data displayed in idea.log when trying to deploy the static resource using the Tooling Api seems corrupted. Base64 data displayed when retrieving the static resource Body (via SELECT Body FROM StaticResource, etc.) is okay.

You can see the corrupted base64 data in the tooling-api-idea.log attached to the issue. I can attach the actual base64 data retrieved from salesforce, for comparison, if need be. My opinion is that the IC mechanism used to encode the file is doing something to corrupt/encode it in some strange way.

[Scott Wells]

Okay. That's what I thought but wanted to be sure. Let me clarify some of the other comments you've added:

• Do I understand correctly that you see the same corruption when you deploy via Workbench? So you're able to reproduce the issue outside of IC with these files?
• The message about not being able to determine the file type is due to the ambiguous content type of application/octet-stream in the meta.xml. That's basically just saying that it's binary data but nothing about the true file type. If you want to make it clear to IC the file is a zip, use application/zip. I doubt that's at all related to the real underlying issue, but it would affect how the file type is interpreted by the IDE.
• When you do force deployment of this file via the Metadata API vs. the Tooling API, does the same problem occur? Trying to isolate the issue to a particular API and/or deployment path in IC. And I apologize if we've already discussed this point in detail before, but I'm catching back up on the details of our previous attempts to diagnose the issue.

[Marko Cekol]

Do I understand correctly that you see the same corruption when you deploy via Workbench? So you're able to reproduce the issue outside of IC with these files?

I thought that was the case, but upon further inspection it seems that I have been using the corrupted base64 data. When using the data taken straight from Salesforce everything works as expected through the workbench.

When you do force deployment of this file via the Metadata API vs. the Tooling API, does the same problem occur? Trying to isolate the issue to a particular API and/or deployment path in IC. And I apologize if we've already discussed this point in detail before, but I'm catching back up on the details of our previous attempts to diagnose the issue.

Yes, there is no difference between Tooling or Metadata api. Base64 data is only visible in the Tooling api idea.log file.

[Scott Wells]

Okay, thanks for confirming. So just reviewing where we are, this happens consistently for you from IC regardless of the API used to deploy the file. With the same file I've been unable to reproduce the issue, so it doesn't seem to be the file itself. The problem seems to be based on some environmental difference. As you've pointed out, it seems to occur when the file is loaded by IC and encoded for transfer.

I just looked at the source, and for Tooling API deployment, IC simply loads the raw bytes from the file and uses Java's standard JAXB base64 encoder. It then either POSTs or PATCHes the associated REST resource with that content depending on whether it's a create or update respectively. For Metadata API deployment IC simply loads the raw bytes from the file and adds them to a Zip archive for submission to the Metadata API's deploy() call.

Given that this happens with both deployment paths, my best guess is that something is messing with the bytes for this particular static resource as they're loaded from the filesystem into memory. That would mean that the incorrect base64 encoding is a downstream consequence of having the wrong bytes. Otherwise the same problem wouldn't occur during Metadata API deployment where no explicit encoding takes place.

Bytes are loaded from the filesystem using a UTF-8 encoding. Extending that best guess, I'm thinking that the file in your filesystem is somehow encoded otherwise and loading with UTF-8 is causing the issue.

I guess another thing we could try is I could send you one of my own binary static resource files that works on all my my machines (two Windows, one Mac, one Linux) and we could see if it exhibits the same behavior. If so, that definitely supports the environmental difference theory.

[Scott Wells]

I'm going to try something...I'm going to update the code that loads these files to try to use the inferred file encoding instead of UTF-8. I'll post a test build here shortly with that change for you to check out.

[Scott Wells]

Okay, here's a build that tries to use the current file encoding when loading it instead of assuming UTF-8. Let's see if it changes the behavior for you at all.

[Marko Cekol]

I installed the attached plugin, tried both the Metadata and the Tooling API, but the end result is the same. The uploaded file is still corrupted.

Perhaps you could attach your binary static resource and I could try with that?

[Bojan Zunko]

Can confirm a similar issue happend to me - was deploying a static resource to an org and it ended up quite smaller then the file was originally. Further the same static resource uploaded manually through the SF UI brought it back to it's normal size.

[Scott Wells]

This is so strange. What I may need to do is instrument things further and write out the exact bytes that I'm sending through the Salesforce API and diff them against the original file. It just doesn't make sense that it would behave this way with a static resource file created through one mechanism but not another. Let me see if I can put together a new diagnostic build that I can post here to dig into this a bit more.

[Bojan Zunko]

Ok - great - when you get the build I will try to repeat the process. Unsure if uploading to a packaging org makes any diffrence but realized that that piece of info was missing from the original comment.

[Bojan Zunko]

Hi Scott. Any news on this build ? Really curious what is exactly going on here :) strange issue and must be hard to pinpoint

[Scott Wells]

I did not get a chance to put together an instrumented build over the past several days. Let me do so over the next few days and I'll post it here. I'm equally curious to figure this one out!

[Scott Wells]

Okay, here's a new build with some additional debug logging and a fix for an issue that might or might not be contributing to the problem you guys are seeing.

Install by downloading the attached build, then open Settings/Preferences>Plugins>Install plugin from disk.

Please make sure that you have the following in Help>Debug Log Settings:

#com.illuminatedcloud.intellij.resourcebundle.StaticResourceBundleUtil
#com.illuminatedcloud.util.IlluminatedCloudZipUtil

Then try to reproduce the issue. If the issue is gone, then the aforementioned fix took care of it. If it's still there, please attach your latest logs so I can see if there's anything telling in them.

[Scott Wells]

The latest build of IC2, 2.0.4.1, includes the potential fix but not the more extensive logging. I would recommend that you stay with the build attached here until we get new logs to help with diagnosis.

[Scott Wells]

Issue tracker grooming. If this is still an issue, please feel free to reopen, ideally with a concrete reproduction scenario.