Connection reset error

Issue #848 resolved
Stephan Wissel created an issue

Working behind a proxy that's not very stable. Created a new project and send loading (all) into the background. Periodically I get "Connection reset" dialog box (with a lengthy error in the log - see attachment).

2 questions: - What happens when I click OK? Reset of the loading and start over, retry, skip or ?? - Could that error dialog be suppressed and retried automatically? Currently I need to watch the IDE - quite distracting

Comments (16)

  1. Scott Wells repo owner

    Right now when there's an issue communicating with the server, the overarching operation stops. I'm inclined to agree that if the issue is just a network connectivity issue and not a failure response from the server, it should prompt to retry (or if it's just in a polling loop, retry silently). I'll take a look and see what it would mean to instrument these client/server interactions with a little more graceful handling of network errors.

  2. Stephan Wissel reporter

    Hi Scott, thx for the comment. I would very much be in favor of a "silent retry" - with a configurable retry threshold. I'm currently working in a very crappy network environment (don't ask) and struggle to download a rather large (classic, not SFDX) org. So far I fail on frequent network resets

  3. Scott Wells repo owner

    Would you do me a favor? Would you capture a debug log of the operation with these types of failures and send it to me so I can see what kind of response/error you're getting in these cases? Here's how to set up debug logging:

    http://www.illuminatedcloud.com/support/debuglogging

    I just don't want to blindly retry on any given exception because some do truly indicate failure. It'd be good to get an idea of what's happening so I can be more selective.

  4. Scott Wells repo owner

    Thanks! I'll take a look this weekend and see if I can make this more resilient for you. I'll let you know if I need anything else.

  5. Scott Wells repo owner

    I'm starting to take a look at this and will get you a test build here as soon as I have one ready. In the interim, looking at your logs, it appears that you're running pretty lean on memory:

    2018-01-11 14:23:38,384 [4043850]  ERROR - intellij.openapi.progress.Task - Java heap space 
    java.lang.OutOfMemoryError: Java heap space
    

    I'd recommend that you give the IDE more breathing room as documented here:

    https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties

    Whether I'm using IntelliJ for Salesforce development, Java development, or whatever, the first thing I do is bump the heap to about 1.5-2GB. All of those indices that it creates, moving big metadata archives back and forth, etc., can create pretty sizable spikes in required memory. Given the machines we have these days, might as well make this class of problems just go away!

    Anyway, I'll let you know what I find on silent or attended retries of these failed client/server interactions.

  6. Scott Wells repo owner

    Okay, looking through the supplied logs, I see four general classes of issues while talking to the server:

    2018-01-11 12:44:34,797 [2411292]   WARN - xf.phase.PhaseInterceptorChain - Interceptor for {urn:partner.soap.sforce.com}PartnerService#{urn:partner.soap.sforce.com}query has thrown exception, unwinding now 
    org.apache.cxf.interceptor.Fault: Unmarshalling Error: Connection reset 
    
    2018-01-10 14:45:42,370 [5591346]   WARN - xf.phase.PhaseInterceptorChain - Interceptor for {urn:partner.soap.sforce.com}PartnerService#{urn:partner.soap.sforce.com}describeSObjects has thrown exception, unwinding now 
    org.apache.cxf.interceptor.Fault: Unmarshalling Error: Premature EOF 
    
    2018-01-11 12:59:35,749 [3312244]   WARN - xf.phase.PhaseInterceptorChain - Interceptor for {urn:partner.soap.sforce.com}PartnerService#{urn:partner.soap.sforce.com}query has thrown exception, unwinding now 
    java.util.NoSuchElementException
    
    2018-01-11 12:59:35,751 [3312246]   WARN - .IlluminatedCloudExceptionUtil - Fault string, and possibly fault code, not set 
    javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set
    

    Of those, the first stands out as clearly being due to an unstable connection and the second as likely as well, so I've added auto-retry logic (up to 5 times) for those. The latter two make me more nervous, though. They could very well be due to a truncated result from the server as a result of a connection issue, but they could just as well be due to an issue with the payload that the server is sending or the client not properly processing a valid server response. I hesitate to add auto-retry logic around those in the first pass.

    Let's do this. I've put together a test build that hopefully takes care of the former two. Why don't you install that build and bump up your IDE's heap size, then see whether that helps. If the problem is still occurring, I'll have you provide a new log, perhaps even turning on verbose communication with the server that will log every SOAP interaction so that I can see the SOAP envelopes that are resulting in those issues. If it does turn out those are a side-effect of the same underlying issue, I'll add the other two to the list of those that should be retried, though I may add a new connection config option like "Unstable connection" and only retry on those if that option a checked.

  7. Scott Wells repo owner

    Here's the test build. Download the archive and install it using Settings>Plugins>Install plugin from disk. It does include a few other fixes above and beyond the current official release (2.0.0.6), but nothing major.

    As noted above, go ahead and bump your max heap size just to be sure that you're not having issues due to insufficient memory.

    Once you install it, regenerate your OST and perform other operations that have led to this issue for you. Let me know whether this resolves the issue, improves the situation, etc. My guess is that it will improve the situation but not 100% resolve it. Assuming that's the case, please add the following under Help>Debug Log Settings:

    #com.illuminatedcloud.intellij.builder.ForceComBuilderUtil!trace
    

    restart the IDE (this is the only log setting that requires a restart), and reproduce the issue. This will result in INSANELY verbose logging because it includes the full SOAP interactions with the server. As a result, use Help>Collect and Show Logs in Explorer/Finder/Files to collect all logs and email them to me for review. Then remove the line you added above and restart the IDE once again to ensure that super-verbose logging is disabled.

    Thanks for all the assistance!

  8. Ruwantha Lankathialaka

    This is appearing in my machine what is the solution to this? should I install the old attached package?

  9. Scott Wells repo owner

    Ruwantha, given that the last activity on this issue was over 2.5 years ago, it's unlikely it's the exact same issue. Feel free to email your idea.log to support@illuminatedcloud.com for review so I can perhaps see more of what's happening. My guess is that it's a network issue or a server issue.

  10. Scott Wells repo owner

    I'm going to resolve this since the last activity on the original issue was so long ago. We'll see what's going on with Ruwantha and perhaps log a separate issue if applicable.

  11. Scott Wells repo owner

    Hi, Ruwantha. Can you please send me a log so I can see the details of what's happening to you now?

  12. Ruwantha Lankathialaka

    Hi Scott,

    I am still unable to get this issue fixed. Is illuminated cloud provide support?

  13. Log in to comment