Pipelines kills idle Maven connections after 5 minutes

Issue #13988 wontfix
Martin Myslík
created an issue

Pipelines started closing idle Maven connections that are idle for more than 5 minutes after the last infrastructure update. This affects any build which is running Maven and takes longer than this amount of time because such build will fail with 'connection reset socket exception'.

Some plugins in the Maven lifecycle are triggered after running integration tests and the only workaround is to force Maven to use a new connection on every request using 'mvn -Dmaven.wagon.http.pool=false clean install'. This, however, affects the build time.

I would like Pipelines to work the same way as they worked before because changes like these are either polluting the build yaml script with ugly workarounds or causing previously passing builds to fail for no reason.

Comments (23)

  1. Matt Ryall staff

    Idle outbound connections from Pipelines will only be maintained by our NAT for 5 minutes. The NAT configuration was introduced recently so we could provide static IPs for people wanting to whitelist Pipelines outbound traffic. The NAT timeout in our AWS infrastructure is not configurable, so connections need to either send data more frequently than every 5 minutes, use TCP keepalive, or reconnect if the connection is reset.

    Although we've just recently added NAT connectivity to Pipelines, the underlying problem seems like a bug with Maven, where it doesn't either 1) use TCP keepalive on its long-lived HTTP connections, or 2) attempt to reconnect if one of its pooled connections are reset during the build. I'd suggest following up with Maven to see if there's an issue open for this.

    You could also see whether enabling TCP keepalive (as mentioned in the AWS link above) is possible through some kind of Maven/Wagon configuration. I couldn't find it from a brief look myself, but you may have more luck.

    This isn't something we can currently fix in Pipelines, so I'm close as "won't fix".

  2. Thomas Turrell-Croft

    @Matt Ryall So any standard Maven build that has a goal that takes longer than five minutes is going to fail on Pipelines?

    Regardless of who's fault it is. The message that I am taking away is that Pipelines can not build Maven projects (without undocumented workarounds).

  3. Matt Ryall staff

    Thanks for the feedback, Thomas. I'm sorry we can't find a way to directly resolve your issue with Pipelines at the moment.

    We are currently running thousands of Maven projects that don't hit this issue, so I wouldn't generalise it the way you have above. Bitbucket Pipelines works fine with Maven for the vast majority of use cases. I agree that it's really unfortunate that we can't just fix this problem and make it cover your build too.

    It might seem harsh to close this issue as "won't fix", but I've found it important to be clear with open source projects when we believe a fault lies with them. (And we're very open to being corrected if we're wrong!) Open source projects have many competing priorities, so knowing which bugs are causing real issues for their users and can't be worked around by a tool vendor like us, can be a good forcing function to get that bug fixed.

    Let me know if you'd like any help following up with Maven. We're happy to report the bug and provide assistance with investigation if there isn't already one tracked by them.

  4. Martin Myslík reporter

    @Matt Ryall Hi Matt. I must admit that I am a bit dissapointed because this issue prevents every Maven build running for more than 5 minutes with an additional goal to finish successfuly without hidden workarounds. This is not a very uncommon scenario and makes Pipelines seem not production ready.

    I am still convinced that this is Bitbucket issue and should be fixed in your NAT configuration. I have opened a ticket with Maven and will try to follow their suggestions. Please, take a look at this issue - it might provide some useful feedback about this topic: https://issues.apache.org/jira/browse/MNG-6199

  5. Thomas Turrell-Croft

    Bump

    Hi @Matt Ryall this whole Pipelines not supporting Maven builds that run for longer than 5 minutes issue is causing me so many problems.

    I've discovered that the Maven release plugin does not seem to honour the maven.wagon.http.pool=false setting. I know this is not your problem but still ... Maven has to be one of the biggest build automation tools out there, I can't believe that Maven users are not important to Atlassian.

    I hear what you are saying about not wanting to work around issues in third party code however at the moment you are simply pushing the work arounds on to your (soon to be paying) customers.

    Any insight you or Atlassian could offer would be very gratefully received.

    Can you take a look https://issues.apache.org/jira/browse/MNG-6199 and offer your thoughts?

  6. Alex Zorin

    Also causing problems here for the last year. Despite workarounds, Maven keeps failing in new and subtle ways.

    We've resorted to running haproxy/socat on loopback interfaces for all external connectivity .

    Not sure using Pipelines is realistic for large Maven projects while the defect still exists.

  7. Dustin Brown

    This is also a big bummer for us. We are enjoying the pipelines but this particular problem is frustrating. We are hitting this problem because we have many unit/integration tests that run(longer than 5 minutes) in between mvn goals that pull down dependencies.

  8. Luca Morettoni

    It's incredible that Atlassian didn't take in account this issue, the only possible workaround to the issue is to use a configuration that consumes more building time! We're starting to look around, after two years nothing is moving!

  9. Luca Morettoni

    Thanks Michael, from the thread seems that the issue is fixed with maven 3.2.0 and above, but there is any requirement of special options on the command line? We're using maven 3.5.4, but without the -Dmaven.wagon.http.pool=false option the build continue to fail.

    Thanks for any hint!

  10. Log in to comment