excessive creation of Ktor-client-apache threads until OOM or user/systen limits are reached

[Christopher Ruwe]

Hi,

first thank you for developing the Bitbucket Linky Plugin. It is a huge help for my daily work.

I am on Ubuntu 18.04 (bionic) with

$ uname -a
Linux mccarthy 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

and

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

On idea-IU-183 with packaged OpenJDK and with system JDK, an excessive number of "Ktor-client-apache" threads may be observed, which eventually render the system unusable.

The logs show messages alike to

2019-01-02 15:31:57,584 [ 619210]  DEBUG -            BitbucketReferences - Probing for Bitbucket Server at 'https://gogs.hb22.cruwe.de:2222/ 
2019-01-02 15:31:57,593 [ 619219]  DEBUG -            BitbucketReferences - Didn't find Bitbucket Server at candidate url 'https://gogs.hb22.cruwe.de:2222/' 
com.atlassian.bitbucket.server.rest.BitbucketServerRestClientException: Error requesting 'application properties' API endpoint

and each time such messages are logged, the thread count increases 600ish. I have attached a log file containing a number of such messages.

Obviously, that situation is put to an end when user or system limits (NOPROC, STACK) have been exhausted. Please note that because on Linux, threads are OS-native threads, NOPROC acually applies.

I can only append one log, I have logs with

2019-01-02 07:43:42,306 [40219358]  ERROR -            BitbucketReferences - IntelliJ IDEA 2018.3.2  Build #IU-183.4886.37 
2019-01-02 07:43:42,306 [40219358]  ERROR -            BitbucketReferences - JDK: 1.8.0_152-release; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o 
2019-01-02 07:43:42,306 [40219358]  ERROR -            BitbucketReferences - OS: Linux 
2019-01-02 07:43:42,306 [40219358]  ERROR -            BitbucketReferences - Plugin to blame: Bitbucket Linky version: 6.1 
2019-01-02 07:43:42,306 [40219358]  ERROR -            BitbucketReferences - unable to create new native thread 
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.start(CloseableHttpAsyncClientBase.java:83)
    at io.ktor.client.engine.apache.ApacheEngine.<init>(ApacheEngine.kt:14)
    at io.ktor.client.engine.apache.Apache.create(Apache.kt:19)
    at io.ktor.client.HttpClientKt.HttpClient(HttpClient.kt:27)
    at io.ktor.client.HttpClientJvmKt.HttpClient(HttpClientJvm.kt:9)
    at io.ktor.client.HttpClientJvmKt.HttpClient$default(HttpClientJvm.kt:15)
    at com.atlassian.bitbucket.server.rest.BitbucketServerApiConfig.<init>(server-api.kt:38)
    at com.atlassian.bitbucket.server.rest.BitbucketServerApiImpl.<init>(server.kt:19)
    at com.atlassian.bitbucket.server.rest.BitbucketServerApi$Companion.invoke(server-api.kt:25)
    at com.atlassian.bitbucket.linky.rest.DefaultBitbucketRestClientProvider.bitbucketServerRestClient(rest-client-provider.kt:61)
    at com.atlassian.bitbucket.linky.discovery.DefaultBitbucketServerProbe$probe$2.invokeSuspend(probe-server.kt:74)
    at com.atlassian.bitbucket.linky.discovery.DefaultBitbucketServerProbe$probe$2.invoke(probe-server.kt)
    at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:84)

which show a situation in which the rest of the system is not able to fork any process anymore.

I have first observed such behaviour after the Dec17th IDEA update (which might coincide with a Linky-update, I do not know). I would be surprised if the problem was present before, because it takes few hours to throw the fork-bomb at my system, grinding it to a halt, when idling and considerably less when actively working.

Thank you very much for your time and consideration.

[Daniil Penkin]

Hello @cjr_at_yes,

Wow, thanks for a thorough issue investigation! This is definitely a nasty flaw, I'm sorry for causing such problems.

Linky is, no doubt, in charge for threads from Ktor. This is HTTP client library I switched to recently: first version powered by Ktor was released on 13th December, so the date you referred to matches. Being my pet project where I often try out various libraries and techniques, and together with lack of proper auto-testing, Linky is not very stable between major versions. Sucks sometimes, but this is how I keep it going :)

In addition to forking suspiciously high number of threads and leaving them around I can also see another issue from the log you attached: Linky is probing the same host again and again, and does that way too often.

Fixes for all these things are now top priority for me. May I ask you to clarify some details of your environment (this might help a bit with narrowing down the issue): do you normally have a single IntelliJ project with multiple repositories in it or do you work with multiple projects with a single repository in each?

While I'm working on a fix, the best option for you is to downgrade to version 5.2 I guess (direct download link). I'll let you know when 6.3 is out, or you can simply wait for IntelliJ to suggest you installing it.

Sorry again for the inconvenience.

Cheers, Daniil

[Christopher Ruwe]

Hi,

thank you very much for your speedy response.

To clarify the environment as you asked:

I usually have projects per "topic", so one project for my puppet modules, another for a distributed system of microservices, and so on. Accordingly, I might have up to thirty git repositories registered per project and in addition might have multiple projects open at the same time. I admit this may be unusual, but I work in the role of "resident DevOps Engineer", so I often need to do many thinks in parallel.

Thanks again for your work for Bitbucket Linky! Cheers, Christopher

[Daniil Penkin]

Thanks for the kind words :)

Cool, thanks, this confirms my assumption about one aspect I found in your log. This bit is easy to fix, but I still need to look deeper into the threading issue.

If all goes to plan, I'll push the fixes out of the door within a week.

Cheers, Daniil

[Daniil Penkin]

Hi @cjr_at_yes,

Just a heads up, I've just released version 6.2 with a fix for the issues you faced.

Thank you for the patience!

Cheers, Daniil

[Christopher Ruwe]

And thank you for the fix! Thanks and cheers!

[Daniil Penkin]

Issue #67 was marked as a duplicate of this issue.

[Daniil Penkin]

Removing version: 6.1 (automated comment)