Socket-Timeout

[Niels Anonymous]

We have a larger commit and I found a lot of WARN ServiceRunner [jigit.indexer.JigitIndexer] SocketTimeoutException occurred when trying to get commit in the log.

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
        at api.client.http.ApiHttpRequester.parse(ApiHttpRequester.java:93)
        at api.client.http.ApiHttpRequester.withResultOf(ApiHttpRequester.java:66)
        at api.client.http.ApiHttpRequester.withResultOf(ApiHttpRequester.java:53)
        at jigit.client.gitlab.GitLabRepositoryAPI.getCommit(GitLabRepositoryAPI.java:47)
        at jigit.indexer.api.gitlab.GitLabAPIAdapter.getCommit(GitLabAPIAdapter.java:37)
        at jigit.indexer.IndexingWorker.indexFromCommit(IndexingWorker.java:128)
        at jigit.indexer.IndexingWorker.indexRepoBranch(IndexingWorker.java:97)
        at jigit.indexer.IndexingWorker.handleBranchMode(IndexingWorker.java:66)
        at jigit.indexer.IndexingWorker.call(IndexingWorker.java:52)
        at jigit.indexer.IndexingWorker.call(IndexingWorker.java:20)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Is it possible to increase the timeout time?

[Dmitrii Apanasevich]

Is it possible to increase the timeout time?

Try increasing value of "Commit request timeout" parameter of a repo at the config page. Click Actions -> Edit to edit this parameter.

[Niels Anonymous]

Perfect! Sorry I forget to look in the configuration.

[Tyler Wright]

Hi All,

I'm having this same issue. I get to about 16k commits for a specific group of repos and then it stops. Checking the logs, I see the same error: "SocketTimeoutException occurred when trying to get commit." I've tried a mixture of settings but haven't found a way of getting all of the commits successfully. Any advice would be greatly appreciated!

Thanks!

[Dmitrii Apanasevich]

Hi Tyler,

The only way to tackle the exception is to increase "Commit request timeout", as suggested above.

Do you get the exception for one and the same commit? Commit hash and repo name should be mentioned in the message.

[Tyler Wright]

Hi Dmitrii,

I've increased the timeout to 600 yet the same results occur. Should I increase even more? Also, this happens for many commits - we're missing quite a few.

Thanks!

[Dmitrii Apanasevich]

600 seconds is a very long period. Usually, problems can occur when Jigit is trying to get a diff of a huge commit (which contains large objects).

Maybe it's a network issue?

[Tyler Wright]

I doubt it. The two have a 10Gb pipe between them and they aren't even utilizing anywhere close to 1/4 of it.

[Dmitrii Apanasevich]

Ok. Let's check it in a direct way. Could you call an HTTP API with the following command from command line at JIRA server? This command does the same as Jigit does.

curl -i --header "PRIVATE-TOKEN: {PUT_HERE_YOUR_SECRET_TOKEN}" "https://gitlab.com/api/v4/projects/{PUT_HERE_YOUR_REPO_NAME}/repository/commits/{PUT_HERE_COMMIT_HASH}"

Placeholders should be replaced with your data. You can use any commit, which caused the exception. Just get its hash from JIRA log. Repo name should be encoded, for example with this service. Command will look like this

curl -i --header "PRIVATE-TOKEN: mysecret" "https://gitlab.com/api/v4/projects/groupname%2Fsubgroupname/repository/commits/g8g2fg28rdh2hf2hrd7h28fh82efg82e8cg"

Let's see how much time it needs to get a commit from your Gitlab instance.

[Tyler Wright]

Looks like it took under a second.

tylwright@twright-mbp  ~  time curl -i --header "PRIVATE-TOKEN: <token>" "http://<local GitLab URL>/api/v4/projects/<group>%2F<repo>/repository/commits/<commit ID>"

HTTP/1.1 200 OK
Server: nginx
Date: Mon, 02 Jul 2018 14:37:55 GMT
Content-Type: application/json
Content-Length: 645
Connection: keep-alive
Cache-Control: max-age=0, private, must-revalidate
Etag: W/"7f6c2aa401b4c2b26127f657bb72832c"
Vary: Origin
X-Frame-Options: SAMEORIGIN
X-Request-Id: ab7998d4-a5e8-43c0-a3f6-97139a0595e6
X-Runtime: 0.164126
Strict-Transport-Security: max-age=31536000

{"id":"<commit ID>","short_id":"<short commit ID>","title":"<title>","created_at":"2018-05-22T14:01:27.000-04:00","parent_ids":["<parent commit ID>"],"message":"<message>","author_name":"<user>","author_email":"<email>","authored_date":"2018-05-21T16:06:07.000-04:00","committer_name":"<user>","committer_email":"<email>","committed_date":"2018-05-22T14:01:27.000-04:00","stats":{"additions":140,"deletions":12,"total":152},"status":null,"last_pipeline":null}

curl -i --header "PRIVATE-TOKEN: P4HDg7nPatejxxHr422W"   0.01s user 0.00s system 4% cpu 0.188 total

[Dmitrii Apanasevich]

Looks acceptable. Do you still get such exceptions in JIRA log?

Actually, this kind of exceptions is not a big problem for Jigit. It just stops indexing for 2 minutes and then starts again. It could be a problem if you saw the same commit hash in the log again and again. This situation would mean that indexing had been stuck with a specific commit and cannot index it.

[Tyler Wright]

I haven't seen the timeout in a while but not all commits are accounted for. There are hundreds that are missing still. I see this in the logs now:

2018-07-05 10:25:46,556 Caesium-1-2 ERROR ServiceRunner     [jigit.indexer.JigitIndexer] JigitIndexer::execute - ExecutionException. Cause:
    at jigit.ao.CommitManagerImpl.create(CommitManagerImpl.java:70)
    at jigit.ao.CommitManagerImpl.persist(CommitManagerImpl.java:179)
    at jigit.indexer.PersistStrategyFactoryImpl$PersistAllDataStrategy.persist(PersistStrategyFactoryImpl.java:50)
    at jigit.indexer.IndexingWorker.indexFromCommit(IndexingWorker.java:150)
    at jigit.indexer.IndexingWorker.indexRepoBranch(IndexingWorker.java:97)
    at jigit.indexer.IndexingWorker.handleBranchMode(IndexingWorker.java:66)
    at jigit.indexer.IndexingWorker.call(IndexingWorker.java:52)
    at jigit.indexer.IndexingWorker.call(IndexingWorker.java:20)
2018-07-05 10:26:51,980 jigit-indexer-1 ERROR ServiceRunner     [jigit.indexer.JigitIndexer] Got an error while indexing repository <group>/<repo> and branch <branch>
    at jigit.client.gitlab.GitLabRepositoryAPI.getCommit(GitLabRepositoryAPI.java:45)
    at jigit.indexer.api.gitlab.GitLabAPIAdapter.getCommit(GitLabAPIAdapter.java:37)
    at jigit.indexer.IndexingWorker.indexFromCommit(IndexingWorker.java:128)
    at jigit.indexer.IndexingWorker.indexRepoBranch(IndexingWorker.java:97)
    at jigit.indexer.IndexingWorker.handleBranchMode(IndexingWorker.java:66)
    at jigit.indexer.IndexingWorker.call(IndexingWorker.java:52)
    at jigit.indexer.IndexingWorker.call(IndexingWorker.java:20)

[Dmitrii Apanasevich]

Hi Tyler,

First of all, sorry for the late response.

As I can see it's not a SocketTimeoutException. It's definitely a new problem. Unfortunately, stacktraces you gave are too short. Do you trimmed them or it's all you can see?

In any case, please, open a new issue and we'll continue there.