[wait-for-feedback] Random http status 403

Issue #242 closed
Thomas Güttler created an issue

We get random http status 403 messages.

It happens roughly in 1 of 50 git clone commands.

Here is the output of our client (pip (Python install tool) started from jenkins):

Obtaining foowork from git+https://source.example.com:40443/repos/foowork@release/2016.371#egg=foowork (from -r /tmp/tmp1jaIlG/requirements.txt (line 12))
Cloning https://source.example.com:40443/repos/foowork (to release/2016.371) to ./src/foowork
Complete output from command git clone -q https://source.example.com:40443/repos/foowork /home/foowork_eins_di2468/src/foowork:

---------------------------------------- error: RPC failed; result=22, HTTP code = 403
fatal: The remote end hung up unexpectedly
Command "git clone -q https://source.example.com:40443/repos/foowork /home/foowork_eins_di2468/src/foowork" failed with error code 128 in None

Has anybody a hint why this fails sometimes?

How can I debug this?

Version: Kallithea==0.3.2

Comments (7)

  1. Andrej Shadura

    Try looking into your server-side logs. You may or may not need to add debug = true and tune log levels if your logs aren’t verbose enough.

  2. Thomas Güttler reporter

    Tonight one CI failed again. Here are is the logging output:

    I renamed our internal repos to "repowhichfailed" and to "otherrepo".

    There are several request coming in in the same second.

    2016-08-17 23:47:58.589 DEBUG [kallithea.lib.middleware.simplegit] Repository path is /home/kallithea/repos/otherrepo
    2016-08-17 23:47:58.589 DEBUG [kallithea.lib.middleware.simplegit] Checking locking on repository
    2016-08-17 23:47:58.592 DEBUG [kallithea.lib.middleware.simplegit] pathinfo: /repowhichfailed/git-upload-pack detected as Git True
    2016-08-17 23:47:58.599 DEBUG [kallithea.lib.middleware.simplegit] Extracted repo name is repowhichfailed
    2016-08-17 23:47:58.608 DEBUG [kallithea.lib.middleware.simplegit] Anonymous access is disabled, running authentication
    2016-08-17 23:47:58.609 DEBUG [kallithea.lib.middleware.simplegit] Not enough credentials to access this repository as anonymous user
    2016-08-17 23:47:58.609 DEBUG [kallithea.lib.middleware.simplegit] Running PRE-AUTH for container based authentication
    2016-08-17 23:47:58.840 DEBUG [kallithea.lib.auth] checking VCS protocol permissions set([u'repository.read']) for user:readonly repository:repowhichfailed
    2016-08-17 23:47:58.841 DEBUG [kallithea.lib.auth_modules] Authentication against [u'kallithea.lib.auth_modules.auth_internal', u'kallithea.lib.auth_modules.auth_ldap'] plugins
    2016-08-17 23:47:58.841 DEBUG [kallithea.lib.auth] Permission to repo: repowhichfailed denied for user: readonly @ PermissionMiddleware
    

    The strange thing: "Permission to repo: repowhichfailed denied for user: readonly @ PermissionMiddleware"

    Some seconds before a different CI run was able to access the repo:

    [Wed Aug 17 23:46:26 2016] [error] 2016-08-17 23:46:26.131 DEBUG [kallithea.lib.auth] checking VCS protocol permissions set([u'repository.read']) for user:readonly repository:repowhichfailed
    [Wed Aug 17 23:46:26 2016] [error] 2016-08-17 23:46:26.132 DEBUG [kallithea.lib.auth] Permission to repo: repowhichfailed granted for user: readonly @ PermissionMiddleware
    

    Do you have a clue ?

    I guess it is related to multiprocessing, since there are several CI runs accessing the repos at once.

  3. Log in to comment