Issue #4 new

Max # of sockets

ppolv
created an issue

Hi,

I'd been using lhttpc (great work!) with a minor patch that limits the number of simultaneous requests to a given endpoint. It has worked fine so far, but isn't really well tested outside my specific scenario.

Anyway, I thought it might come handy for others, or perhaps to include it on lhttpc if the patch is good enough.

The idea was to modify lhttpc_manager to delay the return of gen_server:calls for sockets if there are > MAX active clients at a given endpoint. It will return (either a valid socket or inform the client that is allowed to continue and open a new connection) only when one of the active clients finish.

The tricky part is how to detect that a client has finished the requests. The attached patch don't rely on 'done' messages (those are used to return a socket to the pool), but instead links to the client and uses the 'EXIT' messages to detect a finished client process.

cheers! pablo.

Comments (3)

  1. Tamas Nagy

    Hi Pablo,

    Sorry for the late response. One of the developer of lhttpc is moving to another country so it took some time to discuss this.

    We certainly agree that this is a nice feature to have however we would like to keep the lhttpc_manager as lightweight and simple as possible.

    As this feature might not be necessary for all users of the library it would make sense to keep it separate from the manager and maybe even from the lhttpc:request/x functions.

    Would you like to rewrite to patch into a separate module which would act like a simple semaphore? It could work something like this:

    lhttpc_limiter:down(URL),
    lhttp:request(...),
    ....
    lhttpc_limiter:up(URL)
    

    The down call would block if the max number of connections is reached. Of course the limiter would still need to link to the requester process which could die before releasing the semaphore.

    If this becomes a widely used feature we could still roll it into the API of lhttpc later on (as an option perhaps).

    Regards, Tamas

  2. ppolv reporter

    Hi, agree, lhttpc_manager was getting quite complex after the proposed changes. Your approach is certainly simpler :).

  3. Log in to comment