Pull requests

#2 Merged
Repository
jerryji jerryji
Branch
default
Repository
basti basti
Branch
default

added retry decorator to api.call

Author
  1. jerryji
Reviewers
Description

Making api.call() more robust to battle common Amazon timeout error.

Monkey patch in work --

""" 80) ASIN: B000PSP4YO Nautica Men's N07545 Leather Square Analog Watch <urlopen error timed out>, Retrying in 3 seconds... <urlopen error timed out>, Retrying in 3 seconds... Page 9 of 56 81) ASIN: B000PDD108 Momentum Women's 1M-SP01G2 Atlas Green Dial Black Nautica Leather Watch """

Jerry

Update:

Hi Rahlf,

I've made changes to the retry decorator according to your comments --

  1. retry() is now moved into api.py

  2. The retry-able exceptions are hard coded, only "timed out" from urllib2.URLError and socket.timeout for the time being

  3. I took a look at your unittests but most of them fly above my head, but I have tested the retry live many times

Jerry

Update 2:

In order to make this nice functionality available for the next release, I've added it to contrib. It may still be fully integrated into the API in the future.

Sebastian

  • Learn about pull requests

Comments (2)

  1. Sebastian Rahlf repo owner

    I like the idea!

    May I offer a few thoughts?

    1. I think the decorator would better live in api.py because it probably won't be needed anywhere else.
    2. Why do you pass the exception to be caught to the decorator? Wouldn't it be simpler to manage them in a constant (btw I'm a big fan of those)? Then you could do something like
                    try:
                        return f(*args, **kwargs)
                    except Exception, e:
                        if e.__class__ not in self.RETRY_EXCEPTIONS:
                            raise # re-raise
                        ...
    

    Any chance you could also provide one or two unittests?

  2. Sebastian Rahlf repo owner

    Come to think of it, it might better live in the amazonproduct.contrib package as subclass of API.

    I'm thinking of something along the lines of:

    class RetryAPI (API):
    
        #: Max number of tries before giving up
        TRIES = 5
    
        #: Delay between tries in seconds
        DELAY = 3
    
        #: Between each try the delay will be lengthed by this backoff multiplier
        BACKOFF = 1
    
        def _fetch(self, url):
            """
            Retrieves XML response from Amazon. In case of a timeout, it will try
            :const:`~RetryAPI.TRIES`` times before raising an error.
            """
            attempts = 0
            delay = self.DELAY
    
            while True:
                try:
                    attempts += 1
                    return API._fetch(self, url)
                except urllib2.URLError, e:
    
                    # if a timeout occurred
                    # wait for some time before trying again
                    reason = getattr(e, 'reason', None)
                    if isinstance(reason, socket.timeout) and attempts < self.TRIES:
                        time.sleep(delay)
                        delay *= self.BACKOFF
                        continue
    
                    # otherwise reraise the original error
                    raise
    

    BTW: Please use the Reply link below