Primary EPA SRS Lookups broken- errors

[Paul Stasiuk]

Sometimes getting DNS errors.. sometimes not working at all. Need to test and investigate more fully.

[Paul Stasiuk]

Will create unit tests to the EPA query and attempt to replicate the issues in a number of scenarios:

1. 1000 Calls to the EPA done linearly.
2. 1000 Calls to the EPA done via threads of 100.

[Paul Stasiuk]

Threaded query:

class MakeThreadedEPAQuery(threading.Thread):
    def __init__(self,query_text):
        threading.Thread.__init__(self)
        self.query_text = query_text

    def run(self):
        from epa_query import query
        query(self.query_text)

def make_massive_queries():
    from time import sleep
    num_requests = 10000
    query_text ='Phosphate-phosphorus'

    for i in range(0,num_requests):
        sleep(2)

        t = MakeThreadedEPAQuery(query_text)
        t.start()

Creates the error:

Exception in thread Thread-93:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "threaded_import.py", line 73, in run
    query(self.query_text)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/epa_query.py", line 282, in query
    parse_file = make_query(1,query_params,"",logfile)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/epa_query.py", line 60, in make_query
    os.remove(file_location)
OSError: [Errno 2] No such file or directory: 'scripts/requestxmls/340.xml'

EPA Query Run 10000 times linearly.

if __name__ == '__main__':
    num_requests = 10000

    for i in range(0,num_requests):
        print i
        query_results = query('Phosphate-phosphorus')
        print query_results

Completed successfully. The previous error was caused by a file-locking issue of an unruly script.

[Paul Stasiuk]

Since all of the queries went through with the unit testing, I'll test using the actual upload process.

[Paul Stasiuk]

Getting this proxy error:

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "threaded_import.py", line 29, in run
    check_import(self.importid)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/xml_parsing_backed.py", line 338, in check_import
    write_status_file(importid)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/xml_parsing_backed.py", line 373, in write_status_file
    query_results = query(parameter_to_query)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/epa_query.py", line 282, in query
    parse_file = make_query(1,query_params,"",logfile)
  File "/Users/pstasiuk/git/cacw_deploy/import_heart/epa_query.py", line 58, in make_query
    response = urllib2.urlopen(req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 502: Proxy Error

[Paul Stasiuk]

Current status: SOL.

Will test on different service and check for computability that way.

[Paul Stasiuk]

Resolved after hearing back from the EPA: Hi Paul,

Are things working for you today? I got confirmation that the Network Solutions problems should all be resolved as of yesterday evening. Also, I just copy/pasted your query, used my security token and rant the query 5 times. All requests worked and I got the responses in 7-12 seconds.

Also, although maxRows is a valid parameter, it’s ignored by most Exchange Network services including SRS.

Andrew Hampton | Senior Consultant | CGI Federal | 234.567.9403