Develop: Update.py failing

Issue #22 new
Anonymous created an issue

Update is failing on the develop branch.

Win10x64 Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32

[18:17:30.637] [update ] [ INFO] Loaded 11199488 row(s) of EDSM systems data to DB... [18:17:34.566] [update ] [ ERROR] Error cleaning up temporary file C:\Users\greg\edts_develop\edtslib\data\edts.dbfww5png2.tmp Traceback (most recent call last): File "C:\Users\greg\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 583, in _readinto_chunked n = self._safe_readinto(mvb) File "C:\Users\greg\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 630, in _safe_readinto raise IncompleteRead(bytes(mvb[0:total_bytes]), len(b)) http.client.IncompleteRead: IncompleteRead(2336 bytes read, 16384 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "update.py", line 10, in <module> a.run() File "C:\Users\greg\edts_develop\edtslib\update.py", line 217, in run dbc.populate_table_systems(self.import_json_from_url(edsm_systems_path, cur_edsm_systems_local_path, 'EDSM systems', self.args.batch_size, is_url_local=self.args.local), True) File "C:\Users\greg\edts_develop\edtslib\db_sqlite3.py", line 229, in populate_table_systems self.insert_or_replace_systems_edsm(many, cursor = c, mode = 'REPLACE') File "C:\Users\greg\edts_develop\edtslib\db_sqlite3.py", line 217, in insert_or_replace_systems_edsm c.executemany('{} INTO systems VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)'.format(mode), self._generate_systems_edsm(many)) File "C:\Users\greg\edts_develop\edtslib\db_sqlite3.py", line 182, in _generate_systems_edsm for s in systems: File "C:\Users\greg\edts_develop\edtslib\update.py", line 302, in import_data_from_url line = util.read_stream_line(stream) File "C:\Users\greg\edts_develop\edtslib\util.py", line 135, in read_stream_line return _read_stream_line_inner(stream) File "C:\Users\greg\edts_develop\edtslib\util.py", line 129, in _read_stream_line_inner_py3 def _read_stream_line_inner_py3(stream): return stream.readline().decode("utf-8") File "C:\Users\greg\edts_develop\edtslib\thirdparty\gzipinputstream.py", line 96, in readline self.fill(len(self._data) + 512) File "C:\Users\greg\edts_develop\edtslib\thirdparty\gzipinputstream.py", line 47, in fill data = self._file.read(BLOCK_SIZE) File "C:\Users\greg\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 449, in read n = self.readinto(b) File "C:\Users\greg\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 483, in readinto return self._readinto_chunked(b) File "C:\Users\greg\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 594, in _readinto_chunked raise IncompleteRead(bytes(b[0:total_bytes])) http.client.IncompleteRead: IncompleteRead(0 bytes read)

File Sizes

Comments (3)

  1. Andy Martin repo owner

    I haven't been able to reproduce this I'm afraid, under a similar environment (although on Windows 7 rather than 10).

    Does update.py --download-only work, or does it also fail to download?
    If it does work, you could try a combination of update.py --download-only followed by update.py --local and see if that works correctly.

    If --download-only doesn't work, could you try running update.py --print-urls and see if those files download correctly via a web browser? If so, you can then put the downloaded files into the edts/edtslib/data directory and run update.py --local to import them, so you can use it while we investigate further.

  2. gregmcdougall
    • python -OO update.py --download-only: This worked just fine
    • update.py --local: This worked using the previously downloaded dataset

    Does this suggest a problem with the interaction between the update scripts and a file being downloaded live? I'm loathe to experiment further for a few days as I now have a working dataset, and I'm keen to get on and use it

  3. Log in to comment