1. Rufus Pollock
  2. wdmmgext

Commits

AnnaPS  committed 830c2ed

[wdmmg][xs] switching from sqlalchemy commit() to flush() in an attempt to reduce memory usage

  • Participants
  • Parent commits d8912e5
  • Branches default

Comments (0)

Files changed (1)

File wdmmgext/load/departments.py

View file
     for row_index, row in enumerate(reader):
     #for row_index in range(header_row+1,sheet.nrows):
          if commit_every and row_index%commit_every == 0:
-             print "Committing before processing row %d" % row_index
-             model.Session.commit()
+             print "Flushing before processing row %d" % row_index
+             model.Session.flush()
          #row = [sheet.cell(row_index,col).value for col in range(0,sheet.ncols)]
          row = [unicode(r.decode("mac_roman").strip()) for r in row]
          # Some spreadsheets (Welsh Office) have incomplete rows. Ignore these.
                      value=util.get_or_create_value(key, \
                            unicode(slugify(name)), unicode(name))
                   ))
+    # Commit at the end of every file.
     if commit_every:
         model.Session.commit()
 
+
 dept_dictionary = { 
 'A': 'Administration',
 'AGO': 'Attorney General\'s Office',