Matt Chaput committed c2f1f05

Fixed add_postings_to_pool() to use Cursor.flatten() instead of Node.flatten().
Node.flatten() caused recursion error on very long words.
Cursor.flatten() is iterative instead of recursive.
Fixes issue #366.

Comments (0)

Files changed (2)


         for fieldname, fieldobj in self.schema.items():
             if (fieldobj.separate_spelling()
                 and reader.has_word_graph(fieldname)):
-                for word in reader.word_graph(fieldname).flatten():
+                gr = reader._get_graph()
+                cursor = gr.cursor(fieldname)
+                for word in cursor.flatten():
                     # Adding a post where docnum=None marks it as a separate
                     # spelling word
                     add_post((fieldname, word, -1, -1, emptybytes))


     assert c.format_string(highlight.UppercaseFormatter()) == "dworska"
+def test_very_long_words():
+    import sys
+    length = int(sys.getrecursionlimit() * 1.5)
+    strings1 = [u(chr(i) * length) for i in range(65, 70)]
+    strings2 = [u(chr(i) * length) for i in range(71, 75)]
+    ana = analysis.StemmingAnalyzer()
+    schema = fields.Schema(text=fields.TEXT(analyzer=ana, spelling=True))
+    ix = RamStorage().create_index(schema)
+    with ix.writer() as w:
+        for string in strings1:
+            w.add_document(text=string)
+    with ix.writer() as w:
+        for string in strings2:
+            w.add_document(text=string)
+        w.optimize = True
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.