Ned Batchelder avatar Ned Batchelder committed afd13f1

Adopt the faster loop from issue #1. Also fix the quotes in a docstring so doctest works.

Comments (0)

Files changed (1)

         Yields pairs (`name`, `tokentext`).
 
         """
-        while text:
-            eaten = 0
-            for match in self.regexes[self.state].finditer(text):
-                for name, toktext in match.groupdict().iteritems():
-                    if toktext is not None:
-                        tok = self.toks[name]
-                        new_state = tok.next
-                        eaten += len(toktext)
-                        yield (tok.name, toktext)
-                if new_state:
-                    self.state = new_state
+        end = len(text)
+        state = self.state
+        regexes = self.regexes
+        toks = self.toks
+        start = 0
+
+        while start < end:
+            for match in regexes[state].finditer(text, start):
+                name = match.lastgroup
+                tok = toks[name]
+                toktext = match.group(name)
+                start += len(toktext)
+                yield (tok.name, toktext)
+
+                if tok.next:
+                    state = tok.next
                     break
-            text = text[eaten:]
+
+        self.state = state
 
 
 class JsLexer(Lexer):
     
     >>> lexer = JsLexer()
     >>> list(lexer.lex("a = 1"))
-    [("id", "a"), ("ws", " "), ("punct", "="), ("ws", " "), ("dnum", "1")]
+    [('id', 'a'), ('ws', ' '), ('punct', '='), ('ws', ' '), ('dnum', '1')]
 
     This doesn't properly handle non-Ascii characters in the Javascript source.
 
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.