Python 3.7: RuntimeError: generator raised StopIteration

Issue #1457 new
Miro Hrončok
created an issue

See PEP 479 -- Change StopIteration handling inside generators.

Test failures:

$ make test 
Pygments 2.2.0 test suite running (Python 3.7.0)...
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../pygments/lexer.py:427: FutureWarning: Possible nested set at position 1
  return re.compile(regex, rflags).match
............................/pygments/lexer.py:427: FutureWarning: Possible nested set at position 11
  return re.compile(regex, rflags).match
......................................................................................................................./pygments/lexer.py:427: FutureWarning: Possible set symmetric difference at position 275
  return re.compile(regex, rflags).match
....................................................................................................E.................................................................................../pygments/lexer.py:427: FutureWarning: Possible nested set at position 4
  return re.compile(regex, rflags).match
.................................................................................................................................................................................................................................................................................................................................................................E..........................................................................................................................................SS..SS...........SS..SS...............................................................................................................................................................F...........................................................................................................................................................E.................................................E..........................................................................................................................................................................
kpathsea: Running mktexfmt latex.fmt
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /home/churchyard/.texlive2018/texmf-config/web2c/fmtutil.cnf
mktexfmt [WARNING]: double mention of pdfcsplain/pdftex in /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
mktexfmt [WARNING]: double mention of pdfcsplain/pdftex in /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
mktexfmt [WARNING]: double mention of mf/mf-nowin in /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
mktexfmt [WARNING]: double mention of tex/tex in /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
mktexfmt [INFO]: writing formats under /home/churchyard/.texlive2018/texmf-var/web2c
mktexfmt [INFO]: --- remaking latex with pdftex
mktexfmt: running `pdftex -ini   -jobname=latex -progname=latex -translate-file=cp227.tcx *latex.ini' ...
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (INITEX)
 restricted \write18 enabled.
 (/usr/share/texlive/texmf-dist/web2c/cp227.tcx)
entering extended mode
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/latex.ini
(/usr/share/texlive/texmf-dist/tex/generic/tex-ini-files/pdftexconfig.tex)
(/usr/share/texlive/texmf-dist/tex/latex/base/latex.ltx
(/usr/share/texlive/texmf-dist/tex/latex/base/texsys.cfg)
./texsys.aux found


\@currdir set to: ./.


Assuming \openin and \input 
have the same search path.


Defining UNIX/DOS style filename parser.

catcodes, registers, parameters,
LaTeX2e <2017-04-15>
hacks, control, par, spacing, files, font encodings, lengths,
====================================

Local config file fonttext.cfg used

====================================
(/usr/share/texlive/texmf-dist/tex/latex/base/fonttext.cfg
(/usr/share/texlive/texmf-dist/tex/latex/base/fonttext.ltx
=== Don't modify this file, use a .cfg file instead ===

(/usr/share/texlive/texmf-dist/tex/latex/base/omlenc.def)
(/usr/share/texlive/texmf-dist/tex/latex/base/t1enc.def)
(/usr/share/texlive/texmf-dist/tex/latex/base/ot1enc.def)
(/usr/share/texlive/texmf-dist/tex/latex/base/omsenc.def)
(/usr/share/texlive/texmf-dist/tex/latex/base/t1cmr.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/ot1cmr.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/ot1cmss.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/ot1cmtt.fd)))
====================================

Local config file fontmath.cfg used

====================================
(/usr/share/texlive/texmf-dist/tex/latex/base/fontmath.cfg
(/usr/share/texlive/texmf-dist/tex/latex/base/fontmath.ltx
=== Don't modify this file, use a .cfg file instead ===

(/usr/share/texlive/texmf-dist/tex/latex/base/omlcmm.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/omscmsy.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/omxcmex.fd)
(/usr/share/texlive/texmf-dist/tex/latex/base/ucmr.fd)))
====================================

Local config file preload.cfg used

=====================================
(/usr/share/texlive/texmf-dist/tex/latex/base/preload.cfg
(/usr/share/texlive/texmf-dist/tex/latex/base/preload.ltx)) page nos., x-ref,
environments, center, verbatim, math definitions, boxes, title, sectioning,
contents, floats, footnotes, index, bibliography, output,
===========================================
Local configuration file hyphen.cfg used
===========================================
(/usr/share/texlive/texmf-dist/tex/generic/babel/hyphen.cfg
(/usr/share/texlive/texmf-dist/tex/generic/babel/switch.def)
(/usr/share/texlive/texmf-dist/tex/generic/hyphen/hyphen.tex)
(/usr/share/texlive/texmf-dist/tex/generic/hyphen/dumyhyph.tex)
(/usr/share/texlive/texmf-dist/tex/generic/hyphen/zerohyph.tex)
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/loadhyph/loadhyph-cs.tex
EC Czech hyphenation patterns (Pavel Sevecek, v3, 1995)
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/conversions/conv-utf8-ec.t
ex)
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/patterns/tex/hyph-cs.tex))
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/loadhyph/loadhyph-sk.tex
EC Slovak hyphenation patterns (Jana Chlebikova, 1992)
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/conversions/conv-utf8-ec.t
ex)
(/usr/share/texlive/texmf-dist/tex/generic/hyph-utf8/patterns/tex/hyph-sk.tex))
)
 ) )
Beginning to dump on file latex.fmt
 (preloaded format=latex 2018.7.4)
5116 strings of total length 70571
45581 memory locations dumped; current usage is 144&44451
3427 multiletter control sequences
\font\nullfont=nullfont
\font\OMX/cmex/m/n/10=cmex10
\font\tenln=line10
\font\tenlnw=linew10
\font\tencirc=lcircle10
\font\tencircw=lcirclew10
\font\OT1/cmr/m/n/5=cmr5
\font\OT1/cmr/m/n/7=cmr7
\font\OT1/cmr/m/n/10=cmr10
\font\OML/cmm/m/it/5=cmmi5
\font\OML/cmm/m/it/7=cmmi7
\font\OML/cmm/m/it/10=cmmi10
\font\OMS/cmsy/m/n/5=cmsy5
\font\OMS/cmsy/m/n/7=cmsy7
\font\OMS/cmsy/m/n/10=cmsy10
3633 words of font info for 14 preloaded fonts
24 hyphenation exceptions
Hyphenation trie of length 13518 has 494 ops out of 35111
  248 for language 4
  63 for language 3
  2 for language 1
  181 for language 0
0 words of pdfTeX memory
0 indirect objects
No pages of output.
Transcript written on latex.log.
mktexfmt [INFO]: /home/churchyard/.texlive2018/texmf-var/web2c/pdftex/latex.fmt installed.
mktexfmt [INFO]: Disabled formats: 26
mktexfmt [INFO]: Successfully rebuilt formats: 1
mktexfmt [INFO]: Not selected formats: 19
mktexfmt [INFO]: Total formats: 46
mktexfmt [INFO]: exiting with status 0
..................................................................................................................................
======================================================================
ERROR: test_basic_api.test_lexer_classes(<class 'pygments.lexers.sql.PostgresConsoleLexer'>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../pygments/lexers/sql.py", line 350, in get_tokens_unprocessed
    line = next(lines)
  File "../pygments/lexers/sql.py", line 284, in __next__
    return next(self.iter)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../pygments-main/__venv__/lib64/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File ".../pygments-main/tests/test_basic_api.py", line 69, in verify
    tokens = list(inst.get_tokens(test_content))
  File "../pygments/lexer.py", line 191, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
RuntimeError: generator raised StopIteration

======================================================================
ERROR: test_basic_api.test_lexer_options(<class 'pygments.lexers.sql.PostgresConsoleLexer'>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../pygments/lexers/sql.py", line 350, in get_tokens_unprocessed
    line = next(lines)
  File "../pygments/lexers/sql.py", line 284, in __next__
    return next(self.iter)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../pygments-main/__venv__/lib64/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File ".../pygments-main/tests/test_basic_api.py", line 96, in verify
    ensure(inst.get_tokens('a\nb'), 'a\nb\n')
  File ".../pygments-main/tests/test_basic_api.py", line 90, in ensure
    concatenated = ''.join(token[1] for token in tokens)
  File ".../pygments-main/tests/test_basic_api.py", line 90, in <genexpr>
    concatenated = ''.join(token[1] for token in tokens)
  File "../pygments/lexer.py", line 191, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
RuntimeError: generator raised StopIteration

======================================================================
ERROR: test_examplefiles.test_example_files(<pygments.lexers.ShenLexer>, 'test.shen')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../pygments/lexers/lisp.py", line 2330, in _process_declaration
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../pygments-main/__venv__/lib64/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File ".../pygments-main/tests/test_examplefiles.py", line 108, in check_lexer
    for type, val in lx.get_tokens(text):
  File "../pygments/lexer.py", line 191, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
  File "../pygments/lexers/lisp.py", line 2280, in _process_declarations
    self._process_declaration(declaration, tokens):
RuntimeError: generator raised StopIteration

======================================================================
ERROR: test_examplefiles.test_example_files(<pygments.lexers.PostgresConsoleLexer>, 'psql_session.txt')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../pygments/lexers/sql.py", line 350, in get_tokens_unprocessed
    line = next(lines)
  File "../pygments/lexers/sql.py", line 284, in __next__
    return next(self.iter)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../pygments-main/__venv__/lib64/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File ".../pygments-main/tests/test_examplefiles.py", line 108, in check_lexer
    for type, val in lx.get_tokens(text):
  File "../pygments/lexer.py", line 191, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
RuntimeError: generator raised StopIteration

======================================================================
FAIL: test_examplefiles.test_example_files(<pygments.lexers.XorgLexer>, 'xorg.conf')
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../pygments-main/__venv__/lib64/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File ".../pygments-main/tests/test_examplefiles.py", line 120, in check_lexer
    raise AssertionError('round trip failed for ' + absfn)
AssertionError: round trip failed for .../pygments-main/tests/examplefiles/xorg.conf
-------------------- >> begin captured stdout << ---------------------
--- 

+++ 

@@ -16,7 +16,7 @@

         Identifier "Screen"
    Device "Card0"
    DefaultDepth    24
-   SubSectionSub     "Display"
+   SubSection     "Display"
        Depth       24
    EndSubSection
         Option "UseEDIDDpi" "False"

--------------------- >> end captured stdout << ----------------------
>>  raise AssertionError('round trip failed for ' + '.../pygments-main/tests/examplefiles/xorg.conf')


----------------------------------------------------------------------
Ran 2005 tests in 11.328s

FAILED (SKIP=8, errors=4, failures=1)

Comments (3)

  1. Miro Hrončok reporter

    Here is a fixer. Will try my best to submit a PR, but usually Bitbucket makes it hard for me.

    diff -r 7941677dc77d pygments/lexers/lisp.py
    --- a/pygments/lexers/lisp.py   Mon Mar 13 19:16:03 2017 +0000
    +++ b/pygments/lexers/lisp.py   Wed Jul 04 18:03:07 2018 +0200
    @@ -2327,13 +2327,13 @@
                 token = Name.Function if token == Literal else token
                 yield index, token, value
    
    -        raise StopIteration
    +        return
    
         def _process_signature(self, tokens):
             for index, token, value in tokens:
                 if token == Literal and value == '}':
                     yield index, Punctuation, value
    -                raise StopIteration
    +                return
                 elif token in (Literal, Name.Function):
                     token = Name.Variable if value.istitle() else Keyword.Type
                 yield index, token, value
    diff -r 7941677dc77d pygments/lexers/sql.py
    --- a/pygments/lexers/sql.py    Mon Mar 13 19:16:03 2017 +0000
    +++ b/pygments/lexers/sql.py    Wed Jul 04 18:03:07 2018 +0200
    @@ -347,7 +347,10 @@
                 # Emit the output lines
                 out_token = Generic.Output
                 while 1:
    -                line = next(lines)
    +                try:
    +                    line = next(lines)
    +                except StopIteration:
    +                    return
                     mprompt = re_prompt.match(line)
                     if mprompt is not None:
                         # push the line back to have it processed by the prompt
    
  2. Log in to comment