Problems generating SWIG docstrings when compiling DOLFIN 2017.2.0 on Arch Linux

[Sigvald]

I have problems building (from source) DOLFIN 2017.2.0 (stable I believe) on Arch Linux with Python 3.6.4. I get to 92% in the installation and then get:

[ 92%] Built target copy_swig_files
[ 92%] Swig source
/home/sigvald/Downloads/dolfin/src/dolfin/build/dolfin/swig/modules/mesh/module.i:165: Error: Unable to find 'dolfin/swig/mesh/docstrings.i'
/home/sigvald/Downloads/dolfin/src/dolfin/build/dolfin/swig/modules/mesh/module.i:166: Error: Unable to find 'dolfin/swig/generation/docstrings.i'
/home/sigvald/Downloads/dolfin/src/dolfin/build/dolfin/swig/modules/mesh/module.i:167: Error: Unable to find 'dolfin/swig/geometry/docstrings.i'
/home/sigvald/Downloads/dolfin/src/dolfin/build/dolfin/swig/modules/mesh/module.i:168: Error: Unable to find 'dolfin/swig/refinement/docstrings.i'
/home/sigvald/Downloads/dolfin/src/dolfin/build/dolfin/swig/modules/mesh/module.i:169: Error: Unable to find 'dolfin/swig/graph/docstrings.i'
make[2]: *** [dolfin/swig/modules/mesh/CMakeFiles/_mesh.dir/build.make:110: dolfin/swig/modules/mesh/modulePYTHON_wrap.cxx] Error 1
make[1]: *** [CMakeFiles/Makefile2:825: dolfin/swig/modules/mesh/CMakeFiles/_mesh.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Apparently something's up with the generation of docstrings.i, which I believe should be made by the script generate-swig-docstrings.py. When I try to run this I get:

DONE parsing C++ with doxygen
--------------------------------------------
Generating python docstrings in directory
  swig_dir = /home/sigvald/Downloads/dolfin/src/dolfin/dolfin/swig
Parsing doxygen XML files in /home/sigvald/Downloads/dolfin/src/dolfin/doc/doxygen/xml
....................................................Traceback (most recent call last):
  File "./cmake/scripts/generate-swig-docstrings.py", line 146, in <module>
    generate_docstrings(dest_dir)
  File "./cmake/scripts/generate-swig-docstrings.py", line 115, in generate_docstrings
    swig_header=copyright_info)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/generate_api_rst.py", line 189, in parse_doxygen_xml_and_generate_rst_and_swig
    namespaces = parse_doxygen.read_doxygen_xml_files(xml_dir, ['dolfin', 'ufc'])
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 756, in read_doxygen_xml_files
    item = NamespaceMember.from_compounddef(c, name, kind, xml_file_name)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 250, in from_compounddef
    mitem = NamespaceMember.from_memberdef(m, mname, mkind, xml_file_name, item)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 173, in from_memberdef
    item._add_doc(mdef)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 265, in _add_doc
    description_to_rst(dd, self.docstring)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 645, in description_to_rst
    description_to_rst(child, lines, indent, skipelems, memory)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 646, in description_to_rst
    add_text(child.tail)
  File "/home/sigvald/Downloads/dolfin/src/dolfin/doc/parse_doxygen.py", line 637, in add_text
    lines[-1] += tl[0]
TypeError: can't concat str to bytes

Is this a DOLFIN bug or is it something with my Swig or Python versions that's off?

[Sigvald]

On second thought, this should perhaps have gone into the Q&A before posting an issue here. Sorry about that.

[Tormod Landet]

This seems like a Python3 bug, which is a bit strange since it builds (well, used to build) on CI. Anyway, I did most likely write that code, and now it has been removed. You can probably quite easily (?) fix the script, or just have it fail to build docstrings. There is some logic quite early in this script or the script that calls it that will output empty docstrings if the doxygen magic fails (or an incompatible doxygen version is found). Make it take that code path.

The doxygen magic was quite hard to maintain and I assume this bug report will be closed with WONTFIX since the code is now removed (both the doxygen parser and the whole SWIG wrapper, good riddance). You may miss the docstrings, but so does everyone who uses master at the moment ...

Sorry for (potentially?) introducing a Python3 error. Hope you can hack around it

[Sigvald]

I don't know the reason, but lines[-1] in add_text() in parse_doxygen.py occasionally is of type bytes instead of string. Simply converting it to string when this happens seems to do the trick. The docstrings seems to work, too.

--- parse_doxygen.py    2018-02-20 10:09:01.595734780 +0100
+++ parse_doxygen.py.new    2018-02-20 10:09:35.768496224 +0100
@@ -634,6 +634,8 @@
     def add_text(text):
         if text is not None and text.strip():
             tl = text.split('\n')
+            if isinstance(lines[-1], bytes):
+                lines[-1] = lines[-1].decode('utf-8')
             lines[-1] += tl[0]
             lines.extend([indent + line for line in tl[1:]])
             if text.endswith('\n'):

If you want I can do a pull request with this tiny fix, but I'm not sure which branch it should go into and how the workflow is? Or you can just take the code if you want to.

[Jan Blechta]

As Tormod said, SWIG and related documentation generation has been deleted from development version. We don't plan to do patch releases - we don't have maintenance resources for that.

Anyway note that on other systems, build is working: e.g., stable Docker image and I was able to build DOLFIN and documentation for 2017.2.0 by python3, see this commit message.