Ghost mode not set correctly when using MeshEditor

[Jack Hale]

Output of the below script with:

mpirun -n 2 python3 mwe.py

gives:

none
shared_facet
none
shared_facet

i.e. the ghost mode is not set properly on the mesh created with MeshEditor.

from dolfin import *
import numpy as np

parameters["ghost_mode"] = "shared_facet"
mesh = Mesh(mpi_comm_world())
editor = MeshEditor()
editor.open(mesh, 'quadrilateral', 2, 2)
editor.init_vertices(4)
editor.init_cells(1)
editor.add_vertex(0, np.array([0.0, 0.0]))
editor.add_vertex(1, np.array([1.0, 0.0]))
editor.add_vertex(2, np.array([1.0, 1.0]))
editor.add_vertex(3, np.array([0.0, 1.0]))
editor.add_cell(0, np.array([3, 2, 0, 1], dtype=np.uintp))
editor.close()
print(mesh.ghost_mode())

mesh2 = UnitQuadMesh.create(2, 2)
print(mesh2.ghost_mode())

[Jack Hale]

This could be irrelevant to MeshEditor... This also fails:

from dolfin import *
import numpy as np

parameters["ghost_mode"] = "shared_facet"

mesh = Mesh(mpi_comm_world())
print(mesh.ghost_mode())

mesh2 = UnitQuadMesh.create(2, 2)
print(mesh2.ghost_mode())

[Jan Blechta]

Current state is that ghost mode is set on a single place in DOLFIN:

$ grep -r "mesh._ghost_mode = " dolfin/
dolfin/mesh/MeshPartitioning.cpp:  mesh._ghost_mode = ghost_mode;

(I believe I am not lying. Check that.) That makes it very robust and prone to bugs. Empty mesh has effective ghost mode none because it never went through partitioning and has no ghosts. After all check the docstring in Mesh class:

    /// Ghost mode used for partitioning. Possible values are
    /// same as `parameters["ghost_mode"]`.
    /// WARNING: the interface may change in future without
    /// deprecation; the method is now intended for internal
    /// library use.
    std::string ghost_mode() const;

(Notice of "ghost mode used for partitioning".)

The MeshEditor case is much more subtle. But nevertheless the mesh above probably does not have any ghosts. Just run

r = MPI.rank(mesh.mpi_comm())
for c in cells(mesh, "all"):
    print(r, c.global_index(), c.is_ghost())

on it. (If you can prove the converse I admit it is buggy.) I am not completely sure, - it's @chris_richardson 's area - but I believe there are two possible uses of editor. Either distributed, you'd need to use calls with _global suffix in MeshEditor or local, in which case you need to call distribution tools by yourself, how it is done in mesh factories in dolfin/generation dir.

I hope this makes sense.

[Jan Blechta]

And the question is whether the distributed use of MeshEditor thought about ghosting.

[Chris Richardson]

from dolfin import *
import numpy as np

parameters["ghost_mode"] = "shared_facet"


mesh = Mesh(mpi_comm_world())

if MPI.rank(mpi_comm_world()) == 0:
    editor = MeshEditor()
    editor.open(mesh, 'quadrilateral', 2, 2)

    n = 10

    editor.init_vertices(2*n+2)
    editor.add_vertex(0, np.array([0.0, 0.0]))
    editor.add_vertex(1, np.array([1.0, 0.0]))
    for i in range(1, n+1):
        editor.add_vertex(i*2, np.array([1.0, i]))
        editor.add_vertex(i*2+1, np.array([0.0, i]))

    editor.init_cells(n)
    for i in range(n):
        editor.add_cell(i, np.array([i*2+3, i*2+2, i*2, i*2+1], dtype=np.uintp))
    editor.close()

MeshPartitioning.build_distributed_mesh(mesh)

print(mesh.cells(), mesh.coordinates())

print(mesh.num_cells(), mesh.size_global(2))

print(mesh.ghost_mode())

Try this. It could be made nicer to use.

[Jack Hale]

@chris_richardson solution works. Will continue!