Inconsistant local-to-global mapping from a distributed DMPlex object

[Ben Mather]

Hello,

I am attempting to create a parallel matrix from local vertices in an unstructured mesh (the dmplex.py example in the demo folder was very helpful - thanks!) But the LGMap provided from the DM object creates some obscure entries.

My workflow is:

1. Create a DMPlex object from cell list, then distribute using a section defined by 1 field on the vertices.
2. Get local vertices using getCoordinatesLocal()
3. Find node neighbours from the DAG or by re-triangulating with Qhull.
4. Construct a global matrix and setLGMap from the DM.
5. Insert neighbour indices into a global matrix using setValueLocal()

Running the attached script with n processors should return an identical matrix, but alas it does not. I'm not sure if this is the right place to post this, but any help would be greatly appreciated!

Regards, Ben

[Lisandro Dalcin]

@knepley Could you please help us with this one? As you know, I'm still learning to use your little beast...

[Matthew Knepley]

There has been some disagreement about what L2G is actually supposed to do. I would say the right thing to do is either

a) Use the global PetscSection to get global indices directly

or

b) Call MatSetValuesClosure() for any local mesh point

[Ben Mather]

Thanks for your help! I am getting the same indices as the L2G map from the global PetscSection. Since negative indices are off-processor nodes (or ghost nodes) they are ignored in MatSetValue(). Each row in the matrix may contain ghost nodes, but these are not inserted because they have a negative index... therefore I need their global position.

What this means for me is that the rows and columns need to be mapped differently. Is there a method available to retrieve a ghost node's global ordering?

[Matthew Knepley]

1) The global index is -(i+1) for the negative index i

2) There are the DMPlexPoint*() functions

3) We have internal functions DMPlexGetIndicesPoint(Fields)_Internal()

[Ben Mather]

That did it! Here is the code for reference,

lgmap_row = dm.getLGMap()

# create second L2G map
l2g = lgmap_row.indices.copy()
shadow_mask = l2g < 0
l2g[shadow_mask] = -(l2g[shadow_mask] + 1)
lgmap_col = PETSc.LGMap().create(l2g)

# Assign row-col L2G maps to the matrix and insert values using local ordering
mat.setLGMap(lgmap_row, lgmap_col)
mat.setValuesLocal(...)

Many thanks for your assistance, I suspected it was something simple.