The attached patch upgrades the MPI configuration thorn in a number of ways. First, and most importantly, it automatically configures mpi based on information obtained from mpicc when MPI_DIR isn't specified.
Configuration is only disabled by setting MPI_DIR = NONE.
In addition to MPI_INSTALL_DIR, it will look at CACTUS_EXT_INSTALL_DIR. Frank says that at one point a general external directory for applications was discussed.
My feeling is that having the install land in configs is usually not what people want. This version emits an error message if you set MPI_DIR to BUILD, but don't set one of the above install dirs. If you really want MPI under configs, you can set MPI_INSTALL_DIR=CONFIGS.
I anticipate pushback on my MPI_DIR=NONE, and MPI_INSTALL_DIR=CONFIGS suggestions, but I thought I'd suggest them regardless.