I possibly very useful (and simple to implement) debugging help in Cactus would be if the Cactus driver provided some means to detect array accesses out of the array/grid-function bounds. In general that is hard to do (in C, for Fortran there are compiler switches) however a possible useful partial solution might already be to put canary values before and after the user-visible data of grid functions/grid arrays. The flesh/driver could then check after each scheduled routine if any of the canary values were modified and if so output a warning.
Schematically the layout in memory would be
Canary1 data Canary2
and CCTK_VarDataPtr would return a pointer to "data" only. After a scheduled function all we check Canary1 and Canary2 and output an error if they are corrupted. Similarly the IncreaseGroupStorage/DecreaseGroupStorage routines could set/check the canary values.
This would prevent these errors triggering failures at some later unrelated call to malloc or free. glibc's malloc function provides some of this if _MALLOC_DEBUG is set, though I am not sure how well that actually works in practice in particular since eg OpenMP provides its own malloc function.
I don't have an implementation of this right now and am mostly fishing for comments.