Nested groups are not supported now because django-admin-tools generates invalid html ids for them. Nested groups can work sometimes but generally they are broken.
HTML id generation for dashboard modules is improved in this clone: https://bitbucket.org/kmike/django-admin-tools-experiments
I believe the change is backwards-compatible (html elements will even have the same ids as before, except for duplicates)
The commit from this clone occasionally fixes the issue with nested groups.
I don't think that nested groups is a good UI idea, but sometimes they can be useful.