Using the memoir class breaks some of the assumptions made by the sphinxmanual wrapper class. Memoir provides internal implementations of the functionality of classes like fncyhdr, so loading these by default causes conflict. Also, memoir is smart enough to automatically add a TOC entry for the index, so Sphinx ends up duplicating it.
Finally, memoir throws errors when it encounters \rm, \sf, etc. In order to turn these into warnings, you must set the oldfontcommands option in the class. In order to allow this, I added the 'extraclassoptions' attribute to latex_elements.
These should be the minimal changes necessary to allow the use of the memoir class.