The temp file cleanup code is not safe. The file removal in the atexit could be removing a different file that the one created (using the same name). That's unlikely but should be fixed.
Couldn't the auto cleanup provided by tempfile be used, rather than setting the 'delete' keyword to False? Have WriteOnlyWorksheet keep a reference to the NamedTemporaryFile object and have filename() return the .name attribute of it.
If the temp file needs to outlive the WriteOnlyWorksheet object, I would suggest the following. Set 'delete=True', Append the NamedTemporaryFile object to ALL_TEMP_FILES. In the shutdown function, call close() on all files in the list. In the _cleanup() method, call close() and also remove the object from the ALL_TEMP_FILES.
Growing the list without bound is not a good idea since openpyxl could be used in a long running program that generates many files.