When there is a sub-directory in a composite dataset, shutil.copytree() must be used to copy the whole tree structure of the sub-directory at the new location.
Also there is a related issue with self._get_filename(), this method when contructing the real file path it also creates the new file. So when file_name is actually a directory then before copying with shutil.copytree(), the file created from self._get_filename() must be removed.
Are you sure that _get_filename() might create the file? I don't see anything in the code that would cause it. If it's being created somewhere else we probably ought to find that and fix it rather than just remove the file later.
After double checking it, I can say that you are right self._get_filename() does not create the file/dir.
By the way there is self._get_filename() and self.get_filename() which look similar to me, I just noticed it few minutes ago.
And if you see the code written by me self._get_filename() gets used, while the rest code uses self.get_filename().
The only method that seems to create a file is self.create(), maybe it gets called somehow before or within self_update_from_file().
I'm pretty sure we will found out what is actually happening along the way, after some more tests... :)
The creation was occurring via the call to create() in the same method. What I've done is actually skipped directory detection in the object store and stopped the output collection method from passing directories to the update_from_file method. This has been committed in ddb55ff5fd1c.