s3Boto _clean_name is broken and leads to improper treatment of files ending with a '/'

Issue #188 resolved
Dan G created an issue

The problem is that the s3Boto _clean_name and _normalize_name methods chop the trailing slash off of a valid S3 filename. This is caused by using os.path.normpath() in _clean_name which has a "bug" seen here http://bugs.python.org/issue1707768, where the trailing slash gets stripped. It was just never fixed. The safe_join method in _normalize_name also strips a valid trailing slash. In this case the trailing slash should be restored to be able to deal with S3 files created through the aws console and other tools.

This is the root cause of the "missing trailing slash" bugs with a few of them listed here:

An S3 file can have a trailing slash and in fact the S3 console will create a file with a trailing slash when you create a "folder" seen here: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html.

This bug breaks at least these methods:

  • _open
  • _save
  • delete
  • exists
  • size
  • modified_time
  • get_available_name

url and listdir were fixed independently by appending the trailing slash after calls to _clean_name(), _normalize_name() which will need to be reversed so that there aren't two slashes.

Example to reproduce exists() bug:

  1. Create folder in aws console
  2. storage.bucket.list() #see valid s3 file ending with a '/'
  3. storage.exists() #will return false for the valid file

Comments (6)

  1. Log in to comment