Issue #188 resolved

s3Boto _clean_name is broken and leads to improper treatment of files ending with a '/'

Dan Gerszewski avatarDan Gerszewski created an issue

The problem is that the s3Boto _clean_name and _normalize_name methods chop the trailing slash off of a valid S3 filename. This is caused by using os.path.normpath() in _clean_name which has a "bug" seen here http://bugs.python.org/issue1707768, where the trailing slash gets stripped. It was just never fixed. The safe_join method in _normalize_name also strips a valid trailing slash. In this case the trailing slash should be restored to be able to deal with S3 files created through the aws console and other tools.

This is the root cause of the "missing trailing slash" bugs with a few of them listed here:

An S3 file can have a trailing slash and in fact the S3 console will create a file with a trailing slash when you create a "folder" seen here: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html.

This bug breaks at least these methods:

  • _open
  • _save
  • delete
  • exists
  • size
  • modified_time
  • get_available_name

url and listdir were fixed independently by appending the trailing slash after calls to _clean_name(), _normalize_name() which will need to be reversed so that there aren't two slashes.

Example to reproduce exists() bug:

  1. Create folder in aws console
  2. storage.bucket.list() #see valid s3 file ending with a '/'
  3. storage.exists() #will return false for the valid file

Comments (6)

  1. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.