1. Python CFFI
  2. Untitled project
  3. cffi
  4. Issues
Issue #129 resolved

Have bytes(buffer) return the content

Simon Sapin
created an issue

Hi,

On Python 3.x, using bytes() on a ffi.buffer(…) object returns the content. On 2.x, (where bytes is str), it returns something like <_cffi_backend.buffer object at 0x272b780>. Is it possible to change it to consistently get the 3.x behavior?

Context: cairocffi accepts file-like objects, and uses file_obj.write(buffer). This works great when the object is a real file or io.BytesIO and uses the C-level buffer interface, but not so much for file-like objects implemented in Python (such as Django’s HttpResponse) that end up calling bytes() on their argument.

Original bug: https://github.com/Kozea/WeasyPrint/issues/141

I could use file_obj.write(buffer[:]) in cairocffi, but the additional copy kind of defeats the purpose of having buffers in the first place.

Comments (8)

  1. Armin Rigo

    I believe that just saying file_obj.write(buffer) should do the right thing and not do any copy in all cases. Is it wrong? If this ends up writing <_cffi_backend...> into the file, then it's a rather bad bug. (We can discuss if bytes(buffer) should be equivalent to buffer[:] or not, and I'm rather thinking not, but this seems not to be the main issue here. Note that in all possible implementations, bytes(buffer) would make a copy. As far as I can tell, bytes(buffer) is always <_cffi_backend...> right now, in Python 2 or 3, and there is a test for it.)

  2. Armin Rigo

    Ok, sorry, I noticed that indeed bytes(buffer) is different from str(buffer) on Python 3. There is no way I can think of to prevent bytes(buffer) from working in Python 3, so we'll make it work in Python 2 too. (Again, it works by taking a copy; there's no way we can change that.)

  3. Simon Sapin reporter

    The issue is with file-like objects implemented in python. The implementation of the write method is typically like this:

    def write(self, data):
        self.content.append(bytes(data))
    
  4. Simon Sapin reporter

    Yes, these unusual file-like objects will do copies and that’s fine. I’d just rather avoid having file_obj.write(buffer[:]) in cairocffi and do an additionnal copy every time.

  5. Armin Rigo

    Note that we can argue that the problem is your custom file-like object: on Python 3 it works in all cases, but on Python 2 it will not work for other types of buffer that behave like cffi's buffer. We could also raise TypeError on bytes(buffer) in Python 3 by using a custom __bytes__ method, which I'm very tempted to do instead of 65a3bcbcb054... the problem I have is that str(buffer) now gives a very different result in Python 2 or Python 3 :-(

  6. Simon Sapin reporter

    65a3bcbcb054 looks good, thanks.

    Using str() on Python 2 and 3 for a byte-based thing is probably a bad idea anyway. As long as bytes() and repr() are each consistent across python versions, I think this is fine.

  7. Armin Rigo

    Thanks pjenvey for pointing out that str() gives inconsistent results between Python 2 and 3 on regular byte strings. Wrote this in the documentation. Now this issue is really closed :-)

  8. Log in to comment