Issue #6 new

load_all/dump_all do not work well with streams.

created an issue

I have a Python system that sends Yaml records around a networked system.

pyyaml has been working very well for this, but I had to work around a feature defect in load_all to do it.

Consider the following code snippet:

from __future__ import absolute_import, division, print_function, unicode_literals

import time
import yaml

# Setting either or both of these to true makes no difference to the results.
IS_SAFE = False

load_all = yaml.safe_load_all if IS_SAFE else yaml.load_all
dump_all = yaml.safe_dump_all if IS_SAFE else yaml.dump_all

def send():
  for i in range(3):
    print('send', i)
    yield i
    if USE_SLEEP:

for i in load_all(dump_all(send())):
  print('receive', i)

# Result:
#   send 0
#   send 1
#   send 2
#   receive 0
#   receive 1
#   receive 2

For a network transaction, we obviously want the sends and receives to be interleaved.

In my code, I simply handle it by using safe_dump/safe_load, and synthesizing my own Yaml record separators. Pretty easy!

I did this a few months ago... I seem to remember that at the time I did get pyyaml to be a bit more sensible and not try to suck up its generators, but then there was a secondary issue where it waits to send the separator after a record until the next record arrives, which means of course that your last record in a series doesn't appear until you send more data, but I can't find my notes on this. :-(

Comments (1)

  1. Log in to comment