Issue #4 resolved

WebOb 1.0.3 Issues with paste.request.parse_formvars()

derks
created an issue

I recently wiped my virtualenv for a TurboGears 2.1 project, and after setting it up again and re-installing all deps I found my app was broken. Thanks to mcdonc_ in #turbogears he mentioned WebOb 1.0.3 was recently released so I downgraded to 1.0 and the issue was fixed.

A very simple, Psuedo example is:

{{{

!python

from tw.forms import TableForm, TextField, SubmitButton from formencode.schema import SimpleFormValidator

def my_validator(value_dict, state, validator): print value_dict['name']

ValidateSomething = SimpleFormValidator(my_validator)

class MyResourceForm(TableForm): validator = Schema(chained_validators[ValidateSomething(),]) fields = [ TextField('name'), SubmitButton('submit') ]

}}}

Previously (in WebOb 1.0) value_dict contained the keys/values from my form (as expected). However as of WebOb 1.0.3 the value_dict is empty '{}'.

I'm not certain what other info might be helpful, however I can imagine this would have a negative impact on a lot of TurboGears devs and the like.

If I can provide more information about my environment please let me know and I can get that to you ASAP.

Comments (11)

  1. derks reporter

    Hm... I was working on building a test TurboGears 2.1 app to provide a usable demo of the issue... and I can't seem to reproduce the issue currently. Will continue working at this and update shortly.

  2. derks reporter

    I've been unable to reproduce this in a stock TG2.1 app, so I have jumped back into reviewing my own app where this is an issue.

    I've narrowed the problem down to the fact that the following is being wiped:

    environ['webob._parsed_post_vars']
    

    I'm not sure how that relates to everything else... but that is the one thing I can find that is clearly altered... and the affect is, POST vars are not getting to my validators.

    It seems this happens when I call 'paste.request.parse_formvars()'. I have an 'API Key' identity plugin for repoze.who. When 'api_key' is passed via post vars, I pull that out in order to authenticate the request and create the identity. You can see this plugin here:

    https://github.com/derks/tg21test/blob/master/tg21test/lib/identity.py

    If I simply comment out the 'parse_formvars()' call, then webob._parsed_post_vars is unaffected. Again, this is only in WebOb 1.0.3+. I'm hoping that this little bit of info might sparked something to help lead me towards why calling parse_formvars() is erasing webob._parsed_post_vars as currently my app is completely busted with WebOb 1.0.3.

    The app at the following URL fully reproduces what I'm seeing:

    https://github.com/derks/tg21test

    STEPS TO REPRODUCE:

    $ virtualenv-2.7 --no-site-packages ~/env/tg21test
    
    $ source ~/env/tg21test/bin/activate
    
    $ git clone git://github.com/derks/tg21test.git
    
    $ cd tg21test/
    
    $ python setup.py develop
    
    $ paster setup-app development.ini 
    
    $ paster serve --reload development.ini 
    Starting subprocess with file monitor
    Starting server in PID 12338.
    serving on http://127.0.0.1:8080
    
    
    ### from another terminal
    
    $ source ~/env/tg21test/bin/activate
    
    $ python test.py
    Good: {u'errors': {u'test_field': u'Missing value'}, u'data': {}} 
    Bad: {u'errors': {u'test_field': u'Missing value'}, u'data': {}} 
    

    The above shows that, with WebOb 1.0.3, the 'test_field' is not making it to my validator.... essentially, the POST vars are getting wiped.

    With WebOb 1.0:

    $ rm ~/env/tg21test/lib/python2.7/site-packages/WebOb-1.0.3-py2.7.egg 
    
    $ easy_install "WebOb<=1.0"
    
    ### RESTART paster on the other terminal first
    
    $ python test.py
    Good: {u'errors': {}, u'data': {u'test_field': u'some value'}} 
    Bad: {u'errors': {u'test_field': u'Missing value'}, u'data': {}} 
    

    The above is how it is suppose to be... show that the POST vars are making it to my validators, and being validated. The second call gets an error because the 'test_field' is indeed missing.

    NOTE: The following files from the TG2.1 example are notable in that code was added that might be significant:

    tg21test/config/app_config.py
    tg21test/lib/base.py
    tg21test/lib/identity.py
    tg21test/controllers/test.py
    tg21test/forms/test_validator.py
    tg21test/model/auth.py
    

    I don't know if this is triggering a WebOb bug... or something else. I understand this is a lot of data to process... if it is something unexpected in WebOb I hope this helps resolve that. If it is something else and not considered a bug... that is fine, I'm simply hoping that you all might have some clue as to the differences from WebOb 1.0 -> 1.0.3 that might have triggered this behavior. Please let me know if there is anything else I can provide.

  3. derks reporter

    Hi Sergey, thank you for the quick reply. I've tested with 1.0, 1.0.1, 1.0.2, and 1.0.3... the problem I'm seeing starts with 1.0.2, and everything prior to that is fine.

    My application does not rely on environ['webob._parsed_post_vars'] directly, however in testing when printing out the environ... I can see that on < 1.0.2 the POST vars I pass are listed in the webob._parsed_post_vars MultiDict.... in 1.0.2 and above the POST vars I'm passing seem to dissapear and no long show up in webob._parsed_post_vars. My assumption is... whatever populates webob._parsed_post_vars is closer to where the issue is... and if the POST vars aren't making it there... it would be the same reason why they aren't making it to my form validators and controller.

    A friend of my had a thought that... perhaps this piece of WebOb was converted to be a generator object... where once it was iterated over it was done... so calling it again would be empty. Just a thought. I apologize for not being very well versed in WebOb.... My usage is from the higher level of TurboGears, not WebOb directly.

    If there is a better way for me to get the passed POST vars rather than calling paste.request.parse_formvars I'd be more than happy to try that if it would resolve the issue.

    Thank you for your time.

  4. Sergey Schetinin

    Well, you can actually read webob source related to that key -- it's not hard to understand and there's maybe a screenful of code. Basically the value under that key is a cache for parsed POST body. When req.POST is accessed it checks if the wsgi.input is the same as the one that was parsed before and if it is returns the cached value. Otherwise it tries to parse the new input stream (and caches it there).

    I suspect that the problem is related to how the input stream is handled. Generally WSGI app can only read it once, but for the entire middleware stack to work, the input body is usually made seekable so that each layer can seek to the beginning and read the body again. Webob 1.0.2 changed some parts related to this action to expose a very clear API on how to do that. One needs to call req.make_body_seekable() and the req.body_file will be a seekable stream at 0 pos. Before that change in 1.0.2 req.body_file could be anything that came from up the stack.

    What I suspect is happening is that some code was not paying much attention to these issues of input seeking and was working essentially by relying on undocumented idiosyncrasies of libraries around it. After the change some code reads the input and leaves it seeked to the end of stream. Some other code afterwards tries to read and parse it again without seeking it back to 0, which means it gets no data and that results in empty parsed POST.

    If you can show me the entire middleware stack I might find the offending party.

    http://svn.pythonpaste.org/Paste/trunk/paste/request.py

    parse_formvars seems to be involved, but all it does is read the input never seeking it or replacing it with a seekable copy, but that's not the problem. The code that uses it should make sure that the wsgi.input is seekable and marked as such for webob. So I suppose what happens is that it is made seekable, but never seeked back to 0 after reading and not marked as seekable, which makes webob attempt to read it from where it is. (Webob 1.0.1 and before would try to seek all streams anyway, but different streams behave differently therefore the need to mark the stream explicitly).

    Basically the simplest fix is probably just to replace a call to parse_formvars w/ webob.Request(env).str_POST.

  5. derks reporter

    Sergey,

    I really appreciate you taking the time on this. I know my use case is way out of left field. I tried using webob.Request(env).str_POST as you mentioned and it did indeed fix my issues. My code works on both 1.0, and 1.0.3.

    I would have never been able to figure this out without your help. Thank you!

  6. Log in to comment