- Submission size limitations: Problem with current 10mb limit. Proposed no limit. Agreed on a X-header flag that indicates server's POST submission size limit.
- Discussion around usage of 20x response headers. Remains as is for now.
- isIncomplete flag appears to have been removed from spec
[11/22/2011 9:34:58 AM] Yaw Anokwa: alright. submission api time... [11/22/2011 9:35:10 AM] Jonathan Jackson: I disagree [11/22/2011 9:35:44 AM] Yaw Anokwa: it's 6.30 here and i'm ready. whats your excuse? [11/22/2011 9:35:56 AM] Jonathan Jackson: (kidding) [11/22/2011 9:35:58 AM] Anton de Winter: haha [11/22/2011 9:36:15 AM] Anton de Winter: Okay yaw, knock it out of the park [11/22/2011 9:36:34 AM] Jonathan Jackson: side note: anyone have a better wiki idea than bitbucket for these? happy to switch [11/22/2011 9:36:57 AM | Edited 9:37:27 AM] Yaw Anokwa: minor complaint: we mix and match how we name things. xml_submission_file vs isComplete. [11/22/2011 9:38:04 AM] Yaw Anokwa: the 10mb limit is a gae limit. gae limits change all the time. is there a lesser limit that other servers have? [11/22/2011 9:39:04 AM] Yaw Anokwa: not using a 200 is kinda weird. i remember we had an issue with proxies, but if the full location header has the url you are submitting to, you should need a 201. [11/22/2011 9:39:45 AM] Yaw Anokwa: we sure about the xmlns as http://openrosa.org/http/response? [11/22/2011 9:40:01 AM] Yaw Anokwa: we sure about OpenRosaResponse as the tag name? does it fit with what we do with form listing? [11/22/2011 9:40:45 AM] Anton de Winter: It may be better to decide a tag name here [11/22/2011 9:40:51 AM] Anton de Winter: then when we get to FormList we can refer to this [11/22/2011 9:40:56 AM] Anton de Winter: as opposed to the other way around [11/22/2011 9:41:14 AM] Anton de Winter: re: xmlns, that seems like a reasonable url, what did you have in mind? [11/22/2011 9:42:01 AM] Yaw Anokwa: er. i guess i don't have an alternative... [11/22/2011 9:42:33 AM] Anton de Winter: The reasoning behind not using 200 seems ok... I don't know for sure that it's true though [11/22/2011 9:42:49 AM] Anton de Winter: and responding with 20x seems more precise somehow [11/22/2011 9:42:52 AM] Anton de Winter: which is nice. [11/22/2011 9:43:07 AM] Yaw Anokwa: nice and totally not standard. [11/22/2011 9:43:34 AM | Edited 9:43:40 AM] Anton de Winter: not standard as in people don't use 20x at all, or not standard as in it's not w3c spec? [11/22/2011 9:44:04 AM] Anton de Winter: 10mb limit does seem high. Question: what has been your guys experience with submissions from multimedia forms [11/22/2011 9:44:13 AM] Anton de Winter: do you often see submissions approaching that size? [11/22/2011 9:44:27 AM] Anton de Winter: seems like ODK has more experience with submissions including lots of images/other MM content [11/22/2011 9:44:49 AM] Anton de Winter: what kinds of sizes are we talking? [11/22/2011 9:45:01 AM] Yaw Anokwa: not standard in that most people return 200 when everything is ok. [11/22/2011 9:46:14 AM] Yaw Anokwa: no idea about the 10mb limit. that's a mitch question. my guess is that most people would submit a handful of pictures. never audio or video. [11/22/2011 9:46:55 AM] Yaw Anokwa: also, trying to do a 10mb post sounds dangerous. [11/22/2011 9:47:41 AM] Anton de Winter: yeah, so it would make sense to lower it [11/22/2011 9:48:01 AM] Anton de Winter: but, I feel like you cross the **danger** threshold anywhere above, like, 1mb [11/22/2011 9:48:08 AM] Yaw Anokwa: agreed. [11/22/2011 9:48:11 AM] Anton de Winter: after that it's a bit of wild guess [11/22/2011 9:48:14 AM] Anton de Winter: and 1mb is pretty low :/ [11/22/2011 9:48:25 AM] Anton de Winter: esp if you're submitting high quality images [11/22/2011 9:48:33 AM] Anton de Winter: (my phone generates 1.5mb images from the camera) [11/22/2011 9:48:47 AM] Anton de Winter: (although granted, that's at highest quality) [11/22/2011 9:49:18 AM] Yaw Anokwa: i think people want those images downsized.. [11/22/2011 9:49:29 AM] Yaw Anokwa: my gut says 1mb or 5mb seems more reasonable. [11/22/2011 9:49:29 AM] Anton de Winter: Kind of in favour of giving 200 lower priority than 201 or 202 if we're going to get fake ones from proxies/gateways [11/22/2011 9:49:49 AM] Anton de Winter: Yeah, you're probably right about 1 or 5mb. Should be interesting to see what other people think... [11/22/2011 9:50:24 AM] Yaw Anokwa: the 200 issue, again was only because of proxies. but if you have the submission url in your location header, you will know where the 200 is coming from. [11/22/2011 9:51:06 AM] Anton de Winter: If that's guaranteed to circumvent the problem, I'd be in favor of using 200s [11/22/2011 11:04:52 AM] Drew Roos: i would think 201 is appropriate only if we're doing a PUT [11/22/2011 11:05:08 AM] Clayton Sims: Actually, we looked this up and 201 is an appropriate response for a post [11/22/2011 11:05:32 AM] Drew Roos: and regarding size limits, this seems like another 'SHOULD'-type issue [11/22/2011 11:05:43 AM] Mitch Sundt: Actually, I believe our use of Location is a bit bastardized... [11/22/2011 11:05:46 AM] Clayton Sims: Agreed re:size limits [11/22/2011 11:05:51 AM] Clayton Sims: Mitch: Location is definitely not correct [11/22/2011 11:05:52 AM] Drew Roos: and perhaps servers return a 413 (entity too large) if they get a bigger response than they want to handle [11/22/2011 11:05:52 AM] Clayton Sims: But 201 is fine [11/22/2011 11:06:02 AM] Clayton Sims: I think we should be using the xml response to signal creation [11/22/2011 11:06:45 AM] Clayton Sims: Location would also work if we had a good way to do it [11/22/2011 11:06:51 AM] Mitch Sundt: The size limit is definitely a SHOULD, as in the client should make all efforts to limit the size of their post to less than X MB. If you have a large video capture, then it would have to go up as a huge blob, and we don't want to get into defining how to split that, etc. [11/22/2011 11:06:52 AM] Clayton Sims: IE: Actual REST-ful locaiton of the file [11/22/2011 11:08:28 AM] Mitch Sundt: The size limit, while it aids in reliable submissions, since you are using TCP, you do get reliable transmissions of the put (but just a lot of data packets and time to accomplish that on wireless). The main goal of the submission limit was to limit the size of the JVM required to run the server. [11/22/2011 11:08:47 AM] Mitch Sundt: And to apease Google [11/22/2011 11:09:12 AM] Clayton Sims: you can totally stream submissions straight to disk [11/22/2011 11:09:32 AM] Clayton Sims: fitting whole files in memory should never be an expected requirement [11/22/2011 11:09:57 AM] Mitch Sundt: Yes and no. It is far easier to load everything into memory if you are writing out database blobs. [11/22/2011 11:10:20 AM] Mitch Sundt: And on Google, you can't stream to disk. BITA [11/22/2011 11:10:55 AM] Mitch Sundt: The Location usage per w3c is: The Location response-header field is used to redirect the recipient to a location other than the Request-URI for completion of the request or identification of a new resource. For 201 (Created) responses, the Location is that of the new resource which was created by the request. For 3xx responses, the location SHOULD indicate the server's preferred URI for automatic redirection to the resource. The field value consists of a single absolute URI. [11/22/2011 11:11:22 AM] Mitch Sundt: Thats not quite our usage. [11/22/2011 11:11:36 AM] Clayton Sims: Right, Locaiton is fine if you send back the REST-ful uri where the file ended up [11/22/2011 11:12:05 AM] Mitch Sundt: Yeah. We currently don't. [11/22/2011 11:12:10 AM] Clayton Sims: Seems easy enough to do? [11/22/2011 11:12:20 AM] Clayton Sims: I mean, I'm comfortable with the server responding "I can't fit that whole file on disk" with the appropriate response code [11/22/2011 11:12:36 AM] Clayton Sims: I just don't want to prematurely decide that we should start splitting up files or something [11/22/2011 11:13:30 AM] Mitch Sundt: Yes, this is why the 10MB limit was in the std. For the larger wildlife mock-up we did, with 92 image files in the form definition, they all fit within 10MB. [11/22/2011 11:14:08 AM] Mitch Sundt: For Carl's larger questions displayed-as-images work-around for missing fonts, that form had 352 images, and took 3 POSTs to upload to keep everything under 10MB [11/22/2011 11:14:33 AM] Mitch Sundt: Again, I think we do need a limit above which the client should split their submissions. We can't have this unlimited [11/22/2011 11:14:57 AM] Clayton Sims: I'm fine with that regarding submissions consiting of multiple files [11/22/2011 11:14:59 AM] Mitch Sundt: Or we need to require clients have a configuration settings, which I would not want to do. [11/22/2011 11:15:19 AM] Clayton Sims: Just so long as we're not jumping into "take this video file and manually split it up into chunks" [11/22/2011 11:15:23 AM] Mitch Sundt: Agreed -- only for multiple files. If you have larger videos, try to submit, and it might work or fail. Take your chances. [11/22/2011 11:15:35 AM] Clayton Sims: Yeah, that sounds good to me [11/22/2011 11:16:46 AM] Mitch Sundt: On location and the response codes, I prefer to stick with the 201, 202 codes. I can get a Location response formatted. [11/22/2011 11:16:52 AM] Mitch Sundt: TO be restful [11/22/2011 11:16:59 AM] Mitch Sundt: The one use of location we have... [11/22/2011 11:17:46 AM] Clayton Sims: I feel like requiring anything from location is pretty unfortunate, though [11/22/2011 11:18:15 AM] Mitch Sundt: is we initiate communications with the server with a HEAD request, and expect the response to use a Location response to confirm reaching the server. We should really not be doing that on the client. I think the client could look for the OpenRosa header instaed and use that to confirm that it is talking to a server. [11/22/2011 11:18:29 AM] Clayton Sims: IE: If you submit to something.myserver.com/submission and we're currently transitioning to a new website at betasomething.myserver.com/submission it seems like submit shouldn't fail [11/22/2011 11:18:58 AM] Clayton Sims: Yeah, I'm ok with the OpenRosa header being used for that, although I still think that a well-formed XML response is more solid [11/22/2011 11:19:02 AM] Mitch Sundt: Yes, we currently do a strict matching that doesn't support host redirects like that. I'd like to do that better. [11/22/2011 11:19:32 AM] Mitch Sundt: I would prefer the client not require Location anywhere. [11/22/2011 11:19:36 AM] Clayton Sims: agreed [11/22/2011 11:19:52 AM] Clayton Sims: The client should only use location if it wants to, for instance, go re-fetch the file later [11/22/2011 11:20:10 AM | Edited 11:20:11 AM] Clayton Sims: But I don't really even want to guarantee that just yet [11/22/2011 11:20:23 AM] Mitch Sundt: yes [11/22/2011 11:29:59 AM] Mitch Sundt: One thing I added to Aggregate was in the response XML, I returned the instanceID of the submission and the submissionDate as recorded by the server to the client. This is useful for clients other than Collect -- e.g., Briefcase, that manage data sets and need to track correspondences. It helps in situations where the submission lacks a metadata block, so it is not really needed in OpenRosa submissions. submissionDate is a factoid. Not sure any of this is useful. [11/22/2011 11:31:17 AM] Clayton Sims: We have a similar OpenRosa response that we send back that includes a response message, etc. Is there a reason you're using submissionDate rather than the HTTP response's internal date? [11/22/2011 11:32:40 AM] Mitch Sundt: In a multiple-POST situation (lots of media files), it would be the date of the processing of the first of the POSTs. Haven't looked at using std header fields to convey that. [11/22/2011 11:32:53 AM] Clayton Sims: Ah, interesting [11/22/2011 11:33:25 AM] Clayton Sims: I definitely like using the XML response to signal things like "Parts received"/"Parts expected" [11/22/2011 11:33:38 AM] Clayton Sims: with lists of the ids of the posts that are still missing [11/22/2011 11:33:57 AM] Mitch Sundt: I don't do anything that complex. [11/22/2011 11:34:09 AM] Mitch Sundt: Let me check... [11/22/2011 11:37:24 AM] Mitch Sundt: yeah, I insert a <submissionMetadata xmlns="...opendatakit..." ...> tag into the response with the form id, version, and all the metadata the server has added to the data record, so it includes the submission date and the isComplete flag that the server uses, etc. [11/22/2011 11:37:48 AM] Mitch Sundt: These are all attributes of that tag, so they are unrestricted. [11/22/2011 11:38:07 AM] Mitch Sundt: i.e., extensible in an XMLish sense. [11/22/2011 11:39:20 AM] Mitch Sundt: That can be useful for a client that is just moving data around -- to get info on what it just moved from the server interpretting it. [11/22/2011 11:40:22 AM] Mitch Sundt: not sure I'd want to commit to anything other than (formId, version, instanceID) into any standard. [11/22/2011 11:40:41 AM] Mitch Sundt: I think that's all I use. [11/22/2011 11:40:42 AM] Clayton Sims: I'd like a response message to be in the standard [11/22/2011 11:41:13 AM] Clayton Sims: (as a SHOULD, maybe) [11/22/2011 11:41:16 AM] Mitch Sundt: Yes, I like the <OpenRosaResponse> I add the above tag within that in my own namespace. I think that should be allowed in this spec. [11/22/2011 11:41:46 AM] Mitch Sundt: I would like to make it a requirement if the submission had an X-OpenRosa-Version header. [11/22/2011 11:42:44 AM] Clayton Sims: Definitely agree that adding stuff into the response should be allowed. The response should be super extensible [11/22/2011 11:44:45 AM] Mitch Sundt: The extensibility is written up on the https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaRequest Perhaps the Form Submission api page should just reference that in this section. [11/22/2011 11:45:15 AM] Mitch Sundt: Oh, I guess it does. Didn't see the link [11/22/2011 11:46:53 AM] Mitch Sundt: --- [11/22/2011 11:47:26 AM] Mitch Sundt: w.r.t. the *isIncomplete* element, in practice, this isn't necessary as forms are defined right now. [11/22/2011 11:47:32 AM] Mitch Sundt: So we could remove it. [11/22/2011 11:48:13 AM] Mitch Sundt: The only use case is if you allow sets of items that are referenced by one element. [11/22/2011 11:48:51 AM] Mitch Sundt: I have that on form definition uploads where I send up all the images associated with the form (and I do no scanning of the itext fields in the form to confirm that all image files are present). [11/22/2011 11:49:43 AM] Mitch Sundt: I don't think xforms is trending in that direction, so it can be removed. In practice, as I process the submission, I traverse it and record whether or not I have a missing attachment. [11/22/2011 11:50:02 AM] Mitch Sundt: So I know by what I have on hand whether or not it is complete. [11/22/2011 11:50:21 AM] Mitch Sundt: Anyway, I think that should be removed at this time. [11/22/2011 11:50:38 AM] Clayton Sims: Ok [11/22/2011 11:52:48 AM] Mitch Sundt: Should Anton do the editing, or should I, and if so, shoudl I do it now? [11/22/2011 11:53:33 AM] Clayton Sims: I think it would be fine for you to go ahead [11/22/2011 11:53:37 AM] Anton de Winter: sure [11/22/2011 11:53:38 AM] Clayton Sims: The spec's still publically editable [11/22/2011 11:56:49 AM] Yaw Anokwa: from the log, i didn't get a sense for what the decision about the 10mb was. [11/22/2011 11:59:37 AM] Mitch Sundt: I believe that is unchanged. It allows for ~90 images per submission if in jpeg. [11/22/2011 11:59:48 AM] Mitch Sundt: At least that's my experience. [11/22/2011 12:00:33 PM] Yaw Anokwa: and you don't think that's too high [11/22/2011 12:01:54 PM | Edited 12:01:55 PM] Clayton Sims: Yaw: WE didn't set a concrete limit, I don't think [11/22/2011 12:02:13 PM] Clayton Sims: The server will respond if a post is too big, and the devices will try to send out in smaller parts if possible [11/22/2011 12:02:46 PM] Mitch Sundt: I don't have enough field experience to say what to set it down to to improve the transmission side of things. [11/22/2011 12:04:00 PM] Anton de Winter: May be reasonable to phrase it something like, adhere to server responses saying the POST is too big, MUST chop the post up if over 10mb [11/22/2011 12:04:17 PM] Mitch Sundt: On the server side, it seems reasonable. Our server runs in an ~510MB JVM. [11/22/2011 12:05:13 PM] Clayton Sims: Anton: I don't see any reason to overspecify in advance [11/22/2011 12:05:19 PM] Mitch Sundt: Downside to that is that it puts a lot of code in the client, I think. [11/22/2011 12:05:24 PM] Clayton Sims: The mobile devices are extraordinarily unlikely to try to send over 900MB posts [11/22/2011 12:05:37 PM] Mitch Sundt: for the retry. [11/22/2011 12:06:24 PM] Matt Adams: I think 10MB is pretty big for transfers over mobile networks (would be fine over a good Wifi connection). [11/22/2011 12:06:37 PM] Anton de Winter: Hey guys, I'm going to be offline starting in about 45 minutes for the next 5 to 6 hours (possibly until tomorrow) [11/22/2011 12:06:47 PM] Matt Adams: Sorry - haven't really been following this conversation, just my 2 cents. [11/22/2011 12:06:51 PM] Anton de Winter: I doubt there's anything that needs to happen from my end, but please let me know if there is [11/22/2011 12:07:03 PM] Yaw Anokwa: i think there should be a limit for sure, but its not clear that setting it at 10mb makes sense for the mobile side. [11/22/2011 12:07:04 PM] Anton de Winter: (will send out the final vote tally on MetaData for sure though) [11/22/2011 12:07:51 PM] Yaw Anokwa: like could an exponential sizing work? post all, if it fails, split it in half and try again? [11/22/2011 12:07:52 PM] Anton de Winter: re: size limits, we should also bear in mind that we won't be hitting 10+mb submissions very frequently at all... [11/22/2011 12:08:11 PM] Matt Adams: @Yaw, I like that idea. A lot. [11/22/2011 12:08:35 PM] Anton de Winter: same [11/22/2011 12:09:43 PM] Yaw Anokwa: woot. i had a clever idea! [11/22/2011 12:09:55 PM | Edited 12:10:02 PM] Clayton Sims: The server can send back the maximum size [11/22/2011 12:10:06 PM] Matt Adams: @Anton, yeah, 10MB is pretty large - but I like the idea that the protocol is capable of gracefully failing & retrying with parameters that *might* be more likely to succeed. [11/22/2011 12:10:07 PM] Clayton Sims: as part of the "too big" response [11/22/2011 12:10:14 PM] Matt Adams: @Clayton - good idea also. [11/22/2011 12:11:33 PM] Mitch Sundt: I updated the spec with the earlier changes. We should perhaps first ratify the https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaRequest standard, as this now references that. [11/22/2011 12:11:53 PM] Clayton Sims: It seems like we should make sure we've covered all the request/response use cases before approving it [11/22/2011 12:12:01 PM] Mitch Sundt: the only issue with the graceful fall-back is the extra client code for that. [11/22/2011 12:12:01 PM] Clayton Sims: I think there might be more places where it's used [11/22/2011 12:13:32 PM] Mitch Sundt: I think we should assume a maximum initially, then degrade. I think 10MB is a reasonable maximum (with the caveat in all this that if you have a large attachment over 10MB, you always just try it). [11/22/2011 12:13:53 PM] Mitch Sundt: I'm happy to go with a lower maximum, just nothing higher [11/22/2011 12:14:41 PM] Yaw Anokwa: lower is also inefficient as connections in dev world get better. i like exponential even if it increases client code. [11/22/2011 12:14:44 PM] Clayton Sims: Why not just go for whatever the total size is first? [11/22/2011 12:15:07 PM] Clayton Sims: Then use the server response to adjust hte request? [11/22/2011 12:15:11 PM] Clayton Sims: Less code, more accurate [11/22/2011 12:15:48 PM] Yaw Anokwa: i agree with all that clayton. but you still have to have code on the client to cut that request in half. [11/22/2011 12:15:56 PM] Yaw Anokwa: and track which half is done. [11/22/2011 12:16:14 PM] Clayton Sims: Yaw: Oh. What? Nonono. If the POST fails, it fails [11/22/2011 12:16:19 PM] Clayton Sims: you start over [11/22/2011 12:16:45 PM] Clayton Sims: And you don't bother "cutting it in half", you just stop adding files to the request after the next one puts you over hte limit [11/22/2011 12:18:00 PM] Mitch Sundt: My main though is that since you're charged by byte on the transmission, I don't think send-all then back off makes a lot of sense. Of course, I guess that only incurs 2x the bandwidth if you do implement that. [11/22/2011 12:18:35 PM] Clayton Sims: This is also a one-time thing [11/22/2011 12:18:44 PM | Edited 12:18:55 PM] Clayton Sims: The devices can track what the expected max size is [11/22/2011 12:18:54 PM] Mitch Sundt: Ugh. So the client must remember the limitations of the server? Yuk [11/22/2011 12:19:16 PM] Clayton Sims: Better than the client trying to do really, really clever exponential backoffs against a server which has deterministic behavior [11/22/2011 12:19:24 PM] Mitch Sundt: Would the server provide a hint back on the response headers? [11/22/2011 12:19:36 PM] Clayton Sims: I think there's a "Max size" header, yeah [11/22/2011 12:24:43 PM] Mitch Sundt: I don't see a standard one. I could see this working if the "Max size" header, whatever it is, is always returned, even on the HEAD or 20x responses. Then the client can issue the HEAD, and use the returned size to set up the submits. [11/22/2011 12:25:38 PM] Mitch Sundt: If the client just issues the POST, and it 413's, they can look at the header and resend with the lower limit. [11/22/2011 12:25:56 PM] Clayton Sims: Seems legit [11/22/2011 12:35:01 PM] Mitch Sundt: I'm not finding any max-length header. How about X-OpenRosa-Length-Advice ? (since it might be exceeded for individual large attachments) [11/22/2011 12:36:38 PM] Yaw Anokwa: length advice sounds funny. Maximum-Post-Size? [11/22/2011 12:36:51 PM] Clayton Sims: I'm surprised thereisn't an informal header used for this [11/22/2011 12:36:52 PM] Yaw Anokwa: or Max-Post-Size [11/22/2011 12:37:05 PM] Mitch Sundt: ditto. surprising. [11/22/2011 12:39:55 PM] Mitch Sundt: or Accept-Body-Length ? [11/22/2011 12:40:19 PM] Yaw Anokwa: s/Accept/Max [11/22/2011 12:41:04 PM] Mitch Sundt: or perhaps Accept-Content-Length -- this is in analogy to the use of Accept-Charset, Accept-Encoding, etc. [11/22/2011 12:41:32 PM] Mitch Sundt: Max also works. [11/22/2011 12:41:47 PM] Yaw Anokwa: ahh. makes sense. i'd be ok with acccept-content-length [11/22/2011 12:42:19 PM] Mitch Sundt: OK. I'll edit the spec to add X-OpenRosa-Accept-Content-Length as a header etc. [11/22/2011 12:42:48 PM] Clayton Sims: I like Accept-Content-Length, yeah [11/22/2011 1:05:26 PM] Mitch Sundt: Ok. I updated the wiki [11/22/2011 1:43:13 PM] Yaw Anokwa: when is voting? [11/22/2011 6:10:48 PM] Jonathan Jackson: If we're happy with where it is now, just call a vote. [11/22/2011 6:10:54 PM] Jonathan Jackson: I think Anton went home so hasn't had time to work on this. [11/22/2011 7:05:46 PM] Matt Adams: My turn to read it and throw a wrench into things ;) [11/22/2011 7:06:46 PM] Matt Adams: I should be looking at the submissions standard, right? [11/22/2011 7:24:44 PM] Mitch Sundt: yes [11/22/2011 9:29:29 PM] Matt Adams: Thanks, Mitch.