Thoughts about necessary refactorings in JR core

• Split FormInstance into two objects -- FormInstanceTemplate which contains all the structural information about the instance (node names, hierarchies, constraints, repeatability, repeat templates) and FormInstance which contains only the data that can be affected from filling out the form (answers, # repetitions, and current relevancy)

• FormInstance needs an API better suited to how the form engine needs to interact with the instance. For example, it is amazingly difficult right now to answer the question 'how many repetitions currently exist for this node/reference'. I feel like there are several other situations where answering a simple question actually takes 4-5 low-level APIs calls with several gotchas.

• Support for XPath predicates - absolutely necessary for any sort of large dataset usage.

• Support for external FormInstances - We need to be able to store large datasets in the file system or not-in-the-form generically and pull data out from them. We need both a way to (Without a DOM!) access XML instances using our xpath, and a way to build that connection.

• Cascading relevancy for instance nodes should be calculated dynamically. Currently it uses a 'propogate-downward-upon-update' model which causes hard-to-trace bugs

• Need to take a hard look at the level of static vs. dynamic typing in the instance. Originally everything was static, but things have become more dynamic with calculates, itemsets, and such. As a result, things are kind of a mess between the two and can lead to peculiar behavior. The xforms spec treats almost everything dynamically. Static is nice, though, because it lets us do lots of compile-time validation.
  • (ctsims) static typing is more painful than it's worth in my opinion
    • (droos) but it lets us catch a lot of errors in people's forms that they would have to spend lots of time doing manual testing to catch (which probably won't happen-- they'll just get runtimeexceptions after they've deployed in the field)

• Should we replace the 'on-change' model of revelancy propogation to on-demand? Would simplify things, but I'm not sure it's worth the efficiency hit. (Could also cause problems for stateful xpath functions) (then again, for forms with looooong dependency chains (IMCI), on-change is the efficiency hit)
  • (ctsims) Are we supporting stateful xpath functions? What's the use case? Shouldn't those be triggers in the xform?
    • (droos) there are a few cases where we're using custom xpath functions with side effects. global, increment-per-form counters are the most common use case

• Navigating through a form requires keeping track of several closely-related pieces of information: current instance node, current instance reference, current form index, and current form control/control hierarchy. These are all disparate objects and keeping track of them all is cumbersome. They should all be unified somehow, most likely through a 'smart' FormIndex.
  • (ctsims) Strongly agreed

• FormIndex as linked list.... just say no.
  • (ctsims) The internal representation shouldn't matter, users should only be interacting with FormIndices through other objects.
    • (droos) but i, as a core developer, have to deal with internal representations, not just APIs. and if the internal representation is unfriendly, adding new core features is very hard. a lot of the current bloat/ugliness of FormDef (and elsewhere that deals with form nav) is due to the representation of FormIndex being at odds with how we use it in practice. it's a turn-off to developers.

• The navigation semantics of the form index are hard to grasp, and nearly impossible to modify from looking at FormDef.incrementIndex/decrementIndex

• Preloaders should be deprecated and replaced with calculated expressions (linked to back-end function handlers)
  • (ctsims) I think they should be replaced with external XML instances, not functions (for non-declarative values that will never change)

• Easier conversion back and forth between xml and internal javarosa representations?
  • (ctsims) We really need to decide whether it's worth keeping our in-memory DOM objects around, or whether we want to just simplify interacting with a dynamic XML document and start using that as the "internal javarosa representation" We duplicate a lot of effort because we have both.
    • i think our internal dom is kind of necessary, especially for heapsize considerations. anyway, this was mostly ODKs complaint and i think we've largely addressed it; just put it on here to see if there's any room for improvement