Consideration of charting strategies and tools

@ghachey @jeremy_kells @rupam_bhardwaj

More a broader discussion than an "issue";

Charting

I spent some time looking at the landscape of charting tools over the last 2 weeks since its been a while since I looked at this space.

I don't want to see us loading a proliferation of packages, but may be worth reviewing what we are doing and implications going forward.

With the introduction of crossfilter, it seems naterual to include dcjs which is purpose built for crossfilter, and you have probably seen the range of very nice demos.

The other general purpose tool we have is plottable.

At the moment, the plottable we are using is from a fork I created some years back. At one point, the internal development at Palantir stopped for over a year, and there was no response to pull requests, which were then all closed when they brought out v2. So we are left disconnected in our fork, but we do in the angular environment make use of the features I put in there.

To further complicate, since then d3 has gone from version 3 to version 4 to 5; and d3 version 4 has pretty much universal breaking changes. So developers of chart packages seem to be deciding if and when they jump over this wall.

Plottable version 3 is now out and refactored for d3 v4. dcjs has gone to v3 which uses d3v4, but are maintaining v2 which uses d3 v3. Our version of plottable is d3 v3 still of course.

So our current situation is we use d3 v3, and the charting libraries (customised) plottable and dcjs v2 on top of that.

We could move to "official" plottable v3, dcjs v3 and d3 v4; but we would have to consider the rework to existing charts (and loss of functionality) in ditching the current customised plottable.

The attraction of the plottable model is that it does allow you to get down to the d3 level (as does dcjs to some degree) so you don't need to hit things that are not supported.

The other aproach, say in nvd3, or plotly, is to provide a hgh level object model; but, if the object model does not expose some functionality you need, there's really nowhere to go.

plotly is new to me, I guess because it has been a commercial project now with an open source arm (think: Jasper). It is entirely data driven, and has heaps and heaps of configuration options. So I'm wondering if this may not provide a more straightforward way to get a chart up and running? You get a lot of nice functionality out of the box here, more than I have seen elsewhere in open source options.

Here's some downsides: plotly expects its data in an array; ie there's no option to get the data values via an "accessor" function. I see that meaning a lot of moving data around from complex objects into arrays (school.surveys.map( (survey) => { etc)} Bit of a pain but not the end of the world.

plotly it is still using d3 v3 and there seems no appetite in the repo to go to d3 v4.

Other issue is that the open source project has not addressed the construction of a .d.ts file. There are 3rd party ones out there, but my experminets with them showed them to be not particularly helpful. This is a bit of an issue when everything is driven by collections of attributes provided on certain objects - there's millions of them, so usable intellisense would be nice. Worst case is that the .d.ts does not actually match the real functionality, so you can create code that would run - but can't compile. :(

I think given that our EMIS users can be expected to have pretty crap internet, we have tried so far to have a system that will work without connection to the internet (so long as you can reach the emis server). So to that end, I don't want to introduce, say, google charts. On the other hand cdn support is nice to use in remote mode. (fyi put ?remote=true on the url when you start emis it will load everything it can from cdns. This is very useful if you are out of FSM or Kiribati, trying to connect to their system - you don;t load the javascript libraries from their servers.)

So I'd welcome other opinions / experiences in this area?

Comments (4)