Error 500 received when POSTing ZINC-formatted data

[Stuart Longland]

Hi all,

Not sure if its something wrong with the ZINC data I'm trying to post, or whether there's an error in the parser, but in essence, I'm trying to implement a Python-based client for Project Haystack and I'm using nodehaystack as the test server implementation.

I've written a Python module that generates the ZINC-formatted data, I borrowed the ZINC test cases from nodehaystack code. The implementation is a clean-room implementation based on the ZINC spec on the Project Haystack website.

https://github.com/vrtsystems/hszinc is the module.

When I post, I get an error 500. This seems wrong. If the data is malformed, I'd expect a 400 error (Bad Request).

This is a tcpdump trace of the traffic:

POST /api/hisWrite HTTP/1.1
Host: localhost:8080
Content-Length: 96
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.8.1
Connection: keep-alive
X-Access-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0NTM0MDg5MzkzMjAsInVzZXIiOnsibmFtZSI6InN1cGVydXNlciIsInJvbGUiOiJhZG1pbiIsInVzZXJuYW1lIjoic3VwZXJ1c2VyIiwicGFzc3dvcmQiOiIkMmEkMTAkcFVoM29CbkRNMjZnMU0xb1Audnd1ZTBEWkI5bmo5Snd4VWlkelBILjVQbmhnaWxwVEpkWUcifX0.AkRExw2q0WSvaSrztmFjhLX85lj_lfJ1xWaBZ-VvYIk
Content-Type: text/zinc; charset=utf-8

ver:"2.0" id:@my.site.submeter.1.energy
ts,val
2016-01-15T06:42:35.727102+10:00 Brisbane,123.45

The response is a 500 with the following stack trace (I added a console.log line to HOp.js so I could see what was being passed to the makeReader function, clearly not a string).

Got POST format text/zinc; charset=utf-8 body:
[object Object]
TypeError: Cannot construct: HZincReader(InputStream). this._data.slice is not a function
    at ReadableStream.read (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/io/Streams.js:24:22)
    at consume (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/io/HZincReader.js:46:18)
    at init (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/io/HZincReader.js:54:3)
    at HGridReader.HZincReader (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/io/HZincReader.js:83:3)
    at HGridFormat.makeReader (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/io/HGridFormat.js:104:12)
    at postToGrid (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/server/HOp.js:172:10)
    at HOp.onServiceOp (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/server/HOp.js:70:38)
    at /opt/node_app/routes/index.js:91:6
    at _op (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/server/HServer.js:84:5)
    at HProj.HServer.op (/opt/node_app/node_modules/widesky-node-haystack/node_modules/nodehaystack/server/HServer.js:77:5)
    at /opt/node_app/routes/index.js:82:16
    at Layer.handle [as handle_request] (/opt/node_app/node_modules/express/lib/router/layer.js:95:5)
    at next (/opt/node_app/node_modules/express/lib/router/route.js:131:13)
    at Route.dispatch (/opt/node_app/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/opt/node_app/node_modules/express/lib/router/layer.js:95:5)
    at /opt/node_app/node_modules/express/lib/router/index.js:277:22

Is there a bug somewhere in my ZINC grid generation that's triggering this? How do I run the built-in tests in nodehaystack?

Regards, Stuart Longland

[Stuart Longland]

Just did the following mods:

diff --git a/server/HOp.js b/server/HOp.js
index 9600e68..08fc239 100644
--- a/server/HOp.js
+++ b/server/HOp.js
@@ -27,7 +27,8 @@ var url = require('url'),
     HUri = require('../HUri'),
     HVal = require('../HVal'),
     HGridFormat = require('../io/HGridFormat'),
-    HZincReader = require('../io/HZincReader');
+    HZincReader = require('../io/HZincReader'),
+    util = require('util');

 /**
  * Programmatic name of the operation.
@@ -168,6 +169,7 @@ function postToGrid(req, res, callback) {
   var r = req;
   // if we have an app then Express is being used and we need the body
   if (req.app) r = req.body;
+  console.log('Got POST format ' + mime + ' body:\n' + util.inspect(r, false, null));
   format.makeReader(r).readGrid(callback);
 };

and got this:

Got POST format text/zinc; charset=utf-8 body:
{}
TypeError: Cannot construct: HZincReader(InputStream). this._data.slice is not a function

I definitely didn't post an empty object, so something isn't passing the post body through like it should.

[Stuart Longland]

Found the issue. Seems when express was initialised, registering a handler for text/* was missed.

I'm thinking if the body were compared with an empty object, that would allow nodehaystack to display a less cryptic error message. (Perhaps error 415, since in this instance, the server cannot accept text/zinc.)

That got me this error:

Got POST format text/plain body:
'ver:"2.0" id:@my.site.submeter.1.energy\nts,val\n2016-01-15T08:52:32.984395+10:00 Brisbane,123.45\n'
Error: Too many digits for milliseconds in time value

That, IMO, should definitely be a 400 error, or better yet, can the timestamp not be truncated by the ZINC parser at the server?

[Shawn Jacobson]

I do not disagree with you on the milliseconds, but this is how the current Haystack community has implemented this. If you look at the project haystack site, it even shows DateTime tokens as having 9 decimal spaces. Unfortunately, this does not affect any of the projects we currently integrate with, and I cannot justify the time to alter this at this time. That being said, I would be happy to review a Pull Request if anyone out there wanted to take a crack at it themselves.

As for the error codes, there are many areas where the error codes could be handled better, but again, its a time issue. I will add to our system that we need to review all error producing sections, but to be frank, this will be an extremely low priority for us. Once again though, we welcome anyone wishing to do a Pull Request and making these better.

Thanks

[Shawn Jacobson]

Added issue for better error messages to internal JIRA app.

[Stuart Longland]

Okay, I realise this is now two years on, but today I stumbled on the reason why I was getting that error regarding timestamp precision:

https://bitbucket.org/lynxspring/nodehaystack/src/60548d00dff3298b6989d52cf296c8fb2e70eb0c/io/HZincReader.js?at=master&fileviewer=file-view-default#HZincReader.js-329:340

That switch block can be mathematically derived… using Math.pow:

> function ms(dp) {
... return 1000*Math.pow(10,-dp);
... }
undefined
> ms(3)
1
> ms(0)
1000
> ms(1)
100
> ms(2)
10
> ms(10)
1.0000000000000001e-7
> ms(6)
0.001
> ms(9)
0.0000010000000000000002

This was a new feature in JavaScript 1.0… something Netscape Navigator 3.0 supported, so I'd be very surprised if a NodeJS version in existence that doesn't support this feature.