Token size is broken (always == 0 or 1)

Issue #51 invalid
Matthew Bossard
created an issue

This is a great library which I was successfully using, but in recent commits I have a problem with calculation of token size. Always the size is 0 or 1 for top-level or nested objects/arrays, and is 1 for Name strings (why would string have a size > 0?)

Given the simple JSON object:

{"foo":1234}

jsmn_parse returns 3, but token[0].size = 1

A simple nested object example:

{"foo":{"bar":1}}

jsmn_parse returns 5, but token[0].size = 1, and token[3].size = 1

I suspect this became broken at commit 84cb579. Token size was correct previously. Is it related to the "implemented key/value hierarchy"? (which is not documented anywhere that I can find)

Line 239-241:

            case ':':
                parser->toksuper = parser->toknext - 1;
                break;

Removing line 240 fixes the issue for non-strict parsing. However, with JSMN_STRICT, this causes other issues which I did not fully characterize.
I do not understand what is being done at line 240 (it probably is related to "implemented key/value hierarchy", but there are no comments or documentation about that).

Comments (4)

  1. Matthew Bossard reporter

    It appears this is probably not a bug, but rather is a misunderstanding on my part! Oops. I'll explain here in case anyone else makes the same misunderstanding...

    The jsmn docs describe the jsmn_parse return value as "number of tokens actually used by the parser", and jsmntok_t .size member as "Number of child (nested) tokens". So, for example, the following JSON...

    {"foo":{"bar":1,"no":"duh"}}

    jsmn_parse returns 7, the absolute total tokens used, as I expected.

    However, for the size of each token, jsmn returns the contextual (according to token type) number of tokens! Not the same as the jsmn_parse return value!

    So...

    token[0].size = 1, since token[0] is Object with 1 JSON member
    token[1].size = 1, since token[1] is Name with 1 JSON value (Name always has 1 value)
    token[2].size = 2, since token[2] is Object with 2 JSON members
    token[3].size = 1, since token[3] is Name with 1 JSON value
    token[4].size = 0, since token[4] is Value String (no child tokens, always 0)
    ...and so on...

    Serge, maybe a simple clarification in the docs would explain this for anyone who might have the same misunderstanding? :) Thanks!

  2. Log in to comment