Two stage-parsing

Issue #10 resolved
Anonymous created an issue

Would it be possible to add a function that does an (optional) parsing which returns the required number of tokens for a given JSON string?

The rest could work as it does now. This way it would be possible to handle input of a previously unspecified length and keep all the benefits of the current library.

Comments (5)

  1. illishar

    This is actually a major issue. The way it is now, you have to guess at your required number of tokens. Each token takes up 16-20 bytes of memory. The recommended size (used by the samples) token array is 256. Means 5120 bytes of ram. (You shouldn't stack allocate that much btw.) And 256 tokens might not be enough or it might be way too much.

    Many ansi c functions returns the required "number". That way you can call the function with a NULL allocation and allocate the required number afterwards. (Eg. snprintf comes to mind.)

    The issue is actually too much for me to overcome, I'm afraid. I'll have to search for something else.

  2. Serge Zaitsev repo owner


    Your sources are very helpful, thanks a lot!

    I made changes similar to what you described in your sources and covered them with some tests (commit #5865d5fa9565). Seems to work so far. Now jsmn_parse always returns the number of tokens used, and caller can safely pass NULL as tokens array - then no actual parsing will happen. So, two-stage parsing can be done as:

    count = jsmn_parse(p, js, NULL, 0); tokens = calloc(sizeof(jsmntok_t), count); jsmn_parse(p, js, tokens, count);

  3. Log in to comment