Tutorial: Chunked Decoding with the JSON library

Occasionally, JSON objects may exceed the constraints of json_io, making it impossible to fully decode them in a single operation. In such scenarios, chunked decoding can come in handy.
Chunked decoding is possible when a list, whether in the form of an array or an object, exists at some level of the JSON object, and each of its elements does not exceed the constraints.
In this tutorial, you will learn how to use the JSON library in an application.

Conventions

The file and class names used in this example are based on a newly created app named NewApp1. Your file names may be different depending on your settings.
The complete source code is available on GitHub

Overview

The idea is to create an application that sends a large JSON object from the client side using the Fetch API as a PUT request to the REST HTTP API implemented by the App Service.

Create a simple UI

Create a button
Add an input filed
Adjust CSS

Make a PUT request and attach a JSON object as the body using the Fetch API from the Java Script client side.

The JSON object can theoretically be infinitely large, as long as the individual elements within it do not exceed 64kB, due to the json.h limit.
The JSON object must contain a list(array/object), to use chunked decoding

Implement the REST HTTP API on the App Service to receive the request made by the Fetch API.
Implement PUT on REST API
Define DATA_SIZE_MAX 256

Is used to receive a fixed chunk of data from the REST API, which is similar to what a chunked data transfer would look like.
Each time we receive data from the REST API, we pass it as the size of the bytes we want to receive.
Can be customized, should not exceed 64kB

Use the json.h library to parse the JSON object
Loop through the JSON object and process the complete elements
Save the incomplete elements using write from the json.h library. This will append the next chunk of data.

Preparations

Example JSON structure file, which will be used as a reference point throughout this tutorial.
Starting with the app template, we will open NewApp1/apps/inno-newapp1.js and remove all of the predefined functionality so that it looks like this.

var inno = inno || {};
inno.NewApp1 = inno.NewApp1 || function (start, args) {
    this.createNode("body");
    var that = this;

    var colorSchemes = {
        dark: {
            "--bg": "#191919",
            "--button": "#303030",
            "--text-standard": "#f2f5f6",
        },
        light: {
            "--bg": "white",
            "--button": "#e0e0e0",
            "--text-standard": "#4a4a49",
        }
    };
    var schemes = new innovaphone.ui1.CssVariables(colorSchemes, start.scheme);
    start.onschemechanged.attach(function () { schemes.activate(start.scheme) });

    var texts = new innovaphone.lib1.Languages(inno.NewApp1Texts, start.lang);
    start.onlangchanged.attach(function () { texts.activate(start.lang) });

    var app = new innovaphone.appwebsocket.Connection(start.url, start.name);
    app.checkBuild = true;
    app.onconnected = app_connected;
    
    function app_connected(domain, user, dn, appdomain) {
    
    }
}

In addition, you can remove the unnecessary code of the generated example in the files NewApp1/NewApp1.cpp and NewApp1/NewApp1_tasks.cpp.

Step by step

UI

Now back to JavaScript, we create a simple UI to be called inside app_connected(..), 1 button and an input field.
The inputfiled takes a JSON object for example this one and once the button is pressed we use the fetch api to make a PUT Request the App Service which contains the inputfield data as body. After the PUT request you should get an error as response because we haven't implemented the REST API yet.
A simple implementation could look like this:

function app_connected(domain, user, dn, appdomain) {
    sendDiv = that.add(new innovaphone.ui1.Div("align-content:center; display:flex; justify-content:center;"));
    inputField = sendDiv.add(new innovaphone.ui1.Input("width:150px; height:25px; font-size:15px; margin-top:90px;", "", "Data", "", "text", "inputStyle"));

    sendButton = sendDiv.add(new innovaphone.ui1.Div("height:25px; padding-top:5px; padding-bottom:25px; margin-top:120px;", "Send", "button").addEvent("click", () => {
        // Make the PUT request
        fetch(start.originalUrl + '/json', {
            method: 'PUT',
            headers: {
                'Content-Type': 'application/json',
            },
            body: inputField.getValue()                       // The data inside inputfield is part of the body
        })
            .catch(error => {
                console.error('Error:', error);
            });
    }));
}

The button uses the pre-generated div.button CSS class, the width of the buttons is too large, to change that, remove these lines from NewApp1/apps/inno-newapp1.css

width:calc(100% - 20px); 
left:10px;

REST API

To receive data from the Fetch API, we need to modify the ConfigInitComplete function in the NewApp1 class to listen on the path "/newapp1/json".
This can be done by adding the line "webserverPlugin->HttpListen("json", new class RestApi());" to the function, as shown below:

void NewApp1::ConfigInitComplete()
{
    delete currentTask;
    currentTask = nullptr;
    webserverPlugin->HttpListen(nullptr, nullptr, nullptr, nullptr, _BUILD_STRING_);
    webserverPlugin->HttpListen("json", new class RestApi());                           // added this line
    webserverPlugin->WebsocketListen();
    Log("App instance initialized");
}

The ResApi class that we added is not defined yet so:

We need to define the RestApi class to handle requests.
The RestApi class should have a constructor, destructor, and should inherit the WebserverPluginHttpListenResult from UWebserverPlugin to handle the requests.

Let´s add it to NewApp1/NewApp1.h

class RestApi : public UWebserverPlugin {
public:
    RestApi();
    ~RestApi();
    void WebserverPluginHttpListenResult(IWebserverPlugin* plugin, ws_request_type_t requestType, char* resourceName, const char* registeredPathForRequest, ulong64 dataSize) override;
};

Next we implement the RestApi class that we created created inside NewApp1/NewApp1.cpp:

RestApi::RestApi()
{
}

RestApi::~RestApi()
{
}

void RestApi::WebserverPluginHttpListenResult(IWebserverPlugin* plugin, ws_request_type_t requestType, char* resourceName, const char* registeredPathForRequest, ulong64 dataSize)
{
    if (requestType == WS_REQUEST_PUT) {                // Wait for a PUT request
        plugin->Accept(new RestApiPut(this));           // Send the request to the PUT API class
    }
    else {                                               
        plugin->Cancel(WSP_CANCEL_NOT_FOUND);
    }
}

The RestApiPut class that will be handeling the PUT requests, is not defined yet:

Define a DATA_SIZE_MAX macro with a value of 256. This is used to receive a fixed amount of data from the PUT request.
Defines a class called RestApiPut, which inherits from another class called UWebserverPut.
The RestApiPut class has a constructor that takes a pointer to a RestApi object as an argument, and a destructor.
The RestApiPut class overrides several functions of the UWebserverPut class that are used to handle various events that occur during the processing of a PUT request.
The RestApiPut class has two member variables: a pointer to a RestApi object named "rest", and a character array named "data" with a size of DATA_SIZE_MAX*2 + 1.
The "data" array is twice the size of the incoming data to ensure that there is no buffer overflow. NOTE: Since the JSON library can theoretically handle a single element up to 64kB, it is best practice for the data array to be twice as long, which is 128kB.
There is also a size_t variable called "fill" that is used for chunked decoding.
"last" is used to check if the last chunk of data has been received, will be used with IWebserverPut::Recv().If all data had been read, the WebserverPostRecvResult() will be called with nullptr as buffer and 0 as len.

Let´s add it to NewApp1/NewApp1.h

#define DATA_SIZE_MAX 256
class RestApiPut : public UWebserverPut {
public:
    RestApiPut(class RestApi* rest);
    ~RestApiPut();
    void WebserverPutRequestAcceptComplete(IWebserverPut* const webserverPut) override;
    void WebserverPutRecvResult(IWebserverPut* const webserverPut, void* buffer, size_t len) override;
    void WebserverPutRecvCanceled(IWebserverPut* const webserverPut, void* buffer) override;
    void WebserverPutSendResult(IWebserverPut* const webserverPut) override;
    void WebserverPutCloseComplete(IWebserverPut* const webserverPut) override;

    class RestApi* rest;
    char data[DATA_SIZE_MAX * 2 + 1];
    size_t fill;
    bool last;
};

Now we simply implement the defined RestApiPut class:

Implement the constructor that takes a pointer to a RestApi instance and initializes the rest, fill and last member variables.
Implement the destructor, which is an empty destructor.
To handle the PUT request and its data, implement the WebserverPluginHttpPutResult virtual functions:

The WebserverPutRequestAcceptComplete function begins receiving data and stores the size of the incoming data in the RestApi instance.
The WebserverPutRecvResult function processes the received data in chunks of DATA_SIZE_MAX bytes and appends them to the data buffer until all the data has been received. It then sends a 202 OK response with no content data.
Even if the received data contains non-JSON elements, it is still received but not processed.
The WebserverPutRecvCanceled function is an empty function.
The WebserverPutSendResult function is called after the response is sent, and it closes the connection.
The WebserverPutCloseComplete function is called after the connection is closed, and it cleans up the RestApiPut and the IWebserverPut instance.

RestApiPut::RestApiPut(class RestApi* rest)
{
    this->rest = rest;
    this->fill = 0;
    this->last = false;
}

RestApiPut::~RestApiPut()
{
}

void RestApiPut::WebserverPutRequestAcceptComplete(IWebserverPut* const webserverPut)
{
    // start receiving data
    webserverPut->Recv(data, DATA_SIZE_MAX);                        // Recive 256 Byte of data
}

void  RestApiPut::WebserverPutRecvResult(IWebserverPut* const webserverPut, void* buffer, size_t len)
{
    if (!buffer || !len) {      //Check if this is the last chunk of data
        last = true;
    }
    if (len) {
        //...
    }
}

void  RestApiPut::WebserverPutRecvCanceled(IWebserverPut* const webserverPut, void* buffer)
{
}

void RestApiPut::WebserverPutSendResult(IWebserverPut* const webserverPut)
{
    // response is sent, close
    webserverPut->Close();
}

void RestApiPut::WebserverPutCloseComplete(IWebserverPut* const webserverPut)
{
    // clean-up
    delete webserverPut;
    delete this;
}

JSON chunked decoding

When using the JSON object from the beginning of the tutorial, the data is too large to be decoded all at once by the json.h library. It's limit, as mentioned earlier, is 64kB. So we split the data into bytes, in our case 256Byte. Since the data we split may not be complete,
we will use chunked decoding to decode the data and process it for each chunk. An example of what 256 bytes of data from the sample JSON looks like:


            {"data":[
                {"id":0,"name":"Item 0","description":"This is item 0","price":65,"available":false,"tags":["tag1","tag2","tag3"]},
                {"id":1,"name":"Item 1","description":"This is item 1","price":65,"available":false,"tags":["tag1","tag2","tag3"]},
                {"id":2,"name":"I

We can clearly see that the last element is not a valid json object. A simplified representation of chunked decoding is shown below:

Decode the data. Decoding returns an error because the JSON data is incomplete
Find the list in the decoded data with get_array or get_object
Iterate through the list until you encounter an element with the JSON_FLAG_INCOMPLETE flag.
Do an incomplete encoding of the data, using write with the incomplete parameter set to the list (object or array). This will encode only the elements that are not yet complete, along with the element they are embedded in.
Get the next chunk, append it to the encrypted data, and repeat the previous steps.

Before we implement the code inside "if (len) {...}" and go through it, here are some notes:
- "data" has double the size of the received data to avoid buffer overflow
- "fill" is used to define the number of chars of incomplete data, which is the length of the incomplete data and is 0 for the first data.

// decode
data[fill + len] = 0;
class json_io json(data);
json.decode();
word root = json.get_object(JSON_ID_ROOT, 0);
// get array of objects
word a = json.get_array(root, "data");

So we first set the terminating 0 into the data buffer, after the first data length. "data" looks like this:

{"data":[
    {"id":0,"name":"Item 0","description":"This is item 0","price":65,"available":false,"tags":["tag1","tag2","tag3"]},
    {"id":1,"name":"Item 1","description":"This is item 1","price":65,"available":false,"tags":["tag1","tag2","tag3"]},
    {"id":2,"name":"I'\0'

Then we decode the data buffer, get the root element "{...}" and inside it we get the data array.

Now we check if the array "a" is valid, we go through it using a for loop. If the current element of the loop is complete, we can process it. This goes on until the loop finds an incomplete element, at which point it exits. An element is marked as incomplete if the "json.get_object" is invalid. So the for loop in this case works like this:

Get first JSON element "element = json.get_object(a, 0);"
Check that the element is not empty or incomplete "element != JSON_ID_NONE && !(json.get_flags(element) & JSON_FLAG_INCOMPLETE);">>>
Get next element "element = json.get_object(a, element)"

if (a != JSON_ID_NONE) {
    word element = JSON_ID_NONE;
    for (element = json.get_object(a, 0); element != JSON_ID_NONE && !(json.get_flags(element) & JSON_FLAG_INCOMPLETE); element = json.get_object(a, element)) {
        // get object properties
        const char* name = json.get_string(element, "name");
        // process the data
        // ...
        debug.printf("Name:%s", name);
    }

}

So if the loop breaks out and an incomplete element is found, means that the next chunk of data which we will be reciving is part of the incomplete element. That's why we need to fill the data buffer with the incomplete element,
and pass the address after the last char of the filled data. If the JSON has reached its end but there's some invalid data, that data will still be received but not processed, so when all the data is received, we send a 202 OK response.

if (element != JSON_ID_NONE || flags & JSON_FLAG_INCOMPLETE) {              // check if the end has been reached
    char tmp[DATA_SIZE_MAX * 2 + 1];                                        // temporary buffer to save the incomplete data
    char* x = tmp;                                                          // pointer to temporary buffer
    debug->printf("Buffer before write: %s", tmp);
    json.write(0, x, array);                                                // saves element before incomplete data inside the array, the incomplete data and the terminating 0 to the tmp buffer
    debug->printf("Buffer after write: %s", tmp);
    fill = x - data;                                                        // calculate the length of the incomplete data by subtracting the 2 memory adresses, which is faster then strlen(tmp)
    debug->printf("Fill size: %d", fill);
    memcpy(data, tmp, fill + 1);                                            // copy the incomplete data to the data buffer
}

// Check if there is remaining data from the inital request, which may or may not be JSON data
if (!last) {
    // append next chunk to the end of the incomplete data
    webserverPut->Recv(&data[fill], DATA_SIZE_MAX);
}
else {
    // send 202 OK response with no content data
    webserverPut->SetResultCode(WEBDAV_RESULT_OK, 0);
    webserverPut->Send(0, 0);
}

As mentioned in the comment, the write function also saves the element before the incomplete element, so the data buffer would look like this after the first call:

{"data":[{"id":1,"name":"Item 1","description":"This is item 1","price":65,"available":false,"tags":["tag1","tag2","tag3"]},{"id":2,"name":"I

And the "&data[fill]" passed to the "Recv" function will be the last char, so when the next chunk of data comes in, it will be added to the right place.

The complete code of the WebserverPutRecvResult function would look like the following:

void  RestApiPut::WebserverPutRecvResult(IWebserverPut* const webserverPut, void* buffer, size_t len)
{

    if (!buffer || !len) {                  //Check if this is the last chunk of data
        last = true;
    }
    if (len) {
        // decode
        data[fill + len] = 0;           // Write the terminating 0 after the incomplete data length + the new data
        class json_io json(data);
        json.decode();
        word root = json.get_object(JSON_ID_ROOT, 0);
        // get array of objects
        word array = json.get_array(root, "data");
        if (array != JSON_ID_NONE) {
            word element = JSON_ID_NONE;
            for (element = json.get_object(a, 0); element != JSON_ID_NONE && !(json.get_flags(element) & JSON_FLAG_INCOMPLETE); element = json.get_object(a, element)) {
                // get object properties
                const char* name = json.get_string(element, "name");
                // process the data
                // ...
                debug->printf("%s", name);


            }
            if (element != JSON_ID_NONE || flags & JSON_FLAG_INCOMPLETE) {              // check if the end has been reached
                char tmp[DATA_SIZE_MAX * 2 + 1];                                        // temporary buffer to save the incomplete data
                char* x = tmp;                                                          // pointer to temporary buffer
                debug->printf("Buffer before write: %s", tmp);
                json.write(0, x, array);                                                // saves element before incomplete data inside the array, the incomplete data and the terminating 0 to the tmp buffer
                debug->printf("Buffer after write: %s", tmp);
                fill = x - data;                                                        // calculate the length of the incomplete data by subtracting the 2 memory adresses, which is faster then strlen(tmp)
                debug->printf("Fill size: %d", fill);
                memcpy(data, tmp, fill + 1);                                            // copy the incomplete data to the data buffer
            }
            else
                fill = 0;                                                               // Because the end of the JSON has been reached we start reciving date from the start   
        }
    }
    // Check if there is remaining data from the inital request, which may not be JSON data
    if (!last) {
        // append next chunk to the end of the buffer which contains the incomplete data
        webserverPut->Recv(&data[fill], DATA_SIZE_MAX);
    }
    else {
        // send 202 OK response with no content data
        webserverPut->SetResultCode(WEBDAV_RESULT_OK, 0);
        webserverPut->Send(0, 0);
    }
}

Conclusion

In conclusion, this tutorial provided a detailed guide on how to use the JSON library to decode large JSON objects using chunked decoding.
We learned how to create a small application with a button and an input field to send and receive data through REST API from the client to the App Service.
Finally, we learned the steps involved in chunked decoding, including how to find the list within the decoded data and how to encode incomplete data using the write function.
By following the steps outlined in this tutorial, developers can successfully handle and transfer large JSON objects using chunked encoding and decoding in their applications.