Dec 18 2008

6

Compressed JSON Requests

Category: UtilitiesTags: , ,

I rarely find myself in a place where I am writing javascript applications that use AJAX in its pure form. I have long abandoned the ‘X’ and replaced it with ‘J’ (JSON). When working with Javascript, it just makes sense to return JSON. Smaller footprint, easier parsing and an easier structure are all advantages I have gained since using JSON.

In a recent project I found myself unhappy with the large size of my result sets. The data I was returning was tabular data, in the form of objects for each row. I was returning a result set of 50, with 19 fields each. What I realized is if I augment my result set I could get a form of compression.

// uncompressed
JSON = {
  data : [
     { field1 : 'data1', field2 : 'data2', field3 : 'data3' },
     { field1 : 'data4', field2 : 'data5', field3 : 'data6' },
     .....
  ]
};
 
//compressed
JSON = {
    data : [ 'data1','data2','data3','data4','data5','data6' ],
    keys : [ 'field1', 'field2', 'field3' ]
};

I merged all my values into a single array and store all my fields in a separate array. Returning a key value pair for each result cost me 8800 byte (8.6kb). Ripping the fields out and putting them in a separate array cost me 186 bytes. Total savings 8.4kb.

Now I have a much more compressed JSON file, but the structure is different and now harder to work with. So I implement a solution in Mootools to make the decompression transparent.

Request.JSON.extend({
 
	options : {
		inflate : []
	}
 
});
 
 
Request.JSON.implement({
 
	success : function(text){
		this.response.json = JSON.decode(text, this.options.secure);
		if(this.options.inflate.length){
			this.options.inflate.each(function(rule){
				var ret = ($defined(rule.store)) ? this.response.json[rule.store] : this.response.json[rule.data];
				ret = this.expandData(this.response.json[rule.data], this.response.json[rule.keys]);
			},this);
		}
		this.onSuccess(this.response.json, text);
	},
 
	expandData : function(data,keys){
		var arr = [];
		var len = data.length; var klen = keys.length;
		var start = 0; var stop = klen;
		while(stop < len){
			arr.push( data.slice(start,stop).associate(keys) );
			start = stop; stop += klen;
		}
		return arr;
	}
 
});

Request.JSON now has an inflate option. You can inflate multiple segments of your JSON object if you so desire.

Usage:

new Request.JSON({
       url : 'url',
       inflate : [{ 'keys' : 'fields', 'data' : 'data' }]
       onComplete : function(json){}
});

Pass as many inflate objects as you like to the option inflate array. It has an optional property called ’store’ If set the inflated data set will be stored in that key instead.

The ‘keys’ and ‘fields’ expect strings to match a location in the root of your JSON object.

Tags: , ,

6 Responses to “Compressed JSON Requests”

  1. Oskar Krawczyk says:

    Interesting approach indeed – hard for me to grasp how the actual compressed code looks like. I’ll give this solution a try at some point as more and more of the projects I’m working on are being migrated (by me of course) from A to J.

    Ps. I’m no native speaker but I think there are some spelling mistakes and redundant words in the article – one of the reasons why I don’t post stuff late at night :-)

  2. nwhite says:

    @Oskar : I’ll post a live example with some code showing the changes made on the server side as well.

    Thanks for the heads up, I’ll revise the wording. I was in a rush to meet my girlfriend at the movies :$

  3. electronbender says:

    Just the other day i had re-formated my json string the same way.
    Only it didnt occur to me to deflate it afterwards.
    Good job:)

  4. Daniel says:

    You said you saved 8.4kb, but what was the original size of the object?

  5. nwhite says:

    @Daniel - My original response size was around ~22kb. Using this technique my average requests dropped below 14kb. In the particular app that I am referencing I also served the response compressed using gzip. This got my total size down to 4-5kb.

  6. Guillermo Rauch says:

    I think it makes more sense (API wise) to send a compression flag (like a custom header returned along with the JSON response), and make it transparent to the developer that the (un)compression is taking place.

    See http://www.xulplanet.com/references/objref/XMLHttpRequest.html#method_getAllResponseHeaders

Leave a Reply