Fly together Forever

Over the weekends, I spent some time on Dojo Objective Harness (DOH) to fix a reported bug: in some cases the global progress bar overshoots 100%. This is obvious in the dojo test suite, the progress bar goes beyond 900%.

After some digging, it turns out the algorithm to calculate the global progress bar has a flawed assumption: the number of tests in each group is known before the tests start. That’s not the case if nested test suites are used, such as the case in dojo._base.test.

The fix is quite simple as well: rather than trying to create a portion in the global progressbar for each test, we only create one for each group.

Another problem noted in the original bug report is overflow of test group progressbar. I never noticed it before, but keeping a close eye on each group progressbar when running dojo test suite does reveal it. By removing tests in the dojo test suite, I isolated the problem to dojo.date and dojo.date. The problem is the test group progress bar is not updated at all due to the fact that they don’t have a html page (instead both dojo.date and dojo.data test cases only use javascript files). The fix for this turns out to be quite straghtforward as well as soon as I know what causes it.

After the bug fixes, I feel like to add a new feature to DOH, clickable test cases which will hilight the relevant log line after the test finishes. Searching a failture/error in the log pane is a real pain especially if you have lots of tests generating huge number of messages. The log line will be hilighted according to whether the test failed or succeeded. (Note: the hilighting requires dojo, so if DOH is used without dojo, you won’t be able to see the hilighting animation.)

Similarly, when clicking on the global progressbar, the test group associated with that portion of the bar will be scrolled into view (for any failed test group, a tooltip is also attached to the global progressbar portion to indicate which test group it is). This is useful when locating a failed test group.

In addition, a tfoot is added to the test list table after all tests finish, which contains a summary of the test run. Check out the online demo in our nightly. This will be part of dojo 1.3 release.

In a desktop environmen, menus (be it context menu or menu in menubar) have a column which display the accelerator key (or shortcut key in windows). In the 0.4 dojo Menu widget, we used to have that support, but it was not ported to 1.x.

After seeing Bill’s commit to add a real MenuBar to dijit, I think it’s the time to bring back this feature. For now, see the test_Menu.html test file for a demo.

A new property is introduced for dijit.MenuItem, called accelKey, of type String. It can be set either in markup or in the arguments passed to the dijit.MenuItem constructor (the demo contains examples for both of them). In addition, dynamically setting it is also supported, such as:

//item is a dijit.MenuItem object
item.attr("accelKey","Ctrl+W");

As mentioned in the demo, this accelKey is only a string, you can basically pass in any string you want. However, when the combination is pressed, it won’t actually trigger the action associated with the menu item: that has to be done explicitly elsewhere (such as dojo.connect to body’s onkeydown event, and detect whether a recognized accelerator key is pressed, if so, invoke the corresponding event handler).

By default, dijit.form.DropDownButton requires a dropdown menu specified when initializating. However, with a little bit of patching via subclassing it, we can easily achieve lazy loading of the dropdown menu: only creating it the first time a user clicks on the button. This will speed up the initial loading time of the application, especially if a lot of these type of buttons are present on the page.

Here is the core of how to achieve lazy loading:

dojo.declare("MyDropDownButton", dijit.form.DropDownButton, {
	dropDown:1,
	_openDropDown: function(){
		if(this.dropDown===1){
			this.dropDown=new dijit.Menu();
			var items=[{label:'One'}, {label:'Two'}, {label:'Three'}];
			dojo.forEach(items,function(args){
				this.dropDown.addChild(new dijit.MenuItem(args));    
			},this);
		}
		this.inherited(arguments);
	}
});

The trick here is to set dropDown to something which evaluates to true so that when clicked the first time, DropDownButton will proceed to call _openDropDown, where we have a chance to create the real dijit.Menu we want to show. Without line 2 above, _openDropDown won’t be called because DropDownButton thinks there is no attached dropdown menu.

As you may have already figured, in order to use this, the Menu has to be created programmatically.

While trying to add setting clipboard content support to doh.robot, I encountered a very strange bug and spent several hours of trail and failure without going anywhere. I am using java 1.6 update 11 directly from Sun. Basically, I have two problems both of which are described here: copying HTML data to windows clipboard from java does not work, with the following symptoms

1. when puting data into the clipboard with a RepresentationClass other than java.lang.String does not work properly with any applications (it is the same issue as described in a java bug report, which is supposed to be fixed in java 1.6 update 11)
2. even if you do use java.lang.String as RepresentationClass, the pasted html only appear correctly in IE, not in safari (it contains extra garbage characters), nor in FF (nothing is pasted at all).

The problem as described in the mentioned webpage, is due to buggy support of windows HTML clipboard format in java. One workaround is suggested in the same article by modifying some internal stuffs in the supporting class in java with the source files available for download. However, the provided code fails to run against my java 1.6 (although it does compile fine), probably due to changed internal mechanisms.

(more…)

After introducing dojo.reload in my previous post, I talked with kriszyp in IRC, and he asked whether dojo.reload can actually modify already created objects from dojo.declare-d classes: in most web 2.0 apps, after the app is loaded, there are lots of instances of classes already created, and using dojo.reload to reload their definition won’t actually update any of these existing instances.

As I do not have an idea to actually implement that, I do come up with another “workaround” (if you like): rather than propagate changes to existing objects on the page, we can actually try to ask the app to reload itself without reloading the whole page. We don’t want to add in any special code in any frontend apps to handle this sort of “self reload” mechanism just to speed up development. Instead, a generic way should be available without any extra logic support in the application layer.

With the help of dojo and with one assumption, I did come up such an implementation to support that. The one assumption is that, all existing objects you care about in your applications are all dijits. If this assumption can be met, what we really need to do in order to reload an application is to:

1. call onunload handlers (in case any are registered with dojo)
2. destroy all dijits in the page
3. reset the document.body.innerHTML to what it looks like when the page is first loaded (before any js actually modifys it)
4. call all onload hanlders registered with dojo (such as dojo.parser etc.)

Item 3 can be achieved via a xhr call to retrieve current page from server and figure out the content within <body> tag. Item 4 requires a bit of extra work, because once all the onload handlers are called by dojo, they are removed. So in order to actually rerun them, we have to remember the original list of handlers.

Of course, as you may already figured, the method mentioned above won’t be able to run any modified onload handlers (even after they are dojo.reload-ed), however, it does cover most of other cases.

The one assumption mentioned above is not actually true in most web 2.0 applications using dojo, in such cases, any non-dijit objects have to be destroyed somehow so that they can be recreated, but that logic has to be application specific. On the other hand, in order to prevent memory leaks in browsers (espcially IE), it is always a good idea to have a destroy function for your objects to clean itself up. So if you are not worried about any thing other than dijits, then the method mentioned above should work reasonably well, while if you do want to re-create other non-dijits stuffs, you can call some application specific cleanup routines (which as I mentioned above would be good to have in any case).

You can find the code in dojoc.util.loader, the function is actually called dojo.reloadPage(). The normal use case for this is:

1. dojo.reload any modules you modified and want to test;
2. dojo.reloadPage to actually test them without waiting for reloading all js files/reloading your browser window.

Note: dojo.reloadPage only works if it is included in the initial page load, if you dojo.require dojoc.util.loader after the page is loaded, dojo.reloadPage won’t work (see comment at the beginning of the file to know why)

dojo.reload now also resides in dojoc.util.loader, and compared to the previous version, two additional features are add:

• Partial module name match: now you just need to type in a partial name of the module you want to reload, if it only matches one module already loaded, that module will be reloaded. If there are multiple matches, it will list all possible modules and error out.
• Automatically remove cache of dijit template: if a module references widget template files (using templatePath), these cached templates will be cleared automatically, so next time you create such a dijit, the new template will be loaded from your sever directly.

A new function to reload css files is also included in dojoc.util.loader, dojo.reloadCss, which also supports partial name match. If it is given no argument, it will reload all css files in the page (with a href attribute, of course), similar to ReCSS feature in the firebug lite shipped with dojo core.

Warning: use at your own risk.

For the past several months, I am developing frontend against a bare cherrypy server which also serves all static files, like javascripts/images etc. Without building frontend code (so there are lots of small js files to load), it takes more than 20 seconds to reload the unbuilt frontend application I am working on (which is the main driver behind my hacking on dojo.reload).

Use Apache as proxy server

Today I decided to shield an apache proxy server in front of the cherrypy server to off load all static files serving duties from the latter, in the hope of speeding up reloading speed of the app.

The apache proxy should be setup so that it directly serves any files under /debug and /release, all other requests are dynamic and should be handled by cherrypy server. In addition, our backend app sometimes use HTTP redirects to direct client to a new page. In Apache configuration file, all this can be achieved by:

ProxyPassMatch ^/(?:debug|release)/.* !
ProxyPass / http://127.0.0.1:8000/
 
ProxyPassReverse / http://127.0.0.1:8000/
 
Alias /debug /var/www/htdocs
Alias /release /var/www/htdocs/release

Note: the /debug directory is the unbuilt frontend code, while the /release points to the built frontend code (by default, dojo will put the built version under release directly as peer of dojo dir).

With the above settings, cherrypy is nicely sitting behind the Apache server without worrying about any static files, and the reloading time of the unbuilt frontend code now reduces to about 4 seconds, which is a dramatic improvement.

Try nginx instead

nginx is normally considered to be a faster reverse proxy server than apache, so I want to give it a try.

In nginx configuration file, proxy_pass is used to pass the request to a backend server, while it does not use proxy_pass_reverse, instead the equivalent in nginx is proxy_redirect directive. As long as your host domain name is properly set up, the following nginx directive is equavalent to the above apache directive:

location /debug/ {
    alias /var/www/htdocs;
}
location /release/ {
    alias /var/www/htdocs/release;
}
location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_redirect default;
}

More info on proxy_redirect can be found in official documentation.

Impression of Nginx compared to Apache

While I don’t want to do any thorough comparsion of the two reverse proxy servers, I just tried each of them several times and monitors the net panel output in firebug. It seems, nginx delievers more consistent performance: for the same page, apache sometimes deliver it in 1 second, sometimes in 4 seconds, while nginx always delivers it in 1 second. Thus I guess I will just keep using nginx at least for now.

When working with FAT client web 2.0 application, it is normally quite slow whenever you need to reload your application because you are developing against unbuilt code (the source files are changed frequently, so you don’t want to build your frontend code every time a modification happens). As a frontend developer, you are using Firefox + Firebug, with console/script tabs enabled (sometimes net tab in firebug as well), which makes the loading time of a page with lots of javascript files even more slower.

While we are used to ajax ways of updating part of a page without reloading the entire webpage, no one seems to apply the same methodology to reloading javascript source files.

Dojo has the powerful dojo.require mechanism which can pull in javascript source files on the fly, and it is smart enough to cache any source files which are already downloaded. However, it does not provide any way to force reloading of a js file.

After digging into dojo.require, I came up with a simple function to overcome this very issue: basically, it will unset any memory dojo has about that downloaded source file (or module, as called in dojo.require), and dojo.require that module again.

Here is the source code for this function, which is called dojo.reload for now:

dojo.reload=function(d){
	//remove the module namespace from its parent namespace
	var sym=d.split('.');
	if(sym.length>1){
		var pname=sym.slice(0,-1).join('.');
		var prop=sym[sym.length-1];
		var parent=dojo.getObject(pname,false);
		if(parent){
			console.log('delete property',prop,'on object',pname,':',delete parent[prop]);
		}
	}
 
	//remove _loadedModules entry so that dojo.require will re-load it
	delete dojo._loadedModules[d];
 
	//tidy up _loadedUrls so that dojo._getText will reload it
	sym=dojo._getModuleSymbols(d);
	var relpath = sym.join("/") + '.js';
	//copied from dojo._loadUri
	var uri = ((relpath.charAt(0) == '/' || relpath.match(/^\w+:/)) ? "" : dojo.baseUrl) + relpath;
 
	delete dojo._loadedUrls[uri];
	dojo._loadedUrls.splice(dojo.indexOf(dojo._loadedUrls,uri),1);
 
	//set cacheBust to make sure we load latest file
	dojo.config.cacheBust=+new Date;
 
	dojo._loadreloaded=d;
	if(!dojo._reloadhotkey){
		dojo._reloadhotkey=dojo.connect(document.documentElement,'onkeydown',function(e){
			if(e.keyCode===dojo.keys.F6){
				dojo.reload(dojo._loadreloaded);
				dojo.stopEvent(e);
			}
		});
	}
	console.log('reloading module',d);
	dojo.require(d);
};

Just copy and paste this function somewhere to your frontend code base, and when you want to reload a particular module, just type dojo.reload('mymodule.myfile') in Firebug console, and the latest version of your module will be downloaded and ready to be used/tested on immediately. In order to make sure the browser does not use one from its cache, cacheBust will be set to current timestamp to make sure it is loading from server directly. Another fine touch in the dojo.reload function is that, if you just want to reload the module you last reloaded, simply hit F6 which saves you some type.

WARNING: this is only meant for development, don’t put this in your deployed code base. This is only tested in firefox 3, but it should work in other browsers as well. I think this will only work with the default loader and it will fail if you want to use it in xdomain case. At last, the famous phrase: use at your own risk.

Edit: fixed mangled greater than symbol (thanks slackorama). changed all this->dojo so that no matter what this function is called, it can always correctly unset any “dojo memory”. - December 2, 2008 at 7:51 am

As pointed out by Jon’s blog post, simply use the dojo/dijit objects on the top window within an iframe won’t work for dojo APIs in most cases, because “the top level dojo object has no Out of the Box knowledge of the iframes or their content”.

Actually, dojo has two APIs in the core to support running code in the context of an iframe: dojo.withGlobal and dojo.withDoc. However, they do not have much attention, as most of their usage is only seen in the dijit.Editor code. Actually, they were introduced for the sole purpose of dojo.widget.Editor (which is the dojo 0.4 Editor widget) for dojo 0.4 in the first place.

Both of them have identical signature:

dojo.withGlobal(/*Object*/globalObject, /*Function*/callback,/*Object?*/thisObject, /*Array?*/cbArguments)

In order to use them, you can wrap your code in a function first, like this:

function iframefunc(){
    dojo.byId(‘foo’);
    dijit.byId(‘foo’);
    dojo.query(‘a.foo’);
}

and call the function as this in an iframe:

dojo.withGlobal(window,iframefunc)

You can also call dojo APIs directly use dojo.withGlobal. Say you want to run dojo.byId(‘foo’) in an iframe, you can do:

dojo.withGlobal(window,'byId',dojo,['foo'])

Note that, the above code can also run correctly in the context of the top window.

The differences between dojo.withGlobal and dojo.withDoc is the first argument: if you want to only change document of a function call operates on, you can use dojo.withDoc instead and pass it with the document of an iframe.

As an open source editor, komodo editor is targeted at dynamic programming language editing, which supports javascript, php, python and perl, among others.

One of the features for those supported dynamic languages is auto-generation of a list of available functions defined in a script. However, when editing dojo based javascript files, komodo can not pick up any classes declared using dojo.declare and it does not support dojo.extend either.

Today saw a post from Steve on his blog about ” Fast String Multiplication with Regular Expressions, Exponential Concatenation, and Binary Interpolation“. Basically, it deals with how to fast multiplicate a string n times in javascript.

When I first saw the post, only 3 algorithms were presented. Looking at the 3rd algorithm, I think it can receive some improvement if exponential concatenation is used more efficiently. By the time I implement the improved version, Steve added a 4th one to his post, which is quite similar to what I have here:

function mul5(str, n) {
 	var r=[];
 	while(n){
 		if(n%2)
 			r.push(str);
 		str+=str;
 		n >>=1;
 	}
 	return r.join('');
}

The main difference is that it is more compact. My tests show that, in Firefox and Safari 3.0.4, mul5 is as fast as mul4. However, similar to mul4, mul5 is about 30% slower than mul3 in IE (mult5 performs as good as mul4 in IE).

The reason of slowdown in IE compared to mul3 is this part: str+=str. IE does not handle that direct string concatenation well if the string is quite long. What if we use array.push there as well? After several fail-and-error, I found out one solution which will actually boost the performance for IE:

function mul5(str, n) {
 	var r=[],t=[str];
 	while(n){
 		if(n%2)
 			r.push.apply(r,t);
 		t.push.apply(t,t);
 		n >>=1;
 	}
 	return r.join('');
}

As shown above, the trick is to use array.push.apply(array,anotherArray) to add all elements in anotherArray to array. This IE version is as good as mul3, or even slightly faster than it.

However, this modification will slow down other browsers. We can use conditional compilation support in IE to work around this. The merged version is like this:

function mul5(str, n) {
	var r=[];
	/*@cc_on //for IE
	var t=[str];
	while(n){
		if(n%2)
			r.push.apply(r,t);
		t.push.apply(t,t);
		n >>=1;
	}
	return r.join('');
	@*/
	while(n){
		if(n%2)
			r.push(str);
		str+=str;
		n >>=1;
	}
	return r.join('');
}

If using dojo, the conditional compilation should be changed to just test dojo.isIE. (conditional compilation is not supported by dojo builder, so it will be lost when building dojo)

Try it yourself here. Changes compared to the original one:

1. mul1 and mul2 are slow, so removed from the test
2. the number of repetition and the length of the input str are slightly different from the original one
3. A table is used to present the result, rather than a list