Friday, May 15, 2009

Browser javascript performance vs. Adoption

There was an article on Slashdot today about how browser javascript speed seemed to be inversely correlated to the adoption of the browser. The core of the article was this press release from Futuremark.

Sadly lacking was a hands-on analysis of the numbers. The sample size is pretty small (5 browsers), but still, some analysis would still be called for, so here goes:

First a plot of the data set[1].



At a glance, it doesn't look terribly correlated, but, there is a hidden outlier here. If you remove the third sample, Opera, you get a plot that looks more like a smooth function.



Indeed, if you take the logarithm of this dataset, you find an almost straight line (!)



If you look at the logarithm of the full dataset, it's pretty evident that something is wonky about the Opera point.



For the reduced data set, a linear regression of Performance vs. log(Adoption) as a+b*x yields

a = 0.619
b = -0.00447

A statistical analysis indicates that the probability for such a strong correlation by random chance is 0.062%. The standard deviation is 0.079, and the coefficient of determination R2 is 0.999. For the full data set, R2 is 0.89.

So, the adoption as a function of performance in the benchmark would according to this hypothesis be

Adoption(Performance) = e0.619 - 0.00447*Performance


Comparing this with the data sets, it appears very likely that the hypothesis is true.



Of course, the data set is very small, so it's plausible that this is just a random coincidence, the same way one may get a descending sequence of numbers when throwing a die 4 times without some physical rule dictating that dice only generate descending sequences of numbers.

Some final thoughts: Even though correlation is not causation, my guess is that more popular browsers, being older, need more maintenance. Their large user base will also lead to more time being spent patching security flaws. On the other hand, the smaller upstarts need a competitive edge to stand a chance against the bigwigs, motivating the developers to create faster code.

As for Opera, you can either interpret them as having unusually small adoption for it's speed, or being unusually slow for it's tiny adoption. The former makes more sense than the latter. A factor that should be taken into consideration is that almost all the other browsers either have massive campaigns surrounding them, or ship as default with some operating system. So maybe Opera is simply under-advertised?



The analysis was performed with the statistical functions of a TI-84+, and the plots were generated with gnuplot.

[1] You can find the actual numbers in the press release, I don't want to steal their thunder.

2 comments:

  1. I'm not sure the count of visitors to the FutureMark site is a good proxy for browser popularity. How does JS performance correlate to data from NetApps for instance?

    ReplyDelete
  2. you definitely prove the point about "lies, big lies and statistics" ;)

    ReplyDelete