Asking about a system’s performance is usually an invitation to a fight, not to useful data. But I did want some idea of how JXCore, a fork of node.js that runs on mobile OS’s, performed on iOS and Android. See here for more info on JXCore. Since the work I’m doing heavily involves PouchDB I decided to take the PouchDB performance tests and run them in six different environments and compare the results. Over all, one should expect node.js on a modern phone (Android or iOS) to be around 10x slower than on a PC. Which honestly, isn’t that bad when you think about it. I want to give a shout out to Brian Lambert for making the iOS tests happen and to Oguz Bastemur from Nubisa for helping us past various bugs. Oguz was very patient with Brian and I and we really appreciate it.
1 The machines
For the tests Brian and I used the following machines:
- A Lenovo T440P laptop. This is an Intel core i7-4900MQ @ 2.80 Ghz. It has 4 physical cores and 8 virtual cores thanks to hyper threading. It also has 16 Gigs of RAM.
- A Samsung Galaxy S4 running Android 4.3 with Quad-core 1.6 Ghz Cortex-A15 and 2 Gigs of RAM.
- An iPhone 5 running iOS 8 with a Dual Core 1.3 Ghz Swift (ARM v7) processor and 1 Gig of RAM.
- An iPhone 6 plus running iOS 8 with a Dual-core 1.4 GHz Cyclone (ARM v8) processor and 1 Gig of RAM.
2 The test setup
The test was to run the PouchDB perf tests using an in memory database. I eventually will want to run these tests against LevelDB but for now I didn’t want to deal with all the hassle of getting LevelDB to work against LevelDown as a JXCore native add-on. So instead I used memDown. This means that the tests are really just testing the Javascript engine and libuv, e.g. the stuff node.js does. It’s not testing the IO bus to local storage. We’ll get that covered when we put in LevelDB support.
2.1 Running the test on Node.js on a PC
- Run “npm install express pouchdb memdown express-pouchdb”
- CD’d into node_modules/pouchdb and ran “npm install” (this puts in the dev dependencies)
- Went inside of pouchdb/tests/performance/index.js and replaced
var PouchDB = process.browser ? window.PouchDB : require(’../..’);
with
var PouchDB = require(’../..’).defaults({db: require("memdown")});
I then created the following as a node.js file and run it with node:
var express = require("express");
var pouchDBApp = express();
var PouchDB = require("pouchdb");
var InMemPouchDB = PouchDB.defaults({db: require("memdown")});
// Set up our test server
pouchDBApp.use("/", require("express-pouchdb")(InMemPouchDB, {mode: "minimumForPouchDB"}));
// Node tests by default will look at http://localhost:5984
pouchDBApp.listen(5984);
require(’pouchdb/tests/performance/index.js’);
2.2 Running the test on Android
I actually ran the test against two different node.js for Android code bases. One is just plain old node.js and v8 run along the lines described here. The other is JXCore. You can get the JXCore project here. Make sure to read the readme. I actually ran the JXCore project twice. Once using SpiderMonkey and once using V8.
Note that a bug in JXCore causes the Android perf numbers to be worse than what they will eventually be. The issue is that JXCore can run in two modes on Android. In one mode JXCore can run on the main thread but you have to keep calling it to run a loop. In the other mode JXCore just spawns a thread and lets the node.js code run to its heart’s content in that thread. The problem is that right now there is a bug that keeps the “thread” mode from properly seeing Node.js packages and so require() doesn’t work right. So for this test we just hammer the “loop” API and this adds overhead. Once the require bug is fixed we should see even better perf.
2.3 Running the test on iOS
You can see Brian’s code here. A similar bug to the require() bug described in the Android section also applied to iOS. But thankfully there was a quick and easy work around and so the perf one sees here is an honest picture of what one should expect on an iOS device. Also note that we can only run SpiderMonkey on iOS (no v8 support) and that when running SpiderMonkey on iOS we can’t use IonMonkey, the Firefox JIT’er. So in general the performance on iOS should not be as good as an equivalent Android device.
3 The test results
I provide the raw results in a table below but honestly I don’t think the actual timings are all that interesting. To me what’s more interesting is the relative performance of node.js on mobile versus the PC. Below is a totally unscientific average of the relative performance of the various PouchDB tests versus PC.
| Platform | Relative Average Performance to PC |
| Node.js/V8 on Android | 12x |
| JXCore/SpiderMonkey on Android | 31x |
| JXCore/V8 on Android | 14x |
| JXCore/SpiderMonkey on iPhone 5 | 26x |
| JXCore/SpiderMonkey on iPhone 6 Plus | 13x |
Take these results with a very large grain of salt but they give a sense for how the perf looks. JXCore, with v8, on Android does a reasonable job of keeping up with plain ole v8 on Android. And JXCore’s numbers should get better once the require bug is fixed. But in either case running node.js on your Android phone is likely to be about 10x slower than a PC. Although to be fair I’m using a pretty old Android handset. I’d love to see the numbers on a Samsung Galaxy S5 or even better the upcoming S6. That would be a fairer comparison to the iPhone 6 Plus.
iPhone 5 is going to be really painful. SpiderMonkey without a JIT’er isn’t a pretty sight. So we should be looking for a 30x slowdown and I’m trying not to even think about what an iPhone 4 will look like. The iPhone 6, again without JIT, is on par with a two generation old Android handset. Given the JIT limitation that’s actually pretty impressive to me. So the future is bright for node.js on iOS but the path to get there could be ugly.
Here are the raw measurements. All numbers are in ms.
| Test name | Node.js/V8 on PC | Node.js/V8 on Android | JXCore/SpiderMonkey on Android | JXCore/V8 on Android | JXCore/SpiderMonkey on iOS (iPhone 5) | JXCore/SpiderMonkey on iOS (iPhone 6 Plus) |
| basic-inserts | 314 | 3167 | 6091 | 3572 | 4888 | 2776 |
| bulk-inserts | 858 | 12241 | 24427 | 11595 | 22396 | 10637 |
| basic-gets | 439 | 6914 | 13833 | 6487 | 13581 | 6869 |
| all-docs-skip-limit | 2449 | 40674 | 83278 | 47646 | 101229 | 46247 |
| all-docs-startkey-endkey | 157 | 1940 | 5845 | 2293 | 5573 | 2529 |
| pull-replication-one-generation | 459 | 6845 | 23896 | 7901 | 14022 | 6656 |
| pull-replication-two-generation | 1484 | 19551 | 45959 | 22277 | 30288 | 15322 |
| temp-views | 1360 | 16498 | 36803 | 18215 | 31513 | 14968 |
| persisted-views | 33 | 376 | 1405 | 443 | 1039 | 477 |
| persisted-views-stale-ok | 28 | 271 | 823 | 294 | 695 | 328 |
| basic-attachments | 1161 | 8124 | 16022 | 9381 | 12246 | 6285 |
I was trying to track a ‘possible’ multi-core performance difference between an ARM and Intel processor. Found (a bit old but can be useful) http://www.computingcompendium.com/p/arm-vs-intel-benchmarks.html . Seems there is up to 8X difference on multi-core performance which is very important for this benchmark result (io threads, tasking thread, … )
Considering 10X-12X ratio given by the article and 8X loss due to CPU perf, we still have a room for ~30% gain here.
Thanks for all these hard work.
BTW; using ‘Desktop’ PouchDB perf tests on ‘mobile’ devices was something made me ‘scared’ when I heard it at first. I’m so happy JXcore is mature enough to handle this task literally on mobile devices + different platforms. We suffered from the mobile interface bugs though, but hey, we are working very hard on JXcore to make it stable on mobile and we hopefully will! :)
I was super happy to see JXCore ‘just work’ with the test once we got past a few packaging bugs.
As for perf, our scenarios are particularly well suited to a multi-core CPU. Our hope is to use LevelDB which is multi-thread safe but not multi-process safe. So in theory we should be able to use a LevelDB native plug in while running multiple JXCore threads and let them share the load. This is not something we could do with Node.js because with Node we would have to run in separate processes and LevelDB wouldn’t work there. Even if we put in mutexes we would still have a bunch of extra overhead due to LevelDB’s caching. To make this work on Node.js we would have to do something like IPC. With JXCore though it should ‘just work’.
What worries me is the iPhone. Using the numbers from we have:
iPhone 4 – A4 – 200 – 200
iPhone 4S – A5 – 210 – 400
iPhone 5 – A6 – 700 – 1200
iPhone 5S – A7 – 1400 – 2500
iPhone 6 & 6 plus – 1620 – 2900
In our tests the iPhone 5 was roughly 1/2 the speed of the iPhone 6. Which the numbers above more or less supports. Which means that an iPhone 4 or 4s is going to be somewhere around 8x slower than an iPhone 6! Eeek! I wish I knew what the iPhone 4/4s market share is. Can we get an iPhone 5 to the point where it’s “good enough” or do we have to get things to work on the iPhone 4? I honestly don’t know.
I believe ‘4s’ is no longer a problem. I could not find anything that I can certainly say this or that but seems like ~30% wouldn’t be surprising. (for 4S and below)
I said it’s not a problem because 4S has still enough beef to run many jxcore app scenarios. For example, ‘basic-inserts’ test inserts 1000 docs. into db and queries back 10 times (100 docs each time) This test takes aprox 2800 ms on iPhone 6. (~350 docs second)
Lets say iPhone 4s does the same thing for ~45 docs second. It’s amazingly fast for a single user application. Why? The server (supposed to host tens of thousands only 80X faster than iPhone 4S).
It’s also possible to come up with a cross argument indeed. Still majority of the scenarios will be fitting into 4S’s capabilities perfectly. And the best thing is, number of active 4S users will be continuing to drop significantly.