Smart phones are showing up in the poorest of countries. Even the Internet is showing up but it’s still quite expensive. But for a reasonable price we can deploy Wifi based local mesh infrastructures that can let people run applications on their smart phones and communicate locally with people around them. We have the technology! Below I explain what that technology is and why it’s all Serval's fault.
Internet access is getting better, but we have a long way to go
I was reading the 2016 ICT “Measuring the Information Society Report”. It tells a really exciting story (no, seriously). The first place to go is Chart 1.3 and look specifically at Mobile-broadband subscriptions. What this chart shows is that Mobile-broadband subscriptions are exploding everywhere in the world. Furthermore in the case of least developed countries (LDCs) as well as developing countries it absolutely crushes fixed-broadband subscriptions. The world is getting its Internet via phones [A] [A] I have to admit I’m not 100% sure this is right. Looking at the numbers households with a computer in LDCs and developing countries is pretty close to mobile-broadband subscriptions. So it’s possible that people are buying mobile dongles to connect their PCs rather than buying smart phones. and one can reasonably expect that most of those phones are some kind of smart phone since Internet on feature phones is typically so limited as to not be terribly useful.
But there is a problem, especially among LDCs. As we can see in Chart 4.14 the cost of mobile broadband is really high for LDCs and even developing countries (compared to local income). Chart Box 4.1 shows us the threat. It illustrates how in many of the poorest countries the majority of users do not have what are called full cost plans. These are plans that let the user access any part of the Internet, anytime they want. Instead we see a lot of zero rated and service specific plans. A zero rated plan doesn’t cost any money but it only allows access to a tiny handful of sites. A service specific plan generally costs some money but allows access to a larger curated set of sites. These plans do not allow access to the Internet in any reasonable sense of the term. Rather they just use the Internet as a delivery mechanism for the modern equivalent of a closed content provider. This isn’t the Internet. This isn’t the kind of access that will spark new services and grow economies giving opportunities to people all over the world to create value. Only open Internet access can do that.
Opening up local opportunities
Imagine that you live in a village somewhere in an LDC. You have managed to get a smart phone (the penetration rate only goes up) but the cost of actually using it on the Internet is prohibitive. You might have a zero rated or service specific plan or even if you have a full cost plan you can only use it for limited time. In other areas like refugee camps even if you came with a smart phone your ability to use it, especially on the Internet will be limited to non-existent.
More and more the right to global Internet access is being recognized as a human right. But there is a lot of road between here and giving everyone meaningful access to the global Internet. In the meantime we can use the devices in people’s pockets to help build up an infrastructure that can let them at least use the Internet locally with any apps they want to do whatever they want without having to ask a carrier for permission first.
One Wifi Endpoint
Imagine a village with one Wifi endpoint. It may or (most likely may not) be connected to the Internet. But it is connected to some form of power, possibly a solar array, a car battery or even mains power. In the simplest case we can write applications for smart phones that can leverage the Wifi endpoint to enable phone to phone communication. That is, if two people are connected to the same Wifi endpoint then their phones can communicate peer to peer and transfer information. Enabling this scenario is what the Thali Project is all about. We have the technology. But we can do better.
Now add local storage
Let’s imagine that user A and B want to talk but they aren’t at the Wifi endpoint at the same time. If the Wifi endpoint had some form of storage then user A could leave a message that user B could later drop by and pick up. There is nothing particularly fancy here. We are just talking about a Wifi endpoint that has some kind of storage where users can leave (encrypted/identity protected) messages. But we can do even better than that.
Now add multiple wifi endpoints plus low latency meshing
If Wifi endpoints are within range of each other then we can easily mesh them together automatically with no user intervention using software from the Serval Project. Serval supports two different flavors of mesh. One is the usual low latency mesh. In this scenario a user is attached to Wifi endpoint A which is within range of Wifi endpoint B where the other user is. The two users will be able to automatically communicate thanks to Serval arranging a connection between them. And yes, this all works with an arbitrary number of Wifi endpoints meshed in arbitrary ways. But still, we can do better.
Now add store and forward meshing
The other flavor of mesh Serval supports is store and forward meshing. So a user could leave a message at Wifi endpoint A and later on the other user could come to Wifi endpoint B and retrieve the message in one of two ways. Either Wifi endpoint B would connect to Wifi endpoint A and retrieve the stored message. Or, Serval supports oracles that can automatically push data around. In that case Wifi endpoint B might have preemptively retrieved messages on Wifi endpoint A and stored them locally. This later ability is important because connectivity can be very unreliable. For example, imagine that Wifi endpoints A and B are on opposite sides of a road. During the night and afternoon the road might not be used much but in the morning and evening it can become very busy. Depending on the height of the Wifi endpoints they might lose contact with each other during the busy times. So having logic that can detect and try to compensate for this behavior is important.
So now we have an ability to automatically create and maintain a local Internet using nothing but Wifi endpoints. But yes, we can do better! But first we need to discuss cost.
How many Wifi endpoints do we need?
In looking up Municipal Wifi I typically see estimates of 25 endpoints per square mile. This works out to each Wifi endpoint having a range (read: radius) of 600 feet or so. With a high gain antenna and assuming some reasonable installers 600 feet isn’t unreasonable for Wifi.
But we are looking to use reasonably cheap hardware and antennas (say 3db omni directional) and those will usually not get much beyond 300 feet outdoors in a “real world” environment. So as I previously calculated this would require 138 Wifi endpoints.
In reality we don’t expect to provide complete cover. The expectation is that we would have hot spots and then try to lay out the Wifi endpoints so they form arms to extend out the mesh. But still, we are talking about a lot of Wifi endpoints.
And yes, we can do better!
Now add 900 Mhz radios
For around $80 we can add 900 MHz hardware like the RFD 900 to our Wifi endpoint. 900 MHz radios can support up to 250 kbit/sec. Not screaming fast but not horrible. But what is really powerful about 900 MHz is that its lower frequency lets it go for much longer distances. Depending on how much one wants to spend on the antenna ranges can go crazy distances (many miles) but using low dBi omni directional antennas I’m told that outdoor ranges tend to be under a mile. Just to be safe lets call it 0.25 miles. In that case we could cover a square mile with 5 900 Mhz radios.
Or not.
There’s a catch actually.
The catch is that the 900 Mhz radios are sharing the 250 Kbps bandwidth. So if there are say 10 transmitters then each would in theory get 25 Kbps. The Serval team has run some experiments with a simple Aloha style protocol and they found that in practice if there are 10 transmitters within range of each other who are all trying to chat then each gets an effective bandwidth of around 256 bps. Yes, bps not Kbps.
To avoid this awful fate we have to use directional antennas to set up point to point links. Which means we need directional antennas and we have to have someone on site who decides where to point the antennas, configures the channels and fixes things when units get broken, antennas get mis-directed, etc.
But even directional antennas aren’t a cure all. The RFD 900 for example only supports a single channel at a time. So if endpoint A and B are configured to talk then they are both sharing that single channel which means each is getting less than 250 kbps since it’s a single shared channel.
However we can potentially fix that. If we used cheaper radios than the RFD 900 we could afford two 900 Mhz radios per endpoint. In which case our range would be reduced (the radios are cheaper for a reason) but we could run two channels. One to send and one to receive. That way each point to point link would have the full 250 Kpbs bandwidth.
Still, there is nothing particularly wonderful about 250 Kbps. Moving 1 Megabyte of data over a 250 Kbps link takes 33 seconds! And of course all these channel configurations and directional antennas mean we are no longer in a “fire and forget” mode. People are going to have to maintain the system which reduces the probability of success.
My guess is that it makes sense to sprinkle at least some 900 Mhz radios in with the other radios. If we can train anyone on site to maintain the things it will significantly increase the range of the mesh and make it much more resistant to interference. And if we can’t get anyone on site to configure things then we can at least create a reasonably long range emergency communication network.
Now, after this downer, let’s see if we can’t do a lot better.
Now add Whitespace Wifi
Let’s forget 256 bps or even 250 Kbps. What about 40 Mbs? And forget maybe 1/2 a mile. How about 1 - 10 miles with excellent multipath support (e.g. signals can bounce of things like buildings)? How’s that for a mesh backbone? Well it exists and it’s called Whitespace Wifi. It’s not legal everywhere and it is still silly expensive at around $1000 for a pack of three units. But it’s still very workable for our scenarios. Just those three units could cross connect villages over extended distances and help the mesh to expand very quickly.
Who is going to pay for all of this?
So I was reading BRCK's blog and they mentioned the Kakuma Refugee Camp in Kenya. This led me to find a map of the camp. The camp looks to be roughly 13 km (8 miles) long and 6 km (4 Miles) wide. This actually overstates the camp’s size by a lot because there is a big bulge at the top and a long part at the bottom. But let’s go with that rough size.
Just to build a spine that runs the length of the camp would require 8*5280/520 = 81 Wifi endpoints. But this is where the 900 Mhz radios really come in handy. With an effective range of 0.25 of a mile we can reduce that number down to 8*5280/2286=18. Again, if we have directional antennas and someone to maintain them.
And a camp like this which is long and thin is a picture perfect scenario for Whitespace Wifi. It even appears that Whitespace Wifi is legal in Kenya. With Whitespace Wifi we can easily connect the entire camp long range with three units ($1000 or so) and then fill in hot spots with Wifi units.
The population of the camp is 163,192 but let’s take the statement in the blog article that 70% of the boys in the center had never used smartphones or tablets before. So let’s say that 30% of the camp have smart phones (I know, that is ridiculously high, but bear with me). We can reasonably assume there is no more than 1 smart phone per family. I found on the UNHCR site a statement that of 40,692 Somali refugees there were 9,738 households then if we assume the same ratio for the general population then the camp should have 163192*9738/40692 = 39,053 households. Of which 0.3 * 39,053 = 11,716 we are positing have smart phones. If we assume a load per Wifi endpoint of 100 phones then we need 117 Wifi endpoints just to support the population. Let’s round that up to 200 Wifi endpoints to cover the unexpected.
So tricking out a camp like this would cost 200 Wifi endpoints * $300/Wifi endpoints + $1000 to boost with Whitespace Wifi = $61,000.
That is the cost for a single camp and yes, we can probably make that happen through a number of different philanthropy sources.
But of course I’m missing the real point aren’t I? The cost isn’t just the hardware. Where is the electricity going to come from?
Refugee camps don’t have regular supplies of electricity. In fact here is a picture I found of a business in the camp that sells electricity in the Kakuma Refugee Camp. And in fact the UNHCR’s own guide says that their field offices in the camp don’t have electricity.
There are also other ugly issues to think carefully about. For example, how does one prevent theft of the Wifi units? They are worth serious money.
My suspicion is that in a realistic scenario we would probably be working with someone like the UNHCR to provide Internet access to their field offices throughout the camp and creating hot spots in the camp. This would require fewer units but would also require paying for some form of electricity. Either generators or even solar cells if we are just focused on the Wifi equipment.
But one suspects that a smaller foot print with camp wide coverage (which Whitespace Wifi can easily handle) plus power will cost around the same $60,000.
There is no magic. We can’t wave a wand and turn a refugee camp into a first world city. But we do have the technology to start making a difference and the beauty of this approach is that it’s incremental. We can start with one field office and then add another and another. We don’t have to come up with all the money at once and the cost “step function” (e.g. the cost of adding a little more capacity) is small. This is compared to a cellular infrastructure where the minimum cost of each tower once one considers cabling, power, hardware, etc. is going to be in the millions and each new tower costs as much.
APPENDIX - What about Wifi Direct or BLE or Bluetooth or other local radio technologies?
In theory we could write apps that could let people communicate without even the Wifi endpoint. Phones have lots of radios and in theory we can use those radios to allow handsets to communicate point to point without any external infrastructure like the Wifi endpoints. We have been working hard with these technologies for years as part of the Thali project and the results are not promising.
In iOS land multi-peer connectivity framework (MPCF) does appear to work reasonably well. We can usually get around 10 Mbps out of it which is more than workable. But not many people in LDCs or even developing countries are running around with iPhones. And MPCF only works with other iOS devices, not Android phones. Given the dominance of Android phones this makes the technology less than useful for our scenario.
In Android land there are a number of peer to peer radio technologies we could use. They include Wifi Direct, BLE and Bluetooth.
We started Thali trying to use Wifi Direct discovery since it’s supported on just about all Android phones and we gave up. It often doesn’t work between two phones from the same manufacturer much less with phones from different manufacturers.
This lead us to think about Bluetooth discovery but we gave up on that too because Bluetooth discovery is energy intensive and is based on a poling framework that doesn’t scale well. It was designed for a device that pairs with a small group of other devices. Not for building a general peer to peer infrastructure.
So then we were left with BLE. The good news is that BLE works reasonably well. The bad news is that to use BLE we really need to be both a peripheral and central at the same time which is only supported starting in Bluetooth 4.2 hardware. All flag ship and most high end phones support Bluetooth 4.2 but many mid range and lower range phones do not. So our reliance on it means we basically wrote off the majority of existing Android phones. In another 4-5 years this probably won’t matter but for today it’s a problem.
However BLE is designed for low bandwidth connections. It’s not intended for moving around serious data.
For moving around serious data we have two options, Wifi Direct or Bluetooth.
While Wifi Direct discovery doesn’t seem to work, Wifi Direct group capabilities (the part that lets phones connect and move data) does seem to work [B] [B] Although we haven’t done enough testing to really be sure. We thought Bluetooth worked o.k. until we really started to use it and found the endless problems.. But there are problems. First off, Android requires a user dialog to confirm joining a Wifi Direct group. This means we can’t run in the background, it means we have to have our users deal with a system presented dialog that means nothing to them and we haven’t figured out if there is a way to try again if the user accidentally hits no on the permission dialog. Another alternative is to create a Wifi Direct group and then connect to it using Wifi Infrastructure mode. The motivation for doing this is that there are no idiotic dialogues when hijacking the phone’s entire Internet infrastructure, only when using a Wifi Direct group (you can’t make this stuff up). But this has obvious security problems. It also complicates the living day lights out of the architecture. Imagine we have four phones next to each other that all want to chat. Each phone can only connect to a single other phone via Wifi Infrastructure mode. But one phone can be connected to by multiple other phones. So we need to use BLE to come up with some kind of leader election algorithm. It’s all doable but it’s a pain.
So the route we went (unfortunately) was Bluetooth. While Bluetooth discovery is a mess, using Bluetooth for moving data via RFCOMM is pretty straight forward and it doesn’t have any of the security or architecture issues that Wifi Direct over Infrastructure mode has.
Unfortunately it doesn’t work worth a damn.
Some phones, like the Samsung Galaxy S7 seem to do a reasonably good job of supporting RFCOMM. But even it runs about 11x slower than Bluetooth’s theoretical maximum bandwidth of 3 mb/s and that’s if we turn off the other 2.4 GHz radio users like Wifi and BLE. The Nexus 6P, which was Google’s flagship phone when we ran the tests, runs 38x slower! It’s just completely nuts. And even then random inexplicable failures of Bluetooth are common. It’s a mess.
In theory if we could get phone manufacturers to give a damn about actually testing their radios and get Google to remove the insane permission dialog for Wifi Direct then all of these problems are solvable. But nobody seems interested in solving them today. So for some of Thali’s industrial scenarios we still care about BLE/Bluetooth because we can potentially get phones specifically designed and/or tested to meet Thali’s needs. But for a scenario like the one described in this article we have to work on everything and for today that means normal Wifi, not peer to peer radios.
“Imagine we have four phones next to each other that all want to chat. Each phone can only connect to a single other phone via Wifi Infrastructure mode. But one phone can be connected to by multiple other phones. So we need to use BLE to come up with some kind of leader election algorithm. It’s all doable but it’s a pain.”
We have actually solved this in our P2P stack (https://github.com/croconaut/cpt). It was a HUGE pain but we have both the Wi-Fi discovery and connectivity working reasonably well. We’re still deciding whether to open source it or not but if you’d be interested to try it out and adopt it afterwards, it would definitely encourage us in that decision. ;)
I’d love to chat. You can reach me at yarong @ [insert enormous corporation whose name rhymes with icrosoft here].com. BLE works pretty well for discovery. So for us the main issue is connectivity. Bluetooth is a screaming nightmare on Android.
For those interested in even more gory information about why using local radios on Android for P2P doesn’t work well please see http://thaliproject.org/androidWirelessIssues/
Yep, we’ve been there, incl. experiments with Bluetooth (fortunately I didn’t know about BLE at that time). A few comments:
“phone type ID and a phone identifier (that we can’t set programmatically)” — you can but only on non-Samsung (and perhaps others?) devices, using an undocumented API call.
“The Wi-Fi Direct Service Discovery stack had a habit of just stopping to work after some period of time. And once it stopped working the only way to get it working against was to reboot the phone” — this is because you’ve already messed up the stack. If you do clean discoveries, the maximum I had to ever do is to restart Wi-Fi alone.
“We encountered random disruptions in the working of the main Wi-Fi stack when we used Wi-Fi Direct Service discovery for any extended period of time. This caused havoc in our test infrastructure where we use Wi-Fi to coordinate tests.” — what a clean discovery means is part of my hard-earned know-how ;)
“We could play a nasty little trick. What we could do is have one phone create a Wi-Fi Direct group” — this is what we’ve ended up with. It’s comfortable, easy, reliable, mixing Wi-Fi Direct and STA plays well together.
“When the second phone connects to the first phone as a Wi-Fi Access Point all communication going over Wi-Fi on the second phone, including other applications, will now go to the first phone” — this is even worse, when the first phone has enabled mobile data, you can literally drain whole data plan using this. So the AP must have a flag like “don’t connect to me, I will connect to you” (and if both are online = on mobile data, use an application server for communication). Another option is to disable mobile data for a while (using an undocumented API call)
Setting Phone ID in Wifi Direct – Yeah, hidden APIs are things we do when we have no choice, but it’s not something we can rely on. What do we do for phones that don’t support the hidden API? And since Samsung is dominant amongst Android handset manufacturers an approach that doesn’t work there isn’t terribly useful to us. :(
Wi-Fi Direct Service Discovery – Yup, I have no idea what clean discovery means. But if you care to share I’d be happy to learn. :)
Mobile Data Attack – It’s scenarios like this that made the AP hotspot approach less than desirable.
Honestly at this point I’m ready to just give up on local P2P for Android and instead try to rely on an external device. Something like https://thenextweb.com/insider/2016/01/15/ocean-is-an-amazing-battery-powered-wireless-server-that-fits-in-your-pocket/ would have been perfect but it doesn’t seem to exist anymore.
I’m going to send you an email. There are other cool HW solutions, like this one: http://www.gotenna.com/ … frankly, at some point I’ve come to the same conclusion, mobile phones are very unreliable, you can hardly control what’s happening and let’s face it, if you can have several kilometers with a hw antenna, why would you bother with couple of 100 meters.
Just for other readers, we discussed Gotenna in email and I directed Miro to http://www.goland.org/thalimesh/#toc-Subsection-1.10 which explains that Gotenna is really slow, a closed platform and can’t legally form meshes. Nevertheless I do agree that this is a form factor that could perhaps solve the problems we keep running into. What’s so sad is that the problems we see with Android radios are clearly bugs. They could be fixed if anyone gave a damn.