So the idea is to train people to write code that is useful to them locally. If we have the Wifi mesh infrastructure we previously discussed then this provides an infrastructure that applications can run over to enable collaboration and communication. And none of this precludes access to the Internet. The apps should be able to leverage the Internet when it is available but remain fully functional when it’s not. This is beyond the “Offline First” model. This is the “always available” model.

So if I want to exchange a status update with someone who is reachable via my mesh then I should be able to do so even if I’ve run out of Internet minutes or don’t have any Internet access at all.

This is where Thali comes in. It provides a peer to peer infrastructure that enables applications to exchange information over the Wifi endpoints directly without any Internet in a secure/authenticated/authorized manner.

But if we are going to train people to write Thali apps we have to figure out what their development environment is. The first thing it that when the developer is writing code we have to assume they are not connected to the Internet. Even if there is Internet, the bandwidth is likely to be incredibly limited. So however we plan on having developers we train in LDCs write code for Thali, all the tools and infrastructure they need will have to show up on a thumb drive.

Right now my working assumption is that any training done in villages, towns, etc. in LDCs will be done by people who will show up with some basic equipment. They will have the aforementioned thumb drive pre-loaded with all the development tools needed to build a Thali application. They will also show up with some cheap PC hardware to enable people to write code with a keyboard and some kind of reasonably sized monitor. The assumption is that the students will get to keep the PCs when they finish the course.

Yes, we could enable Thali development right on the phone itself. It is doable. The front end is Javascript/HTML/JS which is all interpreted and the back end is Node which is interpreted. So we could put an editor on the phone and let people just edit the code of an app they side loaded on their Android phone. That way we wouldn’t need those PCs. But writing code on a phone isn’t easy and while I suspect we’ll eventually go there I think we should start with the PC.

So as a practical question the challenge is - what’s on the thumb drive?

This is a variant on the “what books would you take if you were going to be stranded on a desert island?” question.

This model isn’t sustainable because for one thing the Thali build environment is MacOS only and “cheap PC” and “MacOS” don’t go together. In theory the build system should work just fine on Windows and Linux but it isn’t tested very often there.

So the challenge is - do we fix the install system for Thali so it can be installed as it was intended, as a NPM module into a Cordova project? Or do we just build a Cordova skeleton with the Thali bits already installed (we are well set up to do this) and put that on the thumb drive?

For example, we could ship an APK that is a Thali app that just has some tiny default content and a predefined key. We could then enable the dev to take over the app and swap out our content with their own HTML/Javascript/CSS and node_modules. That way they would never need to compile an Android app. But this brings in issues. For example, how do they distribute the app? These problems are solvable of course (here’s a hint, Thali’s heart is PouchDB, a sync engine).

But I can’t avoid thinking that the closer we stay to the chrome the less things are likely to blow up. I also like the idea of empowering the students to do anything they want, not just what we let them.

So starting with Android Studio and the Android SDK seems like a good idea.

The next challenge is - how should the developer edit HTML/Javascript/CSS? What about Node?

There are a ton of open source IDEs that support HTML/Javascript/CSS and Node. Which is best? I honestly don’t know. I can’t stand front end programming so I mostly just need a Node editor and I use IntelliJ and VSCode for that. But whatever. We need to pick something, what should it be?

And keep in mind that this all has to work while the user is 100% offline. So, for example, NPM really, really wants to connect to the net. Thankfully there are work arounds that we already have experience with for other reasons like Sinopia. But are there any other gotchas hiding here with these IDEs that will make them non-functional if they are offline?

What cordova plugins should we include?

What about NPM? NPM libraries tend to be reasonably small. Maybe we include the top 100 libraries? What about their docs? We would need to automate pulling them in.

For example, let’s say a developer wants to make a social networking app. Do we really want everyone that a user trusts able to sync all records on that user’s phone from all other user’s the developer trusts?

Or let’s say that a developer wants to make a market application to let people advertise items or services they want to sell/trade. With the way things work now someone could edit someone else’s ad!

We have to have a more sophisticated model. We’ve known about this problem for a long time in Thali but have been ignoring it. Now we can’t. See here for lots more details.

Solving this problem does go back to our previous questions about how we want to build a Thali app. If we empower people with Android Studio then they would be building a stand alone Android APK which can be distributed. The good news is that once some hidden settings are flipped on a phone it can just download and install an APK from anywhere. So if we are going with APKs then we need to think about where the APK is hosted. Do we post it and download it to the Serval infrastructure? Does this mean we run HTTP servers on the serval endpoints? If so then we have to hook those servers into Serval’s mesh infrastructure so the app imagines can be moved around by Rhizome.

Another alternative is that people can exchange apps directly via the Wifi endpoint. In which case the HTTP server would be on the phone and we already have one of those. But it means that people can only get the apps from someone with the app instead of posting it to Serval infrastructure. Or maybe we do both?

But now how do we update the APK? I’m not even sure if the APK system is smart enough if we side load to recognize that an APK is a new version of an older APK it already has. We need to test that.

Another alternative is to use a system like F-Droid. It is an open source project that is intended to allow for the managed distribution of applications on Android. It supports local Wifi. It understands how to authenticate apps. It understands versioning/updates. But now we are requiring users to download this one app, F-Droid, in order to get a store that can be used locally to get all the other apps. Is that too big a barrier to entry? Also how do we hook F-Droid up to Serval’s infrastructure? Keep in in mind that the Serval Mesh Extender endpoints are really just OpenWRT Linux. So we can do things like run a HTTP server there and possibly connect that server to Rhizome so that F-Droid sees something it might recognize. I suspect we would still have to teach F-Droid how to look for these HTTP servers not to mention hooking them up to Rhizome.

A third option is that we have a single Thali application and inside of it are basically sub-applications, each one contains the Javascript/CS/HTML files for that specific app. Now we can distribute apps as just synching. But this brings up some very ugly security issues since we don’t have good ways to isolate node apps in particular from each other. Do we really want to create our own sand box system or use Android’s system which is based on having separate apps? Btw, even if we decide to make the apps into sub-apps and use sync to distribute there are still real challenges. What happens if we are getting an application update and the sync fails 1/2 way in? Now the existing app as it exists on our phone is in a 1/2 baked state. We can play versioning games with folders to fix this but then we need a way to validate if a folder is complete (easy enough btw) and we need to eat up extra space. Still really doable but we should be aware of the challenge.

There, having read that now we can talk some details. The first issue is - do we try to use IP or Rhizome over Serval?

If we knew we were building a tiny mesh I would be tempted to use IP bridged over MDP/MSP. Right now, for example, if one sends a multicast UDP packet it won’t be forwarded out over the IP to MDP bridge but we could fix that. But I think the IP approach is a mistake. The problem is that Thali’s existing discovery mechanism (using beacons) was strictly intended for point to point hyper local discovery and/or discovery that we knew would only involve a relatively small number of devices. But that isn’t the case here. We are intentionally trying to build an infrastructure that will scale to at least the hundreds of thousands of people (eventually).

To scale to that level we can’t use a discovery mechanism, such as beacons, which depends on everyone advertising who they are looking for and then expecting every person in the system to examine every advertisement (including multiple beacons and lots of expensive crypto) to see if the advertisement is for them.

So we have two (not necessarily mutually exclusive) choices. One choice is to focus on low latency meshing and adapt a system like Serval’s Cooee to let us discover whenever someone we want to talk to is currently available. We would need to play some tricks so that Cooee doesn’t turn into a way to discover people’s location. For example, if everyone has a permanent ID then when I want to find someone I can try to send them a message via Cooee and see where it goes. A much better solution is to have a constantly rotating ID. But then when I want to find someone I have to know their current ID which requires a separate discovery mechanism. This is all doable but I would argue that we would be better off focusing on store and forward. Other than for real time audio/video we can use Serval’s store and forward service Rhizome to meet all out needs.

The real question is - how do we make Rhizome work? There are two questions we have to answer.

First, how do we address Rhizome packages?

Second, how do we hook up Thali to Rhizome?

For the first question I suggest for the moment we stick with simplicity even though it is not secure and we will have to do better in the future. For now I think packages should contain the public key of the sender and receiver. We can even include a beacon to prove that the sender is who they say they are and that the receiver field wasn’t altered and can produce the session key used to encrypt the content. This isn’t a good long term solution because it exposes too much traffic analysis information but it’s a good place to start.

For the second question we need to change how we think about PouchDB. Normally we just focus on real time data synchronization. That is, after all, what CouchDB’s protocol (the one PouchDB uses) is designed for. But that doesn’t work in a store and forward mesh like Serval. So instead what we have to do is switch to a stream based sync protocol.

Imagine that user A updates their local PouchDB with some content that is addressed to user B. For simplicity sake let’s assume that user A has never talked to user B before (at least in the context of the current application). In that case user A would run a filter on their PouchDB change log starting at the first record (since they haven’t talked to user B before) and filtering out any records that user B doesn’t have permission to see. The result will probably be the single record that user A wanted to send to user B. We would then output this record (and any associated attachments), combine it with a header containing a (probably encrypted) change ID (e.g. what record in the change log this lookup got to) and this would then be encrypted, signed and stuck into a Rhizome package with a manifest identifying user A as the sender and B as the receiver.

Later on when user B is looking for Rhizome packages whose manifest are addressed to them they will pull down user A’s package and after validating it then would replay its contents against their local DB. Once this is done user B would create a new Rhizome package addressed to user A that just confirms they received and executed the content. That confirmation package from B to A would include the change ID that user A sent in their package.

When user A gets the confirmation package then user A would record that user B has confirmed it has received content up to that change ID. That way in the future when user A has new content they will only need to send content from the change ID that user B confirmed.

This begs the obvious question - what happens if user A sent a package to user B starting at change ID 0, doesn’t get a confirmation and now wants to send new information to user B? In theory we could use the versioning capability of Rhizome packages to replace the original package to B with a new package to B containing the updated information. We could even be really smart about this. For example, we could include the original package content (so we don’t have to recalculate it) and then append our new content. But this would minimally require downloading the original content again which isn’t cheap or easy. And it brings up questions regarding the package format, how does it handle these kind of multipart messages? And how do we secure it all? So more likely is that A will output a new package with a reference to the previous package specifying they have to be run in order.

Now Rhizome package delivery is best effort. It’s perfectly possible that the first package that A sent to B got lost and that the second package arrives. In that case B has some options. One option is to play the content in the second package. It will be incomplete of course but it could still be useful. But equally likely is that B has to generate a package back to A telling A that B hasn’t been able to find the first package so A needs to try again.

And yes, this is messy. And yes it’s going to require a careful state machine and some non-trivial code. But it isn’t brain surgery. If we get the states right then we will be fine and the hard part will be deciding on optimizations once we have the basics right.

Another fun question is - do we turn off Thali’s Wifi discovery/CouchDB support if we are using Serval? My guess is that the answer is no. If someone we want to exchange information with is right there on the same endpoint there is no need to push everything through Rhizome. But it is an interesting question.

And wait, it gets better. How many identities does the user have? Do they have a different identity for every app? Because if they don’t then we have to share a user’s private key between apps in order to let them use the same identity across apps. No problems there right. :)

In addition there is a resource issue here. If one has two or three Thali apps on a phone then are they all running Node in the background, pinging the local Serval mesh extender end points? That is probably not ideal from a battery perspective.

In Thali we have always talked about the Thali Device Hub (TDH). This was a vision of having a single trusted application that would hold the user’s private key and would handle all network traffic. It would act as a neutral place to keep data so that the same data (such as phone books but also other information such as messages, calendar events, etc.) would be available to all apps with the right permissions on the phone.

Now to be fair this problem relates to our application distribution model. If we decide to make each Thali app a stand alone APK (mostly to benefit from Android’s security model) then we have this problem. If we decide to go with the “sub-app” model where Thali apps are just directories with web and node content that are all loaded from the same central application then this problem is less pressing but the model itself is ridiculously insecure.

But even if we build the TDH how do we make it work? Do we program Thali APKs to be stand alone and only if there is a TDH on the phone would we use the TDH? How do we recognize the TDH? If the user first downloads a Thali application, creates an identity and then downloads a TDH do we transfer the private key from the app to the TDH? Do we instead use a key mesh where the TDH generates a different key and we then produce an attestation where the keys say they trust each other? That later approach isn’t as nuts as it sounds. For example, if a user has 3 Thali apps and we don’t have a TDH anywhere then in theory the 3 apps could point at each other with reciprocal links they could publish to people in their application specific networks and inform them of the uniformity of identity. Or, put in something slightly closer to English, app A and B on the same user’s phone could each generate a signed statement where App A says “I public key A am the same identity as public key B” and vice versa. If a different users gets both assertions then (and only then) can they treat the identities in Apps A and B for that users as being the same. We could even support sync right on the device where apps can share their phone books with each other. In other words instead of having a single global TDH we just build a little sync mesh right on the device. The benefit of this approach is that it doesn’t require the user to download and configure yet another application.