A CouchDB replication endpoint in PHP
CC Attribution - NonCommercial 2.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Nordmann, Kore
A CouchDB replication endpoint in PHP This talk shows how I implemented a replication endpoint for the CouchDB replication protocol in PHP. This makes it possible to use about any backend as a synchronization point for Offline-First HTML5 applications powered by tools like PouchDB or TouchDB. You will also be able to replicate a CouchDB database into a MySQL database or vice-versa. The growing amount of endpoints for this protocol in different languages and environments suggests that the CouchDB replication protocol might evolve as a standard for eventual consistent multi-master replication. ······························ Speaker: Kore Nordmann Event: FrOSCon 2014 by the Free and Open Source Software Conference (FrOSCon) e.V.
 Good morning, everyone. Welcome to the first day at Frostgon and to the PHP subconference. We are holding here in this room number six. First, it's me talking about a contribution and point in PHP. Before I start with the talk, I want to say one thing. We have one talk at 1630, I think, which got cancelled due to sickness. We got it replaced already with two new talks because it was a two-slot talk. PHP and Arduino by Thomas Weynart and at 1745, I think, it's PHP, NG, keep and stack allocation by Johannes Schluter. If you didn't see that in the schedule yet, we got two replaced talks there. So, again, welcome to the PHP track and let me start with a contribution and point in PHP. So, this sounds kind of insane, I guess, to you too. Let me start why I would do something like this. So, we all got this smartphone, right? Who doesn't have a smartphone here? Oh, two. Okay. Almost everybody got a smartphone. The nice thing about the smartphone is that you have access to a number of apps which you can use for a lot of different things like Twitter, Facebook, and for example, there's a schedule app for the Frostgon schedule where you get even live updates of the schedule and stuff like this. Most of those apps, except for very few rare cases, depend on some kind of internet connection. They will access some kind of server. Twitter, for example, will, of course, fetch the tweets. Facebook app will, of course, access the Facebook servers to read new messages or whatever you're using Facebook for. And then, when we are using those apps, I experienced that quite often because I'm a consultant quite often on the road and even in foreign countries where I do not have data access. So, problems with the internet connection usually lead to some non-functional apps. So, I basically can't use my phone when I'm in some other country, for example. It got better and better with new European-wide data plans and stuff like that, but it still sucks to be in a different country. For example, I was in Switzerland in May, I think, which is not included in the European data plans. And again, my phone was basically a brick because, yeah, no internet connection. It sucks. So, most apps nowadays are still mostly unusable without an internet connection. Even the basic navigation sometimes does not really work. So, what can we do about that? There are some technologies which solve this problem really, really nicely. I will explain how this works in a bit. And if you're using CouchDB on the server and, for example, CouchDB on a client, you can have eventually consistent data storage. So, you can use the application on your phone like it would be online, but it is not. And so, the application will sync back with the server magically at some point in time. So, it will be eventually, it will be consistent again. So, it's a false friend in German. It will be schließlich konsistent, not eventually consistent. So, there's a false friend involved here. So, we will have a definite consistent state at some point in time. This already works and it's a nice thing which is nice to use and stuff like that. I won't talk about this. What I did was writing a PHP wrapper in front of CouchDB, which speaks the same API as CouchDB does to make it possible to synchronize the different kinds of phones which are already using one of those implementations of the CouchDB protocol with the PHP backend. The thing why I did that is, I get back to that in a second, and one thing which you can do with that is actually replacing the CouchDB in the backend and using any technology, any backend stack you have there and you do not need to rely on a CouchDB. So, it's a really nice database. I do not advocate against using CouchDB, but I wanted to be able to replace that for some reasons I will get back in a second. So, this is what the talk is about. And this is actually my use case. I have a front end which I want to be offline capable, offline first. I want to be able to use it in offline state. I want to synchronize it in the same eventual consistent state with some backend application. And this backend application, since I developed mostly in PHP, is of course written in PHP. And since this is PHP track, it makes sense to talk about that here. So, use cases. No. First client implementations and server implementations. The CouchDB replication protocol is already implemented by quite some server clients and even replicators. I will talk about a little more depth about all those. So, for example, there is this client PouchDB, I already mentioned by the logo, which is a full, almost full CouchDB implementation in the browser, which can work with indexed DB and local storage, I think. So, you have a CouchDB virtually in your browser where you can just store data, put data, get data out, run those macro use functions and stuff like that. And this CouchDB in your browser can synchronize back with a real CouchDB and can replicate back and forth with a real CouchDB. Houdy is a different thing where it does not really use the CouchDB replication protocol, but it almost uses that and it makes it really, really simple to develop offline first web applications. There will be a talk later today about Houdy. I have it on a later slide. So, if you want to learn more about that, you should go there. Then there is TouchDB, which implements, they implemented this natively for Android and for iOS. I am not sure about the current state. I think there is only an iOS implementation left, or there is also an implementation of the CouchDB protocol. You can synchronize some iOS application against the CouchDB server. There are some server implementations, of course. There is Apache CouchDB, then there is Cloud and PouchDB server, then there is a small Python server, and then there are a bunch of them like, Apache CouchDB itself can of course also replicate the stuff, and again, Replicpy and some other things too, which might or might not entirely work. So, we already have a bunch of different implementations, so we could use those implementations in the stack already. On the front end, on the back end. So, again, why do I want to do that? One idea could be, you have an existing PHP application, and want to put a front end on this PHP application to write a mobile app with PouchDB, TouchDB or something like that, and make that PHP application offline capable. So, you can replicate the data in your mobile app against, from and back to this PHP application. What we also could do is, replicate a MySQL database into a CouchDB or WiseWriter, replicate from a CouchDB into a MySQL database, and any other database you might be able to connect with PHP. The thing I want to use this for, or why I initially wrote this stuff, is I have a CouchDB backend, and I could just replicate against this CouchDB, but I want to adapt the data, which is in the CouchDB, based on some current circumstances, like some user scope and stuff like that. I want to modify the data and pass it only after that to the client. So, I need some modification stack. This is why I implemented this PHP CouchDB replication endpoint to load data from CouchDB, and then we modified and replicated back to the client, which will use something like CouchDB, or already is using CouchDB. So, this is the idea. We want some offline capable client, HTML5 something, to be able to replicate data from and back from our backend in an eventual consistent state. The interesting thing is, how does it work? At least this is interesting to me. So, I now will get a little bit more in depth in regarding protocols or distributed storages. I'll start with a very, very simple case, and to show how this kind of eventual consistent replication can work. It's very simple if you have a single master's case. So, we have one single source of truth where our data is primarily written, and we have a bunch of clients which just read data and do not write data on this client itself. So, no multi-master setup. Basically, you could consider this a master's slave setup using the common database terms. This is a simple case of eventual consistent replication. We are using this in quite some setups where we distribute data across several nodes. This is really failure resistant. So, if you have a distributed system like microservices, like a service oriented architecture, this kind of protocol is a very simple thing to use to distribute this data. So, what do we do in this case? I introduce that because it's a good base for understanding the couch-tube replication protocol. So, what we are doing here, we have some kind of replicator which has the job to ensure the data is replicated. We have this source of truth which is some kind of database, and we have the client who wants to display the most current data. The replicator starts with asking the client, what is the last revision of data you saw? So, what is the last data you saw? You will return null in case you did not see any data yet or some kind of revision hash. There's something we need to remember about those hashes, but I get back to that in a second. Afterwards, we just add the source of truth or master. Please give me all updates since this kind of revision hash. You can limit this number of updates if you like, but that's basically it. This will return a bunch of updates together with a revision of each update, and then we will pass on those updates to our client. This client will write those updates to the disk, but we need to remember two things to make this work in the same way. First, the revisions on our single source of truth or on our master must increment strictly monophonic to be able to have a defined sorting of those revisions. If you mix up the order of updates, depending on the data we are replicating, bad things can happen, like data loss and stuff like that. If I insert A after B, A must have a higher revision than B, and it must not be the same revision. Otherwise, it will not work in the same way. We implemented this mechanism in distributed systems with some of our customers, and at some point, the developers tend to omit some updates or try to merge updates, try to throw away old updates. This is really, really dangerous. You should be really careful with that. So do not try something like that, or be really, really smart about that. The other thing is, on our client, we must have some, or in the best case, we have some ACID compatible storage with small transactions or something like that, because if an update fails on the client, like, I cannot store this update, you must not write the revision as stored. Only, I should only store a revision, like, I received that update, if I really were able to store this current update. This is also really important. Otherwise, the client will report a later revision as stored, but it did not reach the update yet, so we have data loss. Those are the only two things which you need to remember if you are implementing something like this. If you do this, you already have an eventually consistent system which works in a really, really sane and stable way with network connection losses and stuff like that. CouchDB uses an advanced version of this, because we got the problem that we don't have a single master, but a multi-master setup. So let's take a look at that, but it's still fairly similar. CouchDB now acts as a source, as a target of the replication, and CouchDB also acts as a replicator. As mentioned before, we have different implementations for each of those nodes. So let's talk about that. Again, we start by asking our target database, what is your replication status regarding this concrete source? So it will contain the last update sequence, which is similar to the last revision I mentioned on the slide before. And if I'm replicating between, let's say, ten different nodes in each direction or whatever, each database will store this replication status for each of the other nodes. Then it's the same as before. We get the changes since this last update from our source. There are some interesting things about this CouchDB changes feed, which we are using for this. For example, this changes feed can be requested in a continuous way, like using long poles, for example, to receive all updates after and after and after, which is a really nice thing for a lot of different things, like just writing some backend service which processes all data, which is passed to CouchDB and stuff like that. This is already a really nice thing to use, but it's also used for this replication by the replicator. And now it gets a little bit more complicated. You see there's more space left here than in our single source of truth example. And we receive a revision diff from the target. What can happen here? I mentioned that we have a multi-master replication, and for example, you can have three different nodes which replicate each against each other. The target might already have received some revisions of some documents since the last revision. So the replicator asks, okay, I got this bunch of documents with this bunch of revisions. Which one didn't you see yet? And this is returned by the target. The target now reports, okay, I didn't see this, this, this, but I already got those three. This is what the target now reports. And then the replicator, the rest is fairly simple. Ask the source, please give me all the documents, which the target did not receive yet, and passes on those documents to the target. Which I omitted here in the diagram is that the replication status after the replication has been finished is stored on the source and on the target. This is a simplified version on how Couch2B replication works. And what's not with this is an invention consistent multi-master replication, and I did not mention that yet, over HTTP. So all this implementation we saw here is done via HTTP as a protocol. This is disturbing and a nice thing. It's disturbing because at least my first association is when I heard here a database replicates its data over HTTP, this sounds fairly inefficient. HTTP has a huge overhead and it seems just wrong. On the other hand, HTTP is a protocol which you can tunnel through every corporate network, and which of course our browsers understand. This is why Couch2B, Couch2B can just replicate a Couch2B. Every browser implements HTTP client, obviously. And so Couch2B can just say Couch2B, okay, please replicate, and those are just HTTP calls. The nice thing again about Couch2B using HTTP as a protocol is that we can reuse our full HTTP infrastructure stack, like reverse catching proxies, like load balancers, like SSL proxies, and stuff like that. So this is also a nice thing about Couch2B using HTTP as a protocol, and another nice thing about this is every language or every development environment of course speaks HTTP in some way. There's an HTTP client included everywhere, so every development environment you can easily connect to a Couch2B, and to this replication protocol, of course. So this is a protocol. Are there any questions regarding the protocol or stuff? Yes. What triggers the replication type? The question is what triggers the replication? You, basically. So we'll show a Couch2B replication comment later, and it's just a post to some URL replicate, and the replication will start. There are two types of replication in Couch2B, one-time replication, where just the current state of the database is replicated to another one, and there's continuous replication. You add the flag continuous to, and it will just replicate as long as it works. And when it does not work, it will hold and continue afterwards. So there's continuous replication, which is a common type of replication you would use with other database, but there's also one-time replication. If you implement it in a browser, for example, using Couch2B, you would try to replicate basically every 10 minutes or every 20 seconds, whatever you like, and sometimes it will work, sometimes it will not work. And every time it will work, you are guaranteed to receive a consistent state or to end up in a consistent state with a server again. Other questions? Otherwise, I will know you with implementation details. So you might want to add another question about the protocol, otherwise I will get even into code. Okay? Let's say you ask for it. I wanted to write this at PHP back-end for a Couch2B replication. The question now is, the protocol is not really documented. This is a bad thing about the protocol. It's not really documented yet. There are some documentation evolving regarding that, but not all algorithms are documented and stuff like that. So I could not implement this against some kind of specification. So my only idea was, let's record a real Couch2B replication and replay that against my PHP endpoint. So the tool I used was Man in the Middle Dump, which I can just start and say, okay, build up an HTTP proxy to that server and record everything and dump everything. And then I pointed Man in the Middle Dump to my Couch2B server and told Couch2B, please replicate to this proxy, and it recorded everything. Then I told Couch2B, please replicate from this proxy and it recorded this again. I built up a set of scenarios and recorded all those scenarios. And I had several examples of Couch2B replication implementations. The next step I obviously did was replaying those dumps against my own implementation, against my own PHP endpoint. Since Man in the Middle Dump uses TNET strings to dump those files, anybody heard of those? I didn't hear of those before. It tries to be a binary safe JSON thingy developed by DJB. And of course, there's some PHP library there which parses those things, so I could just read them with symphony responses, a request out of it, and put it into my endpoint. So this was fairly simple. There are some things you need to remember when doing such things. For example, there are a lot of random IDs like the revisions or some dates in the HTTP headers and stuff like that. You must replace those, of course, because the date of replication will not match the date when I replace those things. So I replace some headers, I remove some headers and stuff like that. Afterwards, I just fixed the failures. I made every replay work and win. The nice thing about that is I started with a really, really simple case. I created one document in the source database in one revision and said, replicate that. It's a really trivial case. I got that working and then I added another version. I added a conflict and resolved the conflict and stuff like that to make it more complex and more complex. And while doing that, I, of course, gained quite some understanding of the CouchDB replication protocol and was able to refactor my code I wrote for the first one and refactor it and refactor it in a better state. So I ended up with something which is fairly understandable code which solves all those problems. The thing I want to do and one thing why I did this stuff is to document the CouchDB replication protocol. I want to start, there are some other people who are doing the same with the Python backends who also tried to document all the algorithms which are included here. And I want to contribute to that and document how this replication protocol works. And this is one main thing I want to do with this because the CouchDB replication protocol is now implemented in multiple places. It's a very, very sane thing to do. We have a lot of implementations and thus we should document such a protocol, of course. Here are two links to, I mean the middle dump, for example, to record and replay HTTP traffic which is really nice, which is a fairly nice tool to do that. And I wrote this replaying of a mid-dump recordings and replacing dates and IDs and stuff like that in certain requests and responses. And I expected that into a small library which you can find in this address if you want to use this, for example, for testing your application or stuff like that. Pull requests are welcome as always and this is just a small library to make this replaying of stuff easier to do. Those are, of course, just integration tests. I put in an HTTP request and ensure that the outcome is what CouchDB expects. What I will need to write is, for example, unit tests for some of the more advanced algorithms. I mentioned in the protocol overview that I have to do some revision-diffing and this is a little bit more complex which you will not be able to test entirely using those integration tests by replaying HTTP traffic. So I will need to write unit tests for such things. And those are not written yet because it's all very much in development right now and it's not stable at all. The APIs are not stable but I will write unit tests for the more complex algorithms in there. If you have a question, don't hesitate to raise your hand. I will try to answer every question while I'm in the talk but we can also have like 50 minutes at the end of the talk where you can add other questions and discuss stuff I talked about and things like that. But don't hesitate to just raise your hand. The implementation I did uses Symphony 2 as an HTTP framework. For me it's the most, or not how to phrase that. It's not the worst HTTP framework in the PHP world right now. You might even say it's the best one, even if it has its flaws. So we can use that, I use that as a base because I did not want to write another HTTP framework. So I'm using the HTTP kernel from Symphony to pass the HTTP request and provide me with the correct request objects and create response objects and stuff like that. The actual logic of the replication is all implemented in absolute framework agnostic control. They do not know nothing about Symphony 2. So you can put in any HTTP framework you want in front of the actual replication logic because I do not want to have a couch to be replication endpoint, the logic of it, bound to some concrete PHP framework. So if you want to integrate that in your PHP stack, you should be fine. I did not test that yet, but you should be fine. Then there is some kind of a repository. The class is currently called storage, which has the job of actually storing all the stuff. So you could say it's a repository and the only repository I actively tested is an in-memory repository. As mentioned, I'm replaying in tests all the HTTP requests against one endpoint. And for my tests, the fastest thing to use, of course, is a very simple in-memory implementation. So all data is just stored in PHP arrays. This is, of course, unusable in any real application. For this demo I wrote on Thursday, Thursday, quickly, I hacked a MySQL endpoint. So I will demo that later, so where I can replicate a couch to be, simple couch to be yet, into a MySQL database. It was fairly easy to write, and it took me like one or two hours or something like that. So with that, you can add, in theory, basically any storage or any backend you want. I would not rely yet on the stability of the API of this class because, yeah, this is under development still. And everything I did there can also be found on GitHub, of course. It's KGNC couch db endpoint. Probably you'll find that if you Google for couch db endpoint or stuff like that. And so if you want to check it out, want to play with it, just go there and clone it and try it. So now I'm getting to the live demo part. If there aren't any questions left, if you have any questions about the live demo, don't hesitate to ask again. So let's see. I have a small problem with the beamer. You cannot see the stuff which is on the leftmost side, which is slightly annoying. But yeah, I hope it will still work. So first. Take a window a little bit smaller. Yeah, that's annoying because the presentation window is set to top most display and then, well, everything fucks itself or something like that. So first I start my couch db endpoint. I can quickly show it's a very, very simple script which we got here. No, it's just HD docs in XP. It's very simple, standard symphony stuff. I just create my endpoint and say endpoint run. It does some request browsing and stuff like that. It's really, really, really simple. So let me start that server. And I reset to the MyScale database, which is used to store documents. So we can recreate all this kind of stuff. So the next thing I do is using the couch db administration back end, which is delivered together with couch db to create a new database. Let's call it source. And now we are creating documents in this database. So for example, let's start with document test one. We had a field here. Hello world. Couch db documents are basically about any JSON structure. So I can put in whatever I want here. And so let's save this document. Yeah, I saved this document. So we can see this here. Again, there's a little bit missing there. We can add another document. Let's call it Frostcon. And adding a field year 2014, for example. So we have a very simple document which contains a year of the current Frostcon. So we now have a simple database which contains just two documents. And this database I now want to replicate into my couch db endpoint. Replication in couch db is absolutely trivial. The comment does not look that trivial yet, but I'm going to explain that. So what we have here, except for the missing part on the very left side, is a couch db replication I want to start. So if I just post to the URL, this is the address of my couch db. It runs on the port 5984. I post the document to the underscore replicate URL. And then I just say couch db. OK, the source is my test database. OK, I should change that to source because I called the database a just created source. And the target is my PHP server. And I called the virtual database master. This is everything I need to do to execute a couch db replication. I should pass the content type application JSON there to tell couch db, OK, this is a JSON request, a JSON body we are receiving here. So let me quickly change that to source. And yeah, the output I pipe into JSON pretty to make it look a little bit more beautiful. And now we should see a replication action. The nice thing is it worked. Otherwise, we would see some strange, long backtrace. This is the bad thing about couch db. If you happen to run into an error, you always get ugly, airline backtraces. It does not look nice. And it's hard to understand. What we can see here, or what you can't see here, is the most important thing. Let me try to move that a little bit. I hope we get this Bima stuff working for the other presenters later. So what we see here is OK, true. This is a nice thing. And we can see some details about the replication like docs written to start and end time and stuff like that. So we see the replication worked. And now let's look if this really the case. So I have this database called couch db. And you can again don't see everything. So is that left most? Can I move further to the left? Good. So let's try that. So I open my score. I access the database couch db directly. So I have a few tables in here, space missing there, which is document, document, update, and revision. The interesting table, of course, is a document table where all documents are stored. So just do a select, ask the risk from document. And what we can see here is I hacked this endpoint together. So it's not the most beautiful thing you can do. But we can see here the document ID, test one, Frostgun, those are the IDs I used. The document revision, it's both the first revision of those documents. Couch db by now uses a revision scheme where the first number is an incremental number. So one is for the first revision. And everything after the dash is a sh1 hash of the actual document. And then as a document, I just store a PHP serialized version of the object. So this is probably not what you want to do in the real implementation of a couch db endpoint, but for the demo, it just works. So this is a document. Document updates just store some update information about sequence numbers and stuff like that. And the revision stores, the revision database table stores the revision state status of each replication. So this is a simple case which already works. What I can show now is, for example, next year we might have an update to our Frostgun document, it's here 2015, I save that document and restart the replication, executing the same comment again, just say replicate again, and I hope hopefully it works again, looks like. So I can go into my SQL database, select again everything from our database, and what we see here is a second revision of the same document. Now it's in our database. We again have the ID Frostgun and we have the second revision here. What can also show now couch db has an HTTP interface. So my replication endpoint of course also has an HTTP interface. What I can do now is accessing my endpoint now. I call the virtual database master. And what couch db delivers by default is some kind of statistical information. Let's call it this way about the database. The most important thing is, we see the document count here. Most stuff which is in here is not really relevant for replication. So in my personal implemented endpoint, I just leave it empty like disk size, data size, I just don't care. And it doesn't really make sense for my MySQL database, so I leave everything at zero. The replication does not care about it, so I do not care about it yet. I might add that information later. What we can also do of course is requesting the Frostgun document. What is returned now is of course JSON because couch db returns JSON. My browser just nicely formats this kind of stuff. So what we can see here is the here information we added, the ID, the revision of the document and some additional information. This information, couch db would not return that information because it's only requested with a certain flag. But in my implementation, the only thing I need to do is this flag is always passed by the replicator. So I do not care about this flag yet and just return this information always. So the return from couch db itself for the same document should look fairly similar. So I just change the URL to the couch db port and the couch db database and the return should be similar except it does not send a proper header here, I see. This is why my JSON plugin does not detect this as JSON and just presents us a plain string. But it's the same content from couch db and from my replication endpoint. So this is the simple thing about the demo. What I can also show is replication of the other direction. So what I can do now is creating a new database, let's call it target and leave the database empty for now and now start the replication again to replicate out of my PHP endpoint into this new couch db database. So the source now is localhost 8080 slash master and the target now is my couch db database which runs on the following 8.5.9.8.5 and I call the database target and I replicate into the target database. What we see here is three replicated revisions and if I now go into the target database I should see two documents here in the versions we had here. So replication in both directions works for my couch db endpoint and what I always call was my PHP backend. What does not work yet is replicating attachments and there's some slight problem I have understanding the parent revisions in case of some kind of conflict which I'm supposed to return. I do not match the couch db algorithm entirely there yet. So it might not behave entirely like your couch db in some conflict states of your database. So conflicts might occur for example. I replicate one document like let's say some wiki text. I have it on one node replicated to another node. Those both nodes are offline for a second or for a while and I edit this concrete document on both nodes in a different way. When I start the replication next time of course the replicator will detect some kind of conflict because the document has been updated in different ways on both nodes. The replicator by default does not know how to handle that. Imagine somebody changed some paragraph in a wiki document. A computer, it's not possible for some computer to know how to merge those two paragraph changes. So what couch db does here is it puts both versions of the documents on each node, marks this document as conflict, and selects the exactly same version of this document on every node. So you have a consistent state again on every node. One version of those documents wins on every node. And you have a consistent state. The documents are flagged as conflict. So you can go as a user and look at all documents which are in a conflict state and resolve this conflict manually, programmatically, whatever suits you. With a wiki document for example where a paragraph is changed in two different ways, you probably want to resolve that manually. In a lot of other cases it's easy to resolve those conflicts in a programmatic way. So if you like, I can also jump into the code, but I will leave that to the quarter of an hour of question time that you left as you end or in about five minutes. So let me summarize what we saw here. I showed you, no, I think that offline first, eventually consistent HTML5 mobile apps are the future. Bingo. Sorry for the buzzword bingo, but I really think this is a case. So I do not believe in native apps. I think HTML5 apps are what you want to go. I think is that you should build every time your apps or your mobile websites, maybe even your real websites, with offline capability in mind and maybe even start with that. So I'm really pissed when I'm, for example, in a train, I'm using my app and it just drops with about everything. I cannot even use a navigation anymore or stuff like that because I'm currently offline because the train, I'm on the way to Hamburg, for example, and I'm always offline there in the train. So I'm really, really annoyed by that. I cannot continue to work in the second I'm offline. It's really important, for example, if you're using something like Google Docs, it did not work offline for a while. I think by now it works somehow. But you could, you just had to stop typing because you were offline for a second. This is really, really annoying. So please, if you build mobile apps, keep that in mind. I want to continue working with my app when it's offline. On the other hand, the synchronization of data between the state you modified while you are offline and the state of the server can be really, really complex. Apple introduced like three years ago something, some kind of, Jacob, you might remember the name of the Apple stuff to synchronize with the server, data with the server. Sorry? iCloud. iCloud, yeah, the iCloud stuff. Yeah, exactly. It does not really work because it does not know about this kind of revision of this kind of eventual consistency and you have to handle this entirely yourself if you want to use that. If you go there, writing an HTML5 offline app, use some kind of eventual consistent mechanism. And to make that easy, use the CouchBeam replication protocol. This is, for me, this is a key feature of CouchBeam. And the unique selling point for CouchBeam is this eventual consistent multi-master replication over HTTP. This is the nice thing about CouchBeam. Everything else is just bonus. So you might want to keep that in mind. To make that easier, try Hoody. Hoody is an approach to make it really, really simple to develop such offline apps. It uses CouchBeam in the backend. You do not need to care about that. But just if you want to prototype or write those offline first apps, just use Hoody. There's a talk, it's called, Coding Your Green, oh, there's an end missing there, by Team Hoody at quarter past three in C117. It's up back there. And you might want to listen to that talk if you care about writing offline first mobile apps. And yeah, take a look at that talk. Those custom implementation, like the PHP backend I just showed, I want to use that to add some additional source to the data and replicate out of CouchBeam. So with those implementations, you sort of are a bunch of implementations already available in Python, in JavaScript, and stuff like that. You might use those to do something additional with the data and do some fun things. But it already works with the plain CouchDB and stuff like Hoody or CouchDB. You can already write offline first mobile apps, which will just work as the data will be consistent over multiple mobile phones, over multiple notebooks, over multiple servers, and stuff like that. This is not only important, by the way, for the case where you have mobile phones and servers, but also important for if you have servers distributed across different continents, you cannot have, it's not saying to have transactions across multiple continents because the runtime of the request and response is just too long, it takes too long for same response time on the web. So you will need eventual consistency even there. You might have heard about the base model and stuff like that. So that's it, basically. What we now have time for about 15 minutes or less, if you want to leave earlier, is for questions, for discussions. I can even jump into the code if you care about that. Everything is online, as I mentioned. The slides will be online at some point. You'll find that I will Twitter the slides, and my Twitter nick is on here. You cannot read that, by the way. So I will link them on Twitter. All the links to the repositories are on there, so you can jump into the code yourself and look at it. But I can also show you something if you have questions. So any questions left? Yes? What is the setup for multi-user scenarios? So the question is, what would be the setup for a multi-user case where you have certain data just for certain users? The answer, let me answer with three points. First, CouchDB. In CouchDB, you usually use one database per user. A CouchDB server easily works with 10,000 of databases. So you can use one database for each user. The next question is then, I will have documents which will be synchronized between different users. Like, for example, the Facebook case, I post a message on the wall of another user. What you can use for that is, again, replication, and CouchDB supports filtered replication. So a replication between user database A and user database B, which just replicates the messages which are intended for user B. This just works with a replication filter and will also just work like this. The only thing you need to do is pass, okay, please use this filter. Second answer is Hoody. With Hoody, Hoody uses this aspect of CouchDB. In Hoody, you register first, and every user gets its own database. And then in Hoody, there is some very, very, very simple JavaScript API in the browser where you can, say, share this document, and Hoody will take care of this replication stuff in the backend. So with Hoody, it will just work. I'm not sure about the stability of this plugin yet for Hoody, because at least three months ago, it was still under development. It might already work. So I'm going to ask Olar and Lena later in the Hoody talk about this. But it's going to just work in Hoody. Point three, PHP, my PHP backend. You can basically use anything you want here. For example, you could parse an HTTP auth header with a username and select a database based on that, or add a pencil to the MySQL queries or whatever you want. With PHP backend, yeah, it works just like, it works in your PHP application. You can do whatever you want to. Does it answer the question? OK, so that's the one reason you want to work with this. It makes it easier, yes. Other questions? Yes, please. You used to implement basically the entire CouchDB protocol, right? You just have this one use case in mind. It's specialized for that, but it's the entire protocol. Yeah, the question is, did I implement the entire protocol? Even I just had this one through the use case. Yes, I'm just stupid. And so the point is, and it happens to me again and again and again, I have this open source project in mind I want to do. I did that for a different project I wanted to implement. And then I think, OK, I need that. I should do this in the right way. And then it starts to consume time. And because I needed this CouchDB protocol, I implemented this HTTP proxy replay stuff, which I also did in the right way. And for that, I had to implement something else. So this is project B in the ABCD stack. And I'm back to B right now because I want to complete the CouchDB endpoint. And then I want to continue start working on project A. And yeah, this is time consuming, but I try to do it the right way and try to contribute documentation and stuff like that. But again, this is even more time consuming than, of course. So yes, I implemented, I hopefully implemented most of it yet, but not everything. My endpoint doesn't support long polling, for example, yet, for offset changes feed. But I probably will even add that. Yes. And did you have to reverse engineer that too, or was that documented? The question is, if I had to reverse engineer that, if I had to reverse engineer most of it, there's some documentation. The documentation focus is different than what I needed. The documentation focus right now is on how to implement a replicator and on how to implement a client. And what I did was implementing something serverish, which is slightly different than something client-ish. I say client-ish and serverish because in CouchDB world, everything is a client as a server, as a master all the time. But so the documentation had a different focus. And some of the algorithms like the religion divvying and stuff like that, it's not documented yet. And I had to reverse, or I still have to reverse engineer parts of that. Yes, it's a lot of reverse engineering. The problem is, Elan code is, I think, the most unreadable code ever. I'm really, I'm similar with a bunch of functional languages. But Elan is a different kind of thing. Elan code is ugly as hell. So it's really hard to read, understand those algorithms from reading the CouchDB code, which is, of course, open source. It's an Apache project. But yes. So it's a little bit of soft code reading and stuff like that. Other questions? Three, two, one. Then thank you and see you next time in the PHP room. Thank you.
 Good morning, everyone. Welcome to the first day at Frostgon and to the PHP subconference.  We are holding here in this room number six. First, it's me talking about a contribution  and point in PHP. Before I start with the talk, I want to say one thing. We have one talk at 1630, I think, which got cancelled due to sickness. We got it replaced already with two new talks because it was a two-slot talk.  PHP and Arduino by Thomas Weynart and at 1745, I think, it's PHP, NG, keep and stack allocation by Johannes Schluter. If you didn't see that in the schedule yet, we got two replaced talks there.  So, again, welcome to the PHP track and let me start with a contribution and point in PHP. So, this sounds kind of insane, I guess, to you too. Let me start why I would do something like this.  So, we all got this smartphone, right? Who doesn't have a smartphone here? Oh, two. Okay. Almost everybody got a smartphone.  The nice thing about the smartphone is that you have access to a number of apps which you can use for a lot of different things like Twitter, Facebook, and for example, there's a schedule app for the Frostgon schedule where you get even live updates of the schedule and stuff like this.  Most of those apps, except for very few rare cases, depend on some kind of internet connection. They will access some kind of server. Twitter, for example, will, of course, fetch the tweets.  Facebook app will, of course, access the Facebook servers to read new messages or whatever you're using Facebook for.  And then, when we are using those apps, I experienced that quite often because I'm a consultant quite often on the road and even in foreign countries where I do not have data access.  So, problems with the internet connection usually lead to some non-functional apps. So, I basically can't use my phone when I'm in some other country, for example.  It got better and better with new European-wide data plans and stuff like that, but it still sucks to be in a different country.  For example, I was in Switzerland in May, I think, which is not included in the European data plans. And again, my phone was basically a brick because, yeah, no internet connection. It sucks.  So, most apps nowadays are still mostly unusable without an internet connection. Even the basic navigation sometimes does not really work.  So, what can we do about that? There are some technologies which solve this problem really, really nicely. I will explain how this works in a bit.  And if you're using CouchDB on the server and, for example, CouchDB on a client, you can have eventually consistent data storage.  So, you can use the application on your phone like it would be online, but it is not.  And so, the application will sync back with the server magically at some point in time. So, it will be eventually, it will be consistent again.  So, it's a false friend in German. It will be schließlich konsistent, not eventually consistent. So, there's a false friend involved here.  So, we will have a definite consistent state at some point in time.  This already works and it's a nice thing which is nice to use and stuff like that. I won't talk about this.  What I did was writing a PHP wrapper in front of CouchDB, which speaks the same API as CouchDB does to make it possible to synchronize  the different kinds of phones which are already using one of those implementations of the CouchDB protocol with the PHP backend.  The thing why I did that is, I get back to that in a second, and one thing which you can do with that is actually replacing the CouchDB in the backend  and using any technology, any backend stack you have there and you do not need to rely on a CouchDB.  So, it's a really nice database. I do not advocate against using CouchDB, but I wanted to be able to replace that for some reasons I will get back in a second.  So, this is what the talk is about. And this is actually my use case. I have a front end which I want to be offline capable, offline first.  I want to be able to use it in offline state. I want to synchronize it in the same eventual consistent state with some backend application.  And this backend application, since I developed mostly in PHP, is of course written in PHP.  And since this is PHP track, it makes sense to talk about that here.  So, use cases. No. First client implementations and server implementations.  The CouchDB replication protocol is already implemented by quite some server clients and even replicators.  I will talk about a little more depth about all those.  So, for example, there is this client PouchDB, I already mentioned by the logo, which is a full, almost full CouchDB implementation in the browser,  which can work with indexed DB and local storage, I think.  So, you have a CouchDB virtually in your browser where you can just store data, put data, get data out, run those macro use functions and stuff like that.  And this CouchDB in your browser can synchronize back with a real CouchDB and can replicate back and forth with a real CouchDB.  Houdy is a different thing where it does not really use the CouchDB replication protocol, but it almost uses that and it makes it really, really simple to develop offline first web applications.  There will be a talk later today about Houdy. I have it on a later slide.  So, if you want to learn more about that, you should go there.  Then there is TouchDB, which implements, they implemented this natively for Android and for iOS.  I am not sure about the current state. I think there is only an iOS implementation left, or there is also an implementation of the CouchDB protocol.  You can synchronize some iOS application against the CouchDB server.  There are some server implementations, of course. There is Apache CouchDB, then there is Cloud and PouchDB server,  then there is a small Python server, and then there are a bunch of them like,  Apache CouchDB itself can of course also replicate the stuff, and again,  Replicpy and some other things too, which might or might not entirely work.  So, we already have a bunch of different implementations, so we could use those implementations in the stack already.  On the front end, on the back end. So, again, why do I want to do that?  One idea could be, you have an existing PHP application, and want to put a front end on this PHP application  to write a mobile app with PouchDB, TouchDB or something like that, and make that PHP application offline capable.  So, you can replicate the data in your mobile app against, from and back to this PHP application.  What we also could do is, replicate a MySQL database into a CouchDB or WiseWriter, replicate from a CouchDB into a MySQL database,  and any other database you might be able to connect with PHP.  The thing I want to use this for, or why I initially wrote this stuff, is I have a CouchDB backend,  and I could just replicate against this CouchDB, but I want to adapt the data,  which is in the CouchDB, based on some current circumstances, like some user scope and stuff like that.  I want to modify the data and pass it only after that to the client.  So, I need some modification stack.  This is why I implemented this PHP CouchDB replication endpoint to load data from CouchDB,  and then we modified and replicated back to the client, which will use something like CouchDB, or already is using CouchDB.  So, this is the idea.  We want some offline capable client, HTML5 something, to be able to replicate data from and back from our backend in an eventual consistent state.  The interesting thing is, how does it work?  At least this is interesting to me. So, I now will get a little bit more in depth in regarding protocols or distributed storages.  I'll start with a very, very simple case, and to show how this kind of eventual consistent replication can work.  It's very simple if you have a single master's case.  So, we have one single source of truth where our data is primarily written,  and we have a bunch of clients which just read data and do not write data on this client itself.  So, no multi-master setup.  Basically, you could consider this a master's slave setup using the common database terms.  This is a simple case of eventual consistent replication.  We are using this in quite some setups where we distribute data across several nodes.  This is really failure resistant.  So, if you have a distributed system like microservices, like a service oriented architecture,  this kind of protocol is a very simple thing to use to distribute this data.  So, what do we do in this case?  I introduce that because it's a good base for understanding the couch-tube replication protocol.  So, what we are doing here, we have some kind of replicator which has the job to ensure the data is replicated.  We have this source of truth which is some kind of database,  and we have the client who wants to display the most current data.  The replicator starts with asking the client, what is the last revision of data you saw?  So, what is the last data you saw?  You will return null in case you did not see any data yet or some kind of revision hash.  There's something we need to remember about those hashes, but I get back to that in a second.  Afterwards, we just add the source of truth or master.  Please give me all updates since this kind of revision hash.  You can limit this number of updates if you like, but that's basically it.  This will return a bunch of updates together with a revision of each update,  and then we will pass on those updates to our client.  This client will write those updates to the disk,  but we need to remember two things to make this work in the same way.  First, the revisions on our single source of truth or on our master  must increment strictly monophonic to be able to have a defined sorting of those revisions.  If you mix up the order of updates, depending on the data we are replicating,  bad things can happen, like data loss and stuff like that.  If I insert A after B, A must have a higher revision than B,  and it must not be the same revision.  Otherwise, it will not work in the same way.  We implemented this mechanism in distributed systems with some of our customers,  and at some point, the developers tend to omit some updates or try to merge updates,  try to throw away old updates.  This is really, really dangerous. You should be really careful with that.  So do not try something like that, or be really, really smart about that.  The other thing is, on our client, we must have some, or in the best case,  we have some ACID compatible storage with small transactions or something like that,  because if an update fails on the client, like, I cannot store this update,  you must not write the revision as stored.  Only, I should only store a revision, like, I received that update,  if I really were able to store this current update.  This is also really important.  Otherwise, the client will report a later revision as stored,  but it did not reach the update yet, so we have data loss.  Those are the only two things which you need to remember if you are implementing something like this.  If you do this, you already have an eventually consistent system  which works in a really, really sane and stable way  with network connection losses and stuff like that.  CouchDB uses an advanced version of this, because we got the problem  that we don't have a single master, but a multi-master setup.  So let's take a look at that, but it's still fairly similar.  CouchDB now acts as a source, as a target of the replication,  and CouchDB also acts as a replicator.  As mentioned before, we have different implementations for each of those nodes.  So let's talk about that.  Again, we start by asking our target database,  what is your replication status regarding this concrete source?  So it will contain the last update sequence,  which is similar to the last revision I mentioned on the slide before.  And if I'm replicating between, let's say, ten different nodes  in each direction or whatever, each database will store this  replication status for each of the other nodes.  Then it's the same as before.  We get the changes since this last update from our source.  There are some interesting things about this CouchDB changes feed,  which we are using for this. For example, this changes feed  can be requested in a continuous way, like using long poles, for example,  to receive all updates after and after and after,  which is a really nice thing for a lot of different things,  like just writing some backend service which processes all data,  which is passed to CouchDB and stuff like that.  This is already a really nice thing to use,  but it's also used for this replication by the replicator.  And now it gets a little bit more complicated.  You see there's more space left here than in our single source of truth example.  And we receive a revision diff from the target.  What can happen here?  I mentioned that we have a multi-master replication,  and for example, you can have three different nodes which replicate  each against each other.  The target might already have received some revisions of some documents  since the last revision.  So the replicator asks, okay, I got this bunch of documents with this bunch of revisions.  Which one didn't you see yet?  And this is returned by the target.  The target now reports, okay, I didn't see this, this, this,  but I already got those three.  This is what the target now reports.  And then the replicator, the rest is fairly simple.  Ask the source, please give me all the documents,  which the target did not receive yet,  and passes on those documents to the target.  Which I omitted here in the diagram is that the replication status  after the replication has been finished is stored on the source and on the target.  This is a simplified version on how Couch2B replication works.  And what's not with this is an invention consistent multi-master replication,  and I did not mention that yet, over HTTP.  So all this implementation we saw here is done via HTTP as a protocol.  This is disturbing and a nice thing.  It's disturbing because at least my first association is when I heard here  a database replicates its data over HTTP,  this sounds fairly inefficient.  HTTP has a huge overhead and it seems just wrong.  On the other hand, HTTP is a protocol which you can tunnel through every corporate network,  and which of course our browsers understand.  This is why Couch2B, Couch2B can just replicate a Couch2B.  Every browser implements HTTP client, obviously.  And so Couch2B can just say Couch2B, okay, please replicate,  and those are just HTTP calls.  The nice thing again about Couch2B using HTTP as a protocol is  that we can reuse our full HTTP infrastructure stack,  like reverse catching proxies, like load balancers, like SSL proxies, and stuff like that.  So this is also a nice thing about Couch2B using HTTP as a protocol,  and another nice thing about this is every language or every development environment  of course speaks HTTP in some way.  There's an HTTP client included everywhere,  so every development environment you can easily connect to a Couch2B,  and to this replication protocol, of course.  So this is a protocol.  Are there any questions regarding the protocol or stuff?  Yes.  What triggers the replication type?  The question is what triggers the replication?  You, basically.  So we'll show a Couch2B replication comment later,  and it's just a post to some URL replicate, and the replication will start.  There are two types of replication in Couch2B, one-time replication,  where just the current state of the database is replicated to another one,  and there's continuous replication.  You add the flag continuous to, and it will just replicate as long as it works.  And when it does not work, it will hold and continue afterwards.  So there's continuous replication, which is a common type of replication  you would use with other database, but there's also one-time replication.  If you implement it in a browser, for example, using Couch2B,  you would try to replicate basically every 10 minutes or every 20 seconds,  whatever you like, and sometimes it will work, sometimes it will not work.  And every time it will work, you are guaranteed to receive a consistent state  or to end up in a consistent state with a server again.  Other questions?  Otherwise, I will know you with implementation details.  So you might want to add another question about the protocol,  otherwise I will get even into code.  Okay? Let's say you ask for it.  I wanted to write this at PHP back-end for a Couch2B replication.  The question now is, the protocol is not really documented.  This is a bad thing about the protocol.  It's not really documented yet.  There are some documentation evolving regarding that,  but not all algorithms are documented and stuff like that.  So I could not implement this against some kind of specification.  So my only idea was, let's record a real Couch2B replication  and replay that against my PHP endpoint.  So the tool I used was Man in the Middle Dump,  which I can just start and say, okay, build up an HTTP proxy to that server  and record everything and dump everything.  And then I pointed Man in the Middle Dump to my Couch2B server  and told Couch2B, please replicate to this proxy,  and it recorded everything.  Then I told Couch2B, please replicate from this proxy  and it recorded this again.  I built up a set of scenarios and recorded all those scenarios.  And I had several examples of Couch2B replication implementations.  The next step I obviously did was replaying those dumps  against my own implementation, against my own PHP endpoint.  Since Man in the Middle Dump uses TNET strings to dump those files,  anybody heard of those?  I didn't hear of those before.  It tries to be a binary safe JSON thingy developed by DJB.  And of course, there's some PHP library there  which parses those things, so I could just read them  with symphony responses, a request out of it,  and put it into my endpoint.  So this was fairly simple.  There are some things you need to remember when doing such things.  For example, there are a lot of random IDs like the revisions  or some dates in the HTTP headers and stuff like that.  You must replace those, of course,  because the date of replication will not match the date  when I replace those things.  So I replace some headers, I remove some headers and stuff like that.  Afterwards, I just fixed the failures.  I made every replay work and win.  The nice thing about that is I started with a really, really simple case.  I created one document in the source database in one revision  and said, replicate that.  It's a really trivial case.  I got that working and then I added another version.  I added a conflict and resolved the conflict and stuff like that  to make it more complex and more complex.  And while doing that, I, of course,  gained quite some understanding of the CouchDB replication protocol  and was able to refactor my code I wrote for the first one  and refactor it and refactor it in a better state.  So I ended up with something which is fairly understandable code  which solves all those problems.  The thing I want to do and one thing why I did this stuff  is to document the CouchDB replication protocol.  I want to start, there are some other people who are doing the same  with the Python backends who also tried to document all the  algorithms which are included here.  And I want to contribute to that and document how this replication  protocol works.  And this is one main thing I want to do with this  because the CouchDB replication protocol is now implemented  in multiple places.  It's a very, very sane thing to do.  We have a lot of implementations and thus we should document  such a protocol, of course.  Here are two links to, I mean the middle dump, for example,  to record and replay HTTP traffic which is really nice,  which is a fairly nice tool to do that.  And I wrote this replaying of a mid-dump recordings  and replacing dates and IDs and stuff like that  in certain requests and responses.  And I expected that into a small library which you can find  in this address if you want to use this, for example,  for testing your application or stuff like that.  Pull requests are welcome as always and this is just a small  library to make this replaying of stuff easier to do.  Those are, of course, just integration tests.  I put in an HTTP request and ensure that the outcome is  what CouchDB expects.  What I will need to write is, for example,  unit tests for some of the more advanced algorithms.  I mentioned in the protocol overview that I have to do some  revision-diffing and this is a little bit more complex  which you will not be able to test entirely using those  integration tests by replaying HTTP traffic.  So I will need to write unit tests for such things.  And those are not written yet because it's all very much  in development right now and it's not stable at all.  The APIs are not stable but I will write unit tests for  the more complex algorithms in there.  If you have a question, don't hesitate to raise your hand.  I will try to answer every question while I'm in the talk  but we can also have like 50 minutes at the end of the talk  where you can add other questions and discuss stuff  I talked about and things like that.  But don't hesitate to just raise your hand.  The implementation I did uses Symphony 2 as an HTTP framework.  For me it's the most, or not how to phrase that.  It's not the worst HTTP framework in the PHP world right now.  You might even say it's the best one, even if it has its flaws.  So we can use that, I use that as a base because I did not want  to write another HTTP framework.  So I'm using the HTTP kernel from Symphony to pass the HTTP  request and provide me with the correct request objects  and create response objects and stuff like that.  The actual logic of the replication is all implemented  in absolute framework agnostic control.  They do not know nothing about Symphony 2.  So you can put in any HTTP framework you want in front  of the actual replication logic because I do not want to  have a couch to be replication endpoint, the logic of it,  bound to some concrete PHP framework.  So if you want to integrate that in your PHP stack,  you should be fine.  I did not test that yet, but you should be fine.  Then there is some kind of a repository.  The class is currently called storage, which has the job  of actually storing all the stuff.  So you could say it's a repository and the only repository  I actively tested is an in-memory repository.  As mentioned, I'm replaying in tests all the HTTP requests  against one endpoint.  And for my tests, the fastest thing to use, of course,  is a very simple in-memory implementation.  So all data is just stored in PHP arrays.  This is, of course, unusable in any real application.  For this demo I wrote on Thursday, Thursday, quickly,  I hacked a MySQL endpoint.  So I will demo that later, so where I can replicate a couch  to be, simple couch to be yet, into a MySQL database.  It was fairly easy to write, and it took me like one or two  hours or something like that.  So with that, you can add, in theory, basically any storage  or any backend you want.  I would not rely yet on the stability of the API of this class  because, yeah, this is under development still.  And everything I did there can also be found on GitHub,  of course.  It's KGNC couch db endpoint.  Probably you'll find that if you Google for couch db endpoint  or stuff like that.  And so if you want to check it out, want to play with it,  just go there and clone it and try it.  So now I'm getting to the live demo part.  If there aren't any questions left,  if you have any questions about the live demo,  don't hesitate to ask again.  So let's see.  I have a small problem with the beamer.  You cannot see the stuff which is on the leftmost side,  which is slightly annoying.  But yeah, I hope it will still work.  So first.  Take a window a little bit smaller.  Yeah, that's annoying because the presentation window is set  to top most display and then, well, everything  fucks itself or something like that.  So first I start my couch db endpoint.  I can quickly show it's a very, very simple script which we got here.  No, it's just HD docs in XP.  It's very simple, standard symphony stuff.  I just create my endpoint and say endpoint run.  It does some request browsing and stuff like that.  It's really, really, really simple.  So let me start that server.  And I reset to the MyScale database, which is used to store documents.  So we can recreate all this kind of stuff.  So the next thing I do is using the couch db administration back  end, which is delivered together with couch db to create a new database.  Let's call it source.  And now we are creating documents in this database.  So for example, let's start with document test one.  We had a field here.  Hello world.  Couch db documents are basically about any JSON structure.  So I can put in whatever I want here.  And so let's save this document.  Yeah, I saved this document.  So we can see this here.  Again, there's a little bit missing there.  We can add another document.  Let's call it Frostcon.  And adding a field year 2014, for example.  So we have a very simple document which contains a year  of the current Frostcon.  So we now have a simple database which contains just two documents.  And this database I now want to replicate into my couch db endpoint.  Replication in couch db is absolutely trivial.  The comment does not look that trivial yet, but I'm going to explain that.  So what we have here, except for the missing part on the very left side,  is a couch db replication I want to start.  So if I just post to the URL, this is the address of my couch db.  It runs on the port 5984.  I post the document to the underscore replicate URL.  And then I just say couch db.  OK, the source is my test database.  OK, I should change that to source because I called the database  a just created source.  And the target is my PHP server.  And I called the virtual database master.  This is everything I need to do to execute a couch db replication.  I should pass the content type application JSON there to tell couch db,  OK, this is a JSON request, a JSON body we are receiving here.  So let me quickly change that to source.  And yeah, the output I pipe into JSON pretty to make it look a little bit more beautiful.  And now we should see a replication action.  The nice thing is it worked.  Otherwise, we would see some strange, long backtrace.  This is the bad thing about couch db.  If you happen to run into an error, you always get ugly,  airline backtraces.  It does not look nice.  And it's hard to understand.  What we can see here, or what you can't see here,  is the most important thing.  Let me try to move that a little bit.  I hope we get this Bima stuff working for the other presenters later.  So what we see here is OK, true.  This is a nice thing.  And we can see some details about the replication like docs written to start  and end time and stuff like that.  So we see the replication worked.  And now let's look if this really the case.  So I have this database called couch db.  And you can again don't see everything.  So is that left most?  Can I move further to the left?  Good.  So let's try that.  So I open my score.  I access the database couch db directly.  So I have a few tables in here, space missing there,  which is document, document, update, and revision.  The interesting table, of course, is a document table where all documents are stored.  So just do a select, ask the risk from document.  And what we can see here is I hacked this endpoint together.  So it's not the most beautiful thing you can do.  But we can see here the document ID, test one, Frostgun, those are the IDs I used.  The document revision, it's both the first revision of those documents.  Couch db by now uses a revision scheme where the first number is an incremental number.  So one is for the first revision.  And everything after the dash is a sh1 hash of the actual document.  And then as a document, I just store a PHP serialized version of the object.  So this is probably not what you want to do in the real implementation of a couch db endpoint,  but for the demo, it just works.  So this is a document.  Document updates just store some update information about sequence numbers and stuff like that.  And the revision stores, the revision database table stores the revision state status of each replication.  So this is a simple case which already works.  What I can show now is, for example, next year we might have an update to our Frostgun document,  it's here 2015, I save that document and restart the replication, executing the same comment again,  just say replicate again, and I hope hopefully it works again, looks like.  So I can go into my SQL database, select again everything from our database,  and what we see here is a second revision of the same document.  Now it's in our database.  We again have the ID Frostgun and we have the second revision here.  What can also show now couch db has an HTTP interface.  So my replication endpoint of course also has an HTTP interface.  What I can do now is accessing my endpoint now.  I call the virtual database master.  And what couch db delivers by default is some kind of statistical information.  Let's call it this way about the database.  The most important thing is, we see the document count here.  Most stuff which is in here is not really relevant for replication.  So in my personal implemented endpoint, I just leave it empty like disk size, data size,  I just don't care.  And it doesn't really make sense for my MySQL database, so I leave everything at zero.  The replication does not care about it, so I do not care about it yet.  I might add that information later.  What we can also do of course is requesting the Frostgun document.  What is returned now is of course JSON because couch db returns JSON.  My browser just nicely formats this kind of stuff.  So what we can see here is the here information we added, the ID, the revision of the document  and some additional information.  This information, couch db would not return that information  because it's only requested with a certain flag.  But in my implementation, the only thing I need to do is this flag is always passed by the replicator.  So I do not care about this flag yet and just return this information always.  So the return from couch db itself for the same document should look fairly similar.  So I just change the URL to the couch db port and the couch db database  and the return should be similar except it does not send a proper header here, I see.  This is why my JSON plugin does not detect this as JSON and just presents us a plain string.  But it's the same content from couch db and from my replication endpoint.  So this is the simple thing about the demo.  What I can also show is replication of the other direction.  So what I can do now is creating a new database, let's call it target  and leave the database empty for now and now start the replication again  to replicate out of my PHP endpoint into this new couch db database.  So the source now is localhost 8080 slash master  and the target now is my couch db database which runs on the following 8.5.9.8.5  and I call the database target and I replicate into the target database.  What we see here is three replicated revisions and if I now go into the target database  I should see two documents here in the versions we had here.  So replication in both directions works for my couch db endpoint  and what I always call was my PHP backend.  What does not work yet is replicating attachments  and there's some slight problem I have understanding the parent revisions  in case of some kind of conflict which I'm supposed to return.  I do not match the couch db algorithm entirely there yet.  So it might not behave entirely like your couch db in some conflict states  of your database.  So conflicts might occur for example.  I replicate one document like let's say some wiki text.  I have it on one node replicated to another node.  Those both nodes are offline for a second or for a while  and I edit this concrete document on both nodes in a different way.  When I start the replication next time of course the replicator will detect  some kind of conflict because the document has been updated  in different ways on both nodes.  The replicator by default does not know how to handle that.  Imagine somebody changed some paragraph in a wiki document.  A computer, it's not possible for some computer to know how to  merge those two paragraph changes.  So what couch db does here is it puts both versions of the documents  on each node, marks this document as conflict,  and selects the exactly same version of this document on every node.  So you have a consistent state again on every node.  One version of those documents wins on every node.  And you have a consistent state.  The documents are flagged as conflict.  So you can go as a user and look at all documents which are in a conflict state  and resolve this conflict manually, programmatically, whatever suits you.  With a wiki document for example where a paragraph is changed in two different ways,  you probably want to resolve that manually.  In a lot of other cases it's easy to resolve those conflicts in a programmatic way.  So if you like, I can also jump into the code,  but I will leave that to the quarter of an hour of question time  that you left as you end or in about five minutes.  So let me summarize what we saw here.  I showed you, no, I think that offline first,  eventually consistent HTML5 mobile apps are the future.  Bingo.  Sorry for the buzzword bingo, but I really think this is a case.  So I do not believe in native apps.  I think HTML5 apps are what you want to go.  I think is that you should build every time your apps or your mobile websites,  maybe even your real websites, with offline capability in mind  and maybe even start with that.  So I'm really pissed when I'm, for example, in a train, I'm using my app  and it just drops with about everything.  I cannot even use a navigation anymore or stuff like that  because I'm currently offline because the train,  I'm on the way to Hamburg, for example,  and I'm always offline there in the train.  So I'm really, really annoyed by that.  I cannot continue to work in the second I'm offline.  It's really important, for example, if you're using something like Google Docs,  it did not work offline for a while.  I think by now it works somehow.  But you could, you just had to stop typing because you were offline for a second.  This is really, really annoying.  So please, if you build mobile apps, keep that in mind.  I want to continue working with my app when it's offline.  On the other hand, the synchronization of data between the state you modified  while you are offline and the state of the server can be really, really complex.  Apple introduced like three years ago something, some kind of,  Jacob, you might remember the name of the Apple stuff to synchronize with the server,  data with the server.  Sorry?  iCloud.  iCloud, yeah, the iCloud stuff.  Yeah, exactly.  It does not really work because it does not know about this kind of revision of this  kind of eventual consistency and you have to handle this entirely yourself  if you want to use that.  If you go there, writing an HTML5 offline app,  use some kind of eventual consistent mechanism.  And to make that easy, use the CouchBeam replication protocol.  This is, for me, this is a key feature of CouchBeam.  And the unique selling point for CouchBeam is this eventual consistent  multi-master replication over HTTP.  This is the nice thing about CouchBeam.  Everything else is just bonus.  So you might want to keep that in mind.  To make that easier, try Hoody.  Hoody is an approach to make it really, really simple to develop such offline apps.  It uses CouchBeam in the backend.  You do not need to care about that.  But just if you want to prototype or write those offline first apps, just use Hoody.  There's a talk, it's called, Coding Your Green,  oh, there's an end missing there, by Team Hoody at quarter past three in C117.  It's up back there.  And you might want to listen to that talk if you care about writing  offline first mobile apps.  And yeah, take a look at that talk.  Those custom implementation, like the PHP backend I just showed,  I want to use that to add some additional source to the data  and replicate out of CouchBeam.  So with those implementations, you sort of are a bunch of implementations  already available in Python, in JavaScript, and stuff like that.  You might use those to do something additional with the data  and do some fun things.  But it already works with the plain CouchDB and stuff like Hoody or CouchDB.  You can already write offline first mobile apps,  which will just work as the data will be consistent  over multiple mobile phones, over multiple notebooks,  over multiple servers, and stuff like that.  This is not only important, by the way, for the case where you have  mobile phones and servers, but also important for if you have  servers distributed across different continents, you cannot have,  it's not saying to have transactions across multiple continents  because the runtime of the request and response is just too long,  it takes too long for same response time on the web.  So you will need eventual consistency even there.  You might have heard about the base model and stuff like that.  So that's it, basically.  What we now have time for about 15 minutes or less,  if you want to leave earlier, is for questions, for discussions.  I can even jump into the code if you care about that.  Everything is online, as I mentioned.  The slides will be online at some point.  You'll find that I will Twitter the slides,  and my Twitter nick is on here.  You cannot read that, by the way.  So I will link them on Twitter.  All the links to the repositories are on there,  so you can jump into the code yourself and look at it.  But I can also show you something if you have questions.  So any questions left?  Yes?  What is the setup for multi-user scenarios?  So the question is, what would be the setup for a multi-user case  where you have certain data just for certain users?  The answer, let me answer with three points.  First, CouchDB.  In CouchDB, you usually use one database per user.  A CouchDB server easily works with 10,000 of databases.  So you can use one database for each user.  The next question is then, I will have documents  which will be synchronized between different users.  Like, for example, the Facebook case,  I post a message on the wall of another user.  What you can use for that is, again, replication,  and CouchDB supports filtered replication.  So a replication between user database A and user database B,  which just replicates the messages which are intended for user B.  This just works with a replication filter  and will also just work like this.  The only thing you need to do is pass,  okay, please use this filter.  Second answer is Hoody.  With Hoody, Hoody uses this aspect of CouchDB.  In Hoody, you register first, and every user gets its own database.  And then in Hoody, there is some very, very, very simple  JavaScript API in the browser where you can, say,  share this document, and Hoody will take care  of this replication stuff in the backend.  So with Hoody, it will just work.  I'm not sure about the stability of this plugin yet for Hoody,  because at least three months ago, it was still under development.  It might already work.  So I'm going to ask Olar and Lena later in the Hoody talk about this.  But it's going to just work in Hoody.  Point three, PHP, my PHP backend.  You can basically use anything you want here.  For example, you could parse an HTTP auth header with a username  and select a database based on that,  or add a pencil to the MySQL queries or whatever you want.  With PHP backend, yeah, it works just like,  it works in your PHP application.  You can do whatever you want to.  Does it answer the question?  OK, so that's the one reason you want to work with this.  It makes it easier, yes.  Other questions?  Yes, please.  You used to implement basically the entire CouchDB protocol, right?  You just have this one use case in mind.  It's specialized for that, but it's the entire protocol.  Yeah, the question is, did I implement the entire protocol?  Even I just had this one through the use case.  Yes, I'm just stupid.  And so the point is, and it happens to me again and again and again,  I have this open source project in mind I want to do.  I did that for a different project I wanted to implement.  And then I think, OK, I need that.  I should do this in the right way.  And then it starts to consume time.  And because I needed this CouchDB protocol,  I implemented this HTTP proxy replay stuff,  which I also did in the right way.  And for that, I had to implement something else.  So this is project B in the ABCD stack.  And I'm back to B right now because I want to complete the CouchDB endpoint.  And then I want to continue start working on project A.  And yeah, this is time consuming,  but I try to do it the right way  and try to contribute documentation and stuff like that.  But again, this is even more time consuming than, of course.  So yes, I implemented, I hopefully implemented most of it yet,  but not everything.  My endpoint doesn't support long polling, for example, yet,  for offset changes feed.  But I probably will even add that.  Yes.  And did you have to reverse engineer that too, or was that documented?  The question is, if I had to reverse engineer that,  if I had to reverse engineer most of it, there's some documentation.  The documentation focus is different than what I needed.  The documentation focus right now is on how to implement a replicator  and on how to implement a client.  And what I did was implementing something serverish,  which is slightly different than something client-ish.  I say client-ish and serverish because in CouchDB world,  everything is a client as a server, as a master all the time.  But so the documentation had a different focus.  And some of the algorithms like the religion divvying and stuff like that,  it's not documented yet.  And I had to reverse, or I still have to reverse engineer parts of that.  Yes, it's a lot of reverse engineering.  The problem is, Elan code is, I think, the most unreadable code ever.  I'm really, I'm similar with a bunch of functional languages.  But Elan is a different kind of thing.  Elan code is ugly as hell.  So it's really hard to read, understand those algorithms  from reading the CouchDB code, which is, of course, open source.  It's an Apache project. But yes.  So it's a little bit of soft code reading and stuff like that.  Other questions?  Three, two, one.  Then thank you and see you next time in the PHP room.  Thank you.
A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__0_59.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__60_191.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__192_1751.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__1752_2375.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__2376_5531.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__5532_7523.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__7524_10379.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__10380_23975.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__23976_24023.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__24024_24179.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__24180_25667.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__25668_26375.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__26376_29903.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__29904_30347.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__30348_33647.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__33648_35567.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__35568_41255.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__41256_41651.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__41652_42815.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__42816_42911.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__42912_42971.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__42972_43211.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__43212_43835.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__43836_44123.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__44124_44495.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__44496_44627.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__44628_44855.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__44856_45383.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__45384_45503.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__45504_45755.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__45756_46379.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__46380_47183.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__47184_47831.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__47832_47939.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__47940_48191.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__48192_50375.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__50376_50627.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__50628_50747.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__50748_50879.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__50880_50963.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__50964_51419.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__51420_51635.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__51636_52511.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__52512_53003.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__53004_53555.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__53556_53759.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__53760_53879.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__53880_53999.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54000_54251.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54252_54347.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54348_54407.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54408_54563.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54564_54659.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__54660_55319.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__55320_55811.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__55812_56159.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__56160_56399.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__56400_56759.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__56760_57131.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__57132_57395.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__57396_58127.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__58128_58523.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__58524_58631.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__58632_58883.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__58884_59003.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__59004_59243.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__59244_59483.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__59484_65831.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__65832_65963.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__65964_75107.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__75108_75239.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__75240_75311.png A_CouchDB_replication_endpoint_in_PHP__FrOSCon14__75312_75355.png