Jump to content
  • 0

(Archived) Workflow for LAN syncing


Pete248

Idea

Some time ago, I've asked, whether it would be possible, that multiple Macs within a LAN, all connected to the same Evernote account, could speed up syncing and save on internet bandwidth, by syncing locally instead of each client syncing individually with the Evernote server. Lets say you have 5 Macs in your LAN and scan a 2 MB PDF into your Evernote account, this creates 12 MB of traffic - the initial upload and then the download on each of the clients.

Somehow the message vanished from the forum (at least I can't find it any more). Dave's answer was, that syncing between multiple local instances of the Evernote database would be a difficult task and that it was currently not planned to implement such a feature.

I rethought my suggestion and want to present some ideas, how this could be implemented. I think this would be as well a real improvement for the Evernote experience on the upcoming iPad - I wrote about this in a separate post.

I meanwhile think, LAN syncing would work like the current syncing with the Evernote server with just one additional step in-between. Let's look, how syncing currently works in principal (Dave, please correct me if something is wrong):

1.) The Evernote client connects to the Evernote server via Internet.

2.) The Evernote client uploads all notes, that are newly created or changed since the last sync.

3.) The Evernote client downloads all notes, that have been created or were changed since the last sync.

To implement LAN syncing, it would be necessary to load a small Evernote demon app on startup or login. All Evernote demons can detect other Evernote demons within the LAN via Bonjour.

A sync would now work as follows:

1.) The Evernote client connects to the Evernote server via Internet.

2.) The Evernote client uploads all notes, that are newly created or have changed since the last sync.

3.) The Evernote client retrieves a list (prob. containing guid, updateSequenceNum, contentHash) of notes to be downloaded

4.) The Evernote client asks via the Evernote demon, whether any Mac on the LAN has already one of these notes (same guid, updateSequenceNum and contentHash), then retrieves it from that Mac via its Evernote demon and finally removes this note from the list.

5.) The Evernote client downloads all remaining notes on the list from the Evernote server.

So just step 4 is inserted into the current process.

In case no new/updated notes are found on the LAN or no Evernote demons are installed on other machines, the sync process works exactly as it does currently. This means as well, implementation of LAN syncing would work as well in mixed environments, where not all clients are already capable of LAN syncing. Thus you could implement this into each platform one by one without the need to roll it out for all platforms simultaneously to prevent, that syncing breaks in mixed environments.

Dave, I'd like, if you could discuss this with you engineers.

Thanks

Pete

Link to comment

4 replies to this idea

Recommended Posts

Evernote's synchronization model is "client/server". The service shows the master state of your account, and each client reconciles its changes directly against the service.

Implementing this correctly is relatively hard, since we need to handle tricky cases like:

Create Notebook B

Move Note 1 from Notebook A to Notebook B

Delete Notebook A

Sync

The client needs to make sure to notify the server about the new notebook creation before the move, and the move before the delete, or else the server can't reconcile the results.

This becomes even harder when you consider that other clients may be making other changes that need to be reconciled with the clients changes. For a taste of this complexity, see:

http://www.evernote.com/about/developer ... m-sync.pdf

This type of "client/server" synchronization protocol is relatively hard, but we followed a few examples of successful client/server systems like IMAP to make it possible.

Implementing correct synchronization in a peer-to-peer fashion goes from "relatively hard" to "really really really hard". Changes that you make on Mac #5 and sync to Mac #4 may be incompatible with changes you make on Mac #3 and sync to Mac #2, and the whole thing becomes a gigantic mess. This is a classic database problem that is virtually impossible in the general case (and just "very very hard" in more constrained cases).

For our consumer Internet service running in the year 2010, we'd rather put our resources into stability and features rather than trying to make a one-time initial sync on your third Mac computer a bit faster.

Thanks

Link to comment

Dave,

Evernote's synchronization model is "client/server" ...

... Implementing correct synchronization in a peer-to-peer fashion goes from "relatively hard" to "really really really hard".

please reread my original post. I didn't talk about a client to client synchronization without access to the Evernote server. My suggestion was to use the same "client/server" model to retrieve the master state from your server and determine what content has to be downloaded from the server, but instead of blindly downloading this content from your server, to check first, whether this content is already available somewhere on the LAN and then get it from there instead. Via content hashes you could double verify, that content on the LAN is identical to the content on your server. Probably I should have named it "LAN assisted syncing" rather than "LAN syncing" to make it clear.

For our consumer Internet service running in the year 2010, we'd rather put our resources into stability and features rather than trying to make a one-time initial sync on your third Mac computer a bit faster.

It is not just a weird request from someone using a 199X internet connection, nor a geeky wish to win a price for the fastest Evernote sync benchmark. ;-)

Even in 2010 internet connectivity in Europe (and it is not much different in the US) is pretty poor outside the big cities. I.e. one of our offices has a single lame 1 MBit internet connection, that is shared by up to 10 computers. As all these machines have to maintain a full time connection to an offsite database server, we use bandwidth management on the router, plus a strict policy, what is allowed to do via the internet connection and what is not, to keep the whole process fluid. To use Evernote it would be necessary to schedule all syncing at night time, which would mean, I would have to automatically power on/off all machines at night time to make them sync with the Evernote server. This is something I do not want to do for several reasons, like security issues, fire protection requirements etc.

Another of our offices even has internet connectivity only via 3G. Traffic is bloody expensive in compare to flatrates (you can't get a true 3G flatrate in Germany from any provider). It is annoying to pay for multiple downloads of the exact same content that is already present on the LAN.

I'm lucky, my office has a 100 MBit downstream internet connection, so I wouldn't benefit from LAN assisted syncing. Really? I've created a couple of notes with 3 MB attachments and measured the bandwidth used while the Evernote client on another machine was downloading the content. Bandwidth was around 50 KB/s, occasionally spiking to 300 KB/s for a fraction of a second. This translates to 0.4 to 2.4 MBit. Compare this to the 1000 MBit I would get, if the content would have been retrieved from the LAN instead.

So until we see true 100 MBit internet speed from your Evernote server to all civilized places over the world (which will take at least another 20 years, likely more), LAN assisted syncing offers a huge speed boost.

Just theoretical blurb, that doesn't translate into the real world? Well, all I can say is, I've seen a dramatic improvement in syncing speed on my Dropbox account, after I've installed a new version of the client, that had exactly this kind of LAN assisted syncing implemented. The Dropbox people obviously realized there is enough potential in LAN assisted syncing to put some effort in implementing this into their clients. Well done!

While the exact mechanism to do a sync might be different between Evernote and Dropbox, the main principle of syncing content stored on local clients with a server is the same. So what the Dropbox people implemented into their clients should work after adoption to the Evernote API as well with your service.

I'm pretty sure, you will have to implement some method of directly accessing content from other Evernote clients on the LAN, especially with the iPad (see my corresponding article).

Dave, you shouldn't dismiss my feature request for LAN assisted syncing prematurely as undoable or potential hazardous for the stability of your service.

I'd like to second my request, that you might discuss this internally with your engineers for the benefit of all Evernote users.

Pete

Link to comment

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...