(Archived) HELP: Long initial sync times

[peterfmartin 221]

A recent exchange in the Mac 3.1.0 RC thread in the Mac forum probably deserves its own discussion, and it sounds like the right place may be here (not in the Mac forum). Here's the exchange:

(7) Slow download speeds

- Something went wrong and I had to re-install Evernote / re-download my data (under dlu's direction), but the "several minutes" that this was supposed to take has been several hours, with only a fraction of my notes populated. I've had the same problem when using offline notebooks on my iPad. I understand that I have an atypically large account (about 12 GB), but I can download 12 GB of data off of any other site relatively quickly. Why is Evernote so slow? A partial solution to this, of course, would be to provide desktop users with the same "offline/online" notebook capabilities that mobile users enjoy. I would actually prefer to be downloading hardly anything at all.

[EDIT:] Going into day two and I barely have 1/3 of my notes downloaded. Doesn't this seem too slow to anyone else?

GM, did you see the recent Blog where they announced that they have done some server upgrades/changes and now they have been able to cut the power per user in half. This sounds great, but maybe they optimized too far on minimizing power. I don't know. Here's what it said:

Besides making the service more responsive, these updates also resulted in a 50% reduction in per user power consumption for new users, which is pretty huge.

Supposed to be *more* responsive, but it doesn't seem that way.

When I sync'd my Wintel laptop today for the first time in a while, it had to add about 300 Notes (all very small, no PDFs).

I was surprised that it took about 15 min to complete the sync. I have a very large pipe (20-30 Mbps).

i didn't see that. thanks. maybe that is the reason.

i have a fast connection as well, and a few gigabytes ought to be no problem. when i tried to make everything offline last month so i could do work while away from the internet, my ipad took several days just to get a tiny fraction of the notes. it's nice to have the feature, but in practice, i cannot use it, because the download speed is so slow.

by this time next year, i guess it will take more than a week to restore my account if an upgrade doesn't install properly

i like the windows version a lot, but being trapped on my netbook while i wait for the download to finish is not fun.

I had a similar experience last week. On Thursday I had to downgrade to Mac 3.0.6 because support wanted me to test whether that would solve a problem I was having with 3.1.0. My account is now 9.05 GB [EDIT: See note below], so I didn't expect the sync to finish in a flash, but it ended up taking over 10 hours, which really shocked me, because that was much longer than previous initial syncs. Has anyone else, on Mac or Windows, experienced initial syncs this long recently?

NOTE: As I explain in this post, I have reason to believe my account may actually be a lot smaller than 9 GB, maybe as small as about half that.

[dlu 628]

Alrighty, I look forward to a decently long thread here... let's see if we can get to the bottom of this

[GrumpyMonkey 4,319]

Hi. As I have posted elsewhere in other threads, I have an extremely long time for initial syncs.

(1)

On the iPad, when I tried to switch a notebook to offline that contained several gigabytes (I forget how many, because the number showing was incorrect) it took several days, and I ended up having to leave for a trip before it finished. Rather inconvenient, to say the least. I have pretty much given up on offline notebooks, because it just takes too long to download (I usually try to keep things off my ipad when I am in town with a connection).

(2)

On Windows it took at least two days.

(3)

On the Mac I had to re-download everything (problem with the database) and it finally finished syncing last night: that was a total of about two and a half days as well.

My account is rather large I guess (6700 notes and several gigabytes), but as I can download several gigabytes of data in just a few minutes from every other site, I don't think the size of the account should have an impact. It appears to me that every note, no matter how big or small, takes a really long time to download. It is painful to watch.

Solutions?

1. Give us parity with the mobile platforms. If I had "offline" notebooks (or better yet, "offline" tags) I would have only downloaded a few notes. Good for me and good for Evernote servers!

2. Speed it up!

[Metrodon 2,184]

When I was testing for you last week Dlu my initial sync was very slow (as I mentioned in our PMs..)

[Yap 0]

The sync between PC and my android phone seems to be very very slow after the recent update. I wish that the developers will put some efforts in their project to fix this major problem.

It sync so long and I got frustrated and CRAZY !

[digitaltoast 2]

I've griped about this at least twice before, but might the problem not be exacerbated by the fact that evernote FORCES you to sync all your data before you can upload a tiny note on, say, a new or temporary laptop or machine?

How many terabytes of data do you think are floating about, unwanted and unused, because Evernote won't do what dropbox and others do, which is offer a "selective sync" option?

I know the suggested workaround is to have a load of different accounts, but, really.. that defeats the object of Evernote, doesn't it?

My point being, if you multiply Evernote's centrally available bandwidth by all the gazillions of gigabytes of note data being synched that people don't actually want, surely that's going to have an impact on overall system speed?

[JMichaelTX 4,117]

My point being, if you multiply Evernote's centrally available bandwidth by all the gazillions of gigabytes of note data being synched that people don't actually want, surely that's going to have an impact on overall system speed?

I don't think that should be a problem, because Evernote only syncs *changes*.

IAC, remember that you do have an option to create local-only Notebooks that are NOT sync'd.

These NB are stored ONLY on your local hard drive, and are NEVER transmitted to the Evernote cloud.

[digitaltoast 2]

My point being, if you multiply Evernote's centrally available bandwidth by all the gazillions of gigabytes of note data being synched that people don't actually want, surely that's going to have an impact on overall system speed?

I don't think that should be a problem, because Evernote only syncs *changes*.

Given that the title of this topic is "Long *initial* sync times", I can't see how that would make a difference here.

IAC, remember that you do have an option to create local-only Notebooks that are NOT sync'd.

These NB are stored ONLY on your local hard drive, and are NEVER transmitted to the Evernote cloud.

Wouldn't that rather defeat the object of wanting to store stuff in the cloud to make it available everywhere? I've already GOT the facility to store music, images and music locally - it's called my hard drive.

Incidentally, interesting that you're using Dropbox for the link in your sig and not evernote... any particular reason?

Not a criticism, just interested.

[JMichaelTX 4,117]

Incidentally, interesting that you're using Dropbox for the link in your sig and not evernote... any particular reason?

Not a criticism, just interested.

I find DropBox to be a better File Mgt tool than Evernote.

So it is much easier to update a file, and add a new file (like a screen shot) to DropBox than to Evernote.

Plus, when it is a file I want to share/display, it displays better from DropBox than the shared Evernote note.

Basically, faster, easier, better in DropBox.

[digitaltoast 2]

Basically, faster, easier, better in DropBox.

Hmmm, this kind of endorses my point, especially with the image sharing changes, it's just easier, quicker and better to use other things to do many of the things I COULD use Evernote for.

It's getting to be a crowded market out there; EN started well with some great clients, but they now really need to start listening to their customers to keep their share, let alone gaining new customers and converting them to pro.

[JMichaelTX 4,117]

Please note that my comments were with respect to file management.

Evernote is far superior for collecting and organizing, and searching information that is primarily not in files, although I do often attach files to a Note for supporting data.

But for files that are changing a lot, or just for a quick screen shot to share with others (like to embed in this forum), I like DropBox.

[Anubha 10]

Solution please! Speed this up! It's a pain to download huge notebooks. Why is the download sooo slow??

[GrumpyMonkey 4,319]

Solution please! Speed this up! It's a pain to download huge notebooks. Why is the download sooo slow??

I posted a thread somewhere in this forum about my speedy initial sync this week. My notes downloaded in few hours compared to an initial sync a few weeks before that stretched out over several days. I am wondering if I just got lucky (perhaps everyone was asleep around the world and I had the Evernote servers all to myself) or Evernote made an improvement. If the latter, I hope someone takes credit for making this needed and very welcome change!

[Anubha 10]

Solution please! Speed this up! It's a pain to download huge notebooks. Why is the download sooo slow??

I posted a thread somewhere in this forum about my speedy initial sync this week. My notes downloaded in few hours compared to an initial sync a few weeks before that stretched out over several days. I am wondering if I just got lucky (perhaps everyone was asleep around the world and I had the Evernote servers all to myself) or Evernote made an improvement. If the latter, I hope someone takes credit for making this needed and very welcome change!

Yeah, that happened with me as well. But I think there isn't an improvement yet and I say this becasue, I also had a very speedy sync compared to what would previously take me days. So I also, at first, thought that it's all fine and dandy now. But turns out, the problem remains. I tried syncing my device earlier this morning, and just for a handful of notes, it still hasn't finished downloading them all! Don't know when can we expect a solution, but I am seriously looking forward to one in the near future...

[GrumpyMonkey 4,319]

Solution please! Speed this up! It's a pain to download huge notebooks. Why is the download sooo slow??

I posted a thread somewhere in this forum about my speedy initial sync this week. My notes downloaded in few hours compared to an initial sync a few weeks before that stretched out over several days. I am wondering if I just got lucky (perhaps everyone was asleep around the world and I had the Evernote servers all to myself) or Evernote made an improvement. If the latter, I hope someone takes credit for making this needed and very welcome change!

Yeah, that happened with me as well. But I think there isn't an improvement yet and I say this becasue, I also had a very speedy sync compared to what would previously take me days. So I also, at first, thought that it's all fine and dandy now. But turns out, the problem remains. I tried syncing my device earlier this morning, and just for a handful of notes, it still hasn't finished downloading them all! Don't know when can we expect a solution, but I am seriously looking forward to one in the near future...

Interesting. Well, I am going to sync my notes to my Windows computer today, and we'll see what happens (there is a backstory to this that explains what I am doing with yet another Windows machine, and why I haven't been able to use Evernote on Windows for a couple of weeks, but that is for another thread).

[Anubha 10]

Solution please! Speed this up! It's a pain to download huge notebooks. Why is the download sooo slow??

I posted a thread somewhere in this forum about my speedy initial sync this week. My notes downloaded in few hours compared to an initial sync a few weeks before that stretched out over several days. I am wondering if I just got lucky (perhaps everyone was asleep around the world and I had the Evernote servers all to myself) or Evernote made an improvement. If the latter, I hope someone takes credit for making this needed and very welcome change!

Yeah, that happened with me as well. But I think there isn't an improvement yet and I say this becasue, I also had a very speedy sync compared to what would previously take me days. So I also, at first, thought that it's all fine and dandy now. But turns out, the problem remains. I tried syncing my device earlier this morning, and just for a handful of notes, it still hasn't finished downloading them all! Don't know when can we expect a solution, but I am seriously looking forward to one in the near future...

Interesting. Well, I am going to sync my notes to my Windows computer today, and we'll see what happens (there is a backstory to this that explains what I am doing with yet another Windows machine, and why I haven't been able to use Evernote on Windows for a couple of weeks, but that is for another thread).

Sure! Do keep us posted, in case you happen to figure anything new pertaining to this issue...

[peterfmartin 221]

I started an initial sync before going to sleep Sunday night. It was done when I woke up this morning—meaning it took around 30 hours. Sigh.

Dlu or other Evernote employees, if activity logs or anything on my end can be of use, please let me know.

[JMichaelTX 4,117]

I started an initial sync before going to sleep Sunday night. It was done when I woke up this morning—meaning it took around 30 hours. Sigh.

Wow. This seems excessive, but I can't really judge without some more data.

Would you mind sharing:

1. Total size (GB) of your Evernote folder
   
2. Total number of notes
   
3. Reinstall, or new computer
   
4. If new computer, brand/model
   

Thanks.

[GrumpyMonkey 4,319]

I started an initial sync before going to sleep Sunday night. It was done when I woke up this morning—meaning it took around 30 hours. Sigh.

Wow. This seems excessive, but I can't really judge without some more data.

Would you mind sharing:

1. Total size (GB) of your Evernote folder
   
2. Total number of notes
   
3. Reinstall, or new computer
   
4. If new computer, brand/model
   

Thanks.

I think the initial sync times are par for the course. I have reported long sync times in various threads in the past, with one exception, where I believe I completed the initial sync well under 48 hours, and I was wondering if Evernote had changed something. I am now thinking that the sync times fluctuate a little, but I just don't know the reasons.

My account is about 12 GB and usually takes a couple of days (leave the computer on syncing the entire time) in most cases, but sometimes more. My iPad takes more than a week, and I have never actually completed a full initial sync with it (I once tried to put everything offline, but had to leave town before I could accomplish it, and have not tried it again since then). I had the same experience with my iPhone. For various reasons, I have had to perform several initial syncs on my Windows machine, and I have not been able to use Evernote there for about a month. All of these are new (within the last few months).

If Evernote is throttling us (this is the only explanation I can think of, because I have plenty of bandwidth, and regularly download/upload several gigabytes of data with ease), then it would probably be in order to distribute the load for the servers (pure speculation here), and keep Evernote running smoothly (not everyone agrees with my hypothesis ). If so, I think that is great, and totally understandable, but I would like to know if this is the reason, and if there are better times for us to attempt syncs like this. Hopefully, as server capacity increases, they will consider changes to make sync times faster. By this time next year, it will easily take me a week to complete an initial sync, and because we have no way to make desktop notebooks toggle offline/online (as we do on mobile) I have to go through the process before I can use Evernote.

[digitaltoast 2]

My account is about 12 GB and usually takes a couple of days (leave the computer on syncing the entire time) in most cases, but sometimes more. My iPad takes more than a week, and I have never actually completed a full initial sync with it (I once tried to put everything offline, but had to leave town before I could accomplish it, and have not tried it again since then). I had the same experience with my iPhone. For various reasons, I have had to perform several initial syncs on my Windows machine, and I have not been able to use Evernote there for about a month. All of these are new (within the last few months).

Would it be easier if, like Dropbox, you could choose which, if any (or all) folders you wanted to sync? Do you NEED that entire 12Gb over all those devices?

Each time I've made this suggestion (many times now!) people come out with stuff like "but adding selective sync would be FAR too expensive", as if added an extra 1 byte flag in the database would take a skilled team of programmers weeks, vs all the complaints about slow initial sync and all the hundreds of unwanted terabytes being downloaded to devices and using bandwidth.

[GrumpyMonkey 4,319]

My account is about 12 GB and usually takes a couple of days (leave the computer on syncing the entire time) in most cases, but sometimes more. My iPad takes more than a week, and I have never actually completed a full initial sync with it (I once tried to put everything offline, but had to leave town before I could accomplish it, and have not tried it again since then). I had the same experience with my iPhone. For various reasons, I have had to perform several initial syncs on my Windows machine, and I have not been able to use Evernote there for about a month. All of these are new (within the last few months).

Would it be easier if, like Dropbox, you could choose which, if any (or all) folders you wanted to sync? Do you NEED that entire 12Gb over all those devices?

Each time I've made this suggestion (many times now!) people come out with stuff like "but adding selective sync would be FAR too expensive", as if added an extra 1 byte flag in the database would take a skilled team of programmers weeks, vs all the complaints about slow initial sync and all the hundreds of unwanted terabytes being downloaded to devices and using bandwidth.

Hi. Yes, I would like selective syncing, and I have posted quite a bit requesting this feature to be put on desktop clients (mobile clients already have it). Phil Libin (CEO) has already said it is inevitable, so I think it is coming, but we just don't know when.

I don't remember anyone saying that selective syncing would be expensive to implement, but I imagine it would cost something to develop a new feature. Whatever the cost of adding the feature to the desktop clients (again, it already exists on mobile), surely it is offset by the reduction in file transfers.

In my case, I have done several initial syncs lately, and the 60+ GB I must have downloaded over that period could have been reduced to about 1GB if I had selective sync. I only need to have a few files on my drive, because I have regular access to the Internet.

[digitaltoast 2]

I don't remember anyone saying that selective syncing would be expensive to implement, but I imagine it would cost something to develop a new feature. Whatever the cost of adding the feature to the desktop clients (again, it already exists on mobile), surely it is offset by the reduction in file transfers.

In the thread titled How do I NOT sync certain notebooks, apart from cancel premium? I said:

...So I download the beta onto my little palmtop with 8Gb SSD on our 2Mb rural broadband and guess what? The damn thing starts trying to sync gigs of data again!

I just want to upload 10 or 11 screen grabs each day without having to download 3 GB or so of data....

to which Metrodon (2876 posts) replied quite robustly and with some certainty:

Yeah, but the platform it already exists on is a very widely used one. Building the same functionality across the other dozen seems really unlikely based on the affected number of users and maintaining parity is extremely expensive. The reality is that parity will happen for functionality that impacts the most users, bangs per buck. Small subsets of users, expert users who need complex search operators, users with use cases from off the beaten track are less and less likely to be satisfied as Evernote grows. The bulk of Evernote's revenue is going to come from very 'ordinary' users, millions and millions of them - keeping these people happy is where the $$$s are.

Plus, Heather has made their position pretty clear in this thread already.

Incidentally, Heather's "pretty clear position" on syncing clients with limited drive space, low bandwidth, and presumably large accounts like yours is.. don't use the client.

On a netbook, we *always* recommend using Evernote Web. It's that simple.

Additionally, if you want to have specific notebooks synced to your desktop, and only those notebooks, I recommend having two accounts. Selectively share the notebooks you want from your main account to your secondary, and then only load up your secondary account on that machine. Then, you can choose which notebooks you want to sync out of the ones that have been shared. It's not the most elegant of workarounds, but it works.

This has been explained multiple times throughout this thread, in many ways, but I do hope my word will help staunch the misinformation and confusion of the OP.

I don't see anything there, or subsequent, that suggests Evernote have any intention or implementing selective sync. Far from it; the suggestion is just to use the web version.

[GrumpyMonkey 4,319]

Hi. I see.

Well, I agree with Metrodon's point that Evernote ought to focus on developing features that impact a lot of users rather than developing ones that will only benefit a few of them. I can think of exceptions, but overall I think this is the best way forward for the company. I disagree with him that a change like selective syncing would cost Evernote more money, but frankly, neither of us has the data to say anything about this. We are speculating, and ultimately, we are in agreement about how Evernote ought to approach the issue.

I think we have different interpretations of Heather's position. She is offering a workaround under the current system. The suggestions won't work for me (for various reasons-- I have posted about these in other threads on the Web client and shared notebooks), but it might work for others. I don't think she wants us to stop using Evernote

Phil Libin (CEO) has said such a feature is inevitable. They have 100 years to do it, so we might be waiting a while. I am sure they have to prioritize things, and I cannot guess where such a thing would fall on the roadmap.

[Metrodon 2,184]

Grumps

I wasn't saying that selective syncing specifically would cost more money - just that maintaining parity across platforms is expensive.

I am confident that memory (flash and SSD) will continue to drop in price and so larger and faster storage will become more available. Because of this, selective sync becomes a less and less attractive feature to spend money on.

[GrumpyMonkey 4,319]

Grumps

I wasn't saying that selective syncing specifically would cost more money - just that maintaining parity across platforms is expensive.

I am confident that memory (flash and SSD) will continue to drop in price and so larger and faster storage will become more available. Because of this, selective sync becomes a less and less attractive feature to spend money on.

We agree there. Maintaining parity is expensive, but in this case, I am guessing that the ever-increasing account sizes and user base puts more demand on the servers, and the option to do offline/online syncing would perhaps save money in the long run.

I know I am a bit of an oddity with so many devices, but I would guess a lot of users have at least two (one at home and one at work) and few of them want to download the entire database into their work computer (in fact, many are very much not wanting to do this).

SSD will drop in cost. In fact, there was apparently a steep drop recently, far sooner than I predicted in an earlier thread. We live now, though. We don't live in the future, and I don't see the size of storage growing over time. If anything, it is shrinking, as people shift more and more of their work to the cloud, and local storage becomes less of a necessity. The Google Nexus is one example. Predictions of widespread tablet use within the next three years point to smaller SSD sizes as well.

I am not great at predicting the future, especially with innovative companies like Evernote, Google, and Apple radically changing my world every year. All I can say is that the current system could use improvements. In my case (as mentioned before), the slow initial sync times have prevented me from using the Windows client for nearly a month (multiple initial syncs because of issues -- support tickets filed). I am certainly not saying I am the average user, but I am suggesting that one way to alleviate the problem would be to have the offline/online notebook toggle.

As I said, this is something that Phil Libin apparently thinks is coming (based on his comments in the podcast), so it isn't really an issue of whether Evernote should have the ability, but when.

[peterfmartin 221]

I started an initial sync before going to sleep Sunday night. It was done when I woke up this morning—meaning it took around 30 hours. Sigh.

Wow. This seems excessive, but I can't really judge without some more data.

Would you mind sharing:

1. Total size (GB) of your Evernote folder
   
2. Total number of notes
   
3. Reinstall, or new computer
   
4. If new computer, brand/model
   

Thanks.

1. I don't know how to find that. I asked here, and no one responded.

2. A little over 20,000.

3. Reinstall.

4. Nope, same computer I've been using since before I started using Evernote.

EDIT about #1: The total size of my /Library/Application Support/Evernote folder, which I backed up before deleting it to do the initial sync, was about 17.5 GB. But, as I explained in the linked post, that can't be the size of my account, since I've only been a premium user for nine months.

[JMichaelTX 4,117]

I think the initial sync times are par for the course. I have reported long sync times in various threads in the past, with one exception, where I believe I completed the initial sync well under 48 hours, and I was wondering if Evernote had changed something. I am now thinking that the sync times fluctuate a little, but I just don't know the reasons.

. . .

If Evernote is throttling us (this is the only explanation I can think of, because I have plenty of bandwidth, and regularly download/upload several gigabytes of data with ease), then it would probably be in order to distribute the load for the servers (pure speculation here), and keep Evernote running smoothly (not everyone agrees with my hypothesis http://discussion.ev...se/#entry147903).

GM, as I mentioned at the beginning of this thread, in early May 2012 Evernote made some major server upgrades/changes and now they have been able to cut the power per user in half. This sounds great, but maybe they optimized too far on minimizing power. Since they made that change, we all have noticed very significant increase in sync times, especially initial sync time for a new or clean reinstall.

It is not clear whether the increase in sync time is by design, or is an unintended consequence. Since you mentioned throttling, it does make me wonder if Evernote has done this on purpose in order to prepare for the large number of expected users from China and India. It is pretty clear to me that Evernote has been focused on growth for some time. I'm all for growth, but not at the expense of performance or service for existing users.

[GrumpyMonkey 4,319]

I don't know what to think, because I don't have enough information. I am only speculating about the causes. It could be throttling, or it could be something else. I would just like to know why the initial sync is so slow, and/or have it sped up