Sync failed with errorCode=BAD_DATA_FORMAT

[Rob Freundlich 146]

In the last couple of days, my syncs have started failing with errorCode=BAD_DATA_FORMAT. Here's the log of my last sync attempt:

Evernote for Windows 6.11.2.7027 (307027) Public
OS: Windows 10.0.0
User: ME
Memory: 193 MB
Created: 2018/07/22 11:19:39
---
Bootstrap info: loadedFresh
<bootstrap_info><profile name="Evernote"><settings account_email_domain="m.evernote.com" cardscanUrl="https://cscan.evernote.com/cardagain" enable_facebook_sharing="1" enable_gift_subscriptions="1" enable_google="1" enable_linkedin_sharing="1" enable_shared_notebooks="1" enable_single_notesharing="1" enable_sponsored_accounts="1" enable_support_tickets="1" enable_twitter_sharing="1" marketing_url="https://evernote.com" service_host="www.evernote.com" support_url="https://help.evernote.com/hc/"/></profile></bootstrap_info>
---
11:19:05 [INFO   ] [4084] [34132] 100% Retrieving lock status for 11 notes from "www.evernote.com/s26"
11:19:05 [ERROR  ] [4084] [34132] 100% EDAMUserException: errorCode=DATA_REQUIRED parameter="authenticationToken"
11:19:05 [INFO   ] [4084] [35396] Client synchronization started
11:19:05 [INFO   ] [4084] [33180] 0% Loaded updateCount: 139041
11:19:05 [INFO   ] [4084] [33180] 0% Usage Metrics: sessionCount=0
11:19:05 [INFO   ] [4084] [33180] 0% SyncState: devices used: 15
11:19:05 [INFO   ] [4084] [33180] 0% Client is up to date with the server, updateCount: 139041
11:19:06 [INFO   ] [4084] [33180] 0% Updating server items
11:19:06 [INFO   ] [4084] [33180] 100% Uploading 262 item styles
11:19:06 [ERROR  ] [4084] [33180] 100% EDAMUserException: errorCode=BAD_DATA_FORMAT parameter="preferencesToUpdate"
11:19:06 [INFO   ] [4084] [35396] Client synchronization finished, status: failed
11:19:06 [INFO   ] [4084] [35396] * elapsed time: 1s
 

Thoughts?

[EdH 1,670]

I would contact Evernote support directly. 

If you want to do some troubleshooting, sort your notes by date modified and see which have the black dot (list view) and that will show which ones haven't been uploaded. It generally syncs oldest to newest, so maybe export the oldest note and delete it, then go to the trash and delete again. See if that clears it up. You can reimport the note when done.

But other than that, I don't see anything in your log you posted that will tell you exactly which note(s) is causing the problem. But 262 item styles suggests a lot of stuff that hasn't been uploaded.

[Eldorado 26]

Contacting EN support directly is a good suggestion for such specific sync problems. But they will beg you for updating from 6.11 to current 6.13.14 since have been some fixes around sync problems in the last versions... I would try this first before wasting time while trying to play with sync options and own tests ;-)

Or (if you have time):

• Is it really a couple of notes that will not sync?
  • try to show notes in Top List format (Ctrl-F7), RClick in headlines and enable display of "Sync" information.
  • Then click on "Notebooks" in left pane to show all notes and click on column "Sync" to move unsynced notes to top of list
  • If there a no unsynced notes, You may have an other problem...
     
• Do you use many Styles with Notebooks, Tags and Shortcuts?
  • My question comes from a log entry "Uploading 262 item styles" - but it shows a 100% completion (so this seen not to be the problem)
     
• Last try: "preferencesToUpdate"...
  • Do You have enabled or changed setting of  "on demand sync" or some other tricky (sync) options?
  • If I remember correct, there has been some trouble with these options in younger past...

[Rob Freundlich 146]

I just updated to the latest version, and the problem still happens. Nice tip about the List format - I didn't even know about the Sync information there! That yielded some very interesting results: there are a bunch of notes with a black outlined circle in the Sync column, and if I try to click one of them, an equivalent note elsewhere in the list, without the black circle, will be selected. The equivalent note has the same title and metadata, but different created and updated dates.

Yup, time to ask support about this one.

[Rob Freundlich 146]

I have engage customer support on this, and the problem is that I have too many styles defined. Removing styles from some of my tags solved the problem. Here is the email I sent to support in response:

 

Quote

 

Removing styles from a bunch of tags solved my sync problem. I am extremely unhappy with this solution:

1. Evernote encourages me, as a user, to use tags heavily as an organizational tool, and so I have done so.
2. Having a large number of tags can sometimes be unwieldy, and so I have created tag hierarchies using "." as a separator. For example, I have the following tags:
   • school
   • school.2015-2016
   • school.2016-2017
   • school.2017-2018
   • school.activity
   • school.high_school
   • school.middle_school
   • school.college
   • school.conference
   • school.fencing
   • school.form
   • school.homework
   • school.newsletter
   • school.reportCard
   • school.sports
   • school.trip
3. I have certain tag hierarchies that I want to stand out, and so I have set up colors for them. I would like my "school" hierarchy to be colored purple.
4. However, since tag hierarchies are not really supported in Evernote, I can't just assign purple to my "school" tag and have it inherited by all of the other tags in the hierarchy. So I have to assign it to all 16 tags in the hierarchy. I've just used up 16 style slots instead of 1.
5. If I look at my tags view, with all of my hierarchies collapsed, I see 6 tags that have styles attached to them. As for notebooks, I have 2 with styles attached.
   • So if you fully supported tag hierarchies, with style inheritance, I would have 8 styles in use. Instead, because you don't, I have hit the style limit.
   • In fact, I only have 110 top-level tags, and 84 Notebooks. So even if I assigned a style to every top-level tag and every notebook, I would only need 194 styles. But because you don't fully support tag hierarchies with style inheritance, this is not an option for me

Do you understand why I am extremely unhappy with being told that having a sync problem is my fault for having too many styles, and that in order to sync, I have to remove styles from my already sparsely-styled system?

 

How can you, as a company, address this?

1. Support tag hierarchies fully in your product, including style inheritance. This is something that customers have been asking for for years. It would solve not only this problem, but many others.
2. Eliminate the 250-style limit.
   • Seriously - where does this come from? It's not a because you're using a single byte somewhere as a style ID, because bytes hold 256 values (unless you've got 6 reserved styles).
   • But you store everything in a database. Who uses single bytes to store things in a database?
   • And if you are using a single byte, go big and double it up - use two bytes, increasing the limit to 65K styles. We'll thank you for it!

 

It will be interesting to see what kind of response I get.

[dconnet 529]

23 minutes ago, Rob Freundlich said:

It will be interesting to see what kind of response I get.

I can't say what the plans are (I'm just a dev - that's above my pay grade!), but I can say the styles are something the Windows team hacked in. There's a limit of 250 because it's using a generic storage (that is synced) that only has 250 slots. And these slots are used by other things too, like shortcuts. I'm surprised it causes a sync issue - that's something we'll have to look into...

[Rob Freundlich 146]

I got a response this morning:

 

Quote

 

Hi Rob,

Thanks for getting back to me. I'm glad that your Evernote account syncs normally again.

I totally understand your frustration about the tag hierarchy and that the tag style of the main tag is not passed on to the nested tags. Your suggestions that you shared with me in your previous email do make sense to me and I think that passing on the style to the nested tags would be a very practical feature to have in Evernote.

This is why I am pleased to forward your two ideas below to our product development team, that reviews feedback submitted by our users frequently.

1. Support tag hierarchies fully in your product, including style inheritance. This is something that customers have been asking for for years. It would solve not only this problem, but many others.
2. Eliminate the 250-style limit.

Thank you again for the time to state your suggestion and your ideas about how tags could be handled within Evernote. I really appreciate this and hope that a feature update will address your request about the tags and the styles.

If you have any further questions or concerns please feel free to reach out.

Best regards,

(I'll leave out the person's name to protect their privacy)

 

 

OK, I'll accept that. It's a very appropriate response, and the kind of answer I'd feel comfortable giving to one of my own users.

I'm a little skeptical, given how long the tag hierarchy thing, especially, has been around, but I'll give them a chance on this one.

 

[jefito 5,589]

16 hours ago, dconnet said:

but I can say the styles are something the Windows team hacked in.

Once again, no good deed goes unpunished.  

 

...

On the notion of "supporting tag hierarchies fully": whatever that means, it should be optional. Not all Evernote users want that. Not all users organize all of their tags to indicate semantic hierarchy; tag names can be overloaded to mean different things in different contexts, just as normal words can be. Sometimes I'd like search to recognize the tag hierarchy, similar to the kinda/sorta way you can do with the Windows application's "Automatically select child tags" option (this is also a Windows team hack), but more often I don't. One of the reasons that Evernote appeals to me is that it doesn't force me into a tree-shaped box: I can organize by description, rather than by structure, which suits me because a note may pertain to multiple areas of interest. So sure, add in the capability for styles, searches, etc. to be used hierarchically; just don't make me use them.

[gustavgi 308]

11 hours ago, Rob Freundlich said:

I got a response this morning:

 

 

OK, I'll accept that. It's a very appropriate response, and the kind of answer I'd feel comfortable giving to one of my own users.

I'm a little skeptical, given how long the tag hierarchy thing, especially, has been around, but I'll give them a chance on this one.

 

I told you about the 255 style limit over two years ago though, when the feature was relatively new, and nothing has changed since then unfortunately.

I ran into the synch problem/limit a couple of days after the feature was implemented, and I've been wishing for the limit to be lifted since then (I would hate if Evernote implemented any form of tag inherentence though).

But the feature is and has been a Windows-only icing on the cake feature as it is though, so I am also just glad that they implemented it in the first place.

[Rob Freundlich 146]

23 hours ago, jefito said:

On the notion of "supporting tag hierarchies fully": whatever that means, it should be optional. Not all Evernote users want that. Not all users organize all of their tags to indicate semantic hierarchy; tag names can be overloaded to mean different things in different contexts, just as normal words can be. Sometimes I'd like search to recognize the tag hierarchy, similar to the kinda/sorta way you can do with the Windows application's "Automatically select child tags" option (this is also a Windows team hack), but more often I don't. One of the reasons that Evernote appeals to me is that it doesn't force me into a tree-shaped box: I can organize by description, rather than by structure, which suits me because a note may pertain to multiple areas of interest. So sure, add in the capability for styles, searches, etc. to be used hierarchically; just don't make me use them.

Absolutely! I'm a huge fan of options, especially for something like this.

[Rob Freundlich 146]

14 hours ago, gustavgi said:

I told you about the 255 style limit over two years ago though, when the feature was relatively new, and nothing has changed since then unfortunately.

I find this whole 250-style limit to be very bizarre. I'm a software developer, so I'm going to take that one data point and reverse-engineer the most likely design from it, laying out my logic along the way.

Point 1: There's a style limit. From this, I will draw the following conclusions about the design:

1. There is an entity in the Evernote system called a Style

2. A Style describes color, font attributes (bold, italic, etc) and has a unique identifier

3. Style entities are stored in a separate file or table from other entities

4. An entity that can be styled, such as a notebook or tag, has a reference to its Style entity - most likely the Style entity's unique identifier.

Point 2: The limit on styles is 250. From this, I will draw the following conclusions about the design:

5. 250 is very close to 255, which is the number of unique things that can be identified by a 1-byte value. Therefore it is very likely that the unique identifier for a Style entity is a 1-byte value.

6. The file or table storing Style entities is, for some reason, limited to 250 (or, more likely, 255) entries.

These aren't the only possibly conclusions, but they are logical, and fit the facts we have been provided with. Also, they are pretty minimal and simple, and in general, simple and minimal are often (but not always) the best.

Based on these conclusions, plus the fact that Evernote is known to be built on SQL database technology, I will now sketch out what I'd expect to find in the database.

#1: A table called something like STYLES, with an identifier column called something like ID, of type CHAR(1) (which is a single byte type), and columns for color and font attributes. For some reason, this table will have a maximum number of rows set to 250, if that is even possible. I don't know enough about databases to say whether it is.

#2: A table called something like NOTEBOOKS, with, among other things, a column named something like STYLE_ID, which is an ID value from the STYLES table.

#3: A table called something like TAGS, with, among other things, a column named something like STYLE_ID, which is an ID value from the STYLES table.

Again, not the only possible implementation, but it's logical and fits the conclusions. Also again, they are simple and minimal, which often are the best.

So now I'll close Evernote, fire up my DBBrowser for SQLite, and take a look at the Evernote database structure to see how I did with my reverse engineering. Here's what I actually find:

1. NO styles table at all.

2. A table called notebook_attr that pretty clearly contains all of my notebooks. Interestingly, it includes a column called item_color and another called item_style.

• For notebooks I haven't assigned a color to, item_color is -1
• For notebooks that I have assigned a color to, item_color is the decimal value of the color's hex value
• For notebooks I haven't set any font attributes for, item_style is null (the SQL equivalent of "no value")
• For notebooks I have set any font attributes for, item_style is a decimal value

3. A table called tag_attr that pretty clearly contains all of my tags. Interestingly, it includes a column called item_color and another called item_style, whose values are similar to the values in notebook_attr (although the "no font attributes" value for item_style here is 0 rather than null.

Interesting. The color and font information are stored with each notebook and tag, not in a separate table. So from a data storage perspective, there is no reason whatsoever for there to be a limit on styles.

So where does this limit come from? Are they somehow condensing the styles into a structure like I reverse-designed as part of the sync process? If so, what on earth for?!?!? Why not just sync the information as-is?

I'm baffled. And frustrated.

 

[jefito 5,589]

5 hours ago, Rob Freundlich said:

Interesting. The color and font information are stored with each notebook and tag, not in a separate table. So from a data storage perspective, there is no reason whatsoever for there to be a limit on styles.

So where does this limit come from? Are they somehow condensing the styles into a structure like I reverse-designed as part of the sync process? If so, what on earth for?!?!? Why not just sync the information as-is?

Per @dconnet's comment, it sounds like it's not a data storage limit (i.e., in the database); it appears to be related to a storage used in the service (the hint is "that is synced")

On 7/26/2018 at 6:36 PM, dconnet said:

I can't say what the plans are (I'm just a dev - that's above my pay grade!), but I can say the styles are something the Windows team hacked in. There's a limit of 250 because it's using a generic storage (that is synced) that only has 250 slots. And these slots are used by other things too, like shortcuts. I'm surprised it causes a sync issue - that's something we'll have to look into...

Here's a guess, based on a brief amount of poking around the SDK: Application Data (see https://dev.evernote.com/doc/articles/application_data.php). One thing that caught my attention is the following (i boldfaced the BAD_DATA_FORMAT mention): 

Quote

Since applicationData is of limited size and is available to all third-party developers, it’s possible that adding new data may cause the field's value to exceed the the 4kb limit. In this case, an instance of EDAMUserException is thrown with the BAD_DATA_FORMAT error code.

Not saying that this is it, but on a casual inspection, it's fairly lading...

[Rob Freundlich 146]

1 hour ago, jefito said:

Per @dconnet's comment, it sounds like it's not a data storage limit (i.e., in the database); it appears to be related to a storage used in the service (the hint is "that is synced")

On 7/26/2018 at 6:36 PM, dconnet said:

I can't say what the plans are (I'm just a dev - that's above my pay grade!), but I can say the styles are something the Windows team hacked in. There's a limit of 250 because it's using a generic storage (that is synced) that only has 250 slots. And these slots are used by other things too, like shortcuts. I'm surprised it causes a sync issue - that's something we'll have to look into...

I didn't catch on to that before. Interesting. It does pretty much invalidate my reverse-engineered design.

I'm still baffled, though, as to why the actual implementation I saw in the database would get funneled down to any kind of slots for syncing. The notebooks and tags literally each have their own color and font attribute information attached to them in the database. If I were designing a sync protocol for that, I'd just have it send each notebook or tag's color/font attribute information with each notebook or tag so it gets compared/conflict-resolved like any other field. It seems like trying to turn them into some sort of named/slotted things that get synced separately would be an awful lot of extra complexity.

 

1 hour ago, jefito said:

Here's a guess, based on a brief amount of poking around the SDK: Application Data (see https://dev.evernote.com/doc/articles/application_data.php). One thing that caught my attention is the following (i boldfaced the BAD_DATA_FORMAT mention): 

Quote

Since applicationData is of limited size and is available to all third-party developers, it’s possible that adding new data may cause the field's value to exceed the the 4kb limit. In this case, an instance of EDAMUserException is thrown with the BAD_DATA_FORMAT error code.

Not saying that this is it, but on a casual inspection, it's fairly lading...

That's note-specific, not notebook- or tag-related, so I doubt styles live there. Plus it's a 4K limit, which doesn't seem to relate to a 250-style limit, unless they're allocating a specific amount of memory per style and leaving room for other things.

[jefito 5,589]

19 minutes ago, Rob Freundlich said:

That's note-specific, not notebook- or tag-related, so I doubt styles live there. Plus it's a 4K limit, which doesn't seem to relate to a 250-style limit, unless they're allocating a specific amount of memory per style and leaving room for other things.

You might be right about that; I didn't take the time to dig much further than that this afternoon. That's not to say that the Windows application didn't hack a connection to this facility and use it to store non note-specific stuff. But I'll peek a little closer when I get a chance, not that it's all that critical. But curiosity is an itch that must often be scratched...