Lost attached files

[gazumped 11,661]

Just had cause to refer to a note I created a couple of years ago and instead of the receipt I expected to see I got a 'broken link' icon and a 202b note. I looked online and got the same thing. The note has no history apart from its creation date.

I sorted all my notes by size and checked the first few hundred under 500b - at least half have the same icon, and a lot of the titles end with .PDF or .JPG so they were imports from my local folder. That's around 300 notes with missing attachments. The dates range over the past 22 months.

I do have database backups, but don't know when this happened - apart from sometime during the period - so the overhead of finding originals of these notes is probably far worse than managing without them, unless I find something really important missing. (I'm over 9,000 notes and counting now.)

Heather suggested I run the /debugmode database cleanup to try and solve a different issue (an upload count mismatch) so I'll give that a shot and see if anything changes. Meantime and if practical I recommend that you check your database for very small notes of 300b or less and just do a quick check to make sure they have all the content you were expecting...

If the Evernote R&D guys can figure out a viable way to check for corrupted notes it might be a comforting extra for the toolbox too.

Screengrab of the offending note and icon below, or shared version here..

[JMichaelTX 4,117]

That's around 300 notes with missing attachments. The dates range over the past 22 months.

This is extremely disturbing. When a smart Evernote veteran like you has lost this many attachments it is grave cause for concern.

I hope that Evernote can restore your missing attachments.

And I hope they can clearly identify what caused the loss and fix it.

Or if it is something that can happen on the user's end, tell us exactly what happened and how to avoid it.

I wish you the best of luck in recovering your missing attachments.

[dlu 628]

I'll sync up with Heather & customer support to see if they have any idea. Please keep us updated

[James M. 15]

Yikes!

[Owyn 457]

I took a look at the shared note. The "en-media" section was not complete. You should check the contents with the ENML Editor.

The fact that there is no note history suggests something has got mixed up with the service resource storage.

Just a SWAG as I can't look at your original note. Just my saved copy.

[gazumped 11,661]

Thanks for the supportive comments - I did a number on my local database: optimise / reindex / recreate note images (which took a while) and sync / restart just for luck. No observable change to my missing bits.

Tried the ENML editor as suggested by Owyn - the full note content is as follows:

<?xml version='1.0' encoding='utf-8'?>

<!DOCTYPE en-note SYSTEM "http://xml.evernote....pub/enml2.dtd">

<en-note>

<en-media hash="7b46d5514c1e9ed05b5b3a8aec1eb813" type="image/jpeg" />

</en-note>

It's now very late o'clock and I have a meet tomorrow later today, so I'll check some of the other notes for content later and put in a support request.

Unless there's a magic wand handy I don't see any missing pics being restored anytime soon, but it's important to understand what happened here - false modesty aside I'm pretty certain I wouldn't s.c.r.e.w up quite so many scans.

[GrumpyMonkey 4,319]

When I was wandering through my database the other day, I came across a note with a blank PDF inside. It was cool, because I had a backup copy, but I thought for a moment that Evernote had lost my data. I went back through Time Machine to the fateful day that I forwarded the email + attachment to myself, and it turns out that it never got into my database in the first place. Something went wrong somewhere.

Otherwise, I have not run across any problems. It might be worth it to go back a couple of weeks or months in Time Machine (if you have it) to see where things went wrong. The cause of your problem sounds totally different than mine (whatever mine was), but you can still use the same method to at least gather a bit more information about it.

[gazumped 11,661]

Thanks GM - unfortunately only Windows spoken here, no TM available. I have checked a few more notes in the ENML editor - see below - and set up a new tag "lost file" to flag the ones I found. The most important thing IMHO is to check whether "this" (whatever it was) is still happening, or whether it was an upgrade one-off that won't recur. I'm reasonably cool with the lost files - if I didn't find out they were missing for nearly 2 years, they're clearly not critical; but I'm very worried about another 8,500+ files, many of which I currently love and cherish quite a lot...

From the ENML editor -

http://www.evernote.com/shard/s2/sh/14ddeb90-964c-465c-af42-f5b9f17e74f4/148a459d160a6df31cd76ed7726d0675

<?xml version='1.0' encoding='utf-8'?>

<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"

>

<en-note>

<en-media hash="96ae513f2ff59501305f8b23ddaf2790" type="image/jpeg" />

</en-note>

http://www.evernote.com/shard/s2/sh/20e384de-fa1b-4f7f-890b-3c3144fc4cfb/195bdb3af1a6ae9cb1121b22d91c6d9e

<?xml version='1.0' encoding='utf-8'?>

<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"

>

<en-note>

<en-media hash="2015935ec7c3cf153e5dce881584d5ea" type="image/jpeg" />

</en-note>

http://www.evernote.com/shard/s2/sh/44395046-22f7-4fdb-bff0-43adaa2d0f7f/c42351e195b7a55e0c9a9052c5bf7e5f

<?xml version='1.0' encoding='utf-8'?>

<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"

>

<en-note>

<en-media hash="b04bd47caf0a592d44eddcd808a09713" type="application/pdf" />

</en-note>

[gazumped 11,661]

Ticket # 16051-113166 now raised

[Guest mrossk]

Have you found the reason for your missing attachments in the meantime? This makes me a little nervous because I have stored some important attachments in EN.

[Nova 1]

I looked at a note I made in 2008, and all the images in the note are missing. I am worried that more data might get lost in the future!

Below, is a screenshot of part of the note and this is how it looks ------->

----------------------------------------------------------------------------------------------------

[gazumped 11,661]

Support ongoing here - a "high level scan" of my database has produced 37 attachments which had been detached from the original note due to database corruption (?). No explanation yet for why there's been any (apparently quite widespread) corruption.

There's no clear way to reassign these attachments back to their original notes yet. I queried the number, since I counted a few hundred notes missing attachments originally. I've now moved 90 files (which might include 37 parent files of the above) to another notebook so they can be scanned individually. Still waiting for the outcome of that.

Am finding it difficult to 'page' through my database note by note, since when (in list view) I hit <downarrow> to do a quick scan of the next note, Evernote will randomly jump back to the very top of the list, losing my place at #100 forinstance and putting me back to #1 - so I have to search back to the last place I recognise and start again from there. Unless that's a feature of my corrupted file, it's a bug that needs attention.

I've expressed my deep concern to Evernote. Seems to me there should be checks and doublechecks on the client or the server for potential error conditions like broken attachment links, or whatever signs this "high level scan" spotted, which would indicate problems immediately and allow 1) -if appropriate and feasible- a restore from latest backup, and 2) a more immediate investigation of the cause. Evernote should also (in my now far from humble opinion) consider whether they include an extra backup feature in the software so that users can take an easy external snapshot of their database on a regular basis.

The only "good" point I can see so far is that important images mostly don't change much over time, so a monthly backup (say) might mean that in the event of a problem you lose some day to day stuff that will be inconvenient, but not the family documents that would be heartbreaking.

For obvious reasons I'm now going to be backing up on a regular basis, and keeping the backups for a year or two.

Meanwhile I continue to spend time I didn't mean to afford trying to work out what happened here, and how to fix it...

#More to follow

And for @Nova (your post came in while I was typing War & Peace)

Check the note in the web client to see if you still have your pictures there

Check the history of the note to see if you can get your pictures back from the history

If you have no joy, please raise a support ticket and let us know what happens!

[Guest mrossk]

@gazumped

Thanks for your information.

I assume that also your web-version does hot show this attachments.

Maybe it could be simply some defect sectors on your harddisk (or a virus)? But this would assume that the attachments are stored all together at nearly the same place in the database bacause it affected only the attachments. And then you shoud see them in the web-version because I think that evernote does not recognize such a defect as a change of a note on your local computer and so should not have synced them back to the web.

In general, important databases shoud only stored on error redundant raid systems because a single sector-failure coud cause very different data loss depending on what data has been stored there.

Maybe that this is a litte oversized but when storing a database on a single harddisk the programmers could do the following:

Everytime when saving a note they generate a hash of the complete note and store it with the note.

Later, when opening the note they could compare the hash and when it is different then show a warning and automatically download the note again from the much more secure web-storage.

This could avoid data loss caused by disk-failure. Especially on "cheap" mobile devices that you can't count on eventually.

Maybe that such checks are already included, this is only an idea from me that could it make more solid.

[Owyn 457]

It should be possible to implement a debug routine to search for orphaned or missing resources.

I don't have immediate access to my test environment but will do a SQLite test later today to confirm feasibility.

PS The ENML is correct. My previous comment was wrong. What is missing is the resource for the hash.

[gazumped 11,661]

@mrossk - My system is (I'm reasonably sure) clear of infections and defects. I'm an experienced user and all my protections are up to date, plus I get paranoid and run extra scans whenever something weird happens like this. The missing data is not present on the web, nor is it in the history of the notes - the problem so far seems to have affected notes that I stored up to 2 years ago (which is more or less as long as I've been using Evernote seriously) and have not since edited. I may or may not have looked at these items since they were set up, but when I come to check now the cupboard is completely bare. The notes are spread over a wide timescale and contain either PDF and JPG attachments.

@Owyn - thanks for your comments; will be glad for anything you can add..

[kvitekp 298]

gazumped,

Please PM me and I'll send you a utility that will scan your database and report all missing resources. Then we'll be able to track them down on the server side and hopefully figure out what had happened to them.

/Peter

[gazumped 11,661]

OK I PM'd peter so we'll see what happens.. current update is that after a long and boring session eyeballing through my notes I seem to have 95 problems, not hundreds - although that's still 95 too many.

There's a bug in scrolling/ paging through the List view that jumps the cursor to the top of the screen every 30 or so notes - I didn't realise this when I started, so could have easily miscounted the errors I saw.

Support have now sent me a spreadsheet with links to the notes and their orphaned attachments, so hopefully I'll be able to reconnect everything soon. This was caused, they say:

In the 4.5.7.7033 Windows Beta Prerelease, there was an error where notes became corrupted shortly after upload. We have not been able to reproduce this on the current release build, nor have we found any other affected users at this time.

So apparently you can all relax.

But I still think that Evernote needs to have some protections built in so that this sort of thing is found and fixed quickly - which comes back to:

1. Some means to flag up any corruption / missing attachment when it happens
   
2. An easy backup system so I can keep serial copies of the database
   

- and please fix that scrolling bug, it's annoying!

[GrumpyMonkey 4,319]

Well, I am glad to see that you have found a solution and are on your way towards getting back to Evernormal

I like your idea about protections to prevent or address such situations. For #1, I think it would be nice if we could have some sort of system for identifying problematic notes. Perhaps Evernote could work out something for the notifications center. For #2, I am not sure how this would work. I have a Mac and have Time Machine (there are Windows alternatives), so I basically have a copy of my Evernote stuff going back for months and months. If Evernote did this for me as well, I wouldn't complain! But, I imagine it would require some major changes.

In the short term, perhaps if Evernote just compared notes with their histories immediately after migrating to a new build, it could flag the problematic notes (#1) and retain an extra copy of them (#2) until you confirm that the migration was successful.

Oh, and as usual, thanks to Peter. He is doing a great job of jumping into these threads and getting stuff done. I think this is the third or fourth time now that I have seen him tackle seemingly thorny problems (list view, instant search, and now this) with ease. Amazing.

[heather 604]

In this particular instance, we don't actually have any good note history on file - in each case, the note was created and corrupted within the first 8 hours, so the only snapshot we have is the "dirty" version.

This happened in one previous version of the Windows client (back in the 3.5 days) but we were able to correct it.

[Owyn 457]

Heather

The original post specified that the lost attachment was from an old note which had not been changed since creation.

That seems like a different case than you described.

[gazumped 11,661]

Hi again - just reconnected 90+ orphaned attachments with their concerned parent notes with much satisfaction all round. So far as I can see normal service has resumed with only one casualty - a PDF attachment that seems to have been corrupted past the point it will open in any of the five different editors/ readers I have available. If there's only that one continuing problem here, it seems like a pretty good result. Peter has provided a utility with which I can check my database again, so I'll run that tomorrow.

[BurgersNFries 2,407]

Gazumped, I've been following this thread with great interest. I've not encountered this problem (that I'm aware of). But as a user of EN beginning with 3.1, I'm concerned. I'm VERY glad the EN staff have been able to help you restore your notes!!!

Peter, I guess as a safeguard, I'd like to run this utility, to check my notes. Would you (or someone) please PM me the instructions to download it? Thanks!

[luckman212 165]

I too stumbled onto this thread and have been following it intently. Great news about the recovery & I admit I was very skeptical that you'd have such a good outcome. Kudos to the EN staff for taking this seriously and obviously handling it with great care.

I would also very much like to run this utility to scan my database, as I've been using it since the early 3.x days and have been through my share of glitches & crashes with the (very) buggy 3.5 series. Would it be possible to PM me the details as well? Thanks.

[JMichaelTX 4,117]

Seems like this Database Check Util is something that we should all have access to.

Perhaps Evernote could provide a download link somewhere on its web site.

Of course, we need both a Windows and a Mac version.

Thanks.

[kvitekp 298]

OK, anyone who wants to check the database for loose ENML resource references feel free to PM me and I'll send you a link to ENNoteCheck download. Please keep in mind that this utility is a console application and as such it is not exactly user friendly, however it does the job done by revealing all resource related inconsistencies in your database. It does not fix them though.

/Peter

[kvitekp 298]

ENNoteCheck.exe uses a bunch of Evernote dlls so it's best to place it into Evernote program files folder (usually C:\Program Files (x86)\Evernote\Evernote) and run it from there specifying path to your database in the command line, for example:

cd "\Program Files (x86)"

ENNoteTest "\Users\kvitekp\Local Settings\Evernote\Databases\kvitekp.exb

/Peter

[kvitekp 298]

Also, you may want to redirect ENNoteCheck.exe's output to a text file by appending "> log.txt" to the command line, then search it for "* missing" or "*mismatc" or just "* mis" ...

Hope this helps...

/Peter

[kvitekp 298]

BTW, that mystique 'uid' number in ENNoteCheck.exe output is the unique id every object in Evernote database is identified with. You can search for note with the particular uid using "uid:" keyword.

For example, if you see this output from ENNoteCheck.exe:

Scanning note uid=1218 "Windows 7 Product Editions Overview"

* missing resource 'd41d8cd98f00b204e9800998ecf8427e' type 'text/xml'

... you can type "uid:1218" in Evernote for Window search box and the note will be selected.

/Peter

[gazumped 11,661]

Thanks /Peter that's extremely useful information - and the existence of this utility goes a long way to reassuring me that my data is safe with Evernote. Any chance we'll see a version of this on one of the main menus at some point, so there's an easy check to confirm that my local database is OK?

(I'm just about to run ENNoteCheck.exe on my own system so I'll let you know the outcome...)

[gazumped 11,661]

Hmmn - just put the utility app in my Evernote folder as suggested, and started up with the full path to my database - what I get is an error

"Application has failed to start because LibSQLite.dll was not found. Reinstalling the application may fix this problem."

Did a search so I could put the EXE file in the same folder as this DLL, but it doesn't seem to be installed anywhere. Am I doing something wrong?

[gavinch 13]

Did you make sure its

C:\Program Files (x86)\Evernote\Evernote

and not

C:\Program Files (x86)\Evernote

Try searching for the file LibSQLite.dll

I found it in my "C:\Program Files (x86)\Evernote\Evernote" folder

[gazumped 11,661]

Thanks and Doh! on both counts - was in the top Evernote folder, and didn't find the DLL file in the lower folder despite a search. When you look, though...

Let's do that again...

[gazumped 11,661]

That's a good utility - checked 9000-and-change notes in about 5 minutes and generated another 13 missing resources, which is frustrating because I spent a long time going through my notes visually in Thumbnail View, making sure none of them showed the dreaded 'broken link' icon. I could have sworn they were all complete, but obviously I dropped the eyeball on these..

/Peter, I'll PM you with the full details to see whether we can get these items back.

Is everyone else's database OK? I just feel so special at the moment...

[Owyn 457]

I got 36 errors from 10,400 notes.

33 of the errors were for the same resource.

[GrumpyMonkey 4,319]

Yikes. Eyeballing mine I only found one problem - an empty PDF that ought to be full. But, I think I mentioned before that it is a forwarded email that seems not to have arrived properly in the first place (based on my copies of the database from that time). Weird, but not sure what the cause was. I'll be interested to see what I find with a more thorough scan (it will take a while, because my Windows install is not right - tickets filed).

[jbenson2 2,147]

I got 25 errors out of 16, 364 notes.

Not sure, what, if anything can be done now.

[jbenson2 2,147]

I got 36 errors from 10,400 notes.

33 of the errors were for the same resource.

How did you confirm 33 of them came from the same resource. From the type section at the end?

The information I get looks like this:

* missing resource '05a8b77e23ab3856056f540a2cc7c036' type 'image/png'

* missing resource 'd41d8cd98f00b202e9800998ecf8427e' type 'image/gif'

* missing resource '7b2733b734635a38a18ccc50455e4c42' type 'application/pdf'

* missing resource 'd41d8cd98f00b204e9700998ecf8427e' type 'application/octet-stream'

Subtotal count rof errors:

5 gif

10 jpeg

1 octet

1 pdf

8 png

[gazumped 11,661]

My report was a list of items like..

Scanning note uid=10396 "car radio instructions.pdf"

·

missing resource 'df1cbbce92d296ed6417883a6b6b03b5' data, type 'application/pdf'

- in one case I had more than one missing resource under the same uid.

Peter asked me to raise a support ticket to search for the attachments on the server, which I now have. A couple of the attachments the support team have already found for me were a couple of years old, so it may still be possible to reconnect with any missing data - and in the interests of finding out why these items dropped out - I'd suggest you raise a ticket and see what happens. I think support are going be to a little busy for a while...

[jbenson2 2,147]

Thanks for the tip Gaz.I decided to concentrate on the note just before the missing resource note.

Based on the sequence of the log.txt, I pulled the UID for the note just before the affected note.

Then I examined these notes for the original posting (from the Note History).

Here is the spread of dates

Sep 3, 2009

Sep 20, 2009

Oct 2, 2009

Mar 6, 2010

Apr 7, 2010

Apr 27, 2010

May 15, 2010

May 27, 2010

May 31, 2010

Jun 2, 2010

Jun 12, 2010

Jul 25, 2010

Jul 25, 2010

Oct 14, 2010

I wonder how many other Evernote users are affected by this?

[gazumped 11,661]

Mine range over a similar timeline, though some are more recent - one of the missing files flagged by Peter's utility was only a month old. The posters on this thread so far have been the more experienced users, so maybe those of us who were more adventurous (or as in my case, just plain greedy for better features) and opted to follow the betas, have been more at risk than most. Or maybe it's those of us with more notes. Anyway I win so far in having about 105 faults, which even so is not a bad error rate in 9,071 notes. We have to have some user checks in the system though to avoid any repetitions...

[jbenson2 2,147]

Mine range over a similar timeline, though some are more recent - one of the missing files flagged by Peter's utility was only a month old.

At first, I thought some of my notes were more recent, so I went to the Note History and pulled out the oldest date for each note on my list.

I have also filed a Support Ticket # 16051-114353.

[jbenson2 2,147]

I examined my Evernote upgrades during the time period my notes were affected. (9/03/2009 to 10/14/ 2010)

The 8 character code at the end of my search is a search code I apply to all Evernote upgrades.

created:20090903 -created:20101014 GQ8WQ75

It looks like there might have been a problem around Sep 2nd and 3rd in 2010.

There were 3 upgrades in less than 24 hours.

I cannot locate another set of upgrades over such a short period of time.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

2009/09/14 Upgraded to version 3.1.0.1225

2010/05/19 Upgraded to version 3.5.4.224 (83106) Beta 2

2010/08/17 Upgraded to version 3.5.6.2757 (96100) Beta 1

2010/09/02 Upgraded to version 3.5.6.2844 (98594) Update 1 (10am)

2010/09/02 Upgraded to version 3.5.6.2846 (98743) Update 1 (5pm)

2010/09/03 Upgraded to version 3.5.6.2848 (987771) Beta 1 (8am)

2010/09/16 Upgraded to version 3.5.6.2883 (100484)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

.

[Guest mrossk]

I also lost an attachment in an offline-notebook. Fortunately not critical but gives me cause for concern

[gazumped 11,661]

Critical or not I'd support-ticket it; let's help EN find out whether there's a weakness in the system somewhere..

[Guest mrossk]

Normally I would do this, but this note is in an offline-notebook and it has its reason that it is only in an offline-notebook.

I have some backups of my database (especially because of using offline-notebooks) but even in my oldest backup this attachment is already broken.

I can only hope that evernote will find this bug or includes the tool ENNoteTest.exe in the "Database optimizier" in the the debug-menu so before a manual backup you can check that the current database is OK.

[FactMan 196]

I haven't found out how to count just the notes with missing resources. However, I do have a lot, and probably more than you, gazumped! To me, this isn't about winning counts for most, though. It is a very useful resource for those older notes. Most of my missing resources are gif's or png's that I would not want to keep anyway. This is giving me the opportunity to clean up the database in a big way. Many of the notes with missing resources that I have found so far are internet files, so I can get back those resources that I do want to keep. I will file a support ticket if I need help on this, but at the moment I don't.

Thanks again, EN, for a useful resource in database maintenance! Thanks to Peter for making it available!

[jbenson2 2,147]

I haven't found out how to count just the notes with missing resources.

The way I got a quick and accurate count was to paste the output from the txt file into Excel.

Column A was a sequential list of numbers, 1,2,3,4, etc, (so that I could return to the original sequence after sorting).

Column B was the raw data pasted in from the txt file.

I did a primary sort on Column B. The 25 * Missing Resource notes were moved to the lop of the 16,000 notes for easy counting.

One of the missing images was a postcard from my son in the Navy. He was staying in Stravanger, Noway and then heading to Tromso (above the Arctic Circle) and points further north. Because he was on a sub, he could not give me any details. The postcard image was impressive and it is part of my memories, but no longer part of Evernote.

I pulled a similar photo off Bing as a replacement.

http://www.evernote....d64a1f998224f4c

,

[JMichaelTX 4,117]

Anyway I win so far in having about 105 faults, which even so is not a bad error rate in 9,071 notes.

Are you kidding?? All notes are not created equal. Could be the one note with missing attachment is an extremely important attachment, which is irreplaceable.

There can be no acceptable loss of data for databases, especially those in which we trust our valuable resources.

I'll just put this on the table: If I can't trust Evernote to store 100% of my Notes without any permanent loss of data (accounting for EN backups/restores), then Evernote ceases to become an acceptable system for me.

Personal backups are not the solution to this problem. Who can afford the storage to keep backups that are many years old?

IMO, Evernote must do the following ASAP:

1. Identify and fix the problem that is causing this data loss
   
2. Restore the missing attachments as much as possible
   
3. Develop and execute a process to frequently (real time, daily?) check for and identify data loss
   
4. Provide a end-user tool, invoked from the Evernote client menu, to check for local database integrity (including attachments)
   

[heather 604]

JMichael -

We did identify and fix the problem that caused the corrupted note display. All this has been mentioned above.

It was isolated to a particular beta build for users on Windows that chose to use it and happened only on specific circumstances (Beta's are not release candidates, and are offered with the caveat that there may be issues with your data should you choose to install them). We also have released a program that allows users to pinpoint the affected files, and we're getting them back for them. No data is lost, it's just that the notes are corrupted locally - the files are still in our system.

I'm very sorry if you're confused by all of what's going on here, or if we haven't been clear or our comments haven't been attentive enough.

So, we're already way ahead of you on points 1 and 2. We have developed the tool for point 4 - incorporating it into a client may take months, so we provide it on the side for users to run on their own. For #3, we actually do this on a daily basis on our own. The nature of this particular issue is that as there isn't actually any data loss, we don't see it in our server checks.

[JMichaelTX 4,117]

More bad news. EN Mac 3.2.0 Beta 2 does NOT "Save Attachments" properly.

Out of 152 Notes which have attachments, it save ONLY 64 attachments in one execution (sorted by Date Created Ascending), and then ONLY 72 attachments in a 2nd execution (sorted by Date Created Descending).

This is unbelievable! I was trying to backup all of my attachments given this data loss issue, and I can't seem to do that using Evernote.

[JMichaelTX 4,117]

JMichael -

We did identify and fix the problem that caused the corrupted note display. All this has been mentioned above.

It was isolated to a particular beta build for users on Windows that chose to use it and happened only on specific circumstances (Beta's are not release candidates, and are offered with the caveat that there may be issues with your data should you choose to install them). We also have released a program that allows users to pinpoint the affected files, and we're getting them back for them. No data is lost, it's just that the notes are corrupted locally - the files are still in our system.

I'm very sorry if you're confused by all of what's going on here, or if we haven't been clear or our comments haven't been attentive enough.

So, we're already way ahead of you on points 1 and 2. We have developed the tool for point 4 - incorporating it into a client may take months, so we provide it on the side for users to run on their own. For #3, we actually do this on a daily basis on our own. The nature of this particular issue is that as there isn't actually any data loss, we don't see it in our server checks.

Heather, thanks for your quick reply.

I guess I am confused. It was not clear to me from the above posts that the problem had been fixed, and that the user's files had been restored/attached to the proper Note.

So is this a EN Win ONLY problem? If so, what versions and dates did this occur?

I am mostly running EN Mac now, and have been for about a year. But occasionally I do use EN Win.

At this point I am afraid to run EN Win. Can you provide a Mac tool that I can check my EN Mac database with?

[heather 604]

Yes, it is a Windows only problem. It occurred in the 4.5.7.7033 beta. There had also been some users affected in one "blink and you missed it" 3.5 beta version in 2010. I'd have to check my records to find out exactly which one it was (@jbenson2), sufficed to say that we caught it quickly then too and all the data is on our servers, even now, for anyone who had it installed and managed to trigger it.

We have no idea what dates it would have occurred, because we can't track who is affected, because we don't have any way to scan for these. As far as our server can tell, the notes are good (we see the resource hash, and we have a matching resource, so, yay).

It's the local client that says "Hey wait a sec, I don't have that resource..."

I'm not entirely sure why you're afraid to run the Windows client. Is there something that you're still confused about as to what you think might be happening here? Did you install that beta version?

[JMichaelTX 4,117]

I'm not entirely sure why you're afraid to run the Windows client. Is there something that you're still confused about as to what you think might be happening here? Did you install that beta version?

I afraid because some losses seem to be very recent:

Mine range over a similar timeline, though some are more recent - one of the missing files flagged by Peter's utility was only a month old.

[heather 604]

4.5.7.7033 was released on 5/28/2012. It is currently 6/24/2012. 4.5.7.714 is the current version.

[JMichaelTX 4,117]

Heather, I think it would be helpful if you would list ALL versions, and their release date, that have caused data loss or disconnect of attachments from their note.

[heather 604]

As I've already mentioned in this thread multiple times, 4.5.7.7033 was released on 5/28/2012, and also, as I mentioned, there was a beta version of 3.5 released sometime in 2010 that I will need to delve into our records to locate exactly which it was. Those are the only two versions of Evernote that have ever caused this issue. Both were also beta versions, and not stable release candidates.

[Guest mrossk]

Thanks for locating the problem and the affected versions. Since both versions are betas and this error does not affect the general user but only users subscribed to the beta-channel I can accept this.

Maybe it has been better when you had informed about this in this forum so that the beta-users had the chance to check their database immediately after you was aware of this. Then this users had a much bigger chance to get their ressources back out of their backups (especially when using offline-notebooks) or to scan their documents or whatever again into evernote.

[jbenson2 2,147]

I also wish Evernote had informed the pre-release users about this known issue.

http://www.idiomques...-under-the-rug/

After finding out about the 2010 problem last week, I turned off the pre-release update feature.

There is no sense risking my data.

The procedure to turn off the pre-release In Windows Evernote:

>Tools

>Options

>General

Remove check mark from "Update to pre-release version when available"

[luckman212 165]

I'm happy and very relieved that after scanning my database I seem to have no missing resources (db has 2,519 notes dating back to 1/24/2009). Whew. (and I was pretty aggressive back in the day with testing beta releases) I just want to make sure I'm using the tool properly:

1. put ENNoteTest.exe in the Evernote program directory — in my case that is C:\Program Files (x86)\Evernote

2. open command prompt and cd C:\Program Files (x86)\Evernote

3. execute ENNoteTest.exe "C:\path\to\my\database.exb" >%TMP%\ennotetest.txt

4. open the %TMP%\ennotetest.txt file with Notepad and scan for the text "* mis"

When I did this I came up clean. Is it safe to say that the bug that caused these missing resources has been squashed as of 4.5.7.7146 (258477) ?

[gazumped 11,661]

I would have thought that you'd have to put the test app in C:\Program Files (x86)\Evernote\Evernote (see previous messages in this thread) - but maybe that's not essential. You might want to make that change and re-test, just to be sure...

And the output file can be copied into a spreadsheet where you should be able to see your 2,519 notes have been checked individually, and will be able to sort any 'mis*' entries to the top. Again see previous messages in this thread.

[luckman212 165]

Yep. I know that's a bit non-standard but when EN started installing itself at ...\Evernote\Evernote\ I thought it was a bug so I forced the Install to just a single layer deep and it's been that way since. I re-ran the tool just now and sorted it in Excel as you suggested and got the same results. Guess I'm just lucky! Screenshot below

[gazumped 11,661]

OK - sorry for doubting your expertise there

Glad you're not affected - by the distinct lack of crowding in this thread (so far) it looks like only a few of the more adventurous souls have been hit. We should have been aware of the risks inherent in Beta trials, so this appears to be a lesson in bug hunting and control for both users and developers. So far Evernote have been top notch in finding my dislocated files and helping me reconnect them (although I'm still waiting to hear about the missing attachments found by ENNoteTest).

I hope it's appreciated over there in the jungle though that users tend to take missing attachments 'quite seriously' - one of my attachments was a spreadsheet called "daily cashbook" and if it had been the live version and not an archived copy I would have been really spiral fixinged. Having serial backups of the database is not exactly a convenient safeguard - without a date for the problem I'd have to load different databases until I found a good copy of the document. And then export/ import it into the current version.

I'm still waiting for an official reaction to the idea that a more user-friendly and enhanced version of ENNoteTest (and a Mac equivalent) should be a permanent inclusion with the desktop client, and that sooner rather than later.

[jbenson2 2,147]

...by the distinct lack of crowding in this thread (so far) it looks like only a few of the more adventurous souls have been hit.

Keep in mind that your original bug report was only posted a week ago on June 19.

Most beta users are not even aware of the problem (unless they've been following this thread).

Even after following your initial instructions to look for small sized notes, I did not see any.

It was not until Peter offered to let users download the EN Note Check program that I realized I had 25 affected notes from back in 2010.

[luckman212 165]

Given how fast the ENNoteTest seems to run (takes about 4 seconds to complete on my system (2600 notes on my "old" Q9650 with a ye-olde-fashioned spinning platter hard drive) I wonder if it wouldn't be a good idea to simply have a future EN release run this integrity test during the initial post-install launch and then inform the user so they could take action. In my mind better to do this sooner rather than later when the original notes might be unrecoverable, or the user desperately needs the information and cannot get in touch with support to assist in retrieving it (EN has been acquired, etc—hey it could happen). The notes in question could be moved to a special notebook, similar to how EN handles conflicting notes now by moving them to an offline notebook.

[heather 604]

Gaz -

You got stuck in the weekend slough. Hopefully should have the rest of them today.

[gazumped 11,661]

~~##Thankyou##~~

[jbenson2 2,147]

Tonight, Evernote Support sent me their results on my missing notes, links, or images.

Total 41

Recovered: 12

Not recovered 29

[JMichaelTX 4,117]

Tonight, Evernote Support sent me their results on my missing notes, links, or images.

Total 41

Recovered: 12

Not recovered 29

JB, thanks for the report.

But, I'm a bit confused:

• I thought the only issue was lost attachments. Does it go beyond that to lost Notes?
  
• I thought Evernote said that they had all of the attachments, and that only the link to the attachment was broken in the Note. Have attachments been permanently lost as well?
  

Also, I think Heather stated that this issue was caused by only 2 Windows Beta versions.

Does this still look right?

[heather 604]

In the Windows 3.5 Beta version back in 2010, there was an additional error in that some attachments did not actually reach our server in the first place. We assigned a GUID to the placeholder resource , but the attachment only existed on that local machine.

This was actually reported on our forums at the time - the most common cause was when people were pasting in content from other programs that included embedded images. It "looked" ok in the local note, but we didn't receive the image.

(This may ring some bells to people who frequented the Windows 3.5 client forums, which were not migrated over here.)

.

[gazumped 11,661]

Well I'm back (so far as I know at the moment) to 100% now, with the latest tranche of recoveries from Support. Thanks to Peter, Heather and Allan and the other support gnomes for finding my bits and pieces. Although we hear the occasional winge about lack of assistance, I have to say that on the occasions when I've had a bijou panic, they've always been as helpful as I could hope.

I know we're going with the "only two isolated incidents" theory at present, and I have no information to contradict it, but I hope that some thought will be applied to implementing an absolute, ironclad, copper-bottomed, sure-fire, fail-safe, assured, cast-iron and pretty well guaranteed means of flagging up any more 'missing' or 'unattached' information to users immediately. If Evernote can generate a suggested heading from note context, it should be able to flash a warning if the same content suggests there should be an attachment.

Even if it generates the occasional false alarm that "attachment expected but not available" I'd feel more secure.

I feel strongly that Evernote should not underestimate how concerned loyal users will feel if the integrity of stored data is even slightly at risk. In many ways having parts of a database corrupted or missing is worse than having the whole account trashed. If the data is gone, you know it's gone; and (if you have a brain cell) there are backups you can use to recover from the loss.

If an attachment to a note goes missing I may not find out until I need that information. And I may not be able to easily recover from its loss if I don't know when that loss occurred. Personal backups going back months may be useless for that specific note, and while my experience has been fairly positive with help from Support, JB still seems to be out some files.

I'm going to continue to use Evernote intensively, with increased precautions; but if there's a recurrence of any mysterious random missing attachments I may need to think about investigating that old promise about how easy it will be to get my data OUT of Evernote.

[jbenson2 2,147]

My personal corrective action steps to limit data base corruption are:

1.) Avoid the beta releases (even if they are bug fixes) and stick only to the fully tested general releases.

2.) Continue making backup database files over the preceding two months.

I hope that Evernote will let us know when they run into this sort of problem in the future.

There is not much I can do 2 years after the fact.

[Mike Wood 139]

@gazumped did you not say that the notes which had missing attachments were also missing on the web? (you could see the notes on the web but the attachments were still missing)

In which case EN could/should be able to identify them & yes check all our online databases regularly (weekly) for this issue and they should notify us if there is an issue. Even if my local database became badly corrupted I would expect to be able to go into the note history online and see the previous version (or is this a recently added feature?)

AND yes we need a utiliity we can run to check the integrity of our local database but I don't think that excuses what mutiple users are now uncovering.

In theory one should be able to junk your main PC re-install EN and the local database would be repopulated from the cloud version (local folders excluded obviously) Is this not right?

[gazumped 11,661]

@Mike Wood - yes, the attachments were missing from the note on my local desktop, my android (which has a downloaded database for offline searching) and the web.

There was no history - I couldn't retrieve the attachments by going back to a previous version of the note. Support found some by (I assume) looking for attachments in my account that didn't have a parent note. The utility in this thread found others though that have taken support a while to retrieve, and I don't know how or where they found them. Much respect to Evernote that they were, eventually, on the system somewhere.

You would think that a regular check of the databases should at least redlight attachments without parent notes, which could spark a deeper integrity check on that account. Evernote are so far silent on the feasibility of such a check, and whether it will be implemented. With 30M+ users however I'd imagine it would not be a trivial undertaking.

And yes, you should be able to restore your local account in full (excluding offline notebooks) by downloading the online version. Support suggested that process as a fix for me at one stage when my local upload stats diverged from the online version. Still haven't done it though - too much hassle (and slight risk) involved. And I've been a bit sidetracked lately by stray attachments.

[Mike Wood 139]

Well they clearly need to do something.... and Heathers comment 'The nature of this particular issue is that as there isn't actually any data loss, we don't see it in our server checks.'

Indicates they need to do different checks!... and it seems everyone who runs the utility is finding some problems?#!

I'm sure this isn't a major issue but clearly its not as resilient as it should be.

[JMichaelTX 4,117]

I know we're going with the "only two isolated incidents" theory at present, and I have no information to contradict it, but I hope that some thought will be applied to implementing an absolute, ironclad, copper-bottomed, sure-fire, fail-safe, assured, cast-iron and pretty well guaranteed means of flagging up any more 'missing' or 'unattached' information to users immediately.

I couldn't agree more. Gaz, I think you have used just about every synonym for "robust" that exists, and yet this can NOT be overstated.

I feel strongly that Evernote should not underestimate how concerned loyal users will feel if the integrity of stored data is even slightly at risk.

This is huge, not only for loyal users, but all users. It would be disastrous if the public were to lose confidence in Evernotes main job of storing our memories.

I'm going to continue to use Evernote intensively, with increased precautions; but if there's a recurrence of any mysterious random missing attachments I may need to think about investigating that old promise about how easy it will be to get my data OUT of Evernote.

Agreed. This is the only prudent course of action.

[heather 604]

@Mike Wood -

My explanation of the issue may be lost somewhere in this thread, but in essence, no, we cannot search for these on the server, because the problem is with the local client.

Even though the display of his note via Evernote Web matched what he saw in the local client (ie, the corrupted display), our server (which is not such a pretty GUI) shows the note (which is just a GUID as containing a resource hash, and that hash can be cross referenced to our resource tables and there is a match. It really is a display issue.

That jbenson2's notes circa 2010 also were missing data due to a completely separate issue (a different type, but also, a display issue) is unfortunate as well.

[gazumped 11,661]

My explanation of the issue may be lost somewhere in this thread, but in essence, no, we cannot search for these on the server, because the problem is with the local client.

Even though the display of his note via Evernote Web matched what he saw in the local client (ie, the corrupted display), our server (which is not such a pretty GUI) shows the note (which is just a GUID as containing a resource hash, and that hash can be cross referenced to our resource tables and there is a match. It really is a display issue.

That jbenson2's notes circa 2010 also were missing data due to a completely separate issue (a different type, but also, a display issue) is unfortunate as well.

Heather, I assume I'm getting confused over this - but your team have managed to send me links to a hundred and some orphaned files which were clearly somewhere on your servers. My "local client" couldn't display the attachments, ditto the web client; and it was your team that found most of the disconnections, so there does seem to be something identifiable somewhere that flags a problem. If that's the case, then klaxons need to go off if it happens again. If that's not the case, then Evernote clients needs a revision that does something similar at local level when attachment links are broken.

[heather 604]

The automatic checks we can do server side and the manual things we can do GUI-side are completely different. Your case was handled by our technicians manually.

And we'll be including the resource check .exe in an upcoming client version.

[JMichaelTX 4,117]

And we'll be including the resource check .exe in an upcoming client version.

This is good.

Will you be providing a Mac version as well?

I've seen quite a few EN Mac users complain about lost data, but I have assumed it was mostly user error since I have not had any obvious losses (to my knowledge). However, I now wonder about these loses, and if I have had some but just didn't notice at the time.

[Mike Wood 139]

@Mike Wood -

My explanation of the issue may be lost somewhere in this thread, but in essence, no, we cannot search for these on the server, because the problem is with the local client.

Even though the display of his note via Evernote Web matched what he saw in the local client (ie, the corrupted display), our server (which is not such a pretty GUI) shows the note (which is just a GUID as containing a resource hash, and that hash can be cross referenced to our resource tables and there is a match. It really is a display issue.

That jbenson2's notes circa 2010 also were missing data due to a completely separate issue (a different type, but also, a display issue) is unfortunate as well.

Ok I don't know enough about your systems.... But it's clear that you could see the exact same problem in the web client.... So I'm sure if you logged into gazumps account using the windows client you would create a local database that recreates the same problem and that newly created local database could be checked by your utility.

So your guys could have done the work for gazumped... Once you fixed the orphans when he next synced all would be good again?

YES or NO?

Obviously I appreciate it may be quicker to let us fix it... But it leaves people like me thinking you aren't in control of your system and that you are actually reliant on finding some of this lost stuff on gazump's local PC, hence he has to fix it!

[gazumped 11,661]

@Mike -

...reliant on finding some of this lost stuff on gazump's local PC, hence he has to fix it!

No - the lost stuff came entirely from Evernote's side - what I got was a set of /shard references to the attachment files which I had to manually re-connect back to the original notes. Don't know if it was possible to do it in a more coherent way, but the simplest seemed to be to download the file to my desktop, find the original note and re-upload the attachment.

@Heather's

And we'll be including the resource check .exe in an upcoming client version.

is a welcome reassurance for all users that 1) the data is safe and 2) Evernote recognises this issue is important. - Thanks, Heather.

[Mike Wood 139]

@gazumped

Good to hear.... Lets hope it turns out the same for everybody with the same issue; fingers crossed!

I'll try and keep quiet until we get feedback from more experienced users like yourself.

[gazumped 11,661]

@Mike

No need for you to keep quiet - your views are at least as important as mine here - any note or attachment can be vital and irreplaceable to any user!

[Mike Wood 139]

How long has the note history been a feature? And how long back does it go.... Is it in theory forever?

We work with a lot of cloud backup solutions for business and you would be surprised how silly they are when you start to drill into the detail of what they do and how they work.... Including some of them deleting files 30 days after you delete them?!

With systems like EN you need something that can verify data integrity regularly as its very likely you may not look at something for years. A bit like Sage or Quickbooks accounts systems do....

What concerns me is this is obvious and is a day one requirement not something you introduce four years after launch....

[gazumped 11,661]

Note history is available from when you set up the note, and (AFAIK) has always been available from Evernote in all its incarnations. It's meant to enable you (if you're a Premium user) to roll back any ill-advised or inadvertent changes that happen to the note. Evernote backs itself up on a frequent basis to ensure that the 'old' versions are still available if required.

Evernote has always been very aware that they're just custodians of our data, and I'm sure they've taken all sorts of precautions against deliberate or accidental interference that no-one will want to discuss publicly for security reasons. Somehow something went wrong - apparently in two Beta builds rather widely separated in time- which split a small number of attachments from notes. I suspect that someone, somewhere with a 'Buck Stops Here' sign on their desk had decided that they had enough protection thanks, and hadn't continued to push the development of more checks and more protections.

We found a hole, and it's going to be patched - welcome to the wonderful world of software development...

[heather 604]

@Mike Wood -

You may be surprised to know this,but Evernote employees would never load up their account on one of our machines except in dire circumstances to troubleshoot an issue, as that would lock a user out of their account temporarily (as we'd need to change their password to do so).And we'd certainly need their consent to do so - we'd never do that on our own.

Additionally, no employee here will *ever* ask a user for their account password.

Your data is your own, and we do not believe in invading user privacy. The way we've retrieved these contents we still maintain the privacy and integrity of the contents of the user's notes.

[Mike Wood 139]

@Heather

I understand that..... and I'm very pleased to see your response and commitment to privacy. My experience of IT in general is rather too long..... I think my subconcious may have been testing your resolve/integrity? (sorry)

[jbenson2 2,147]

Good to hear.... Lets hope it turns out the same for everybody with the same issue; fingers crossed!

I'll try and keep quiet until we get feedback from more experienced users like yourself.

My gut feel - there are probably very few people who know if they are affected. The ones who do; would have to have used the affected Beta versions a couple years ago, then recently sent a PM requesting the EN Check exe program, download it, run it and export the results to analyze for errors. If one does not follow these steps, it is likely that the user(s) are not aware of the problem unless they stumble across one of the notes. I had over 25 issues but could not find any of them until I ran the check program.

[BurgersNFries 2,407]

On 6/27/12, I submitted support ticket 16051-115671 regarding this issue. I included the result of my running the ENNotetest.exe program. I have not yet received a reply from anyone at Evernote that this has been fixed. Your ticket system auto closed this ticket on 7/11/12. I checked two of the bad records & my attachments are still missing. Please advise.

PS - I opened a new ticket with the same info as this post but have not yet received a ticket number... Yes, I did receive the "thank you for submitting a ticket" confirmation notice.

[BurgersNFries 2,407]

Never received a ticket number from when I submitted a ticket from ~10 minutes ago, so I tried again. Got the confirmation page but no ticket number emailed to me. Yes, I checked the junk/spam folder.

[heather 604]

B&F - thanks for bringing this to our attention. Our Support system glitched out and closed your ticket instead of assigning it to an engineer. I'm on it.

[BurgersNFries 2,407]

B&F - thanks for bringing this to our attention. Our Support system glitched out and closed your ticket instead of assigning it to an engineer. I'm on it.

Thank you!

[eithkrp 0]

I've had this same issue with multiple PDFs from back in April showing a broken icon. There is no past history for the note and I cannot get access to the attachments from any of my Evernote clients. A major concern as I no longer have the originals.

[logandb 227]

Hi @eithkrp! The best thing to do would be to lodge a support request. Based on the experience of other users (earlier in this thread), it seems that EN are likely to be able to restore your attachments from their servers.

Good luck!

[jbenson2 2,147]

Yes, a support ticket is the way to go.

Evernote was able to recover some of the missing information, but not all of it.

[Curmudgeona 0]

As a new Evernote user I have been debating link vs. attach, and the history of this issue has certainly made up my mind. At least locally I have all documents synced between two machines and each machine backed up to a hard drive as well as two different online storage services. Overkill? I don't think so. My solution would be expensive if I were a commercial or institutional user but for me the investment in a couple of external terabyte hard drives and online storage plans is well worth the peace of mind.

I thank everyone for the generosity with your time and experience not only in this thread but throughout the forum - seems like a handful of people do most of the info sharing but you have made entry into Evernote a lot easier for me and no doubt for a lot of other people too.

[Janina 0]

Hi guys, it's a bit upsetting to realize that the last post in this thread is dated 2012 - and I just discovered this issue with my Evernote database today. Happened with some of my notes with dates up to Feb 2014. Exactly the same, broken link + 202b note. It is very upsetting. I have logged the ticket 440837; in the mean time, could you perhaps email me the utility so that I could recover my attachments? Thx.

[BurgersNFries 2,407]

Hi guys, it's a bit upsetting to realize that the last post in this thread is dated 2012 - and I just discovered this issue with my Evernote database today. Happened with some of my notes with dates up to Feb 2014. Exactly the same, broken link + 202b note. It is very upsetting. I have logged the ticket 440837; in the mean time, could you perhaps email me the utility so that I could recover my attachments? Thx.

 

 

This is a users message board.  If you need to contact Evernote directly ("send me the utility"), you need to submit a support ticket.

[Janina 0]

 

Hi guys, it's a bit upsetting to realize that the last post in this thread is dated 2012 - and I just discovered this issue with my Evernote database today. Happened with some of my notes with dates up to Feb 2014. Exactly the same, broken link + 202b note. It is very upsetting. I have logged the ticket 440837; in the mean time, could you perhaps email me the utility so that I could recover my attachments? Thx.

 

 

This is a users message board.  If you need to contact Evernote directly ("send me the utility"), you need to submit a support ticket.

 

 Did that, as you can see from, ehhmm, ticket # in the original message; and you do have a point, except for that fact that post 25 on this discussion ( http://discussion.evernote.com/topic/27111-lost-attached-files/?p=146443 ) says that Evernote employee would send this utility on request, via this forum. Quite a while ago though. 

[BurgersNFries 2,407]

Hi guys, it's a bit upsetting to realize that the last post in this thread is dated 2012 - and I just discovered this issue with my Evernote database today. Happened with some of my notes with dates up to Feb 2014. Exactly the same, broken link + 202b note. It is very upsetting. I have logged the ticket 440837; in the mean time, could you perhaps email me the utility so that I could recover my attachments? Thx.

This is a users message board. If you need to contact Evernote directly ("send me the utility"), you need to submit a support ticket.

Did that, as you can see from, ehhmm, ticket # in the original message; and you do have a point, except for that fact that post 25 on this discussion ( http://discussion.evernote.com/topic/27111-lost-attached-files/?p=146443 ) says that Evernote employee would send this utility on request, via this forum. Quite a while ago though.

If you read what you posted, it is not clear that in the ticket, you requested the utility. ("I have logged the ticket 440837; in the mean time") And the post #25 you mention says to PM (private message) the employee.

OK, anyone who wants to check the database for loose ENML resource references feel free to PM me and I'll send you a link to ENNoteCheck download. Please keep in mind that this utility is a console application and as such it is not exactly user friendly, however it does the job done by revealing all resource related inconsistencies in your database. It does not fix them though.

/Peter