95 replies to this topic

[gazumped]

Just had cause to refer to a note I created a couple of years ago and instead of the receipt I expected to see I got a 'broken link' icon and a 202b note. I looked online and got the same thing. The note has no history apart from its creation date.

I sorted all my notes by size and checked the first few hundred under 500b - at least half have the same icon, and a lot of the titles end with .PDF or .JPG so they were imports from my local folder. That's around 300 notes with missing attachments. The dates range over the past 22 months.

I do have database backups, but don't know when this happened - apart from sometime during the period - so the overhead of finding originals of these notes is probably far worse than managing without them, unless I find something really important missing. (I'm over 9,000 notes and counting now.)

Heather suggested I run the /debugmode database cleanup to try and solve a different issue (an upload count mismatch) so I'll give that a shot and see if anything changes. Meantime and if practical I recommend that you check your database for very small notes of 300b or less and just do a quick check to make sure they have all the content you were expecting...

If the Evernote R&D guys can figure out a viable way to check for corrupted notes it might be a comforting extra for the toolbox too.

Screengrab of the offending note and icon below, or shared version here..

Attached Files

•  broken note.png   22.61K   20 downloads

[JMichael]

That's around 300 notes with missing attachments. The dates range over the past 22 months.

This is extremely disturbing. When a smart Evernote veteran like you has lost this many attachments it is grave cause for concern.

I hope that Evernote can restore your missing attachments.

And I hope they can clearly identify what caused the loss and fix it.
Or if it is something that can happen on the user's end, tell us exactly what happened and how to avoid it.

I wish you the best of luck in recovering your missing attachments.

[dlu]

I'll sync up with Heather & customer support to see if they have any idea. Please keep us updated

[James M.]

Yikes!

[Owyn]

I took a look at the shared note. The "en-media" section was not complete. You should check the contents with the ENML Editor.

The fact that there is no note history suggests something has got mixed up with the service resource storage.

Just a SWAG as I can't look at your original note. Just my saved copy.

[gazumped]

Thanks for the supportive comments - I did a number on my local database: optimise / reindex / recreate note images (which took a while) and sync / restart just for luck. No observable change to my missing bits.

Tried the ENML editor as suggested by Owyn - the full note content is as follows:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote....pub/enml2.dtd">
<en-note>
<en-media hash="7b46d5514c1e9ed05b5b3a8aec1eb813" type="image/jpeg" />
</en-note>

It's now very late o'clock and I have a meet tomorrow later today, so I'll check some of the other notes for content later and put in a support request.

Unless there's a magic wand handy I don't see any missing pics being restored anytime soon, but it's important to understand what happened here - false modesty aside I'm pretty certain I wouldn't s.c.r.e.w up quite so many scans.

[GrumpyMonkey]

When I was wandering through my database the other day, I came across a note with a blank PDF inside. It was cool, because I had a backup copy, but I thought for a moment that Evernote had lost my data. I went back through Time Machine to the fateful day that I forwarded the email + attachment to myself, and it turns out that it never got into my database in the first place. Something went wrong somewhere.

Otherwise, I have not run across any problems. It might be worth it to go back a couple of weeks or months in Time Machine (if you have it) to see where things went wrong. The cause of your problem sounds totally different than mine (whatever mine was), but you can still use the same method to at least gather a bit more information about it.

[gazumped]

Thanks GM - unfortunately only Windows spoken here, no TM available. I have checked a few more notes in the ENML editor - see below - and set up a new tag "lost file" to flag the ones I found. The most important thing IMHO is to check whether "this" (whatever it was) is still happening, or whether it was an upgrade one-off that won't recur. I'm reasonably cool with the lost files - if I didn't find out they were missing for nearly 2 years, they're clearly not critical; but I'm very worried about another 8,500+ files, many of which I currently love and cherish quite a lot...

From the ENML editor -

http://www.evernote....cd76ed7726d0675

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote..../pub/enml2.dtd"
>
<en-note>
<en-media hash="96ae513f2ff59501305f8b23ddaf2790" type="image/jpeg" />
</en-note>


http://www.evernote....1121b22d91c6d9e

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote..../pub/enml2.dtd"
>
<en-note>
<en-media hash="2015935ec7c3cf153e5dce881584d5ea" type="image/jpeg" />
</en-note>


http://www.evernote....c9a9052c5bf7e5f

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote..../pub/enml2.dtd"
>
<en-note>
<en-media hash="b04bd47caf0a592d44eddcd808a09713" type="application/pdf" />
</en-note>

[gazumped]

Ticket # 16051-113166 now raised

[mrossk]

Have you found the reason for your missing attachments in the meantime? This makes me a little nervous because I have stored some important attachments in EN.

[Nova]

I looked at a note I made in 2008, and all the images in the note are missing. I am worried that more data might get lost in the future!
Below, is a screenshot of part of the note and this is how it looks ------->
----------------------------------------------------------------------------------------------------

[gazumped]

Support ongoing here - a "high level scan" of my database has produced 37 attachments which had been detached from the original note due to database corruption (?). No explanation yet for why there's been any (apparently quite widespread) corruption.

There's no clear way to reassign these attachments back to their original notes yet. I queried the number, since I counted a few hundred notes missing attachments originally. I've now moved 90 files (which might include 37 parent files of the above) to another notebook so they can be scanned individually. Still waiting for the outcome of that.

Am finding it difficult to 'page' through my database note by note, since when (in list view) I hit <downarrow> to do a quick scan of the next note, Evernote will randomly jump back to the very top of the list, losing my place at #100 forinstance and putting me back to #1 - so I have to search back to the last place I recognise and start again from there. Unless that's a feature of my corrupted file, it's a bug that needs attention.

I've expressed my deep concern to Evernote. Seems to me there should be checks and doublechecks on the client or the server for potential error conditions like broken attachment links, or whatever signs this "high level scan" spotted, which would indicate problems immediately and allow 1) -if appropriate and feasible- a restore from latest backup, and 2) a more immediate investigation of the cause. Evernote should also (in my now far from humble opinion) consider whether they include an extra backup feature in the software so that users can take an easy external snapshot of their database on a regular basis.

The only "good" point I can see so far is that important images mostly don't change much over time, so a monthly backup (say) might mean that in the event of a problem you lose some day to day stuff that will be inconvenient, but not the family documents that would be heartbreaking.

For obvious reasons I'm now going to be backing up on a regular basis, and keeping the backups for a year or two.

Meanwhile I continue to spend time I didn't mean to afford trying to work out what happened here, and how to fix it...

#More to follow

And for @Nova (your post came in while I was typing War & Peace)

Check the note in the web client to see if you still have your pictures there
Check the history of the note to see if you can get your pictures back from the history
If you have no joy, please raise a support ticket and let us know what happens!

[mrossk]

@gazumped
Thanks for your information.

I assume that also your web-version does hot show this attachments.

Maybe it could be simply some defect sectors on your harddisk (or a virus)? But this would assume that the attachments are stored all together at nearly the same place in the database bacause it affected only the attachments. And then you shoud see them in the web-version because I think that evernote does not recognize such a defect as a change of a note on your local computer and so should not have synced them back to the web.

In general, important databases shoud only stored on error redundant raid systems because a single sector-failure coud cause very different data loss depending on what data has been stored there.
Maybe that this is a litte oversized but when storing a database on a single harddisk the programmers could do the following:
Everytime when saving a note they generate a hash of the complete note and store it with the note.
Later, when opening the note they could compare the hash and when it is different then show a warning and automatically download the note again from the much more secure web-storage.
This could avoid data loss caused by disk-failure. Especially on "cheap" mobile devices that you can't count on eventually.
Maybe that such checks are already included, this is only an idea from me that could it make more solid.

[Owyn]

It should be possible to implement a debug routine to search for orphaned or missing resources.
I don't have immediate access to my test environment but will do a SQLite test later today to confirm feasibility.

PS The ENML is correct. My previous comment was wrong. What is missing is the resource for the hash.

[gazumped]

@mrossk - My system is (I'm reasonably sure) clear of infections and defects. I'm an experienced user and all my protections are up to date, plus I get paranoid and run extra scans whenever something weird happens like this. The missing data is not present on the web, nor is it in the history of the notes - the problem so far seems to have affected notes that I stored up to 2 years ago (which is more or less as long as I've been using Evernote seriously) and have not since edited. I may or may not have looked at these items since they were set up, but when I come to check now the cupboard is completely bare. The notes are spread over a wide timescale and contain either PDF and JPG attachments.

@Owyn - thanks for your comments; will be glad for anything you can add..

[kvitekp]

gazumped,

Please PM me and I'll send you a utility that will scan your database and report all missing resources. Then we'll be able to track them down on the server side and hopefully figure out what had happened to them.

/Peter

[gazumped]

OK I PM'd peter so we'll see what happens.. current update is that after a long and boring session eyeballing through my notes I seem to have 95 problems, not hundreds - although that's still 95 too many.

There's a bug in scrolling/ paging through the List view that jumps the cursor to the top of the screen every 30 or so notes - I didn't realise this when I started, so could have easily miscounted the errors I saw.

Support have now sent me a spreadsheet with links to the notes and their orphaned attachments, so hopefully I'll be able to reconnect everything soon. This was caused, they say:

In the 4.5.7.7033 Windows Beta Prerelease, there was an error where notes became corrupted shortly after upload. We have not been able to reproduce this on the current release build, nor have we found any other affected users at this time.

So apparently you can all relax.

But I still think that Evernote needs to have some protections built in so that this sort of thing is found and fixed quickly - which comes back to:

• Some means to flag up any corruption / missing attachment when it happens
• An easy backup system so I can keep serial copies of the database

- and please fix that scrolling bug, it's annoying!

[GrumpyMonkey]

Well, I am glad to see that you have found a solution and are on your way towards getting back to Evernormal

I like your idea about protections to prevent or address such situations. For #1, I think it would be nice if we could have some sort of system for identifying problematic notes. Perhaps Evernote could work out something for the notifications center. For #2, I am not sure how this would work. I have a Mac and have Time Machine (there are Windows alternatives), so I basically have a copy of my Evernote stuff going back for months and months. If Evernote did this for me as well, I wouldn't complain! But, I imagine it would require some major changes.

In the short term, perhaps if Evernote just compared notes with their histories immediately after migrating to a new build, it could flag the problematic notes (#1) and retain an extra copy of them (#2) until you confirm that the migration was successful.

Oh, and as usual, thanks to Peter. He is doing a great job of jumping into these threads and getting stuff done. I think this is the third or fourth time now that I have seen him tackle seemingly thorny problems (list view, instant search, and now this) with ease. Amazing.

[heather]

In this particular instance, we don't actually have any good note history on file - in each case, the note was created and corrupted within the first 8 hours, so the only snapshot we have is the "dirty" version.

This happened in one previous version of the Windows client (back in the 3.5 days) but we were able to correct it.

[Owyn]

Heather
The original post specified that the lost attachment was from an old note which had not been changed since creation.
That seems like a different case than you described.