Duplicate imports from Import folder

[drmrbrewer 2]

I just dropped 25 documents into my Import folder, and 31 new notes have been added as a result.  Turns out that some of the items dropped into the folder now have duplicate entries in Evernote.  Anyone else seeing this?

 

Version details:

10.54.4-win-ddl-public (3978)
Editor: v166.1.20643
Service: v1.66.1

 

[PinkElephant 6,784]

The system tracks what has already been imported. There are situations when an additional import will happen, like editing a file inside of the import folder.

If it has imported twice, maybe the first import had a problem, which may happen if there was a short network problem. If you are sure anything is working fine, just a duplication issue, I would unconnected the current import folder, create a new one and set up a completely fresh link.

If it repeats itself then, either replace the local database (which may have a corruption), or contact support.

To answer the question: No, no such problem since import folders were introduced on the Mac - one of the big positive added features for the Mac with v10. With legacy import folders were only available on Windows.

[gazumped 10,748]

I had that situation happen to me on Windows 11 once - can't remember the circumstances exactly,  but I'd put 20+ files into the folder,  seen them import,  then rebooted for some reason... and saw Evernote tell me that 20+ files had been imported - again.  I generally clear out my Import Folder after the files have been processed - it's a waste of space (and an invitation to confusion) to have the file both imported and separate.  There's a Windows app called DropIt which automates file management - it can (forinstance) check a folder and move or delete any contents,  as can Power Automate and any number of similar applications.  I tend to use a two-part process...  anything in my Import Folder gets redirected to an external drive Import Archive for a few months which is cleared a couple of times a year.  (I use a similar process when scanning - processed paper stays in a tray for a few weeks in case I need it again...)

[drmrbrewer 2]

@gazumped @PinkElephant this wasn't a case where all files in the folder were imported twice, just a small number of them.  I think I dropped in 25 and soon after it showed that there were 31 new files in the database.  There was no reboot or other significant event in between, nor did I edit any files which might have triggered a re-import.  I too delete any files in the Import folder after import (though I'm not 100% sure when it is safe to do so, i.e. when they are fully imported).  In an older version of the Evernote client for Windows, it used to be the case that files were automatically removed from the Import folder after the import was complete, so we didn't have to do this extra step ourselves.  Sadly, this is no longer the case.

[drmrbrewer 2]

This keeps happening and it's getting on my wick. For example I just imported 20 pdfs via the auto import folder, and ended up with 24 notes.  Better to duplicate notes rather than to remove them, but I hate duplication too.

Does anyone from Evernote tech support read these?  How do I report this, given that my only support channel is via this forum?

[s2sailor 1,534]

13 minutes ago, drmrbrewer said:

Does anyone from Evernote tech support read these?  How do I report this, given that my only support channel is via this forum?

For free users I think there is still support through Twitter, or you can join for a month and then submit a ticket.  FWIW, I haven’t experienced this one myself.

[drmrbrewer 2]

What I've noticed is that, in the several minutes following an auto import, the "updated" field keeps randomly changing to "now" for *some* of the notes, without me doing anything to them.  For example, the following list now shows one of the notes was apparently updated 1 min ago, though this definitely wasn't the case because I imported a whole batch of files (around 15 of them) at the same time.

 

Having notes randomly duplicating doesn't inspire confidence that the opposite won't also happen one day, i.e. randomly deleting notes.

[s2sailor 1,534]

Have you only noticed this with large batches of files?  What happens if you only add say 1 or 3?  It should work either way, but would be helpful to know if this only affects large batches.  My import folder is only used for scans, which is one at a time and maybe why I don’t see it.

[drmrbrewer 2]

@s2sailor Because I'm nervous about this problem, and so that I can keep a closer eye on what's happening (e.g. to see if it has been fixed), I've tried to limit my imports to a maximum of 10 or 20 pdf files at a time (even when I have more to import).  But ideally I'd like to be able to import any number at the same time.  It definitely happens with 10 or 20, quite regularly.  I don't think I've ever seen it happen when importing just one (which I did just now), so it may be that anything more than a small number and things start going awry.

[s2sailor 1,534]

4 minutes ago, drmrbrewer said:

so it may be that anything more than a small number and things start going awry.

That is my thought as well.  Seems like a timing issue that gets increasingly exposed as the number of files is increased and it is possible that the new sync mechanism has exacerbated the problem.  Probably best to limit the number of files for the time being and submitting a problem ticket with this detail would be helpful.

[drmrbrewer 2]

Yeah I think the clue might be (as mentioned above) that notes are imported, sit there for a few minutes, and then later just randomly have a more recent update time, as if some process is going on in the background which updates something about some notes (maybe metadata rather than content), without me doing anything, and maybe the duplication happens at the same time.  It's difficult to follow what's going on when it's changing even as you look at it.

[gazumped 10,748]

I had one anomaly with my Import folders a little while (and a couple of updates) ago.  I imported 20-30 files successfully on one day,  then noticed when I logged in the next day that the same files were imported again.  I meant to do something about it,  but got distracted and had just restarted my desktop (again) when I remembered the issue.  Had no choice but to abort the imports as they went through for the third time...

When I became capable of rational thought again,  I checked the Import folder and found my files sitting there,  grinning at me...  I chose a couple of files names and ran a search,  then noted the created date and time for each.  I actually had FOUR duplicates of some files (don't know when that happened...) but it was pretty easy to find the blocks of imported notes around those times.  I deleted all but the first block,  and reminded myself to move all the imported files from the Import folder to an archive folder once processed.  

I left two files in the Import Folder so I could check whether this happened every time - and of course they've stayed there ever since,  having been imported once only.

All of which is a very long-winded way to say I haven't watched details change over time - I just wait to get a '30 notes imported' message,  then move those notes from Import to Archive.  (The Archive is because of the occasional 'unknown file type' error I've also been seeing,  and a way to restore a lost attachment if necessary.)

[drmrbrewer 2]

@gazumped I do wait a short time (e.g. 10 mins) and then delete the files in my Imports, once I'm confident that they have indeed been imported, but even in that time there are already duplicates in Evernote.  So I don't even have to leave them there for days, merely minutes is enough.  Of course, there shouldn't be any duplication of files regardless of whether or not the files stay in Imports or are moved out. 

If only they could add back the "auto delete" feature that used to be available, where Evernote will itself delete files from the Imports folder once imported... maybe that would prevent these duplicates because it happens immediately, not minutes or days later... it has been "COMING SOON" for a long time now:

[gazumped 10,748]

29 minutes ago, drmrbrewer said:

it has been "COMING SOON" for a long time now:

Yeah - Evernote has its own definition of that phrase,  meaning "sometime between now and the heat death of the universe..."  

[agsteele 2,579]

Files placed in an import folder should be added to a note once UNLESS the file is amended, gained a new last saved date or for some reason the Operating System amends the last saved date.

In that case Evernote will treat the amended file as new and reimport it.

This feels, to me, as though the duplicated notes are somehow getting a new saved date.  I'm not clear exactly how @drmrbrewer is creating these notes and saving into the import folder but that would be my first place to start exploring a solution.

[drmrbrewer 2]

@agsteele I'm really not doing anything complicated.  I have an Import folder (on my Desktop), and I have a Holding folder (on my Desktop) which I use to place files (all pdf files) that are ready to import into Evernote when there is space.  Then I just select a batch (e.g. 20) of the files in Holding, Ctrl-x to "cut", go over to Import, and Ctrl-v to "paste".  This is just a move from Holding to Import.  And the auto-import then takes over.  I'm not opening the files, not editing them, not saving them, just moving them into the Import folder.  There is no way that the Last Modified date should be changed, and if it is, it's not me that's doing it.  Once auto-imported, I'm not doing anything in Evernote either, not editing any note, not saving anything... just scrolling up and down the notes list and maybe clicking on a couple of the imported notes to check.

[s2sailor 1,534]

56 minutes ago, drmrbrewer said:

and maybe clicking on a couple of the imported notes to check.

This might be it.  Others have reported, and I've seen as well, the updated date being changed by just viewing.

[agsteele 2,579]

Here's a suggestion. Instead of moving the PDFs from from holding into import, rather copy them. That will leave the original notes and allow you to observe the last edited date of the original PDF.

After the Import process, identify any duplicates inside Evernote. Now check the import folder and compare the dates.

Another means of resolving this might be too empty the import folder after the process is complete. I have a batch file on my Windows machine which I have scheduled to run once a day.

[drmrbrewer 2]

1 hour ago, s2sailor said:

Others have reported, and I've seen as well, the updated date being changed by just viewing

🤨sounds like a bug to me, nothing should be updated just by viewing.  Perhaps as a result, those files are imported again?  Thing is that it only seems to happen on a subset of files for me, maybe three or four.  For @gazumped it seemed to happen on the whole folder full of files?

 

1 hour ago, agsteele said:

Instead of moving the PDFs from from holding into import, rather copy them.

I'll try that next time and observe.

1 hour ago, agsteele said:

empty the import folder after the process is complete

I already do.  Not immediately, but after 10 or so minutes.

[drmrbrewer 2]

This is getting beyond a joke... just one pdf sitting in the auto import folder when I opened Evernote just now, and now I see not just a duplicate import, but a triplicate:

Can anyone suggest any alternatives to Evernote?

 

[gazumped 10,748]

My Evernote updated today,  but the Import folder still shows 'coming soon' for automatic deletion.  Meantime I simply delete files once they're imported.

[drmrbrewer 2]

@gazumped automatic deletion would be nice, but it's not the problem I'm getting increasingly frustrated with here.  The problem here is that a single file in the auto import folder is imported more than once... sometimes even thrice.  Surely it's not that difficult to write code which compares the properties of a file in the auto import folder (name, creation date, modification date) and compares it against what has already been imported into the database, and if there is a match, then don't import it again?

And no, the name, creation date and modification date of the file itself have not changed on the filesystem... same as they were a month ago when I put the file into the folder in the first place:

[gazumped 10,748]

2 hours ago, drmrbrewer said:

Surely it's not that difficult to write code which compares the properties

Actually,  I believe it is - for entirely Windows reasons.  Windows defaults very carefully to files being locked against third party access - so much so that there are some apps around that can forcibly unlock a windows file that has been wrongly reserved for another application.  If you've ever had the message about a USB drive being 'in use' by another application when you try to remove it,  you are probably experiencing that effect.  I'm sure there are ways around it,  but if it was easy I think we'd have auto-deletion already.

Plus it is definitely 'not difficult' to delete (or move) the contents of the Import Folder once the files have been processed.  Agreed auto-deletion will be nice,  but until Evernote release the feature,  your frustrations are self-inflicted.

[drmrbrewer 2]

18 minutes ago, gazumped said:

If you've ever had the message about a USB drive being 'in use' by another application when you try to remove it,  you are probably experiencing that effect.  I'm sure there are ways around it,  but if it was easy I think we'd have auto-deletion already

Again, I'm not talking about auto deletion (that's a complete red herring... yes auto deletion would be nice to have but the program should work without it).  Try to forget about auto deletion for now.  In this context, I'm not asking that Evernote deletes any file in my import folder, merely that it imports anything in there only once.  For that, it doesn't need to delete anything.  It just needs read access to the file, and its properties.  In any case, if the problem really was caused by other applications locking this file (and completely preventing access), surely that wouldn't lead to duplication or triplication... if anything the opposite??

Quote

Agreed auto-deletion will be nice,  but until Evernote release the feature,  your frustrations are self-inflicted.

Once again, I'm not talking about auto deletion.  Surely it's not beyond the capability of a program such as this to ensure that a file that sits in a folder (even forever) is only imported once?  I mean, that's the basis of any decent file synchronisation program (e.g. a backup program).  Are you saying that these if a backup program creates duplicates or triplicates of a file on the backup filesystem, merely because you were so stupid enough to actually retain the original file on the source filesystem, this problem is "self-inflicted"?

Anyone from Evernote care to justify why their program is not fit for purpose?

[gazumped 10,748]

2 hours ago, drmrbrewer said:

Anyone from Evernote care to justify why their program is not fit for purpose?

An amazing number of people seem to consider "not fit for purpose" to mean "It doesn't do what I want it to".  So either find an app that does do what you want,  or change what you're doing so it doesn't have the same effect. 

Evernote have never,  to my knowledge,  shuffled their feet in front of the Principle User's desk and tried in halting terms to explain why they are just not able to meet the exacting demands of their audience.  If you want personal feedback use Support.

There are ways to avoid this issue.  Contact Support and please use the work-arounds until they can fix it...

[s2sailor 1,534]

9 hours ago, drmrbrewer said:

This is getting beyond a joke... just one pdf sitting in the auto import folder when I opened Evernote just now, and now I see not just a duplicate import, but a triplicate:

What version are you using?  I haven't seen this problem since v10.59 IIRC.

[drmrbrewer 2]

7 hours ago, gazumped said:

An amazing number of people seem to consider "not fit for purpose" to mean "It doesn't do what I want it to".

For a notes application, I'd say that being able to import notes is part of its "purpose".

7 hours ago, gazumped said:

So either find an app that does do what you want,  or change what you're doing so it doesn't have the same effect.

I've tried both.  The latter (deleting files after import) didn't work (see above).  Regarding the former, there are several but the problem is that I have many years worth of data tied up in Evernote that I can't easily get out.

The frustrating thing about this is that file importing is such a basic piece of functionality, and it should be really simple to implement in a way that actually works... this really shouldn't be a reason to annoy a user so much that they want to jump ship?  There might be other reasons to move, like not being able to afford the monthly fees or finding the interface, but file import... surely not?  It's not just the fact the this basic function doesn't work, it's also that this raises doubts about whether that note you imported 10 years ago is actually still there, or maybe it just got lost because the application kinda lost track of it.

[drmrbrewer 2]

5 hours ago, s2sailor said:

What version are you using?  I haven't seen this problem since v10.59 IIRC

I update every time I'm offered the choice.  Not sure what it was on when this latest triplication happened, but when I opened the program a moment ago it offered another update and it's now on 10.61.7.

I'll leave a couple of pdfs in my import folder and see how many times they are imported from now on.

[agsteele 2,579]

I don't want to offend by stating the obvious but, just in case something has been missed...

Any changes to a file in an import folder will cause it to be uploaded a further time. Is there anything going on on your computer which is causing the updated flag on the file to change?

Again, just in case, have you tried a complete and deep uninstall? It will mean reconnecting your import folders. It is possible that the glitch that was definitely around with 10.59 or thereabouts might have persisted.

On a Windows machine, uninstall using the free Revo Uninstaller then reinstall using the latest download. There's is a similar a program for Macs.

If you've tried these then my apologies for stating the obvious.

[drmrbrewer 2]

2 minutes ago, agsteele said:

Any changes to a file in an import folder will cause it to be uploaded a further time

See the screenshot in the following comment above:

The file hasn't been touched (at least not according to the "official" file properties) since it was put into the folder.

I'll see if the very latest version has the same problem, else I will try a re-install.  Or maybe just an uninstall. 

Thanks.

[Hydrangeas4ever 0]

I am on a Mac and the same thing is happening (import folder imports in triplicate) both before and after the last update so it isn't just PCs.  

[drmrbrewer 2]

@Hydrangeas4ever sorry to hear that... it's a pretty basic error that shouldn't take this long to fix?  Can't even blame Windows or Microsoft.

[PinkElephant 6,784]

I had it once, but it didn’t repeat itself since. Maybe because I decided to reconnect my Import Folder ? It might be this resets a counter or list, and stops any multiple import.

[drmrbrewer 2]

Well I've gone from a situation where pdfs were being imported multiple times, now to a situation where I dropped two pdfs into my Import folder... and only one of them has been imported!!!  And yes, I have tried disconnecting and reconnecting the Import folder.

[PinkElephant 6,784]

I think support will be the logical next step.

[drmrbrewer 2]

Again... 

[drmrbrewer 2]

In the interests of fairness, I'm here to report that auto import seems to be behaving a lot better in recent weeks.  Perhaps this issue has now (finally) been resolved.  Maybe I can start to get my love back for Evernote.  Famous last words...