How to find duplicate notes?

[vogelap 27]

I've got Evernote 4.5.7.7070 on my PCs. I've noticed that there are several duplicate notes in my ENbase. The dupes seem to have been generated some time ago (I've been syncing and the dupes don't go away).

Is there some way to have Evernote display for me "likely" duplicate notes?

[spg SCOTT 736]

I would suggest sorting by title, and checking to see if there are copies of titles.

The list view is probably the easiest view for this.

[vogelap 27]

I've got ~4000 notes... If there was a way Evernote could help by finding suspects, it would speed things up!

[Owyn 457]

There is no way to find duplicates that I know of.

I usually just clean them up as I find them. This happens most frequently when I am processing my Inbox.

[gazumped 11,665]

OK - how's this for lateral think:

Either

1. sort notes by title
   
2. copy note titles into excel (or your spreadsheet of choice) as a list
   
3. use a simple formula (if a1=a2, then "x") to generate marks against duplicates
   
4. delete all other titles from list (sorting is iffy, because a1 no longer = a2 even if it did originally...)
   
5. find those names in Evernote
   
6. Kill! ()
   

Or:

1. Don't sweat it
   
2. Carry on with normal searches
   
3. If you notice a duplicate kill it then
   

[Parisie 2]

OK - how's this for lateral think:

Either

1. sort notes by title
   
2. copy note titles into excel (or your spreadsheet of choice) as a list
   
3. use a simple formula (if a1=a2, then "x") to generate marks against duplicates
   
4. delete all other titles from list (sorting is iffy, because a1 no longer = a2 even if it did originally...)
   
5. find those names in Evernote
   
6. Kill! ()
   

Or:

1. Don't sweat it
   
2. Carry on with normal searches
   
3. If you notice a duplicate kill it then
   

The second one is the better.

I am often finding duplicate notes. It is mostly (if not at all) my fault because I copied interestings things more than one time in my Evernote und different (!) note titles. If it i an Error of EN it will have the same note title, so it will be easy to find.

my question: Does the first possibility also work with a big number of notes?

[gazumped 11,665]

Does the first possibility also work with a big number of notes?

Depends how many rows your spreadsheet has. As far as I can see it should work with up to 10,000 - 50,000 titles fairly easily, depending on your spreadsheet and the speed and memory of your machine. Excel 2007 forinstance goes up to 1M rows and will highlight duplicates for you via "Conditional Formatting". Spreadsheets are pretty good tools for identifying duplicates. The only practical problem I see is that once you have your list of duplicated entries you're back to purely manual checks to find and delete them.

So I'd go with method 2 unless you're really stuck for space or overwhelmed with duplicates!

[jbenson2 2,147]

sort notes by title

1. copy note titles into excel (or your spreadsheet of choice) as a list
   
2. use a simple formula (if a1=a2, then "x") to generate marks against duplicates
   
3. delete all other titles from list (sorting is iffy, because a1 no longer = a2 even if it did originally...)
   
4. find those names in Evernote
   
5. Kill! ()
   

I thought it would be interesting to look at my note titles from a different perspective, so I gave your suggestion a try.

But I'm stumped on step #1. When I try to paste a few note titles (from Copy Note Links) into a spreadsheet, they all drop into the same cell.

How did you get them to drop into individual cells?

.

[gazumped 11,665]

Exactly the same process I think - List View / Select links / Copy Note Links and then Paste into Excel 2007. All separate lines. Maybe if you dump the list in a word-processor first?

Actually JB you may have hit on a trick I didn't think about - because these are all active links, the deletion process isn't as painful as I thought it would be.

All you need do is delete all but one of the highlighted duplications (I seem to have x4 of one heading!) by clicking the respective links, then deleting the notes from their opened windows. It's not automatic, but it's not exactly a hard slog.

Edit: Although you might want to check that the notes are actually identical and do something less destructive first - like maybe move them to another notebook? Evernote doesn't actually prevent you from having two identical filenames, and maybe you were just amazingly consistent in naming all the notes from one event!

[jbenson2 2,147]

Exactly the same process I think - List View / Select links / Copy Note Links and then Paste into Excel 2007. All separate lines. Maybe if you dump the list in a word-processor first?

Thanks for the reply.

Pasting it into the word processor as the middle man solved the problem.

[BurgersNFries 2,407]

OK - how's this for lateral think:

Either

1. sort notes by title
   
2. copy note titles into excel (or your spreadsheet of choice) as a list
   
3. use a simple formula (if a1=a2, then "x") to generate marks against duplicates
   
4. delete all other titles from list (sorting is iffy, because a1 no longer = a2 even if it did originally...)
   
5. find those names in Evernote
   
6. Kill! ()
   

Or:

1. Don't sweat it
   
2. Carry on with normal searches
   
3. If you notice a duplicate kill it then
   

I vote for #2. However, if you wanted to get totally OC about it, you could take it a step further by comparing not only duplicate titles, but note size. OTOH, I guess that's not totally OC. Because a true OC would say that just b/c the title & note size match, that doesn't mean the notes are exact dups.

[jbenson2 2,147]

I just ran a search for all Political notes during the past 18 months. Came up with 4,600 notes.

Pasted them into a spreadsheet and used the formula =IF(A2=A1;"dupe";" ")

If it was a duplicate title, the word dupe appears

If it was not a duplicate, an invisible space shows up.

So it was easy to find the dupes.

I found about 15 - and most of them were "screen clip" when I captured images. They had different images were not duplicates, so I did not delete them.

But it did help remind me to rename them correctly.

Thanks

[gazumped 11,665]

Would be convenient and helpful to have an Evernote "duplicate finder" that checks for note name and size, since I found that some of my notes are named identically because they're all about one event, but each has a different image / different contents. As JB found though, most of my duplicates were default headings - which it's useful, but hardly vital, to clean up.

When duplicate notes matter is when you find your routine search turns up two notes that are titled differently but still have the same content. If/ when that happens I'll worry about it.

[vogelap 27]

Would be convenient and helpful to have an Evernote "duplicate finder" that checks for note name and size

+1 to that idea!

[idoc 400]

I've got ~4000 notes... If there was a way Evernote could help by finding suspects, it would speed things up!

You would be surprised at how easy it is with the list view. Just get into that view and scroll slowly down. I don't think it would take you more than 15 minutes to eliminate all the duplicates. I did it recently on 2000 notes and it went very smoothly.

[gazumped 11,665]

Actually I'd like to amend my previous suggestion - notes with the same name are relatively common; usually a default of some sort. The same name does not indicate the same content. In fact my main problem is notes with different headings but the same content - when I grab (or upload) the same image twice (or more) times with my somewhat variable wetware supplying different titles each time. A search for exactly the same note size might find quite a bit of duplicated content.

[vogelap 27]

Or, @gazumped, a combination of note names being similar and note sizes being similar... That'd give results, too, I think.

[kertison 0]

Well, currently I'm using Duplicate Files Deleter to find & delete duplicate files. It's pretty effective with multiple settings on how thorough you want to be and it categorizes the results.

[jefito 5,589]

Well, currently I'm using Duplicate Files Deleter to find & delete duplicate files. It's pretty effective with multiple settings on how thorough you want to be and it categorizes the results.

Does this have anything to do with Evernote? Evernote for WIndows stores its content in a database, rather than individual files, so it's likely not to be too effective for the problem described here.

[BurgersNFries 2,407]

Well, currently I'm using Duplicate Files Deleter to find & delete duplicate files. It's pretty effective with multiple settings on how thorough you want to be and it categorizes the results.

Does this have anything to do with Evernote? Evernote for WIndows stores its content in a database, rather than individual files, so it's likely not to be too effective for the problem described here.

Yup, Jeff's right. A duplicate file finder is not going to help anyone find duplicate notes in Evernote.

[J-Mac 19]

I ended up with a lot of duplicates by importing OneNote content into Evernote. If you import at one point in time and then later have a lot more OneNote notes to import there is no good way to only import new notes. So you end up re-importing some of the same notes again. A lot of the same notes again!!

 

I think I'm going to try copy/paste the note titles into Word documents and then compare them using a program called Beyond Compare. All it does is compare separate files and highlight all identical passages. I'll post back if it works.

 

Thank you.

 

Jim

 

 

[gazumped 11,665]

I ended up with a lot of duplicates by importing OneNote content into Evernote. If you import at one point in time and then later have a lot more OneNote notes to import there is no good way to only import new notes. So you end up re-importing some of the same notes again. A lot of the same notes again!!

 

I think I'm going to try copy/paste the note titles into Word documents and then compare them using a program called Beyond Compare. All it does is compare separate files and highlight all identical passages. I'll post back if it works.

 

Thank you.

 

Jim

 

Sounds like a lot of effort for no great return - especially on a lot of files.  Surely you'd do better keeping your notes in one place only - so when transferring from OneNote to EN,  delete the ON originals and rely on EN.

 

For lots of duplicates,  just sort List View into title order and scroll.  If you want to get that list somewhere else,  highlight the titles and copy/ paste into a spreadsheet or WP.

 

There's no great urgency though - all you're doing is making your database slimmer and more efficient.  I find more dups by searching now than any other method.  When my search turns up a few hits,  some of them obvious duplicates,  I clean up then.  You've already suffered the hit on your upload limit by adding the duplicate in the first place. 

 

Better to avoid doing that in the first place...

[J-Mac 19]

Thanks gazumped.

 

I know I should stick with one notes program, but I find that some documents are better done in OneNote. The editor is a lot better if it is something I am composing. Plus sometimes when I paste or clip into Evernote it comes out looking way off, so I pop it into ON instead. Believe it or not, when I move it feom ON into EN it looks perfect, whereas if I clip it into EN to begin with it doesn’t. I don’t know why. Plus I don’t usually put anything sensitive into EN. Not that I use ON for very sensitive content but since it is local only I do put things in there unencrypted that I won't put into EN without encrypting it first. If I get lazy I put it into ON until I have more time to get it into EN.

 

So one way or another I will usually end up with some stuff in OneNote. I agree that it isn't urgent and I can just take my time with it. I just usually want to start something and get it done... Now! Totally unnecessary I guess! I need to get over some of the false urgency that we all hang onto at times. 

 

Thanks again!

 

Jim

[gazumped 11,665]

Patience is a Virtue.  If only I had some. 

[jyoung.usa 3]

Evernote is basically a very handy searchable database product that stores records to be accessed via desktop clients and a web interface.  

 

In databases duplicates happen.  Sometimes LOT of duplicates happen when the sync process goes bad.  Or just too many copies are stored.  Storing and replicating these duplicates needless takes space on client machines, and your servers, chews up bandwidth shuffling these files around, and the $$ to transfer to mobile devices, etc.  Duplicates are the cost that keeps on costing.  

 

PLEASE provide a simple Duplicate search utility that can find and list duplicates and allow deletion as desired.  This functionality is quiet common in other products having replication and duplication issues.  Enough people have been asking and your internal Support Staff likely could use this tool on a very regular basis as well.

 

DUPE CHECK PLEASE

[gazumped 11,665]

This functionality is quiet common in other products

 

And the actual "other products" would be?

[lannykint 0]

Duplicate Files Deleter (http://DuplicateFilesDeleter.com) is a simple, but effective tool to locate duplicate files in one or more selected search paths. It scans the files and compares them based on Byte for Byte Comparison, which ensures 100% accuracy. You can then choose to delete the selected duplicate or original files. The program is multi-threaded and performs scans quickly.

[marianoi 1]

IMHO, this is a must have feature. In the mac I also use DevonThink (LINK). DT has a feature that looks for some sort of semantic similarities between notes and groups suspected duplicates in a smart folder. You can then go there and confirm whether those are duplicates or not. Mendely (LINK), a reference manager for scientific papers, also has a similar feature.  

 

Since EV (at least the premium version) scans all PDF, pictures, and text files, something similar to DT solution could be implemented... If you use EV as an "everything bucket" (as many people do, I guess), you will certainly end up with tons of duplicates.

 

Just my 5 cents...  

 

M

[jbenson2 2,147]

If you use EV as an "everything bucket" (as many people do, I guess), you will certainly end up with tons of duplicates.

 

 

I disagree.

 

I have over 34,000 notes in Evernote with several hundred more added each week.

I run many different types of searches daily. 

 

It is a very unusual day when I will find even a single duplicate.

[jefito 5,589]

I find the occasional duplicate among my notes -- they're almost always web clips that I've done a couple of times without realizing, usually software development reference material, and not the important stuff that I create. So to me, it's a big "So what?" It's not like I'm hurting for storage space. It's not like having the extra copy is a burden to performance. It's not like I'm losing any information. I just delete any duplicates I run across, and move on. Might be a nice third-party tool to have in your kit, but I don't see the 'must have' in this...

[FactMan 196]

Jefito, I agree with you - I have nearly 13,000 notes, and rarely find duplicates unless I have done something silly... Or a web site is repeating material/titles from some time ago. For anyone concerned about security, most of my clips that I control I use an off-line notebook, and transfer the incoming files to the offline notebook at the earliest opportunity. I use Clearly for cleaning up web pages, then clicking on the item in list view and pressing CTRL-spacebar cleans up pretty much everything in my note. I clean up images manually (delete usually, occasionally resize), so addition of resize to the annotation options would be a great advance. I was looking at some older notes yesterday, and EN has come a long way in making it easier to store stuff!

[BurgersNFries 2,407]

IMHO, this is a must have feature. In the mac I also use DevonThink (LINK). DT has a feature that looks for some sort of semantic similarities between notes and groups suspected duplicates in a smart folder. You can then go there and confirm whether those are duplicates or not. Mendely (LINK), a reference manager for scientific papers, also has a similar feature.  

 

Since EV (at least the premium version) scans all PDF, pictures, and text files, something similar to DT solution could be implemented... If you use EV as an "everything bucket" (as many people do, I guess), you will certainly end up with tons of duplicates.

 

Just my 5 cents...  

 

M

 

I have over 63,000 notes in my main account & very rarely come across dups.  But when I do, like Jefito & Factman, it's a "so what?" moment.  I either merge them or put one in the trash, depending upon my mood.  IE, if I'm not entirely sure they are dups, I merge them.  NBD.  But certainly I don't have "tons of duplicates" and this is not a "must have" for me.  One of my "must haves" is having the Windows client be able to handle the large number of notes.  (It doesn't).  Or the iOS client able to handle large accounts.  (It doesn't)  Or have the web client search produce the same results the Windows client does.  (It doesn't).  Or having the note count the same between all clients when you have notebooks shared to you.  (They aren't).  Yeah, those are what I consider "must haves".  But dups?  No big deal.  (shrug)

[CalS 5,280]

On 8/6/2014 at 7:48 AM, marianoi said:

IMHO, this is a must have feature. In the mac I also use DevonThink (LINK). DT has a feature that looks for some sort of semantic similarities between notes and groups suspected duplicates in a smart folder. You can then go there and confirm whether those are duplicates or not. Mendely (LINK), a reference manager for scientific papers, also has a similar feature.  

 

Since EV (at least the premium version) scans all PDF, pictures, and text files, something similar to DT solution could be implemented... If you use EV as an "everything bucket" (as many people do, I guess), you will certainly end up with tons of duplicates.

 

Just my 5 cents...  

 

M

Don't see this as a must have either.  But if you feel compelled:

1. Select All Notes or whatever context
2. Open List view
3. Sort by title
4. Click a title
5. Edit "Select All"
6. Copy
7. Paste into word processor or spreadsheet
8. Do a little Excel arithmetic and you should be in the neighborhood of your duplicates.

End of the day you are only going to have duplicates if you put them there.

[J-Mac 19]

Well, I never called it a "must have" feature. That said, I still think it would be a very nice feature in Evernote. The OneNote import hasn’t changed; as new notes are added into OneNote and if you wish to get them into Evernote, you have to import entire sections - again. No way to only import new notes. I try to get all OneNote notes within certain important sections into Evernote for a couple of reasons. First, it is nice to have important notes in both apps because while searching in OneNote is fast on a Windows PC it is excruciatingly slow on both Android and iOS devices. So for searching when mobile it's great to have them in Evernote as well as OneNote. Second, if I need to print any of these notes I need them in OneNote in Windows (because Evernote's printing function isn't nearly as good on Windows), and I need them in Evernote too because OneNote doesn’t offer printing at all on mobile devices. So basically there are benefits for me to have certain OneNote sections in Evernote, and when I import OneNote sections subsequent times I am creating more duplicates and building up my Evernote database needlessly.

 

Those are the reasons I would like it as a feature. Just me... I do realize others don’t necessarily have the same needs and of course I don’t insist that others should support the idea just because I would like it. (But I really would like it!!    )

 

Thank you.

 

Jim

[peterson 0]

On 2012-6-13 at 10:51 PM, vogelap said:

I've got Evernote 4.5.7.7070 on my PCs. I've noticed that there are several duplicate notes in my ENbase. The dupes seem to have been generated some time ago (I've been syncing and the dupes don't go away).

Is there some way to have Evernote display for me "likely" duplicate notes?

Why don't you try Duplicate Files Deleter? It will do a thorough search of your hard disk and find out the two or more duplicate files of the same file which may be stored at different locations. It will give you a comprehensive list of all those files and you can decide for yourself what you want to do with them. It's easy to use. Hope it will help you.

[J-Mac 19]

Duplicate File Finder can find files, not notes. The notes are all contained within one file - the Evernote database. Jim 

 

[gazumped 11,665]

2 hours ago, J-Mac said:

The notes are all contained within one file

In Windows that's true;  in Mac,  not so much.  Either way any talk of 'duplicate file removers' (especially by first-time posters) is just spam.

[J-Mac 19]

On 6/8/2017 at 7:04 PM, gazumped said:

In Windows that's true;  in Mac,  not so much.

Really? I don’t have a Mac - you mean on Apple Evernote has a file for every note?! Seems awfully wasteful of space.

Thanks, Jim

[DTLow 5,736]

50 minutes ago, J-Mac said:

you mean on Apple Evernote has a file for every note?! Seems awfully wasteful of space.

Regardless, you do not address duplicate notes from the file OS

On the Mac platform, there's a folder for every note, with multiple files

You could say the space is the same, separate files or combine all into a single .exb file, although the .exb file is probably more compressed

I think it's better to work with the separate note files, instead of the massive GB exb file.    Certainly from a backup perspective (Mac Time Machine doing incremental backups)

[Patrick Holmes 7]

To find duplicate Titles in Evernote:  (It's simple logic. It's a shame Evernote doesn't do this for us!!!)

Here are steps that work with Evernote version 7.14 on a Macbook Pro, also using Microsoft Excel for Mac 2011. This is a variation on the "use Excel" suggestion in this thread, but it is April, 2020, so this is more up-to-date than the suggestions below and will work with my current version of Evernote:

(1) Select All notes, then View/Top-List-View

(2) Click on first (or any) note, then Select-All (use menu, or [Command-A] on a Mac or [Ctrl-A] on a PC)

(3) Choose "Create Table of Contents Note" (you may need to scroll to see this option) (see 1st screenshot below)

(4) The "Table of Contents" (TOC) note will open. (Note that each row has a row number followed by a period. We will "clean" that in the the next step using Excel formulas.) Now: Click anywhere inside the note, then Select-All, then Copy (see 2nd screenshot below)

(5) Open Excel, then using columns A, B, and C:
      (a) create the following titles:  A1: "Original Title",  B1: "Cleaned Title", C1: "Duplicate?"
      (b) Paste the copy from the TOC into A2. All the titles should now be in column A. 
      (c) Enter or copy/paste this formula into B2:   =MID(A2,FIND(".",A2)+2,LEN(A2))
      (d) Enter or copy/paste this formula into C2:   =IF(B1=B2,"DUPLICATE","")
      (e) Now copy B2 and C2 formulas down to last row (in my case, that is row 725)
      (see 3rd screenshot, with "Show Formulas" turned on briefly)

(6) Click anywhere in 1st row, then click or turn-on "Filter". Then click on Filter in C1, and select only "DUPLICATE" values. These are your duplicates. (see 4th screenshot)   

1st screenshot:

2nd screenshot:

3rd screenshot:

4th screenshot:

pat holmes

 

 

[gazumped 11,665]

On 4/12/2020 at 10:37 PM, Patrick Holmes said:

this is more up-to-date than the suggestions below and will work with my current version of Evernote

Hi.  Thanks for the update,  but you're adding to a 3-year old thread.  I came to the conclusion a while ago that I had the choice of spending an hour or two searching for duplicate notes every month or so.. but apart from a small reduction in database disk size,  it gained me nothing in storage space or upload limit.  I can (and do) delete duplicates when they pop up in searches or otherwise annoy me,  but otherwise I just get on with business and leave duplicates to emerge at their peril...    

[DTLow 5,736]

On 4/12/2020 at 2:37 PM, Patrick Holmes said:

Here are steps that work with Evernote version 7.14 on a Macbook Pro, also using Microsoft Excel for Mac 2011.

Or any spreadsheet product; MS Excel, Apple Numbers, Google Sheets, ...

>>(1) Select All notes, then View/Top-List-View

and Ascending Title sequence

>>Note that each row has a row number followed by a period. We will "clean" that in the the next step using Excel formulas...  (c) Enter or copy/paste this formula into B2:   =MID(A2,FIND(".",A2)+2,LEN(A2))

This can also be removed in Evernote by turning off the numbered list option