Note and database "hygiene" tips to maximize portability

[tavor 668]

Do you have any tips on maximizing portability of your database and notes?

One thing I've been semi-cognizant of is not allowing my notes database to bloat up because as it relates to portability, if you need to travel, it's easier to travel light. I'm ~5k notes and the db is a little over 4gb. Just sorted by largest size and see that I have plenty of notes over 5mb. I can see right away that some are no longer of any use, so those will get the axe. Others I still want to hold onto, but really don't need the content in EN because I'll only need to access it from my main desktop - for those, I'll print to pdf (or save the content in a different file type if a pdf isn't the best option), delete content from note and replace with a link to the file on my desktop's disk. 

That's one example. Perhaps extensive note linking (notes with links to other notes) isn't great if one wishes to be maximize portability? Workaround for that, that is portable? 

Any other ideas?  Of course not all tips will be practical for all users who have an interest in this topic as our use cases are all different.

[DTLow 5,744]

On 12/11/2020 at 1:41 PM, tavor said:

Do you have any tips on maximizing portability of your database and notes?

My database just under 15GB      
My devices have lots of free space so I carry full offline copies (Mac and iPad)   
I have a designated offline notebook, but that's for quick download

I don't do any curating of my notes

As to portability out of Evernote;

I append the note-id (and other metadata) to the note contents   
This allows replacement of the link function with text search

As part of my backups I run a full export in html format    
My notes are can be accessed without the Evernote interface

edit; As posted by @eric99, I avoid Evernote text encryption

[10gallonhat 20]

Thanks @DTLow You're giving me ideas about how to potentially archive some old items. What do you do about attachments?

[DTLow 5,744]

21 minutes ago, 10gallonhat said:

Thanks @DTLow You're giving me ideas about how to potentially archive some old items. What do you do about attachments?

My only archiving action is to assign tag:!Archive    
This allows archived items to be excluded from searches

With the html export of notes, the note contents are exported as separate note-title.html files   
Attachments are exported in their native format to a note-title-resources folder

[10gallonhat 20]

@DTLowYou ARE giving me ideas. I have stuff in EN that doesn't need to be there anymore, but should be saved in a still accessible, if static, format. I can work with the test note I just exported. I'll need to try it on a group of notes first, but I can see the possibility of doing that and backing up to my NAS. Thankfully I'm pretty good about creating useful note titles and I'm a heavy linker.

[CalS 5,307]

5 hours ago, tavor said:

Do you have any tips on maximizing portability of your database and notes?

One thing I've been semi-cognizant of is not allowing my notes database to bloat up because as it relates to portability, if you need to travel, it's easier to travel light. I'm ~5k notes and the db is a little over 4gb. Just sorted by largest size and see that I have plenty of notes over 5mb. I can see right away that some are no longer of any use, so those will get the axe. Others I still want to hold onto, but really don't need the content in EN because I'll only need to access it from my main desktop - for those, I'll print to pdf (or save the content in a different file type if a pdf isn't the best option), delete content from note and replace with a link to the file on my desktop's disk. 

That's one example. Perhaps extensive note linking (notes with links to other notes) isn't great if one wishes to be maximize portability? Workaround for that, that is portable? 

Any other ideas?  Of course not all tips will be practical for all users who have an interest in this topic as our use cases are all different.

Some housekeeping bits. 

1. If you are going to export to PDF it may make sense to be sure you don't have any duplicate note titles. 
2. Ditto for any PDFs contained in notes. 
3. Also for PDFs in notes it may make sense to be sure the note name and the file name are the same, with the exception of the .PDF in the title.
4. It is easy to save attachments from a group of notes to see what you have and fix if need be.  AutoHotKey is a boon in this effort, voice of experience speaking.

As I have a pile of PDFs in a couple of local notebooks (7,567 as I type) I started looking for alternatives to no more local notebooks.  Typically I don't use Windows index for search, too much overhead if applied universally.  I did find in a test though that limiting the folders to be indexed does not create a lot of overhead.  And the search  and index functions in File Explorer work well, and fast.  So moving my local notebook PDFs to folders is an option.  Obviously not as seamless as EN but very workable as those PDFs are not high use - should EN not change course on local notebooks.  I would not create links within EN, I would just go to Explorer to fund things.

Linking is ugly.  If you keep the link name the same as the note title you at least have a mechanism to do a secondary search to find the note of record. . 

[eric99 1,080]

10 hours ago, tavor said:

Do you have any tips on maximizing portability of your database and notes?

One thing I've been semi-cognizant of is not allowing my notes database to bloat up because as it relates to portability, if you need to travel, it's easier to travel light. I'm ~5k notes and the db is a little over 4gb. Just sorted by largest size and see that I have plenty of notes over 5mb. I can see right away that some are no longer of any use, so those will get the axe. Others I still want to hold onto, but really don't need the content in EN because I'll only need to access it from my main desktop - for those, I'll print to pdf (or save the content in a different file type if a pdf isn't the best option), delete content from note and replace with a link to the file on my desktop's disk. 

That's one example. Perhaps extensive note linking (notes with links to other notes) isn't great if one wishes to be maximize portability? Workaround for that, that is portable? 

Any other ideas?  Of course not all tips will be practical for all users who have an interest in this topic as our use cases are all different.

For portability reasons, I avoid EN encryption as much as possible. I use standard encryptions instead (pdf , zip ...)

 

[gazumped 12,062]

I'm figuring that with over 50K notes,  it's going to be easier to leave stuff in my old Evernote (probably Basic by then) account whatever I may use in future. 

I'll link back to useful content or selectively export specific note / note series if I continue a related product.  It's nice to have a single place to search for stuff,  but another one isn't going to hurt as much as checking through that much of a body of notes to make sure they all made it!

[tavor 668]

On 12/11/2020 at 5:09 PM, DTLow said:

I append the note-id (and other metadata) to the note contents   
This allows replacement of the link function with text search

 

On 12/11/2020 at 10:54 PM, CalS said:

Linking is ugly.  If you keep the link name the same as the note title you at least have a mechanism to do a secondary search to find the note of record. . 

On the issue of note links as it relates to portability, as I understand it (and please correct me if I'm wrong), the issue is that enex files do not include a note's noteID. NoteID is how a link in Note A to Note B identifies Note B. Without Note B's noteID, there is no way for any app that imports enex files (including Evernote itself) to identify the note that is pointed to by the note link in Note A. So you end up with Note A containing a link to nowhere. Is that about right?

Why wouldn't EN, in defining the enex standard, not include the noteID? I guess it's not an issue if you have your exb file backed up, but restoring from enex files would be a minor disaster for someone who makes extensive use of note links.

@DTLow's solution makes sense. If you append noteID to each note's contents, then in my example, after importing your enex files, when your work next brings you to Note A, you see the note link (usually a title name - as per @CalS' suggestion, if you're diligent about this, you can search for that title to find Note B), copy the underlying link's noteID, and search for that noteID to find Note B, and then you can recreate the note link.

@DTLowhow are you appending noteID to each note's contents? AppleScript? I was looking at ENScript (for Windows) and don't see an obvious way to retrieve a note's noteID. You have more scripting knowledge than I do - perhaps you see an avenue of attack that I'm missing? https://dev.evernote.com/doc/articles/enscript.php

Alternatively, I could probably script something in AutoHotKey that cycles through all the notes and appends the noteID to each note.

On a side note, any idea why the note links repeat the noteID 2x? 

[CalS 5,307]

The note title must be accurate whether it is just the title or the ID in the title for this to work. .  Clueless as to how to get ID to title easily on Windows.  

I adopted the same title/link strategy early on to shortcut the losing links on import issues of EN. I use links but not extensively so I am okay with it.  And I’d rather keep titles clean, personal thing. FWIW 

[tavor 668]

4 minutes ago, CalS said:

The note title must be accurate whether it is just the title or the ID in the title for this to work. .  Clueless as to how to get ID to title easily on Windows.  

I adopted the same title/link strategy early on to shortcut the losing links on import issues of EN. I use links but not extensively so I am okay with it.  And I’d rather keep titles clean, personal thing. FWIW 

I wouldn't put the noteID in the title (too ugly and in your face all the time, when it would only be needed very rarely), but rather append it to the end of the note. 

[CalS 5,307]

20 minutes ago, tavor said:

I wouldn't put the noteID in the title (too ugly and in your face all the time, when it would only be needed very rarely), but rather append it to the end of the note. 

My bad, misunderstood. 

[DTLow 5,744]

2 hours ago, tavor said:

@DTLowhow are you appending noteID to each note's contents? AppleScript? I was looking at ENScript (for Windows) and don't see an obvious way to retrieve a note's noteID.

As you posted, the note-id is the last section in the internal note link
I use Applescript to retrieve the note link, and parse before pasting

[tavor 668]

1 hour ago, DTLow said:

As you posted, the note-id is the last section in the internal note link
I use Applescript to retrieve the note link, and parse before pasting

Would you mind posting the script? Maybe I can do something similar using ENscript for Windows. If I can access note properties using ENscript, that would probably be easier than doing this via AutoHotKey.

[Paul A. 676]

2 hours ago, tavor said:

 

On the issue of note links as it relates to portability, as I understand it (and please correct me if I'm wrong), the issue is that enex files do not include a note's noteID. NoteID is how a link in Note A to Note B identifies Note B. Without Note B's noteID, there is no way for any app that imports enex files (including Evernote itself) to identify the note that is pointed to by the note link in Note A. So you end up with Note A containing a link to nowhere. Is that about right?

Why wouldn't EN, in defining the enex standard, not include the noteID? I guess it's not an issue if you have your exb file backed up, but restoring from enex files would be a minor disaster for someone who makes extensive use of note links.

I'm not that familiar with the internals of EN links, but I do expect Evernote to do something related to improving links in the not-too-distant future. I saw an interview with the CEO who was asked about Roam Research, and he implied that Evernote would do its own spin on bidirectional links down the road... I'm hoping this is on the 2021 roadmap.

[DTLow 5,744]

34 minutes ago, tavor said:

Would you mind posting the script?

Here's a sample using AppleScript on a Mac

Keystrokes could also be used to retrieve the note link

[tavor 668]

1 hour ago, Paul A. said:

I'm not that familiar with the internals of EN links, but I do expect Evernote to do something related to improving links in the not-too-distant future. I saw an interview with the CEO who was asked about Roam Research, and he implied that Evernote would do its own spin on bidirectional links down the road... I'm hoping this is on the 2021 roadmap.

Roam and Obsidian have changed the game with respect to note linking. Evernote is a laggard here, so I expect they will enhance functionality.

This thread is more concerned with recreating note links. For whatever reason, EN does not include each note's unique identifier in the enex file. This is a problem for users who are making backups of their notes in enex format, expecting that if there's a problem with their database, they can restore by importing the enex files. They can restore, but all their note links will be lost. In the context of this thread, this creates a portability problem for users who use note links. No app (including EN itself) can recreate those links, so users making extensive use of note links may feel somewhat trapped because their data isn't as portable as EN has led them to believe. So in this thread we are discussing ways of getting around that.

[Paul A. 676]

32 minutes ago, tavor said:

Roam and Obsidian have changed the game with respect to note linking. Evernote is a laggard here, so I expect they will enhance functionality.

This thread is more concerned with recreating note links. For whatever reason, EN does not include each note's unique identifier in the enex file. This is a problem for users who are making backups of their notes in enex format, expecting that if there's a problem with their database, they can restore by importing the enex files. They can restore, but all their note links will be lost. In the context of this thread, this creates a portability problem for users who use note links. No app (including EN itself) can recreate those links, so users making extensive use of note links may feel somewhat trapped because their data isn't as portable as EN has led them to believe. So in this thread we are discussing ways of getting around that.

Got it; thank you for clarifying.

Hopefully they fix that problem, too! Data portability is important, and while it often isn't perceived this way, it can be a competitive benefit. When people are reassured that they can get their data out easily, they may be more comfortable committing to using your platform in the first place (and using it extensively, in a way that justified paying for a Premium subscription).

[tavor 668]

37 minutes ago, Paul A. said:

Data portability is important, and while it often isn't perceived this way, it can be a competitive benefit. When people are reassured that they can get their data out easily, they may be more comfortable committing to using your platform in the first place.

This cuts both ways. For certain when you are a small competitor, you better have data portability, otherwise users with thousands of notes at larger players like Evernote will be reluctant to try your app.

But what happens when you are the giant in the space? Do you really want to have excellent data portability, when it means you could lose more users than you could gain? Most of your new users are new to digital note taking or only have a small number of notes, so data portability isn't critical for them to try your app.