Highlighting of search terms within notes

[Limehouse 4]

Hello,

I'm brand new to Evernote (and I'm just trying it out at the moment to see if I should commit to it) so should I be posting this question in the wrong place, or in the wrong way, or breaking any other 'conventions', please accept my apologies and tell me how I should be doing it please - thank you.

I like the way that search works within Evernote but I seem to have found a small anomaly and was wondering if anybody can tell me how to overcome it.  Having entered quite a few (50+) notes I searched for the phrase "to do list" [keying it into the search box exactly as that - i.e. the three words surrounded by double quotes] and the results showed four notes.  Within two of the found notes I quickly spotted the phrase for which I was searching as it was highlighted within an orange framed box.  However, in the other two notes there was no such highlighting (making me think that the phrase was perhaps not there).  On closer examination though, I found that these latter two notes contained the words "'to do' list" [i.e. the same phrase but with the two words "to do" also surrounded by single quotes] but it was not highlighted in the same way.

I like the fact that the search algorithm found these other two notes (that's quite powerful) but is there any way that I can get the system to highlight the phrase "'to do' list" (i.e. with the single quotes) as well as the phrase "to do list" (without the single quotes).

I've noticed that many answers on this website start by asking the question "Which version you are using?" - I've tried this on Windows, Android and the web - and they all work in the same way.

Can anybody help me please - thank you.

 

 

[rezecib 98]

@Limehouse Looks like the highlighting works as expected in the Mac client (highlighting both occurrences), but not in the web client (and I guess not Windows and Android, which I don't have test setups for myself). So this is a client bug, not something you're doing wrong.

To give some more insight into the mechanics of this, search retrieval and highlighting are done separately-- so in terms of search indexing/retrieval, both "to do list" occurrences will be tokenized into "to", "do", and "list"; searching "to do list" will match both because they both have the same tokens in the same order. Highlighting is implemented separately by each client, done on opening the note with the search context still active. And it looks like some clients are not using the correctly tokenized terms to do that highlighting.

Edit: a workaround you can do is to add some redundant terms to the search, such as:

"to do list" to do list

this will obviously over-highlight, but at least it will catch the cases that are being missed here.

[CalS 5,310]

On 4/14/2018 at 3:48 AM, Limehouse said:

I like the fact that the search algorithm found these other two notes (that's quite powerful) but is there any way that I can get the system to highlight the phrase "'to do' list" (i.e. with the single quotes) as well as the phrase "to do list" (without the single quotes).

EN ignores the ' in the search so do' is the same as do.  FYI, the underscore _  is the only special character that is searchable, do_ is different that do.

[Limehouse 4]

On 17/04/2018 at 1:23 AM, CalS said:

EN ignores the ' in the search so do' is the same as do.  FYI, the underscore _  is the only special character that is searchable, do_ is different that do.

Thanks for that - what you say explains why it finds the two extra notes (which is a good thing and I would always want it to keep doing that); however, if "do'" is the same as "do" why doesn't it highlight "do'" (with the eye-catching orange frame) in the same way as it highlights "do"?  Do you think that this is a bug maybe?

[CalS 5,310]

You are welcome.  Got me on the highlighting.  I did a quick test and highlighting seems to work for me.  Sometime EN loses the highlighting when you do something after the search.  Try redoing the search to see if the highlighting returns.  Could be a bug of sorts.

[Limehouse 4]

Hi CalS, Thank you for persisting with this.  Mine still doesn't work; any idea on what I might be doing wrong?

[jefito 5,598]

Does https://dev.evernote.com/doc/articles/search_grammar.php help? Specifically the Section "Matching Literal Terms". And even more specifically: "Multiple whitespace and/or punctuation characters in the quoted phrase or the note will be compared as if they were a single space."

[CalS 5,310]

1 hour ago, Limehouse said:

Hi CalS, Thank you for persisting with this.  Mine still doesn't work; any idea on what I might be doing wrong?

 

No problem.  My search was specifically just do as the term of the search, not the whole phrase.  I'm guessing if you just search for do only the do in do' will highlight.

[Limehouse 4]

@rezecib thank you for your answer; it is reassuring on a number of levels: (a) that it is a 'bug' and not something that I'm doing wrong; (b) that somebody from Evernote is monitoring the messages in this forum (not something that every software publisher does), and; (c) that the answer came so quickly.  In fact, I should like to thank everybody who contributed - coming onto this forum has been a pleasant experience.

Presumably (given that you are an Evernote employee), I won't need to report this issue elsewhere and that it will be fixed someday (but recognising that it is not something which stops the system working - so it may take a while).

Thanks again everybody

[DTLow 5,745]

On 2018-04-14 at 3:48 AM, Limehouse said:

how to overcome it

Having identified the bug; you could modify your search to     "to do' list" "to do list"
You'd have to do it for each variation.
Not a great solution, but this will get you the highlighting.

 

[Limehouse 4]

Hi @DTLow,  Thank you for that but (as @rezecib said above), the problem isn't in the search algorithm (which works), it is in the separate highlighting algorithm - and the bug is such that neither your nor @rezeczib's workaround solutions work I'm afraid (but would work if the bug didn't exist) - but, then again, if the bug didn't exist then this post wouldn't have been made either :-).  However, it's really not a problem because, having received @rezecib's explanation, I now understand the limitations and should therefore always be able to find some way around it until such time as the bug is fixed.  As somebody new to Evernote though, I am a bit surprised that text (in what, after all, appears to me to be a very text based application) cannot be searched (and highlighted) using Regex or something similar.  Don't get me wrong, I really like the Evernote search grammar - it is instinctive and natural (but it also lacks precision - so I couldn't distinguish between "to do list" and "'to do' list" even if I wanted to do so).  I would have thought that one could enter something like regex:[characterstring] into the search criteria and have Evernote perform a Regex search on your notes.

Thanks again to everybody though

Limehouse

[DTLow 5,745]

The search feature is documented at   Evernote Search Grammar

Definitely not regex based.

[jefito 5,598]

30 minutes ago, Limehouse said:

I would have thought that one could enter something like regex:[characterstring] into the search criteria and have Evernote perform a Regex search on your notes.

Wishful thinking meets the spec, a common occurrence. Regex would be nice, as well as any other number of enhancements, but that's not the reality at this time. My (semi-informed) hunch is that the search language as it is today is intended to foster quick searches over many notes (probably via some sort of specialized indexing data structure) so that you can get reasonable search times at UI speeds. Regexes over a large number of notes probably wouldn't give you that. 

[rezecib 98]

One of the major issues with regex at scale is that it's pretty easy to unintentionally write regex with exponentially scaling runtime with the size of the document. Standard regex libraries also don't usually support safe early termination (e.g. if it runs too long, kill it). So in the realm of possible search improvements full regex is quite unlikely to be added. Even Google doesn't support regex in their search, as far as I can tell.

We do, however, allow a final wildcard on search terms (and this gets added automatically). So for example "list*" will match "list", "listing", "lists", etc.

I did have a hack week project to add a more powerful search mode, but unfortunately it ended up being blocked by some clientside query processing. But hopefully that can be circumvented eventually...

[jefito 5,598]

16 hours ago, rezecib said:

Even Google doesn't support regex in their search, as far as I can tell.

As a sidelight on this: https://webapps.stackexchange.com/questions/19673/is-there-a-way-to-search-in-google-using-regular-expressions-regex led to 

 

[Limehouse 4]

Hi @rezecib , thank you for your (for a second time in this thread) very cogent explanation and also @jefito for your insightful comments and pertinent video - I'm afraid that I'm one of those "vanishingly small" group of nerds mentioned in the video who actually likes Regex .

Whilst I agree with @rezecib's argument that "it's pretty easy to unintentionally write regex with exponentially scaling runtime with the size of the document", and especially so where, with Google, the set of objects to be searched would be "the whole of the WWW" (and thus I fully understand that they will never implement it) whereas, in the case of Evernote, it would be limited to "the whole of my notebooks" (a much smaller set of things - but I'll keep trying ).   I should also be happy for it to be recognised that a Regex search might temporarily require scarce resource and therefore could take some time to complete but it would still be good if it could ever become available.

My suggestion would be that, a Regex term could never be included in search criteria which also contain the "any" modifier (they would be declared mutually exclusive) and it also could not take on the negating ("-") property (because any negation should always be able to be covered within the Regex search string itself).  In this way Evernote could first conduct the search ignoring the Regex term (which it does extremely well now, as there is no such facility) and once the matching notes have been selected (again, exactly as now) they would be displayed.  However, because a Regex term was included in the criteria, they would be displayed with a 'banner' similar to the one which says "this note is waiting to be synchronised" (i.e. the triangle in the upper right corner with the 'up arrow' in it) but with a different symbol (say, a whirling circle or an hourglass) to say that the search is waiting to complete.  A low priority background task could then set running on the displayed notes and they would gradually have their banner removed from them (if they contained the Regex term) or would be eliminated from the list (if they didn't contain the Regex term).  In this way, things shouldn't grind to a halt - just a thought

Talking of synchronisation, I've noted an unexplained piece of behaviour there which I'll raise in a separate thread.

Thanks again for a stimulating debate

@Limehouse