Search restriction request - exact match

[AndyDent 97]

Something that drives me utterly crazy is that Evernote desktop (both Mac and Windows) has a stupid search - putting things in quotes does not force an exact match.

This is contrary to the way the API and the web interface work. 

It is contrary to the way the documentation says it works, but there is no desktop-specific documentation explaining this difference.

This may not be much of an issue for trivial users. I have nearly 40K notes and frequently want to search on exact matches.

I have at times given up and used the web UI because desktop returns too many hits.

There seems to be some very serious quality engineering going on in Evernote - I'm very impressed by the speed improvements recently in v7.

So, please give us a way to search like you have documented, with exact matching.

My most recent attempt to clarify this was ticket #2765560 back in Feb 2019 at the end of which I received a clear message.

I've just heard back from our developers on the "Search Grammar" issue. From my understanding this appears to be an issue due to confusion between searching from the API vs. searching from the Evernote client.

According to our product engineers, the behavior described below is exactly how our API and client always worked.

1. When searching for the word foo through the API (that's important!), you will only find full words. foo will be matched, foobar won't. That aligns with what we have in the API documentation.

2. When searching for the word foo through any Evernote client(including Mac), the actual submitted search query that will hit our API will be foo* — this way you will find both foo and foobar.

According to our engineers, this behavior was always like that in all official Evernote clients.

Note that, earlier in the discussions, we had again gone around the circle where I'd said search didn't work as documented, people referred me to the search grammar pages (standard helpdesk response) and I had to prove that it didn't work. It would be funny if it wasn't so painful.

[DTLow 5,736]

5 hours ago, AndyDent said:

Something that drives me utterly crazy is that Evernote desktop (both Mac and Windows) has a stupid search - putting things in quotes does not force an exact match.

I've noticed some inconsistencies with the search, but never considered it a big deal; certainly not something "that drives  me utterly crazy"

My preference is for an exact search, unless I use  wildcards.

[skkippy 12]

Odd. I just checked my 7.10 Evernote running on Mac OS X 10.14.5. A search for invoic returns over 200 results; a search for "invoic" returns only 2, excluding notes that contain the word invoice.  As expected, the quotation marks forced exact match.

Footnotes: The 2 exact matches were found as words in images (incorrectly as it turns out - the actual images contained the full word).  I also checked case-sensitivity: as expected both searches were case-insensitive.

[PinkElephant 8,103]

Did the check myself, cutting away the last letter of „Rechnung“ (invoice in German).

Similar result, searching for Rechnun created several hundred hits, searching for „Rechnun“ only 7.

Maybe this results from OCR where there may be alternative spellings saved, or where this exact spelling resulted from a OCR by accident. There is OCR software that will save different options when it had problems to identify a word.

Then I have tried with Premium and „Premi“. Here it was 99 to 0 (zero).

So I think that the hyphen do really work for exact matches, as described, at least on the Mac client.

[gazumped 11,664]

Not a Mac user,  so no input on this really... but my attitude,  when things don't work out exactly as advertised,  would be to find a way to work around the problem. 

So if searching for "foo" (including the quotes) gives an inaccurate response,  I'd add a tag to the correct hits so that "tag:foo" (no quotes this time) gives me the correct result. 

Yes,  things should work as advertised;  frequently they do - but there are glitches and there are bugs in all software.  We have to learn to live with them - even when people work to fix things, the fixes take time to implement.

[PinkElephant 8,103]

Tagging means that I know in advance I am going to search for it one day. Works fine, as long as I know. But often I will not know in advance, and then reliable search is a must. The initial post opens a valid question, that advanced search does not work as advertised.

But my own quick test leads to another assumption: Search works as advertised / described. 

• Text only means text within context, not necessarily exact match
• Text in hyphens means exact match

The residue of unexplained search results IMHO is related to faulty OCR, not to the search function itself.

This is my impression from a brief test described above on the Mac client. When I find some time, I may repeat this on the Windows and iOS clients, just to learn for myself.

Conclusion: If we want to be sure to find what we are looking for, we should make sure the OCR is performing when saving our stuff. Or don’t OCR ourselves, and leave the OCRing to EN on the server. When a document comes already loaded with OCR content, EN will not OCR it again. Thus we can ourselves create the messy search we are complaining about afterwards.

[AndyDent 97]

5 hours ago, PinkElephant said:

But my own quick test leads to another assumption: Search works as advertised / described. 

• Text only means text within context, not necessarily exact match
• Text in hyphens means exact match

 

Where is use of hyphens advertised/described? It doesn't work for me. Are you talking about a really specific example where this worked or am I missing something?

The documentation I linked to is the standard Evernote search help page which clearly states using quotes to limit a search. It is the top hit when you Google Evernote Search.

At the bottom of that page is a link to the Search Grammar page.

I erupted with frustration a while back because the standard helpdesk response is to direct people to that page even though it is only correct for the web client. Neither of those pages states that there is any difference with desktop or mobile client search.

In one of my more recent support ticket responses I got the impression I should have been cognisant of the wording on the Search Grammar page that it says Cloud API.

Users do not care if you are documenting an API.

If an API describes behaviour that is not applicable to end user apps, it should be very difficult for them to find and have a clear disclaimer at the top. (more lines of rant elided...)

Back to the search problem

Here's the test I used in reporting the problem - search for china "chan"

That's a good test because if the quotes are working it should ignore common words such as changes and changed.

On the web, I get 27 notes (out of my 40K), indicating that the web client uses the API directly and thus works as documented.

On Mac and Windows desktop, iOS mobile clients: 43 notes found, with the highlighting indicating it is indeed doing a partial match on chan in changes.

Interestingly, my Android client found the correct 27 notes but I wonder if that was because I seldom use it and it did the search whilst it was part-way through a major metadata sync, so would have had to call the API for the search.

Note, my frustration is not because I have to skim 27 vs 43 notes but because in some cases I'm visually scanning hundreds of false positives, having to read through their content. That's a massive waste of time.

So my workflow is often:

• search in Mac client
• realise there are massive numbers of hits
• swear
• open the web client
• repeat search,  noting drastically smaller list
• stay in web until I've finished my task 

 My workflow if I'm away and trying to use mobile clients involves a lot more swearing if need to search on small terms.

Why am I so obsessed by, and upset with this?

1. It's really broken - there is no replacement for being able to restrict searches other than having to use a web client, which is not always an option. A major product point is that Evernote should work offline.
2. It's neither well-known within Evernote support nor is it documented, because the wrong help advice is given. I had to push and push to get someone to talk to developers to realise I was right and the docs didn't reflect reality.
3. As a developer for over 35 years, who's worked on multiple database engines, query interfaces and data visualisation products, I'm professionally embarrassed on behalf of Evernote. This is shoddy, indefensible work.

 

[AndyDent 97]

5 hours ago, PinkElephant said:

Conclusion: If we want to be sure to find what we are looking for, we should make sure the OCR is performing when saving our stuff. Or don’t OCR ourselves, and leave the OCRing to EN on the server. When a document comes already loaded with OCR content, EN will not OCR it again. Thus we can ourselves create the messy search we are complaining about afterwards.

Just to clarify things, for anyone reading that comment, I am not talking about OCR.

Evernote search is fundamentally broken as I described at length for original content typed into notes and text clipped from the web.

I agree OCR is not perfect and sometimes I've seen slightly off interpretation of words on photos. This is not the same thing.

[AndyDent 97]

13 hours ago, skkippy said:

Odd. I just checked my 7.10 Evernote running on Mac OS X 10.14.5. A search for invoic returns over 200 results; a search for "invoic" returns only 2, excluding notes that contain the word invoice.  As expected, the quotation marks forced exact match.

Footnotes: The 2 exact matches were found as words in images (incorrectly as it turns out - the actual images contained the full word).  I also checked case-sensitivity: as expected both searches were case-insensitive.

I think I have an explanation for our different results.

I'm fairly sure, after reading your and PinkElephant's post, that you are only talking about hits on OCR documents. That's a very interesting point of difference, which may indicate these searches are using the server API rather than local indexes.

Do you have any plain text notes which verify these results? What happens if you type in a plain note with invoice in it then do a search on "invoic"? 

Note that the results I quoted in my message above are from testing I repeated as I wrote the message, on the current desktop clients. I'm definitely not working from old samples or memory. During the last tech support thread I triggered complete re-indexing to see if there was a related problem. This is highly repeatable for me.
 

[PinkElephant 8,103]

Frustration about support I can follow. First level asks 3-5 questions to you for each one you’ve asked to them. Often it would be an endeavour taking several Pomodoros to get all the information, so I sometimes do not follow up. But they showed some methods to me of how to make live screen copies from my devices, which sort of balances the bill 😉

They have good staff on 2nd level, and sometimes one of the devs will be called - then it usually is o.k.

My database (and use) of EN is mostly pdf-attachments embedded in notes with no or only few additional content. Most of it is tagged. Thus I heavily rely on OCRed docs. Many docs are in German, some in English. OCR in my case derives from

- own scan, OCRed partly with the Nuance Paperport SW delivered with my ix500 last year, part only scanned and OCRed by EN on the server. You can not tell these 2 apart, only if you dig into the pdf itself. EN will not alter the original file when doing their OCR, so if the file contains OCR-information, it is from the Nuance process.

- Scanned and partly OCRed with ScannerPro on my iPhone 6S+. Same issue as before, plus these docs were all heavily corrected in their perspective by the scanning app, plus these docs contain more irregular paper sizes/shapes than the nice A4s done on the ix500.

- Clipped pages from the web, most of them not by the webclipper but clipped on my iPad through the share-print-topdf-share workflow.

- some pdfs from Goodnotes 5, handwriting with OCR from the app

- some pdfs I’ve received as pdfs though mail or the web, send untouched into EN

- little notes with text written into the note itself

- nearly no other docs than pdfs (Office-type)

Using hyphens to mark an exact match comes some of natural to me from other services, so I’ve simply tried it. I am used from the old days to build Boolean search strings instead of todays „type in the words and hit enter“. The hits in the pdfs are highlighted, so I can see visibly what they believe to have found. What is amazing to me is that there are often hits in the very fine print like the footer of official letters or the legal stuff at the rear of pages in light blue on lighter blue paper.

This leads to additional docs found where you would not expect the searched string, like you search for invoice, and it finds an order confirmation printed on stationary that at the rear in the legal section talks about their invoices as well. For you this may look like a false hit, for the search it is a valid hit because it is „there“.

When I find some time, I will play around with Windows and iOS as well. From the basic design of the clients, I would expect the desktops to search in the local database, and the mobiles on the server. Maybe I will go offline 😱 a bit to verify.

[CalS 5,280]

11 hours ago, AndyDent said:

search for china "chan"

Did this and I can verify your results, with some additional weirdness.

I got 8 notes on the Windows desktop and 5 notes on the web (web beta and IOS got the same five results).  However, only one of the notes is in both lists.  Not sure on "chan", all of the notes returned in my case had attachments, PDF or Office.  I created a note with chan and China in it.  It seems for text only notes "chan" works.  Changing chan to chances eliminates the note from the search results. 

So maybe something funky with attachments, which could be OCRish for PDFs and not sure what for Office.  Which is  bad in my view based upon the number of notes I have with attachments.  I know for a fact that the EN server PDF OCR engine does not perform as well as the one used by ScanSnap.  Reported some years back and got confirmation from EN staff.

[skkippy 12]

11 hours ago, AndyDent said:

Do you have any plain text notes which verify these results? What happens if you type in a plain note with invoice in it then do a search on "invoic"? 

Hello Andy,

I created a note titled test with only the word invoice as content. Search behaves as expected: invoic finds the note; "invoic" does not. Evernote 7.10 OS X 10.14.5. 

[lisec 283]

I know that search excludes some characters from a search (things like @mary or @terry will be found when searching for mary or terry even though they begin with the @ sign) but how can I force the exact match "@mary" or "@terry"?  I was using @mary and @terry tags but they are too infrequent so I decided to put them inside the note instead, specifically keeping the @ at the beginning to filter for the mary and terry I know rather than have to go through hundreds of web articles with references to mary and terry.  Is there a way to force a search for "@mary" or is this also the result of the way Evernote searches and I'm up the creek...?

Update: ok, turns out that if I use an underscore (_mary) I can search for that without have to use the exact match quotation marks.  I can definitely go ahead and make these changes however I'm not sure I should invest that much time doing it if it turns out it is a fluke or a bug or something that Evernote will change. Anyone know why underscore seems to be the only exception?

Edited June 28, 2019 by lisec
updated with new info

[gazumped 11,664]

Hi.  I "know" that Evernote will only search for upper and lower case alpha + numeric characters and the underscore "_" at the start of a search term.. but I'm blessed if I can find any documentation to support that knowledge.  Evernote being,  well Evernote,  I don't think there are any cast-iron guarantees whether that will ever change - but it seems unlikely in the short term.  FWIW I use 'special' keywords like Xmary or Xterry if I need to establish a searchable key,  but I know that others use underscores for memorable tags...

[CalS 5,280]

Yup, underscore is the ONE special character that IS searchable.  

[lisec 283]

Ok, thanks guys. That's too bad for me, because I am so used to having @ precede a person's name, # precede a place and other such symbols I use throughout my tags for easy reference. Would have been nice to use them in the text as well.  I guess I have to choose: use a letter like gazumped does or use the _ and hope Evernote doesn't change its mind. 

[CalS 5,280]

53 minutes ago, lisec said:

Ok, thanks guys. That's too bad for me, because I am so used to having @ precede a person's name, # precede a place and other such symbols I use throughout my tags for easy reference. Would have been nice to use them in the text as well.  I guess I have to choose: use a letter like gazumped does or use the _ and hope Evernote doesn't change its mind. 

You can use the special characters with tags, they are searchable there.  I use = for people, ! for TSW,  for projects and _ for fiscal years.  I use _ in notes for things like _Paid.  FWIW. 

[lisec 283]

5 minutes ago, CalS said:

You can use the special characters with tags, they are searchable there.  I use = for people, ! for TSW,  for projects and _ for fiscal years.  I use _ in notes for things like _Paid.  FWIW. 

Yeah, that's what I do now but I have a few people who are only tagged in about 8 notes and I'd rather just stick @terry inside the note instead of using up a tag. Or at  least that was the plan. To be consistent in my searches I'll have to either keep the tags I was going to eliminate, or change the tags to reflect whatever method I use in the notes (ie _terry in the note and _lucie as a tag, so that any search with _name will show up, whether in a note or in a tag). 

[CalS 5,280]

1 hour ago, lisec said:

Yeah, that's what I do now but I have a few people who are only tagged in about 8 notes and I'd rather just stick @terry inside the note instead of using up a tag. Or at  least that was the plan. To be consistent in my searches I'll have to either keep the tags I was going to eliminate, or change the tags to reflect whatever method I use in the notes (ie _terry in the note and _lucie as a tag, so that any search with _name will show up, whether in a note or in a tag). 

Got it.  May be more work and bad feng shui but you you could do something like _pTerry.  NOPE, yuck just as I typed it. 

Or just burn a tag for @terry.  Not the worst thing that ever happened in the world. That way an @ in a tag drop down will show all people.  No confusion, plus if Terry becomes more popular no changes have to be made.  

[DTLow 5,736]

3 hours ago, lisec said:

instead of using up a tag

The limit is 100,000 tags.  I'm not concerned with using up tags

>>any search with _name will show up, whether in a note or in a tag

I prefer to select tags from the list 

>>ie _terry in the note and _lucie as a tag

 

Or was that terri and lucy

Also, it's easy to rename tag-names, harder for "in the note"

 

 

[lisec 283]

On 6/28/2019 at 8:59 PM, CalS said:

Or just burn a tag for @terry.  Not the worst thing that ever happened in the world. That way an @ in a tag drop down will show all people.  

On 6/28/2019 at 9:05 PM, DTLow said:

The limit is 100,000 tags.  I'm not concerned with using up tags

 

All true, but my scrolling finger can only do so much on my Android phone... 😀

[CalS 5,280]

55 minutes ago, lisec said:

All true, but my scrolling finger can only do so much on my Android phone... 😀

Yeah,  with technology.  

With IOS you can search tags....

[DTLow 5,736]

54 minutes ago, CalS said:

With IOS you can search tags....

The scrolling might be easier too.  With a few flicks I can move through my 300+ tags

[lisec 283]

Look at you apple heads showing off....

[PinkElephant 8,103]

Who’s got it, has got it 🍎

And one per day as well keeps the doctor away.