Jump to content

[RESOLVED] (They made things right for the downtime too!) -- Is Evernote having some issues right now? 10/8/2024 (Update: This is completely a me problem, but also an Evernote server side problem)


Recommended Posts

23 minutes ago, mackid1993 said:

@Jon/t can you believe these answers?

Understand. Not an excuse, just an understanding but there's been a sync issue over the last few days and I would imagine a bunch of folk especially on the back-end team have been pulled off other stuff to get it fixed fast and it is fixed now.

I'm surprised its gone on this long. I would say more complicated/lengthy to fixed rather than any sort of incompetence. Everyone I've spoken to is pretty switched on and don't seem to mind hard work.

I really hope it gets sorted quickly for you.

Link to comment
  • Level 5*

They should definitely slow down development and work on the list of open problems. One way of triaging problems is to first work on those affecting the most people, with consideration to severity.

@mackid1993 if your issue is mainly affecting you, that unfortunately keeps you down on the list.

  • Like 3
Link to comment
  • Level 5

Every larger problem starts with a single case ...

I think the engineers should know by now what sort of problem they encountered here. With what little influence we users have on the clients, and even less on the backend, it should virtually be impossible for a user to individually cause a situations as the one encountered here.

This means from my view that everything that has happened in this case can happen to anybody else as well.

  • Like 2
Link to comment
25 minutes ago, Jon/t said:

there's been a sync issue over the last few days

Unfortunately it's not only a sync issue where data from clients have to be synced with the server(s).

In earlier times, clients did their job with no problem when offline: If I added a Tag to a note, the note was immediately available by searching with "tag:..."

Now it seems that all searches are based on a working online connection: searches that contain "tag:..." or "updated:..." either do not work (no answer) or work incompletely (not based on last changes) 😡.

For me EN10 is completely unusable this time 😤.
Even working completely offline is not possible... 

 

 

  • Like 2
Link to comment
Just now, s2sailor said:

Sounds like a failing shard which would explain the limited scope of the problem.

Exactly. Good thing I take backups. Although I think shards are replicated.

Link to comment
  • Level 5
Just now, mackid1993 said:

Exactly. Good thing I take backups. Although I think shards are replicated.

Which is why I'm creepy. 

Usually you switch to an alternate, and the user won't even notice some piece of silicon is now a brick (or better a pebble). If there is a (working) alternate ...

Link to comment
3 minutes ago, mackid1993 said:

Although I think shards are replicated.

I think so... remembering back to 2017/18 when they moved to Google cloud, I think one of the benefits was automatic duplication of data to prevent issues like this.

One thing I've realised since becoming an EN expert is the software is way more complicated than it should be and one of Spoons big jobs is simplifying it. I suppose that's what they call technical debt.

  • Like 4
Link to comment

No word from support, but suddenly it started working... and this isn't usually when it would work. It's been going for like a half hour now normally. I don't want to say they fixed it until I hear from support though and jinx it.

Link to comment

I'm having this issue as well, and is getting worse. Before the problem was a delay in loading the first note in a tag--now it's all the notes. I'm losing my train of thought waiting for notes to load--and they are small notes. Not a good thing for note-keeping software. I never used to have all these problems!

***

10/25: The latency issue is everywhere on the interface, such as slow loading of the toolbar. Latency also on the app. Yesterday I created a note remotely, and the note can't be accessed. I had to recreate it. Every time I go to update a note I have to wait until it loads--sometimes minutes! What seems to solve the problem is selecting another note--then the note i want loads.

  • Sad 1
Link to comment

New day - new luck? My syncing problem seem to be away. With one exception? Don't know whether it came with the sync probs or with 10.112.2. I've never seen it before:

UpdateDate of notes is not visible in result of searches: Whereas updated notes show correct in notebooks, their UpdateDates are grayed on when listed as search results:

image.png.6ae1bdf663ac02c5b1bbb341d1ccd6cf.png

Web-App (now on 10.112.3) is OK - So I'll wait for Win-App 10.112.>2 before adding a support incident 😉

  • Like 1
Link to comment
28 minutes ago, AlbertR said:

New day - new luck? My syncing problem seem to be away. With one exception? Don't know whether it came with the sync probs or with 10.112.2. I've never seen it before:

UpdateDate of notes is not visible in result of searches: Whereas updated notes show correct in notebooks, their UpdateDates are grayed on when listed as search results:

image.png.6ae1bdf663ac02c5b1bbb341d1ccd6cf.png

Web-App (now on 10.112.3) is OK - So I'll wait for Win-App 10.112.>2 before adding a support incident 😉

The syncing delay should be fixed now.

I've replicated the greyed out updated date issue. Only seems to happen in top or side view. Snippets and card views are OK. I've mentioned it.

  • Thanks 1
Link to comment
On 10/10/2024 at 4:12 PM, mackid1993 said:

Maybe they are doing something... instead of constant 503's I'm seeing this now quite a bit.image.thumb.png.c1b567be13a88284c4ef7355b3e691c3.png

I have been getting this on the web versions from three browsers. Have run through all the obvious steps and still get it. I run my business off evernote, so this is very alarming, since the problem appeared 4 days ago. Submitted a ticket then and getting AI responses. Not good. Already an Obsidian user, am thinking of moving mission critical content over there.

Link to comment
  • mackid1993 changed the title to FIXED: 10/25/2024 (They made things right for the downtime too!) -- Is Evernote having some issues right now? 10/8/2024 (Update: This is completely a me problem, but also an Evernote server side problem)
3 hours ago, bradsayers said:

I have been getting this on the web versions from three browsers. Have run through all the obvious steps and still get it. I run my business off evernote, so this is very alarming, since the problem appeared 4 days ago. Submitted a ticket then and getting AI responses. Not good. Already an Obsidian user, am thinking of moving mission critical content over there.

I've already moved to another solution for mission critical content. I still love Evernote and use it a little, but it's not stable enough for me at the moment to be my go-to. This thread is an example of several similar issues that I have faced over the last year with it.

  • Like 1
Link to comment
44 minutes ago, mackid1993 said:

Some good news.

Fine. May I summarize? <envious>

  • You get premium support (including contact to devs)
  • You got discount on your account
  • You have personally tuned server access

</envious>

But THX! You effort seems to help us all 🙂

Link to comment
11 minutes ago, AlbertR said:

Fine. May I summarize? <envious>

  • You get premium support (including contact to devs)
  • You got discount on your account
  • You have personally tuned server access

</envious>

But THX! You effort seems to help us all 🙂

With all due respect, I literally spent weeks trashing them on Twitter until they finally listened to me. I also sent dozens of logs screenshots and finally a screen recording that got their attention. 

  • Thanks 1
Link to comment
11 hours ago, Jon/t said:

I've replicated the greyed out updated date issue.

FYI: Has gone away without a client update 👍- so it must have been a server problem? 🤔

BTW: For the moment EN-App seems to be much faster - also without a client update 🙂
EN-Web has been updated to 10.112.4 - and feels even faster than EN-App. Great!

... Or have I got new server connected, too? 😉 


 

Link to comment
  • Level 5*
1 hour ago, mackid1993 said:

With all due respect, I literally spent weeks trashing them on Twitter until they finally listened to me. I also sent dozens of logs screenshots and finally a screen recording that got their attention. 

I think this validates the squeaky wheel approach 😀.

Link to comment

Seems like they may still have to monitor things. From 2:40 to 2:50 PM CPU usage on my shard spiked to 97%. She is trying to figure out now if it's my account or my shard. So it's better, but they are still monitoring and looking into this.

Link to comment

This is an interesting find on the engineer's part:

image.thumb.png.6342ef15587d2fd68462f7b4419859aa.png

She's working at midnight, so hopefully I'll hear from her tomorrow. I wonder why my client is calling deprecated APIs. The plot thickens. Say what you will about the direction of Bending Spoons and Evernote but it's almost midnight in Italy on a Friday and this engineer is still working--- on a problem for one guy. They have some damn dedication to their jobs.

Link to comment

Also, I suspect when the boss says, "I would like you to see if you can fix this problem for this guy," it provides some additional motivation.

Also, given the size of Evernote's US and western hemisphere user base, it wouldn't surprise to learn that they had some people working shifts during US business hours.  I'd be surprised if they didn't, actually.  And Evernote's not their only significant software service, so US hours (or western hemisphere hours to be less egotistical) may be baked into their normal HR strategy across the company.

Glad they're making progress with you.

 

Vinnie

Link to comment
Just now, VincentC said:

Also, I suspect when the boss says, "I would like you to see if you can fix this problem for this guy," it provides some additional motivation.

Also, given the size of Evernote's US and western hemisphere user base, it wouldn't surprise to learn that they had some people working shifts during US business hours.  I'd be surprised if they didn't, actually.  And Evernote's not their only significant software service, so US hours (or western hemisphere hours to be less egotistical) may be baked into their normal HR strategy across the company.

Glad they're making progress with you.

 

Vinnie

As am I. She just got back to me to ask for some more info. She's going to let me know once they figure out what is going on.

  • Like 1
Link to comment
4 hours ago, tired... said:

Thanks for your advice. I guess I also need to get far more persistent and direct, and also active on more platforms, to get the bugs and other problems solved.

If you don't make your voice heard, how can you expect to get anything done? They have 100 million users each with dozens of opinions. A lot of noise gets drowned out. It doesn't help that my issue was confused for a different bug that @Jon/t observed with client performance that was quickly fixed. That delayed me getting help by an entire week due to the confusion. I blasted them on Twitter nearly every day multiple times. I tagged them on Reddit directly as well, they have a support account on Reddit and Federico has an account as well. Two different Experts also were kind enough to flag this issue for me. Finally Federico responded to me one day on Twitter and a few days later I had a much higher up customer service person checking in with me every 2-3 days. Then yesterday they escalated the priority and today I heard from an engineer, we've been going back and forth all day, she's been asking me questions and collecting logs from me while she looks into things on her end. I even asked if they could give me a bit of free service for my trouble without specifying time and they gave me 3 months.

It also doesn't help that they just bought this company 2 years ago and didn't build it from the ground up, they took it on with 100 million users and had to rebuild it from scratch, so there's a disconnect and some inefficiencies between engineering and support that based on this thread we can clearly see are indeed improving with time. I'd say give them another year to refine their processes further and we'll start to see things become more efficient. It's all about having a positive outlook on things and life but also sticking up for yourself when you need to. We are a far cry from December 2023 where it would take 3 weeks to get a reply on a billing question. I needed sales tax refunded once and I waited nearly a month.

  • Like 8
Link to comment
  • mackid1993 changed the title to (ALMOST) FIXED: 10/25/2024 (They made things right for the downtime too!) -- Is Evernote having some issues right now? 10/8/2024 (Update: This is completely a me problem, but also an Evernote server side problem)

Pretty interesting, randomly it seems like activity on my shard is spiking I've been going back and forth with Daniela. She is looking into why my client is calling these deprecated APIs. Hopefully she finds a solution soon, but I'm hopeful now.

Link to comment
2 hours ago, mackid1993 said:

 She is looking into why my client is calling these deprecated APIs.

Maybe your Python Evernote backup? I don't use it myself, is that running in the background?

Link to comment
  • Level 5

The backup job is not running in the background, unless you create a task that executes it.

It is a command line tool. It’s pretty fast, because it saves incrementally only.

Yes, it may access the old API. But I doubt EN engineers could take it for the client by mistake.

Link to comment
17 minutes ago, eric99 said:

Maybe your Python Evernote backup? I don't use it myself, is that running in the background?

Seems like it was the webclipper. Somehow I had a few hundred sessions authorized. She ran a script to clear them out. Not sure how that could have happened.

Link to comment
12 minutes ago, mackid1993 said:

Seems like it was the webclipper. Somehow I had a few hundred sessions authorized. She ran a script to clear them out. Not sure how that could have happened.

I'm still monitoring this thread. You guys have been into some heavy geeking. That webclipper situation is bizarre. All of this is making me think more of alternatives. I've been good for a couple weeks now on EN web but for how long? Plus, it would be nice to have a desktop client again.

  • Like 1
Link to comment
20 minutes ago, mackid1993 said:

Seems like it was the webclipper. Somehow I had a few hundred sessions authorized. She ran a script to clear them out. Not sure how that could have happened.

uh, what does that mean: "a few hundred sessions authorized"? What are these sessions doing to the api when you're not using the webclipper?

Link to comment
  • Level 5
25 minutes ago, mackid1993 said:

Seems like it was the webclipper. Somehow I had a few hundred sessions authorized. She ran a script to clear them out. Not sure how that could have happened.

Do you close your browser tabs, or just let them hang around case you need them again ?

I would expect that closing the tab terminates all pending sessions.

Link to comment
1 minute ago, PinkElephant said:

Do you close your browser tabs, or just let them hang around case you need them again ?

I would expect that closing the tab terminates all pending sessions.

I would expect that closing the webclipper would terminate pending sessions as well

 

Link to comment

I have no idea. She killed them all and they started duplicating again. I think it might have something to do with sync in Vivaldi. She's going to look into it. She told me just to remove the web clipper for now until they can do some research into it. I legit had a couple hundred sessions though. 

Link to comment
8 minutes ago, mackid1993 said:

I have no idea. She killed them all and they started duplicating again. I think it might have something to do with sync in Vivaldi. She's going to look into it. She told me just to remove the web clipper for now until they can do some research into it. I legit had a couple hundred sessions though. 

So it's a webclipper or a vivaldi bug?

Link to comment

I think what was happening was for some reason I had hundreds of webclipper sessions using their deprecated login API. She cleared them out. I just removed my webclipper extension and readded it. Now I can see when I sign out of the Evernote webapp I'm signed out of the webclipper which is how it should work now. I'm currently signed in on two browsers so I have two webclipers signed in. Evernote backup was also using the old API. I singed back into that using evernote-backup reauth --oauth.

  • Like 1
Link to comment
4 minutes ago, mackid1993 said:

No Vivaldi, @Jon/t uses Vivaldi too so I'm wondering if he sees this also. The page is: https://www.evernote.com/AuthorizedServices.action

I do and mentioned it a couple of months ago. From what I was told the account page is still being run by the old monolith but the actual system is on new stuff. The solution seemed to be to wait until the account stuff was rewritten. 

That being said I experienced no issues because of it. Just 100s of web clippers. I don't think it's a Vivaldi thing, just a mismatch of old and new code. 

Link to comment
1 minute ago, Jon/t said:

I do and mentioned it a couple of months ago. From what I was told the account page is still being run by the old monolith but the actual system is on new stuff. The solution seemed to be to wait until the account stuff was rewritten. 

That being said I experienced no issues because of it. Just 100s of web clippers. I don't think it's a Vivaldi thing, just a mismatch of old and new code. 

The weird thing is that I see 100s of web clippers for chrome, which I rarely use, and none for Firefox, my regular browser

Link to comment
  • Level 5*
2 minutes ago, eric99 said:

Thanks for the link: btw I see hundreds of Evernote Web Clippers for Chrome on this page. It's not entirely clear to me what this page really shows

Same here and I don't use Chrome.  I did a revoke all and then rechecked and I have 6 now ... weird. 

Link to comment
7 minutes ago, eric99 said:

The weird thing is that I see 100s of web clippers for chrome, which I rarely use, and none for Firefox, my regular browser

Same. From what I understood it was more of a reporting/display issue than an actual issue, down to the page still being run from the monolith code. I know authorisation is totally different from the old setup so maybe this has something to do with it.

Link to comment
13 minutes ago, Jon/t said:

Same. From what I understood it was more of a reporting/display issue than an actual issue, down to the page still being run from the monolith code. I know authorisation is totally different from the old setup so maybe this has something to do with it.

Since we all see these long lists, maybe the @mackid1993 problem has nothing to do with this after all?

Link to comment
Just now, mackid1993 said:

She just confirmed it's a known bug in their backlog that doesn't affect anything. 

Does that mean that they are still in the dark?

 

Link to comment
39 minutes ago, eric99 said:

Does that mean that they are still in the dark?

 

No it's unrelated to my issue. She looked in their issue tracker about it. She just wants to see why my shard is calling a deprecated API and wants me to avoid using it until she can talk with her colleagues on Monday about it. 

Link to comment
1 hour ago, mackid1993 said:

No Vivaldi, @Jon/t uses Vivaldi too so I'm wondering if he sees this also. The page is: https://www.evernote.com/AuthorizedServices.action

Thanks for the link, you are obviously not alone: 

1 hour ago, eric99 said:

Thanks for the link: btw I see hundreds of Evernote Web Clippers for Chrome on this page. It's not entirely clear to me what this page really shows

I recently had issues with web clipper not loading and your link showed hundreds of authorisations for chrome as well. Funny since I only use chrome-based Brave browser on my Android phone and Safari on my Macs.

-> 

 

There are literally hundreds of them:

image.thumb.jpeg.f9c82331ff92299250f496bf9b607cab.jpeg

Link to comment

revoking them will not work: 

{"success":"false","reason":"csrf_failure"}

 

Edit: after several tries it seemed to have worked at last. 

No more Chrome authentications.

Link to comment
On 10/26/2024 at 8:24 PM, mackid1993 said:

Seems like it was the webclipper. Somehow I had a few hundred sessions authorized. She ran a script to clear them out. Not sure how that could have happened.

*****, I just checked and I have 2300 browser session for clipper for Chrome, which I can not even ever remember using (but I do have it installed).  I use Firefox.

Lets see if that improves the Windows app performance (just started having issues with it).. I doubt it

Link to comment
51 minutes ago, Grant837 said:

*****, I just checked and I have 2300 browser session for clipper for Chrome, which I can not even ever remember using (but I do have it installed).  I use Firefox.

Lets see if that improves the Windows app performance (just started having issues with it).. I doubt it

I was told that after looking into it, that this is a harmless bug. Cosmetic only.

  • Like 1
Link to comment
On 10/27/2024 at 4:39 PM, eric99 said:

The weird thing is that I see 100s of web clippers for chrome, which I rarely use, and none for Firefox, my regular browser

THX for link - just checked, found 2.687 "Evernote Web Clipper for Chrome" but use 99.9% of my time Edge (maybe based on Chrome?). Tried to revoke all but got "www.evernote.com kann diese Anfrage zurzeit nicht bearbeiten. HTTP ERROR 503" 😞 

... 3 times... With the 4th try, all have gone away 👍
But after that my clipper stopped working. Had to restart the browser. But even after that restart there was no clipper server seen 🤔

28 minutes ago, mackid1993 said:

I was told that after looking into it, that this is a harmless bug. Cosmetic only.

We're only very few users here. But some thousand services running for each for some million users is not only ugly cosmetic 🤡

Link to comment
20 minutes ago, AlbertR said:

THX for link - just checked, found 2.687 "Evernote Web Clipper for Chrome" but use 99.9% of my time Edge (maybe based on Chrome?). Tried to revoke all but got "www.evernote.com kann diese Anfrage zurzeit nicht bearbeiten. HTTP ERROR 503" 😞 

... 3 times... With the 4th try, all have gone away 👍
But after that my clipper stopped working. Had to restart the browser. But even after that restart there was no clipper server seen 🤔

We're only very few users here. But some thousand services running for each for some million users is not only ugly cosmetic 🤡

This is the answer I got:image.thumb.png.e4668d4d9494835bebf2a1830c7beac1.png

  • Thanks 1
Link to comment
4 minutes ago, AlbertR said:

Daniela wrote:

  • ... known glitch happening to some users....

Nice - and coincidentally we here are that "some users?" 😉

 

 
 

I don't know Jon said this is all going to be rewritten so I guess that's why it hasn't been fixed. Also I found these webclipper entries really mean web browser. I don't have the webclipper installed and this shows up just when I sign into the web client.

Link to comment
3 minutes ago, mackid1993 said:

... this is all going to be rewritten so I guess that's why it hasn't been fixed.

Maybe - but this assumption leads to an interresting finding: IIRC I've been on AuthorizedServices.action before not-so-long time and did not see such a problem. So their "rewriting" might be the problem 🤔

7 minutes ago, mackid1993 said:

... these webclipper entries really mean web browser ...

Don't know the real architecture behind. But this web site can show only connection end points on server site. Even if this does not mean that there are running (or hibernating) threads or processes, they might occupy sockets that at the end eat server resources...

 

Link to comment
4 minutes ago, AlbertR said:

Maybe - but this assumption leads to an interresting finding: IIRC I've been on AuthorizedServices.action before not-so-long time and did not see such a problem. So their "rewriting" might be the problem 🤔

Don't know the real architecture behind. But this web site can show only connection end points on server site. Even if this does not mean that there are running (or hibernating) threads or processes, they might occupy sockets that at the end eat server resources...

 

I mentioned it a couple of months ago and was told the page is still being served by the old monolith thingy so the data is displayed wrong. The new stuff in the background is working properly.

I think the end goal is to stick all the account stuff in the app itself but this hasn't been sorted yet hence millions and squillions of web clippers.

As far as I know its not a problem as such and the solution is to finish rewriting everything and switch off the monolith.

  • Thanks 1
Link to comment
4 minutes ago, Jon/t said:

I mentioned it a couple of months ago and was told the page is still being served by the old monolith thingy so the data is displayed wrong. The new stuff in the background is working properly.

I think the end goal is to stick all the account stuff in the app itself but this hasn't been sorted yet hence millions and squillions of web clippers.

As far as I know its not a problem as such and the solution is to finish rewriting everything and switch off the monolith.

I think it has a lot to with oauth also, the webclipper and the webapp is one auth token now, so the webapp shows up as the same session as the webclipper.

Link to comment
1 hour ago, mackid1993 said:

I think it has a lot to with oauth also, the webclipper and the webapp is one auth token now, so the webapp shows up as the same session as the webclipper.

and is your initial  problem solved now?

Link to comment
50 minutes ago, eric99 said:

and is your initial  problem solved now?

So it's much better today. She is just trying to figure out why the CPU on my shard spikes at random and why the webclipper calls a deprecated API. Also for some reason attachments at random will lag when opening for me so she's looking into that. So my main issue is solved, just a few more branches that were discovered that she found that she is trying to close off. It seems like I was having more than one issue that seemed like one issue but was actually many issues.

Link to comment
1 hour ago, mackid1993 said:

So it's much better today. She is just trying to figure out why the CPU on my shard spikes at random and why the webclipper calls a deprecated API. Also for some reason attachments at random will lag when opening for me so she's looking into that. So my main issue is solved, just a few more branches that were discovered that she found that she is trying to close off. It seems like I was having more than one issue that seemed like one issue but was actually many issues.

Yes, today I also experienced lagging attachments (minutes)

Link to comment
2 minutes ago, Dave Green said:

I have also seen delays on pages that I have not looked at in a while that have attachments.

This wasn't that. This was nearly every attachment and it was intermittent.

  • Like 2
Link to comment
  • Evernote Staff
18 hours ago, eric99 said:

Yes, today I also experienced lagging attachments (minutes)

Hi Eric, I’m Valeria from the Evernote team. To help us investigate this issue with lagging attachments, could you let us know if you’ve already reported this issue to customer support? If not, you can create a support ticket, and it would be especially helpful if you could attach your activity logs to the ticket (and share your ticket number here). This info will give our developers more insight into what might be causing the delay. Thanks for helping us improve Evernote!

  • Like 2
Link to comment
1 hour ago, Valeria P said:

Hi Eric, I’m Valeria from the Evernote team. To help us investigate this issue with lagging attachments, could you let us know if you’ve already reported this issue to customer support? If not, you can create a support ticket, and it would be especially helpful if you could attach your activity logs to the ticket (and share your ticket number here). This info will give our developers more insight into what might be causing the delay. Thanks for helping us improve Evernote!

Thanks so much for looking at this thread. This was a really wild issue and I'm glad you guys have been able to peg everything down. Daniela has been fantastic by the way!

  • Like 1
Link to comment

This still isn't resolved yet. They are still working. Not going to post updates here anymore. Taking a break from commenting on social media and especially these forums for a while.  I've got to take care of my own mental health right now.

  • Like 3
Link to comment

Enjoy your break.  If it's any help, I will personally find it to be a loss for me to not have you here contributing but, clearly, do what you need. 

FWIW, (and forgive the topic drift) the tools that I use that let me benefit from the expertise here without disturbing my inner peace 😁 are 1) I only come here once a day, 2) I use the "site read" button  when I get bored with repetitive posts (no criticism to posters generally, it can be hard to figure out where to start when one is experiencing a problem), and most importantly the ignore key.  I've fired it once already on today's session, just a few minutes in.  I've seen you stick with a conversation long after I would have jumped out, @mackid1993.  Props to you (and others) for your persistence - but, for me, as soon as a conversation turns personal, criticizing others here or calling Evernote staff names, I'm out.  (The example today (paraphrasing) was the poster who said basically, Evernote is ***** and Evernote staff are idiots.  And nothing positive to offer).  There's just no need for that, IMO, and it drives out positive contributors to the community, like yourself.

(I do see what I think are misunderstandings that can arise from the complexity of English and the challenge posters have whose first language is different - the subtleties of word connotations going back and forth between languages can make some posts seem more aggressive than perhaps the poster originally intended but, even then, a poster who delivers 100% criticism and 0% balance wins a place on my Ignore list.)

Anyway, that's how I "keep calm and carry on" in what can sometimes be an emotionally fraught environment around here.

Hope to see you back soon!

Vinnie

  • Like 5
Link to comment

Fortunately feeling more like myself again. I've been going back and forth with Daniela. Friday she rebuilt my database on the serverside which didn't fix it But yesterday she made some changes on the shard. I inquired and it has something to do with the way the monolith is configured. No CPU spikes for the last 48 hours. We need to monitor all next week because CPU usage is lower on the weekends.

She shared this with me and I thought you guys might find it interesting. You can see my last spike and how it's been quiet ever since.

yLv6llia3Dh60HL0o8mbAqbvi?token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..QAM3O9YBrQ138SBBTohycg.UfoPNQktvsJhXxqXMkdepF_iLFwuNnp91teEgdK3C_-RUJnSyJw643ht-fHO_yCknM0nQgM48b6bDe2OPxHSn-TgjsKFBYvecuDEPk111izgsosggMFaFmFqnEWI_08PWR1EMJ4zhRyG3pJxYhP8OhpTzLgX8MBrGTuVw3GPbfB8KRcByQ7oki-U4Y-KH2F2K2baKYxaHPCOZRwTIYc2Oc1Ld8vmxsN1cTQaGcBxGg2HndbXWGzUApfA72O5rOAhDBn1yGd_sN1mZSjzhG5-_vIvwa5pnVSD_ekEVHcNfnc.yNIG912-xGFq_GUwU11MIg

Link to comment
  • mackid1993 changed the title to NEARLY FIXED 11/3/2024 (They made things right for the downtime too!) -- Is Evernote having some issues right now? 10/8/2024 (Update: This is completely a me problem, but also an Evernote server side problem)
21 minutes ago, PinkElephant said:

Thanks for keeping everybody updated on this.

I still ask myself how a server user session can take such a turn without all alarms going of. Nobody home to read the logs ?

Well something was causing the CPU to spike. I also learned that each shard is a virtual machine, so they are probably running 10s of thousands of these virtual machines as I'm not sure how many users they have on one shard. It seems like the issue lies with the old monolith that they are trying to decommission which seems to be the cause of many of the problems with Evernote. 

Edit: this https://highscalability.com/evernote-architecture-9-million-users-and-150-million-reques/ is old but back in 2011 they had 90 shards. So now they are virtualized on GCP so it's not bare metal like it was back then, so maybe they have a few hundred shards.

Link to comment

I have been watching this thread, as I have to wait for each note to load for 0, but too often 4 to infinite seconds.  The main way to accelerate things is for me to click on another note in a list, and then go back to the other.  That is likely more of a display issue, but I also have notes, mainly those with an image (not big ones), that refuse to load.  I also have the CPU spikes.  It goes away, but only for a short while, if I log out, clear my data, and log in again.

For now, I will wait to see if this performance issue is fixed, but I will eventually put in a support request.

  • Thanks 1
Link to comment
9 hours ago, mackid1993 said:

Edit: this https://highscalability.com/evernote-architecture-9-million-users-and-150-million-reques/ is old but back in 2011 they had 90 shards. So now they are virtualized on GCP so it's not bare metal like it was back then, so maybe they have a few hundred shards.

You'll probably find this interesting.
For anyone not wanting to watch over an hour of tech stuff I wrote up a few interesting bits.

https://www.tamingthetrunk.com/i/150889861/behind-the-scenes-of-evernote

 

  • Like 1
Link to comment
5 hours ago, Grant837 said:

The main way to accelerate things is for me to click on another note in a list, and then go back to the other.

or force a reload:  CTRL-R

  • Like 1
  • Thanks 1
Link to comment
9 hours ago, Grant837 said:

I have been watching this thread, as I have to wait for each note to load for 0, but too often 4 to infinite seconds.  The main way to accelerate things is for me to click on another note in a list, and then go back to the other.  That is likely more of a display issue, but I also have notes, mainly those with an image (not big ones), that refuse to load.  I also have the CPU spikes.  It goes away, but only for a short while, if I log out, clear my data, and log in again.

For now, I will wait to see if this performance issue is fixed, but I will eventually put in a support request.

No, we are referring to CPU spikes on the shard... not my computer.

Link to comment
14 horas atrás, Grant837 disse:

I have been watching this thread, as I have to wait for each note to load for 0, but too often 4 to infinite seconds.  The main way to accelerate things is for me to click on another note in a list, and then go back to the other.  That is likely more of a display issue, but I also have notes, mainly those with an image (not big ones), that refuse to load.  I also have the CPU spikes.  It goes away, but only for a short while, if I log out, clear my data, and log in again.

For now, I will wait to see if this performance issue is fixed, but I will eventually put in a support request.

This is happening to me too.

Link to comment

It seems as if this all came down to some configuration issue with the monolith on my shard. CPU has been stable for 3 days now. She's going to monitor the rest of the week and if no more spikes I should be good.

The last spikes were Friday:ZWinDUjDJCnibOFeFiAiJrdAL?token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..sCoYAT5m0aLtNY_aEXkRuA.jVkHV2jQoZypsv9qRu22mUnVOMngMksP8B8Eli_L6lgP45vuQoLOpEZW3oXPm2F_XZOsuUSR-_DO7dr-ppbw3HQgpsmQfl4vog7AloZ6_FbzN2jLqEHLT7woqKK81hIGJfi69dgbUjmHWAQNkYBAr1eNnsGOl5e7M2vySE86OkKpS82u0K4fbrwHVFg-Jf5MVoubEimm-oaPp5DivUDE8BODirxTxxI_JIEuKAlvhsGyQIZdpw5uQaZY02COKBL9wNImVMFxM7NYeBTb72kNdpxtAjFritQ83Z5xRYkxa68.IG2OQD5XeVyF9lar7CkoVw

  • Like 1
Link to comment
On 11/4/2024 at 6:23 AM, Jon/t said:

You'll probably find this interesting.
For anyone not wanting to watch over an hour of tech stuff I wrote up a few interesting bits.

https://www.tamingthetrunk.com/i/150889861/behind-the-scenes-of-evernote

 

This was a great watch! So yeah the monolith which is running on the 750+ shards which are GCP VMs and the source of my misery should be dead by the end of the year! Good riddance! Which means those shards will be dead too. If you look at the comment from Daniella the webapp relies more on the monolith so when the CPU on the shard spiked the webapp failed for me and only certain functions in the other clients since those APIs likely relied on the monolith but many others don't. I imagine those attachments are still on the shard.

Edit: she confirmed today, attachments are served by the monolith still but should be migrated away soon as the monolith is sunset.

Link to comment

I take it that "monolith" is a polite term to describe what is really a tall pile of spaghetti code with poor documentation.  When I was much younger, I did broadcast engineering for radio stations. I quickly learned when touring a new station client to ignore the front of the equipment rack and look in the back of the rack.  Ususally I would find spaghetti wiring.  Cables running in every direction, none of them labeled, none of them organized.  The solution: rip it all out and start over.  I think Bending Spoons is going through the analogous process for software code.  

Vinnie

  • Haha 1
Link to comment
1 hour ago, VincentC said:

I take it that "monolith" is a polite term to describe what is really a tall pile of spaghetti code with poor documentation.  When I was much younger, I did broadcast engineering for radio stations. I quickly learned when touring a new station client to ignore the front of the equipment rack and look in the back of the rack.  Ususally I would find spaghetti wiring.  Cables running in every direction, none of them labeled, none of them organized.  The solution: rip it all out and start over.  I think Bending Spoons is going through the analogous process for software code.  

Vinnie

In Evernote's case it's an old Java 11 application, really outdated architecture built with a 2010s mentality. The monolith along with these shards will be dead by years end.

This is the CPU usage on my shard for the last 24 hours. Looking solid. Soon this won't even matter as they keep chipping away at what is relying on the shard and what isn't.

The team now is building mostly everything in Python, Typescript, and Node.JS.

AsyVoD1X4uOYq1pxhTwHH3I5H?token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..QsozElTXQ1BlcI4qePlo5A.k7uN2iceAWK9YVD6tu5_yuxTy84sMJUNCUo3b2hRhKsyCcXUAK6eR9Dhhq0Mz05fsvQbEPSNlw8wusQp3PLwY_KLHfBeHRB59lE8HLvq2cQAFQGTVWcJCWcvto1qZyUwqWISb0X8FyJHj7w_muKFmeYQORx-ne0k-Jd6_pPUFaDggBMwDKWs6HND3E685nu6umTgZ61j7aGr--GXTp4ZCFrzOW_Ubw4h-6AEextvLmn6c_vHzaAES9uPmvtXbfoWrBELPvnfGqGj75EYwIS1QtIrvcq1Di22S7lwn6AwgJE.kjFg3YzhEsU_2wSWwilfjQ 

Link to comment
8 minutes ago, eric99 said:

...which explains the poor performance...

😭 I know. Watch the video Jon shared. There was some talk of Rust but for an internal tool not really Evernote. Maybe they'll adopt Rust more in the future. 

Another interesting tidbit is they prefer to hire talent over experience. They want young talented people they can teach. Hence all of the bugs because they are loaded with inexperienced young college grads that make mistakes. That video explains a LOT. 

Link to comment
18 hours ago, mackid1993 said:

Edit: she confirmed today, attachments are served by the monolith still but should be migrated away soon as the monolith is sunset.

Interesting that attachments are still served by the monolith - I appreciate you sharing some of these technical tidbits. I've had infrequent but repeated issues with attachments, and I wonder if that's related to this. If you ever find out when they update the attachments code I'd be interested to know (and hopeful that my occasional attachments issues are resolved).

Link to comment
2 hours ago, mackid1993 said:

😭 I know. Watch the video Jon shared. There was some talk of Rust but for an internal tool not really Evernote. Maybe they'll adopt Rust more in the future. 

Another interesting tidbit is they prefer to hire talent over experience. They want young talented people they can teach. Hence all of the bugs because they are loaded with inexperienced young college grads that make mistakes. That video explains a LOT. 

Thanks for sharing that info. Guess that does away with the hope that this is just a teething phase before EN returns to a stable, reliable state.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.