Will RENT solve RTE problems?

[AlbertR 694]

In blog article At the end of 2023, Evernote’s technology is better than ever @Federico Simionato mentioned

• We’ll release RENT, a new metadata sync process.
• RENT is a counterpart to the collaborative editing initiative from April 2023.

 

I'll come back to some discussions around RTE - the "Real Time Editing" technology for note contents (from April 2023...):

• RTE is the reason for many problems around server API calls that try to modify note content and metadata...

 

So please,

• First solve RTE problems (and|or help Automation Software developers to work around them)
• before releasing new sync methods that may interfere with current workflows 🙏

 

Background (from my point of view - correct me if I'm wrong... 😉)
 

• Before RTE,
  • all syncs have been done by comparing modification dates of the whole note, then replacing the whole note by the new content.
  • This was done in sync processes that where triggered seldom (automatically or on request - but in all cases after completed changes of a note)
     
• With RTE,
   
  • a client may open a so-calls "RTE room" that blocks other clients from updating a note from other clients as long as the one client hold the room open.
    • OK, necessary for the following... 🤔
       
  • Every note editing operation in a client opens the room until this note is removed from display. This allows to send small changes of pieces of the note's content to the server along with every keystroke.
    • This in turn helps to save these small changes on server site in case of a client death or internet breakdown. OK, good idea... 👍
       
  • As long as the room is open for this note other clients may display the note - but are blocked from changing anything because a room cannot be opened once more...
    • This allows to receive small changes from server and display them immediately. Looks nice - but has no real advantage 😉
       
• The bad message with RTE is:
   
  • There is no possibility for API calls (or better said: applications that use the API) to handle closed "RTE rooms"
    • Yep - API hasn't been enhanced i.e. by "WaitForRTERoom()" or similar calls 😞
       
  • Every single API call that tries to change a note requests an "RTE room"
    • As long as the room is available (no client is editing the note), this is not a problem.
    • If the room is not available, the calls simply fails - most often the whole application fails 😞
       
  • Simply re-execute the desired call if it fails,
    • is not a good solution (even after implementing growing wait times)
    • because this results in extensive API usage which is counted and restricted on server site to avoid server overload situations
    • Clients my get errors like "EDAMSystemException:RATE_LIMIT_REACHED"
    • This in turn disables any API usage for an increasing amount of time for this account

 

One example: 

• I've implemented Make (Automation Software) scenarios that react of specific tags like
  • "Forward reminder" :=
    • If tag ">>2" is found
    • then set reminder date in two days and removes the tag...
       
• Whilst editing my notes, this tag is set along with some other changes (i.e. comments within the note text). So...
  • RTE room is opened the first change
  • Scenario is triggered during the editing session because changes are sent to the server immediately
  • but it fails to change the reminder date and to remove the tag...

 

Needless to mention that I've no problems as long as I'm working with Legacy 😀

[eric99 1,077]

4 hours ago, AlbertR said:

• As long as the room is open for this note other clients may display the note - but are blocked from changing anything because a room cannot be opened once more...
  • This allows to receive small changes from server and display them immediately. Looks nice - but has no real advantage 😉 

hmm, this is not what I experience... When I open the same note on my PC and my phone, both can change the note concurrently...

[anfang 38]

3 hours ago, eric99 said:

hmm, this is not what I experience... When I open the same note on my PC and my phone, both can change the note concurrently...

Could AlbertR be referring to split-second simultaneous edits near the same area of text?

I presume the 'room' opens and closes really fast to avoid blocking other clients, and the 'room' is localized to a small edit area (per line/block?). I guessing something like concurrent edits on multiple clients on the same line of text (or worse, the same word) would run into issues?

7 hours ago, AlbertR said:

I've implemented Make (Automation Software) scenarios

For a while I was stunned and thought this was referring to Make (e.g. GNU Make) 😂. Would be pretty hardcore.

[AlbertR 694]

6 hours ago, Cardin said:

Could AlbertR be referring to split-second simultaneous edits near the same area of text?

OK, I'll have a deeper look into this. You may be right because my changes normally touch the begin of note's text (newest changes are always on the top of a note to see the current state of the topic 😉)

6 hours ago, Cardin said:

I presume the 'room' opens and closes really fast to avoid blocking other clients,

This would be great if "to avoid blocking other clients" covers API changes, too. My experience is that API changes (from Make (formerly called "Integromat"), Filterize (now suspended - as a result of increasing problems with EN changes) or IFTTT (that I use rarely)) are rejected with error messages like

• EDAMSystemException: RATE_LIMIT_REACHED: {"errorCode":19,"message":"Attempt updateNote where RTE room has already been open for note: 5c3a5028-3664-4302-b644-783f73eb0a9d","rateLimitDuration":60}

[eric99 1,077]

4 hours ago, AlbertR said:

OK, I'll have a deeper look into this. You may be right because my changes normally touch the begin of note's text (newest changes are always on the top of a note to see the current state of the topic 😉)

This would be great if "to avoid blocking other clients" covers API changes, too. My experience is that API changes (from Make (formerly called "Integromat"), Filterize (now suspended - as a result of increasing problems with EN changes) or IFTTT (that I use rarely)) are rejected with error messages like

• EDAMSystemException: RATE_LIMIT_REACHED: {"errorCode":19,"message":"Attempt updateNote where RTE room has already been open for note: 5c3a5028-3664-4302-b644-783f73eb0a9d","rateLimitDuration":60}

I still have a question about legacy API behaviour:

How were note conflicts prevented when the API changed a note at the same time as other client editors? Wasn't there also a problem?

 

[Jon/t 1,652]

10 minutes ago, eric99 said:

How were note conflicts prevented when the API changed a note at the same time as other client editors? Wasn't there also a problem?

 

I think(?) it locked the note.

[PinkElephant 8,773]

From my understanding it was easier to handle when everything happened on the server. There you can probably lock a data set to avoid conflicting edits. The API should handle this together with the content management (remember, EN notes are basically websites).

The problems on legacy sync occurred when a local copy was changed, while another change took place, and then both synced to the cloud master copy. Since the local edits couldn‘t mutually lock the data set, a conflict resulted. And data set was the whole note.

With RTE the sync is happening in small pieces of data and very fast, so the conflict situation is in general avoided. Probably it builds shadow copies as well, from which a consistent result is reconstructed.

You can still provoke it by taking a client offline, change something there and make a different change elsewhere. Like changing a word in a paragraph on the offline client, but erasing the whole paragraph online. When the offline change syncs, there can still be a conflict.

[AlbertR 694]

FYI: another effect of possible fails of metatdata syncs is described in comment ...-sync-issues-...-tag-changes-not-saving-some-tags-simply-not-syncing-at-all

 

[AlbertR 694]

On 1/2/2024 at 2:07 PM, PinkElephant said:

With RTE the sync is happening in small pieces of data and very fast, so the conflict situation is in general avoided.

Nope - exactly this is the problem (that might explain the last seen effect with tag syncing...)

• BECAUSE sync happens in small pieces changes to tags and content (parts) are "synced" in pieces (between client and server)
  • and apps using the API try to work in parallel on these notes (on other maschines)
• ... changes mix up and will be synced back to clients.
• Thereby older data might come back to clients and will be seen as an automatic "UNDO" operation

This and (the formerly described problems with the "RTE room" blocks) were never seen before RTE came up. If EN would sync explicitely only complete notes - maybe optional - we wouldn't have the problems 😉

On the other hand RENT might solve these conflicts - will it for sure? I know from other projects that syncing even single data packages (notes) together with metadata that covers many packages (tags) without conflicts is a fairly unsolvable problem 🤔.

[Teos2515 4]

I’m not sure if rent alone will fix all the issues with RTE, as it's more of a surface-level solution. The deeper problem lies in how systems are designed and managed. AI could offer a more effective approach by improving processes through automation and personalized solutions. For example, creating AI-based systems that recommend more efficient solutions can optimize workflows. Here's a great resource on how to create an AI-based recommendation system https://ddi-dev.com/blog/case/how-to-create-an-ai-based-recommendation-system/ that explores this concept further. Ultimately, combining AI with practical changes might be a better long-term fix than just adjusting rent.

[Feitz 262]

On 1/11/2024 at 11:55 AM, AlbertR said:

If EN would sync explicitely only complete notes - maybe optional - we wouldn't have the problems 😉

+1

I still see no proper use case for RTE except for a collaborative environment, not for single users anyway.

With a dedicated sync button you finish your work, press sync and you're instantly done.

[eric99 1,077]

1 hour ago, Feitz said:

+1

I still see no proper use case for RTE except for a collaborative environment, not for single users anyway.

With a dedicated sync button you finish your work, press sync and you're instantly done.

that's how I always did with legacy

[Feitz 262]

4 minutes ago, eric99 said:

that's how I always did it in legacy

I didn't want to bring up Legacy because for some this is a no+no, but I never had sync issues with it (7.14 Mac).

[PinkElephant 8,773]

Maybe you didn’t have - or maybe you never checked. When I first installed v10, I had a significant difference in the number of notes between legacy, iOS and web. The largest difference (about 100 notes missing in legacy) were between web and legacy. The difference between web and iOS was an annoying but less alarming single note.

Legacy had several tools to repair the database. I needed to use them all (I think it was 4 cycles) until legacy finally matched up with the web (= server). Since then I question everybody if he BELIEVES everything was ok back then, or if he KNOWS.

[Feitz 262]

35 minutes ago, PinkElephant said:

or if he KNOWS

I suppose I do know. I never had a sync or database issue with 7.14 on any of my three Macs during four years of use whereas since I switched to V10 I was recommended to reinstall 4 or 5 times to mend bug-related issues, lost content and still don't see half of my note previews. At least this is consistent across all my macs (1* intel, 2* apple silicon).

[Alxa 495]

2 hours ago, Feitz said:

+1

I still see no proper use case for RTE except for a collaborative environment, not for single users anyway.

 

Me too. For personal note taking I do not have the same note open on two devices. BUT: If I do a change on desktop, I absolutely expect this change to be reflected on my mobile immediately (push). Come on, we are in the year 2024! When I am firing up Evernote on Android it still needs several seconds to run through all changes done remotely until I finally see (hopefully) the actual note content. Not to speak about images not being shown...attachments shows as "unavailable" on first opening of these remote edits and so on. RENT and RTE in theory come in handy if the clients you are using are always on and running, which mobile clients in real world obviously are not. I hoped RENT would eventually give us a background push sync service for the meta database for mobile. But no. Nothing on the road.

[Feitz 262]

50 minutes ago, Alxa said:

I hoped RENT would eventually give us a background push sync service for the meta database for mobile.

If I could choose between an old fashioned but reliable sync and a modern but flaky service I would choose old fashioned any day...

[PinkElephant 8,773]

Push sync is obviously not the most efficient way to organize a sync. It may be ok for messages (or a messenger). These get special status by the OS - especially that the app must not be open to receive the input on the device.

It is quite obvious that for a notes app these exceptions are not granted. This means as well that Push is not the way to optimize sync between a notes cloud server and its clients.

For a notes app the good old handshake method is still what does the job. What was obvious on the past was that changes to metadata often took hours to reach the clients. RENT rollout is still a young endeavor. We need to see if it improves the throughput time for metadata changes.

 

[Dave-in-Decatur 4,006]

On 9/16/2024 at 2:06 PM, Feitz said:

I still see no proper use case for RTE except for a collaborative environment, not for single users anyway.

Does this count? A few months ago I was working on a note on my phone when the battery died. (Definition of user error.) I turned to my laptop, brought up Evernote, and there were all my new edits, right down to the word I was typing when the battery died. A single user with multiple devices can find RTE useful, maybe particularly for certain PEBCAK situations.

[mackid1993 1,236]

1 hour ago, Dave-in-Decatur said:

Does this count? A few months ago I was working on a note on my phone when the battery died. (Definition of user error.) I turned to my laptop, brought up Evernote, and there were all my new edits, right down to the word I was typing when the battery died. A single user with multiple devices can find RTE useful, maybe particularly for certain PEBCAK situations.

It's also really nice for collaboration in real time.

For what it's worth, pre-RENT I'd have to stare at the tasks list on my phone for 60-90 seconds for it to update with what was marked off on my computer. Now it's more like 10-20 seconds.

[Jon/t 1,652]

On 9/16/2024 at 7:06 PM, Feitz said:

I still see no proper use case for RTE except for a collaborative environment, not for single users anyway.

 

I work on scripts on my desktop PC and then when I move into my little studio all the changes are waiting for me.

With old-skool god awful legacy you'd have you'd click a sync button, wait.... go to the studio, click a sync button... wait.

It now works like modern software. I know quite a few folk who use Evernote in studio environments, especially voice overs.

4 hours ago, mackid1993 said:

Now it's more like 10-20 seconds.

I think the plan is to get it much quicker than that. Just a couple of seconds, if that, depending on networks etc. 

[mackid1993 1,236]

7 hours ago, Jon/t said:

I think the plan is to get it much quicker than that. Just a couple of seconds, if that, depending on networks etc. 

That would be great!

[Paul A. 675]

13 hours ago, Jon/t said:

I think the plan is to get it much quicker than that. Just a couple of seconds, if that, depending on networks etc. 

That's what it should be, so I'm glad that's the goal. Question for you: do you know if RENT is still rolling out slowly? Federico's latest video update implied that it was fully rolled out, but I'm finding my phone still takes quite a while to refresh when I open it after making many changes and new notes on desktop.

[Jon/t 1,652]

48 minutes ago, Paul A. said:

That's what it should be, so I'm glad that's the goal. Question for you: do you know if RENT is still rolling out slowly? Federico's latest video update implied that it was fully rolled out, but I'm finding my phone still takes quite a while to refresh when I open it after making many changes and new notes on desktop.

Yes as far as I know it's all rolled out but there are tweaks happening. They've also just announced a separate team that will focus on performance so expect better things.