Will RENT solve RTE problems?

[AlbertR 579]

In blog article At the end of 2023, Evernote’s technology is better than ever @Federico Simionato mentioned

• We’ll release RENT, a new metadata sync process.
• RENT is a counterpart to the collaborative editing initiative from April 2023.

 

I'll come back to some discussions around RTE - the "Real Time Editing" technology for note contents (from April 2023...):

• RTE is the reason for many problems around server API calls that try to modify note content and metadata...

 

So please,

• First solve RTE problems (and|or help Automation Software developers to work around them)
• before releasing new sync methods that may interfere with current workflows 🙏

 

Background (from my point of view - correct me if I'm wrong... 😉)
 

• Before RTE,
  • all syncs have been done by comparing modification dates of the whole note, then replacing the whole note by the new content.
  • This was done in sync processes that where triggered seldom (automatically or on request - but in all cases after completed changes of a note)
     
• With RTE,
   
  • a client may open a so-calls "RTE room" that blocks other clients from updating a note from other clients as long as the one client hold the room open.
    • OK, necessary for the following... 🤔
       
  • Every note editing operation in a client opens the room until this note is removed from display. This allows to send small changes of pieces of the note's content to the server along with every keystroke.
    • This in turn helps to save these small changes on server site in case of a client death or internet breakdown. OK, good idea... 👍
       
  • As long as the room is open for this note other clients may display the note - but are blocked from changing anything because a room cannot be opened once more...
    • This allows to receive small changes from server and display them immediately. Looks nice - but has no real advantage 😉
       
• The bad message with RTE is:
   
  • There is no possibility for API calls (or better said: applications that use the API) to handle closed "RTE rooms"
    • Yep - API hasn't been enhanced i.e. by "WaitForRTERoom()" or similar calls 😞
       
  • Every single API call that tries to change a note requests an "RTE room"
    • As long as the room is available (no client is editing the note), this is not a problem.
    • If the room is not available, the calls simply fails - most often the whole application fails 😞
       
  • Simply re-execute the desired call if it fails,
    • is not a good solution (even after implementing growing wait times)
    • because this results in extensive API usage which is counted and restricted on server site to avoid server overload situations
    • Clients my get errors like "EDAMSystemException:RATE_LIMIT_REACHED"
    • This in turn disables any API usage for an increasing amount of time for this account

 

One example: 

• I've implemented Make (Automation Software) scenarios that react of specific tags like
  • "Forward reminder" :=
    • If tag ">>2" is found
    • then set reminder date in two days and removes the tag...
       
• Whilst editing my notes, this tag is set along with some other changes (i.e. comments within the note text). So...
  • RTE room is opened the first change
  • Scenario is triggered during the editing session because changes are sent to the server immediately
  • but it fails to change the reminder date and to remove the tag...

 

Needless to mention that I've no problems as long as I'm working with Legacy 😀

[eric99 1,028]

4 hours ago, AlbertR said:

• As long as the room is open for this note other clients may display the note - but are blocked from changing anything because a room cannot be opened once more...
  • This allows to receive small changes from server and display them immediately. Looks nice - but has no real advantage 😉 

hmm, this is not what I experience... When I open the same note on my PC and my phone, both can change the note concurrently...

[anfang 37]

3 hours ago, eric99 said:

hmm, this is not what I experience... When I open the same note on my PC and my phone, both can change the note concurrently...

Could AlbertR be referring to split-second simultaneous edits near the same area of text?

I presume the 'room' opens and closes really fast to avoid blocking other clients, and the 'room' is localized to a small edit area (per line/block?). I guessing something like concurrent edits on multiple clients on the same line of text (or worse, the same word) would run into issues?

7 hours ago, AlbertR said:

I've implemented Make (Automation Software) scenarios

For a while I was stunned and thought this was referring to Make (e.g. GNU Make) 😂. Would be pretty hardcore.

[AlbertR 579]

6 hours ago, Cardin said:

Could AlbertR be referring to split-second simultaneous edits near the same area of text?

OK, I'll have a deeper look into this. You may be right because my changes normally touch the begin of note's text (newest changes are always on the top of a note to see the current state of the topic 😉)

6 hours ago, Cardin said:

I presume the 'room' opens and closes really fast to avoid blocking other clients,

This would be great if "to avoid blocking other clients" covers API changes, too. My experience is that API changes (from Make (formerly called "Integromat"), Filterize (now suspended - as a result of increasing problems with EN changes) or IFTTT (that I use rarely)) are rejected with error messages like

• EDAMSystemException: RATE_LIMIT_REACHED: {"errorCode":19,"message":"Attempt updateNote where RTE room has already been open for note: 5c3a5028-3664-4302-b644-783f73eb0a9d","rateLimitDuration":60}

[eric99 1,028]

4 hours ago, AlbertR said:

OK, I'll have a deeper look into this. You may be right because my changes normally touch the begin of note's text (newest changes are always on the top of a note to see the current state of the topic 😉)

This would be great if "to avoid blocking other clients" covers API changes, too. My experience is that API changes (from Make (formerly called "Integromat"), Filterize (now suspended - as a result of increasing problems with EN changes) or IFTTT (that I use rarely)) are rejected with error messages like

• EDAMSystemException: RATE_LIMIT_REACHED: {"errorCode":19,"message":"Attempt updateNote where RTE room has already been open for note: 5c3a5028-3664-4302-b644-783f73eb0a9d","rateLimitDuration":60}

I still have a question about legacy API behaviour:

How were note conflicts prevented when the API changed a note at the same time as other client editors? Wasn't there also a problem?

 

[Jon/t 1,439]

10 minutes ago, eric99 said:

How were note conflicts prevented when the API changed a note at the same time as other client editors? Wasn't there also a problem?

 

I think(?) it locked the note.

[PinkElephant 8,374]

From my understanding it was easier to handle when everything happened on the server. There you can probably lock a data set to avoid conflicting edits. The API should handle this together with the content management (remember, EN notes are basically websites).

The problems on legacy sync occurred when a local copy was changed, while another change took place, and then both synced to the cloud master copy. Since the local edits couldn‘t mutually lock the data set, a conflict resulted. And data set was the whole note.

With RTE the sync is happening in small pieces of data and very fast, so the conflict situation is in general avoided. Probably it builds shadow copies as well, from which a consistent result is reconstructed.

You can still provoke it by taking a client offline, change something there and make a different change elsewhere. Like changing a word in a paragraph on the offline client, but erasing the whole paragraph online. When the offline change syncs, there can still be a conflict.

[AlbertR 579]

FYI: another effect of possible fails of metatdata syncs is described in comment ...-sync-issues-...-tag-changes-not-saving-some-tags-simply-not-syncing-at-all

 

[AlbertR 579]

On 1/2/2024 at 2:07 PM, PinkElephant said:

With RTE the sync is happening in small pieces of data and very fast, so the conflict situation is in general avoided.

Nope - exactly this is the problem (that might explain the last seen effect with tag syncing...)

• BECAUSE sync happens in small pieces changes to tags and content (parts) are "synced" in pieces (between client and server)
  • and apps using the API try to work in parallel on these notes (on other maschines)
• ... changes mix up and will be synced back to clients.
• Thereby older data might come back to clients and will be seen as an automatic "UNDO" operation

This and (the formerly described problems with the "RTE room" blocks) were never seen before RTE came up. If EN would sync explicitely only complete notes - maybe optional - we wouldn't have the problems 😉

On the other hand RENT might solve these conflicts - will it for sure? I know from other projects that syncing even single data packages (notes) together with metadata that covers many packages (tags) without conflicts is a fairly unsolvable problem 🤔.