Support MSC4222 state_after

[dbkr]

Closes #4532
Fixes element-hq/element-web#28536
Fixes #4528

This implements support for state_after in sync responses, as per matrix-org/matrix-spec-proposals#4222

In this, the events that arrive in the timeline should no longer be added to the state of the room. Instead, all state events arrive explicitly, duplicated, in state_after. Applying the events in state_after gives the state of the room after that sync request. This fixes problems where state events in the timeline should not actually change the state of the room, for example if old state events arrive from disconnected homeservers.

Synapse impl: element-hq/synapse#17888

This will need aggressive testing against synapse both with and without PR 17888. It's quite an invasive change. I opted to require callers of methods that add timeline events to explicitly specify whether they want the timeline events to apply to state since defaulting to either is not really safe, and this helps us catch all the code paths where it happens.

This changes some interface in the js-sdk and so should probably be considered a breaking change (and I've removed some deprecated interfaces while I'm at it).

Changes to public functions:

• SyncApi.injectRoomEvents() - parameters have changed
• Room.addLiveEvents() - addLiveEventOptions parameter is now required
• EventTimeline.addEvent() - toStartOfTimeline is now required; new required option addToState
• EventTimelineSet.insertEventIntoTimeline() - new required parameter addToState
• EventTimelineSet.addEventToTimeline() - deprecated variants removed; toStartOfTimeline is now required; new required option addToState

Checklist

• Tests written for new code (and old code if feasible).
• New or updated public/exported symbols have accurate TSDoc documentation.
• Linter and other CI checks pass.
• Sign-off given on the changes (see CONTRIBUTING.md).

[toger5]

This was tried in the SPA (EC with this branch directly connected to a home server)
We did run into the following issue:

• The call member events were synced on most clients (we assume there the reliable state worked, cannot be proven of course)
• The room members however showed incorrect bahaviours
• different clients had different room member counts
• On Hughs client a room member was there and then disappeared (but still is there on other clients)
• room.currentState.events("m.call.member") also has those missing members (on some clients) - as does room.currentState.getMembers()`
• `room.currentState.events("org.matrix.msc3401.call.member") showed the same events on all clients.

Found issue

This was caused by only storing events that come down through the timeline sync section in indexDB.
So a reload on the client makes you loose all events that you have synced as part of the initial sync and came down through state_after
A client that never reloaded will work fine since it has state_after state in memory and all changes will also update.
That is why we had clients with broken state (clients that were reloaded by the user) and clients that had all state.

[toger5]

There are still issues with the local storage.
The previous comment mentions an issue where,

a reload on the client makes you loose all events that you have synced as part of the initial sync and came down through state_after

This was easily testable by reloading a client and seeing all room names to disappear.

With the fix from @t3chguy this is not the case anymore. But we still see issues after a reload (loading sync from cache).

This can be used to investigate the issue https://pr2767--element-call.netlify.app/

If you are in a call and others join and leave, you reload the client the number at the top left will not be correct. The list of org.matrix.msc3401.call.member also will be incorrect.

[dbkr]

Thanks for continuing work on this. The changes look good as far as I can see, glad the sync accumulator wasn't too much of a big change to get working.

[toger5]

I only have minor comments. Nice that there is also the test to the sync accumulator

[richvdh]

crypto changes look fine

[toger5]

Is there anything else we should wait for before merging this?

This mentions the EW pr needs to be merged first: element-hq/element-web#28398

[t3chguy]

Yes, it will need force merging when I am done with the release process