Connection freeze after a while

[aseyfpour]

Hi Dear Streamrs

I am trying to fetch a large amount of data from about 9 months ago. I developed an app that subscribes to a stream with resent option.
On the first try, I used a wide timeframe for the resent option (from: 9 months ago timestamp, to: now) but it freezes after a while and it doesn't receive any message or throw an exception.
So on the second try, I used a loop to subscribe on a narrow timeframe (an hour), after that It fetches all the messages in this timeframe I will unsubscribe this timeframe and will subscribe to the next timeframe. But in this scenario, it freezes too.
I tried these on different networks and different devices, the result is the same as each other. It is not happening at a specific time or after some amount of messages, it is completely random.
How can I help you to investigate this issue?

[fonty1]

Hi @aseyfpour - Thank you for reporting the issue!

It's quite likely to be a Streamr Network issue as we've been experiencing problems with resends lately. We're rewriting this core resend logic and it's a large engineering effort that is more a less complete, but requiring some testing. Instead of using websockets, the client will use a classic REST request for resends so it should be far more stable.

I would say this issue will resolve in 1-3 weeks and I will update you when the client update arrives.

[aseyfpour]

Thank you @fonty1.

[fonty1]

@aseyfpour while I think it's the resend issue I spoke to in the last message, I forgot to mention you should double check you're running the latest version - as there was a memleak fixed in the latest,

https://github.com/streamr-dev/streamr-client-javascript/releases/tag/v5.2.1

[aseyfpour]

@fonty1 Yes, I'm using 5.2.1. I also tried the 5.3.0 alpha version. Nothing has changed.

P.S: I reported that memleak issue: #233

[aseyfpour]

I also mentioned these network issues when I was verifying @timoxley patch.

[aseyfpour]

@fonty1 Is there any update on this topic?

[fonty1]

@fonty1 Is there any update on this topic?

Hey @aseyfpour, we're finishing up internal testing of the resend upgrade. Fingers crossed for a release this week, thanks for your patience with this one!

[fonty1]

Hi @aseyfpour could you try your resends now? The resend data brokers have been updated. If you have issues, can you be so kind as to report the behaviour and any logs you get.

Thanks!

[aseyfpour]

Hi @fonty1, Sure. I will do that.
Do I need to update my streamr-client? If yes, which version?

[fonty1]

@aseyfpour - please use v5.4.1 but the main change was on the backend, so it hopefully fixed in older versions as well.

[aseyfpour]

@fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
My Subscribe pseudo-code is something like this:

client.subscribe({
	streamId: stream_id,
	resend: {
		from: {
			timestamp: x
		},
		to: {
			timestamp: x + 1hr,
		}
	}
}, async (message, metadata) => {
	count++;
	console.log(`Count: ${count}`)
}).then(() => {
	console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
}).catch((err) => {
	console.log(`Error occured ${stream_id}`)
})

When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).

[fonty1]

@fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
My Subscribe pseudo-code is something like this:

client.subscribe({
	streamId: stream_id,
	resend: {
		from: {
			timestamp: x
		},
		to: {
			timestamp: x + 1hr,
		}
	}
}, async (message, metadata) => {
	count++;
	console.log(`Count: ${count}`)
}).then(() => {
	console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
}).catch((err) => {
	console.log(`Error occured ${stream_id}`)
})

When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).

@aseyfpour - Argh! Sorry to hear that. Can you share the streamId with me? Either here or to matthew.fontana@streamr.network. We can investigate the logs from there to see what went wrong.

[aseyfpour]

@fonty1 - here is my stream id: 94ylzZvOTdqf_kHFkLT2Aw

[harbu]

Hey, you could try to directly request the data from this HTTP endpoint

https://streamr.network/api/v1/streams/94ylzZvOTdqf_kHFkLT2Aw/data/partitions/0/range?fromTimestamp=<START_TIME>&toTimestamp=<END_TIME>

replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.

You also need to pass the following authentication header with the request:

Authorization: Bearer <SESSION_TOKEN>

The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.

[hpihkala]

While the workaround above should work for the time being, we're also investigating the original issue why the resends over websocket get corrupted. Could be an issue with the websocket library we're using.

[aseyfpour]

Hey, you could try to directly request the data from this HTTP endpoint

https://streamr.network/api/v1/streams/94ylzZvOTdqf_kHFkLT2Aw/data/partitions/0/range?fromTimestamp=<START_TIME>&toTimestamp=<END_TIME>

replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.

You also need to pass the following authentication header with the request:

Authorization: Bearer <SESSION_TOKEN>

The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.

Sure, I will feedback to you soon.

[aseyfpour]

@harbu The Rest API is OK. Its connection slows down sometimes but it is much reliable than the Streamr client.

[aseyfpour]

Any update on this?

[fonty1]

@aseyfpour - It's been a bit of a rabbit hole as multiple dependencies have needed to be switched out, I'll see about getting an update from the Network team this week about how much work is left before we can make a new production release.

[fonty1]

@aseyfpour - could you try again now? We've made some very heavy updates to the broker that are very fresh, but may solve the issue. Please give it a go.