Azure IOT cbs negotiate claim error

[ghost]

• The issue is present in the latest release.
• I have searched the issues of this repository and believe that this is not a duplicate.

What happened?

Currently have our chirpstack setup in docker swarm.
Redeployed my chirpstack network server container yesterday and its now going into a boot loop.

This network server follows the Azure QuickStart guide and utilizes an Azure IOT hub and Service Bus.
The error its giving does not seem to have much info on the net but seems to maybe be a dependency issue with AQMP.

This network server had been running fine for 8 months.
I made a post on the forum with this content but it got no traction other than another user who said they got the exact same issue so putting it here to hopefully get some response as we have been unable to use our chirpstack setup.

More context on what i tried

switching to secondary keys for the events_connection_string and commands_connection_string
Regenerated the primary keys and tested with those
Tested the connection strings outside chirpstack and they seem to work fine.
No modification was done to the network server toml before the reboot so this is very strange.
The redeploy did update the version but even reverting that did not resolve the issue.
Updated all chirpstack components to latest versions and still have the same error.

Would appreciate any assistance on the matter.

What did you expect?

I expected Chirpstack network to connect to Azure as it has been doing for several months.

Steps to reproduce this issue

Setup a chirpstack pointing to Azure IOT Hub and Service queue as outlined here.
https://www.chirpstack.io/project/guides/microsoft-azure/

Could you share your log output?

Error
time=“2022-08-09T20:12:07.35505132Z” level=info msg=“gateway/azure_iot_hub: starting queue consumer” queue=us915-gateway-events
time=“2022-08-09T20:12:07.443098005Z” level=info msg=“gateway/azure_iot_hub: negotiating amqp cbs claim”
time=“2022-08-09T20:12:07.446436404Z” level=fatal msg=“gateway-backend setup failed: new c2d session and link error: cbs negotiate claim error: protocol error: received flow without next-incoming-id after session established”

Your Environment

Component	Version
Application Server	v3.17.7
Network Server	v3.16.4
Gateway Bridge	3.14.3
Geolocation	3.3.1

[brocaar]

It could be that something has changed in the way Azure does the authentication with AMQP. The Cloud to Device (c2d) messaging (for sending downlinks to the gateways) happens over AMQP, as there is no Azure REST API call for c2d. In order to connect to the AMQP endpoint, there is some magic happening. I remember I had to reverse engineer this from an other SDK (I believe I took their Python SDK as example), as Azure provides zero documentation on how to do this and their Go SDK does not provide c2d functionality.

Soon (as in the next days), I will start working on implementing this for ChirpStack v4 and I will dive into this further.

[ghost]

Any luck with this @brocaar?

[jburhenn]

We are seeing this issue with some of our hubs as well. The strange thing is the same code works with other hubs. It appears to be occurring in the cbs.NegotiateClaim of the azure-amqp-common-go package.

Has this been addressed in later versions of chirpstack-network-server? Do you know if it's a problem with azure-amqp-common-go or perhaps some configuration issue within the IoT Hub itself?

[brocaar]

Sorry, I do not have an update on this. For ChirpStack v4 I decided to not include the Azure IoT Hub integration For GW <> NS communication, because of the added complexity and issues that have been reported in the past with the Azure SDKs (there is still an integration to the Azure Service-Bus).

I believe there might be some SDK updates, but I have not tried these. Would you be able to try the updated package to see if that solves the issue?

[Opap1993]

Any update about this issue guys?

[brocaar]

No update from my side.

[Opap1993]

Any luck with this @brocaar?

Did you find any solution so far?

[Opap1993]

We have an update regarding our issue, apparently the problem was caused by the IoT Hub (we use Azure IoT Hub configuration).
We had to create a new IoT Hub in order for the Network Server to work again. Just to mention that we had to figure it out on our own since the Azure IoT Hub team did not provide us with much help.

[nparikhqmc]

We have an update regarding our issue, apparently the problem was caused by the IoT Hub (we use Azure IoT Hub configuration). We had to create a new IoT Hub in order for the Network Server to work again. Just to mention that we had to figure it out on our own since the Azure IoT Hub team did not provide us with much help.

Hello, did you have to recreate all your gateways for the new hub, and as a consequence did you need to update the bridge configuration on every gateway? We're facing a similar issue right now in a production environment.