NoPermissionError when using AWS ElastiCache

[epicserve]

When trying to deploy Django Channels for the first time in production, I got the Redis NoPermissionError exception when using the group_send() function.

Example code that would produce the error:

from asgiref.sync import async_to_sync
from channels_redis.core import RedisChannelLayer

redis_url = (
    "rediss://<user>:<password>@master.<endpoint>.use1.cache.amazonaws.com:6379"
)
to_username = 'test'
data = {"type": "push_message", "message": "Testing 1, 2, 3."}
channel_layer = RedisChannelLayer(hosts=[{"address": redis_url, "db": 0, "ssl_cert_reqs": None}])
async_to_sync(channel_layer.group_send)(f"push_message_{to_username}", data)

The Traceback from the Exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/opt/venv/lib/python3.11/site-packages/asgiref/sync.py", line 240, in __call__
    return call_result.result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/venv/lib/python3.11/site-packages/asgiref/sync.py", line 306, in main_wrap
    result = await self.awaitable(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/channels_redis/core.py", line 529, in group_send
    await connection.zremrangebyscore(
  File "/opt/venv/lib/python3.11/site-packages/sentry_sdk/integrations/redis/_async_common.py", line 98, in _sentry_execute_command
    value = await old_execute_command(self, name, *args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 612, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 586, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 633, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 569, in read_response
    raise response from None
redis.exceptions.NoPermissionError: this user has no permissions to access one of the keys used as arguments

I spent a long time debugging this and trying to figure out why it was failing because there wasn't anything I could find in the docs or searching the internet, and using ChatGPT wasn't helping much. It wasn't until I used the redis-cli in production and tried running the ACL GETUSER <username> command, which returned an error that then led me to discover that the user I was using only had the access string on ~app-test/* +@all -@dangerous +keys +info set. Looking at the docs, it says that the default prefix it uses is asgi, which explains why I was getting the error. When I added the prefix to RedisChannelLayer() (e.g., channel_layer = RedisChannelLayer(hosts=[{"address": redis_url, "db": 0, "ssl_cert_reqs": None}], prefix="app-test/")), the group_send() function worked. So I then updated the Django settings with the following:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [REDIS_URL],
            "prefix": REDIS_PREFIX,
        },
    },
}

Is there an explanation that could be added to the docs to make this easier to troubleshoot, or could a try/except be added in the code so that when it raises the exception, it gives a letter a better hint of what is going on? Maybe it could include the key it was trying to write to in the exception and suggest looking at the ACL for the Redis user.

[bigfootjon]

@carltongibson, how do you feel about updating the exception message to hint that maybe a prefix is required?

[carltongibson]

@bigfootjon That might be OK. (I'd be sceptical of adding anything too vendor specific, but an appropriate message here my be generally helpful.)

[epicserve]

@carltongibson I think the error would happen with any vendor's Redis service if the ACL didn't give you permission.

[bigfootjon]

@epicserve how do you feel about submitting a PR that updates the error message to mention common things to check to fix things? (i.e. "make sure you have prefix set correctly..." type things)

[epicserve]

@bigfootjon, I'm fine submitting a PR, but I'm wondering if you could point me in the right direction on how you would write a test that fails this way and assert that the exception message is correct. That seems like the hard part. Do you guys mock Redis or run Redis in docker for tests? If you run Redis in docker, you would have to figure out how change the ACL for a user temporality for the test to fail the way you want.

[carltongibson]

@epicserve — I think in that case I would mock it out, just to always error. We're checking only that our code responds correctly, and overriding redis configs for a single seems... erm... delicate 😅