Scaling metrics beyond CPU monitoring

[marksweb]

I've got a GKE cluster running Nginx & daphne workloads to provide websockets for some chat functionality. (I've not defined ASGI_THREADS, so it's a very default setup)

What scaling metrics do people use for their daphne/channels infra?

Currently my cluster is setup to scale on CPU, but daphne/channels don't really take a lot of CPU power. The pod resources that I'm running are;

    resources:
      limits:
        memory: 350Mi
      requests:
        cpu: 900m
        memory: 250Mi

One potential fix here might be to reduce the CPU resources and lower the scaling threshold, but that's not really how you should do scaling because that's just trying to guess when it might need it.

The django side of things is simple, setting up using redis;

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {"hosts": [f"{REDIS_URI}/3"]},
    }
}

I've been wondering if there are custom metrics that I might be able to gather, similar to getting the queue lengths from celery.

[carltongibson]

Hi @marksweb

The other thing (beyond memory and CPU) to measure then is response time... — when does it start getting slower?

For simple request-response Nginx can upstream response time. If it's long-lived requests, you may need to add some custom timing hooks at key places.

[marksweb]

Thanks @carltongibson, yes, good point, but I think that'd be at the nginx level (I've not tried it before, so may be wrong) and I've got nginx and daphne in separate services (pods) so they scale individually.

That's why I was hoping for some kind of API from daphne to gather the number of connections a particular instance had, or something similar.

Similar to your thoughts there, I did find today that with the google auto scaler I can scale "based on any metric reported by a Kubernetes object in a cluster, such as the rate of client requests per second or I/O writes per second.

This can be useful if your application is prone to network bottlenecks, rather than CPU or memory."

So, quite tied in to GKE, but at least that's what I'm using.

[carltongibson]

A lot there sounds more platform specific than I can speak to.

There's nothing in Daphne per se. I'd start with an ASGI middleware at the first level to add some metrics.

Let me know what you come up with if you can.

[marksweb]

Thanks - I like the idea of digging around ASGI to get some metrics out. I've done a bit of that lately to get celery queue lengths.

I shall put this on the "in between jobs" jobs list. Hopefully I can get this google magic to look at the request rate for now and then build something once the football season is underway.