KeyError: 'rgw.main' after running for a long time

[plutoniumjake]

hello,

I am attempting to use your exporter on our ceph cluster. There are currently over 102000 buckets in the cluster. When I run the script in screen I get 200 returns, but then after a long period of time the script just errors out. Here is the output:

time python radosgw_usage_exporter/radosgw_usage_exporter.py -H http://objects.liquidweb.services -a ${key} -s ${secret}

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): objects.liquidweb.services:80
DEBUG:urllib3.connectionpool:http://objects.liquidweb.services:80 "GET /admin/usage/?format=json&show-summary=False HTTP/1.1" 200 1573666140
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): objects.liquidweb.services:80
DEBUG:urllib3.connectionpool:http://objects.liquidweb.services:80 "GET /admin/bucket/?format=json&stats=True HTTP/1.1" 200 120606221
Traceback (most recent call last):
File "radosgw_usage_exporter/radosgw_usage_exporter.py", line 324, in
main()
File "radosgw_usage_exporter/radosgw_usage_exporter.py", line 313, in main
args.host, args.admin_entry, args.access_key, args.secret_key))
File "/usr/lib/python2.7/site-packages/prometheus_client/core.py", line 50, in register
names = self._get_names(collector)
File "/usr/lib/python2.7/site-packages/prometheus_client/core.py", line 86, in _get_names
for metric in desc_func():
File "radosgw_usage_exporter/radosgw_usage_exporter.py", line 70, in collect
self._get_bucket_usage(bucket)
File "radosgw_usage_exporter/radosgw_usage_exporter.py", line 231, in _get_bucket_usage
if 'size_actual' in bucket['usage']['rgw.main']:
KeyError: 'rgw.main'

real 695m36.585s
user 430m55.932s
sys 2m38.033s

I think if I could get this working on our cluster it will really help us gain insight into trends in usage. Any assistance you can provide would be most appreciated.

[blemmenes]

Hello,

Thank you for the issue report. None of my clusters have that many buckets, I'll have to generate random buckets in test cluster to try to recreate this. I will attempt to do that this week.

Thanks,
Berant

[k0ste]

@plutoniumjake can you start exporter with env DEBUG=1? Then you will see the element where exporter stucks.