rewrite_tag_filter sets message key to null with this installation

[r1ckr]

Hi guys, we are trying to use the rewrite_tag_filter with this docker image and after using it, it is setting the message key to null.

This is the bit of extra config that we are doing:

 # Splitting users logs
   <match kubernetes.var.log.containers.users**>
     @type rewrite_tag_filter
     <rule>
       key message
       pattern /access_log/
       tag access.${tag}
     </rule>
     <rule>
       key message
       pattern /.+/
       tag app.${tag}
     </rule>
   </match>
  
   # These ones are to see the output before sending the message to logzio
   <match app.kubernetes.var.log.containers.users**>
     @type stdout
   </match>

   <match access.kubernetes.var.log.containers.users**>
     @type stdout
   </match>

This produces the following output:

2021-03-11 11:20:47.710023717 +0000 access.kubernetes.var.log.containers.users-9d589d58-mnrqf_kube-namespace_users-452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a.log: {"stream":"stdout","type":"k8s","message":null,"docker":{"container_id":"452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a"},"kubernetes":{"container_name":"users","namespace_name":"kube-namespace","pod_name":"users-5785b46454-mnrqf","pod_id":"2aa0e7ce-b45c-482a-9d58-1c4cf906ebf1","host":"aks-secondary-a228a228-vmss000002","labels":{"app":"users","pod-template-hash":"5785b46454"},"master_url":"https://10.0.0.1:443/api","namespace_id":"9d9cd9bf-869a-4546-a228-71bb773523b2"}}
2021-03-11 11:20:49.359273955 +0000 access.kubernetes.var.log.containers.users-9d589d58-mnrqf_kube-namespace_users-452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a.log: {"stream":"stdout","type":"k8s","message":null,"docker":{"container_id":"452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a"},"kubernetes":{"container_name":"users","namespace_name":"kube-namespace","pod_name":"users-5785b46454-mnrqf","pod_id":"2aa0e7ce-b45c-482a-9d58-1c4cf906ebf1","host":"aks-secondary-a228a228-vmss000002","labels":{"app":"users","pod-template-hash":"5785b46454"},"master_url":"https://10.0.0.1:443/api","namespace_id":"9d9cd9bf-869a-4546-a228-71bb773523b2"}}

As you can see it rewrites the tag without issues, but the message key is then set to null.

If we isolate this config, it works OK and doesn't set to null the message key, the issue is with this image.

[mirii1994]

Hi @r1ckr , can you please specify the image name & version that's causing you this issue?

[r1ckr]

Hi @mirii1994, yeah, we are using logzio/logzio-k8s:1.1.6

[mirii1994]

We moved a couple of months ago to a new image: logzio/logzio-fluentd:1.0.0 and daemonset, which allows configuring fluentd from configmaps and not from the image itself.
Can you please try our latest daemonset & configmaps (you can configure the configmaps with your configuration), and let me know if this solves your issue? (Instructions on how to use the new daemonset/configmaps/image are in this repo and in our docs).

[r1ckr]

Hey @mirii1994, just updated to the latest image, still the same issue. Let me know if you need more info.

[mirii1994]

Hi @r1ckr,
Seems like there is a collision between the rewrite_tags plugin & the record_modifier plugin. We will open an issue on the plugin page.