Skip to content

How to troubleshoot and drill down FluentD configuration issues

Notifications You must be signed in to change notification settings

denniskniep/fluentd-troubleshooting

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

FluentD Troubleshooting

How to troubleshoot and drill down FluentD configuration issues

  1. Place your configs into folder fluentd-config
  2. Add prints to debug the current event at specific points with:
    <filter in.**>
        @type stdout
        @id debug1
    </filter>
    
  3. Add prints of customized messages (see here) at specific points by :
    <filter in.**>
        @type stdout
        @id debug1-with-msg
        <format>
            @type sprintf
            sprintf_format "Debugging the tag:${tag}\n"
        </format>
    </filter>
    
  4. Optional: Enable running with verbose
  5. Start by running sudo docker-compose up --build

Sending Logs to FluentD

FluentCat

Install FluentCat by installing fluentD

or

use FluentCat by execing into the running fluentd container with sudo docker-compose exec fluentd bash

Send log:

echo '{"event":"message"}' | fluent-cat fromfluentcat

Send unparsed FluentBit log:

echo '{"log":"{\"event\":\"message\"}"}' | fluent-cat fromfluentcat

FluentBit

Append to file (watched by fluentbit):

echo '{"event":"message"}' >> ./fluentbit/input-fluentbit.json

Examples

No Problem

echo '{"@timestamp":"2019-02-04T23:09:52.884742296+0100","event":"message"}' >> ./fluentbit/input-fluentbit.json

sudo docker-compose exec fluentd bash

echo '{"log":"{\"@timestamp\":\"2019-02-04T23:09:52.884742296+0100\",\"event\":\"message\", \"foo\":\"bar\"}"}' | fluent-cat fromfluentcat

Invalid date or strptime format

echo '{"@timestamp":"2019-02-04T23:09:52.8847422960100","event":"message"}' >> ./fluentbit/input-fluentbit.json

sudo docker-compose exec fluentd bash

echo '{"log":"{\"@timestamp\":\"2019-02-04T23:09:52.8847422960100\",\"event\":\"message\", \"foo\":\"bar\"}"}' | fluent-cat fromfluentcat

Other Resources

Debug FluentD with VisualStudio Code

  • Follow instructions here
  • Open in VisualStudioCode
  • Install ruby Extension
  • Add debug config by creating file .vscode/launch.json
    {
        "configurations": [
            {
                "name": "Debug Local File",
                "type": "Ruby",
                "request": "launch",
                "cwd": "${workspaceRoot}",
                "program": "${workspaceRoot}/lib/fluent/command/fluentd.rb",
                "args": [
                    "-c", "myFluent.conf"
                    //, "-vv"
                ],
                "env": { 
                    "RUBYLIB":"lib"
                }
            }
        ]
    }
    
  • Know limitation: Stopping VSCode Debugging does not stop fluentd process. Run kill $(ps aux | grep -v grep | grep fluentd.rb | awk '{print $2}') to stop it