Fluentd
Fluentd is an open-source data collector that receives logs and other events, turns them into structured records, and routes them to storage or analysis systems.
itCloud native tools and technologies | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic - Fluentd
Fluentd centers on Fluentd. Applications, operating systems, containers, and network services all produce logs. Those logs often use different formats and live on different machines.
Operate from the project's own resources and APIs. Learn how desired state becomes running state, which status conditions matter, and which dependencies (network, storage, identity, certificates) the control plane assumes.
Upgrades, backups, and credential rotation are part of the product, not optional aftercare. Version skew between clients and servers creates failures that look like application bugs. Pin versions and rehearse rollback.
Convenience features reduce boilerplate and widen blast radius. Enable them when you can observe and reverse the expanded surface. Defaults from quickstarts are starting points, not production policy.
Name owners for upgrades, credentials, and disaster recovery before traffic arrives. Unowned control-plane state becomes an outage with no clear pager. Prefer explicit version pins and tested rollback over floating tags that quietly change behavior between deploys.
Name owners for upgrades, credentials, and disaster recovery before traffic arrives. Unowned control-plane state becomes an outage with no clear pager. Prefer explicit version pins and tested rollback over floating tags that quietly change behavior between deploys.
Name owners for upgrades, credentials, and disaster recovery before traffic arrives. Unowned control-plane state becomes an outage with no clear pager. Prefer explicit version pins and tested rollback over floating tags that quietly change behavior between deploys.
Read the Intro for the mental model. Use the Cheatsheet when you need the resource map. Updates and Upstream track the Fluentd release line that changes these APIs.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.fluentd.org/
Supports
- Fluentd is an open-source data collector for a unified logging layer
- Fluentd unifies data collection and consumption
- Fluentd uses the Apache License version 2.0
- https://docs.fluentd.org/quickstart/life-of-a-fluentd-event
Supports
- Event structure contains tag, time, and record
- Inputs generate events from data sources
- Filters accept, reject, or change the event path
- Matches select outputs
- Labels define separate routing sections
- Buffered output stores events before destination writes
- https://docs.fluentd.org/configuration/config-file
Supports
- Source, match, filter, system, label, worker, and include directive purposes
- Tags act as routing keys
- Tag wildcard and expansion behavior
- Match rules are evaluated in configuration order
- Specific matches must precede broad matches
- A filter after a consuming match does not process those events
- Copy output is used to send events to multiple outputs
- https://docs.fluentd.org/input
Supports
- Input, parser, filter, output, formatter, storage, service discovery, buffer, and metrics plugin categories
- Input plugins retrieve events from external sources
- Core input plugin list
- https://docs.fluentd.org/parser
Supports
- Parser plugins convert input text into time and record fields
- Parsers are used by input and filter plugins
- https://docs.fluentd.org/filter/parser
Supports
- Parser filter parses a string field and updates the event record
- Invalid parsed records can follow the error route
- https://docs.fluentd.org/output
Supports
- Non-buffered, synchronous buffered, and asynchronous buffered output modes
- Buffered outputs use stage and queue areas
- Output plugins support different buffering modes
- Chunk keys can use time, tag, or record fields
- https://docs.fluentd.org/buffer
Supports
- Buffer consists of chunks
- Chunks move from stage to queue before transfer
- Retry wait grows exponentially by default
- Buffer plugins are selected according to system requirements
- https://docs.fluentd.org/configuration/buffer-section
Supports
- Buffer section belongs under a supported output match
- Core includes file and memory buffer plugins
- File buffer is more durable for general workloads
- Chunk keys group events by tag, time, or record field
- Flush, retry, queue-limit, and overflow settings shape outage behavior
- https://docs.fluentd.org/output/forward
Supports
- Forward output sends events to other Fluentd nodes
- Forward output supports load balancing and automatic failover
- Forward output supports at-most-once and at-least-once semantics
- Forward transport supports authentication and TLS settings
- https://docs.fluentd.org/input/forward
Supports
- Forward input receives event streams from Fluentd instances and client libraries
- Forward input uses TCP and supports TLS and authentication
- Forward protocol uses MessagePack internally
- https://docs.fluentd.org/deployment/failure-scenarios
Supports
- Memory-buffered data is lost when the Fluentd process stops
- File-buffered data persists across a normal process restart
- File buffers still fail on disk corruption, I/O failure, or full disk
- Destination outages cause retries until buffer limits are reached
- https://docs.fluentd.org/deployment/high-availability
Supports
- Forwarder and aggregator network roles
- Multiple aggregators provide active and standby destinations
- Forwarders and aggregators can buffer on disk
- Asynchronous delivery has loss and duplication tradeoffs
- https://docs.fluentd.org/deployment/plugin-management
Supports
- fluent-gem installs Fluentd plugins
- External plugins can introduce incompatible dependencies
- Gemfile isolation pins the plugin versions loaded by Fluentd
- Native-extension plugins may need reinstalling after a Ruby upgrade
- https://docs.fluentd.org/input/monitor_agent
Supports
- Core monitor-agent input exposes internal metrics through an HTTP API
- Stable plugin identifiers improve metric attribution
- Metrics include queue size, queued bytes, retries, errors, and slow flushes
- Debug output can expose sensitive configuration values
- https://docs.fluentd.org/monitoring-fluentd/overview
Supports
- Fluentd can expose internal metrics for external monitoring
- Process and port monitoring complement plugin metrics
- https://docs.fluentd.org/container-deployment/kubernetes
Supports
- A DaemonSet can run a Fluentd pod on each selected Kubernetes node
- Node-local placement enables container-log collection
- Log storage destination is a separate design decision
- https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit
Supports
- Fluent Bit is a lightweight telemetry agent in the Fluent ecosystem
- Fluent Bit can collect and forward data to Fluentd
- Fluentd provides a larger Ruby plugin ecosystem and aggregator role
- https://docs.fluentd.org/installation/install-fluent-package
Supports
- Official fluent-package distributions exist for supported operating systems
- Installation choices include packages and platform installers
- https://github.com/sindresorhus/awesome
Supports
- Required starting index used to discover a relevant awesome list
- https://github.com/awesome-foss/awesome-sysadmin
Supports
- Awesome Sysadmin is a curated open-source system-administration resource list
- Its Log Management section includes Fluentd, Loki, rsyslog, and GoAccess
- Loki, rsyslog, and GoAccess were selected as relevant non-duplicate ecosystem destinations
- https://grafana.com/docs/loki/latest/
Supports
- Loki is an open-source logging stack
- Loki indexes log-label metadata and stores compressed log chunks
- Documentation covers deployment modes, configuration, LogQL, and sending logs
- https://docs.rsyslog.com/doc/
Supports
- rsyslog documentation includes installation and first-configuration guidance
- Beginner material covers inputs, rulesets, actions, order, and remote forwarding
- Reference material covers queues and output modules
- https://goaccess.io/man
Supports
- GoAccess is a real-time web-log analyzer and interactive viewer
- GoAccess can render terminal, HTML, JSON, and CSV reports
- GoAccess parses web-server log formats and can read live input