Fluentd
Fluentd is an open-source data collector that receives logs and other events, turns them into structured records, and routes them to storage or analysis systems.
itCloud native tools and technologies | OpenSkills.info
Intro
Fluentd
Applications, operating systems, containers, and network services all produce logs. Those logs often use different formats and live on different machines. Fluentd provides a collection and routing layer between these producers and the systems that store or analyze their data.
Fluentd is an open-source data collector for a unified logging layer. It receives events through input plugins, converts them into a common event structure, and sends them through filters and outputs. This separation lets producers emit data without knowing every destination's protocol or schema.
By the end of this course, you will be able to explain Fluentd's event model, read its routing configuration, and judge its delivery tradeoffs. You will also know where plugins, buffers, labels, and monitoring fit in a production design.
The event is the unit of work
Fluentd processes a stream of events. Each event has three parts:
- A tag identifies the event stream and drives routing.
- A time records when the event occurred.
- A record holds the event data as a structured object.
An input plugin creates events from a source such as a file, an HTTP endpoint, or another Fluentd process. Fluentd then evaluates filters and match rules against each event's tag.
The record is commonly represented as structured data rather than an unparsed line. Parsers can convert text into fields. Filters can select, reject, or change records. Formatters control how an output serializes them.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.fluentd.org/
Supports
- Fluentd is an open-source data collector for a unified logging layer
- Fluentd unifies data collection and consumption
- Fluentd uses the Apache License version 2.0
- https://docs.fluentd.org/quickstart/life-of-a-fluentd-event
Supports
- Event structure contains tag, time, and record
- Inputs generate events from data sources
- Filters accept, reject, or change the event path
- Matches select outputs
- Labels define separate routing sections
- Buffered output stores events before destination writes
- https://docs.fluentd.org/configuration/config-file
Supports
- Source, match, filter, system, label, worker, and include directive purposes
- Tags act as routing keys
- Tag wildcard and expansion behavior
- Match rules are evaluated in configuration order
- Specific matches must precede broad matches
- A filter after a consuming match does not process those events
- Copy output is used to send events to multiple outputs
- https://docs.fluentd.org/input
Supports
- Input, parser, filter, output, formatter, storage, service discovery, buffer, and metrics plugin categories
- Input plugins retrieve events from external sources
- Core input plugin list
- https://docs.fluentd.org/parser
Supports
- Parser plugins convert input text into time and record fields
- Parsers are used by input and filter plugins
- https://docs.fluentd.org/filter/parser
Supports
- Parser filter parses a string field and updates the event record
- Invalid parsed records can follow the error route
- https://docs.fluentd.org/output
Supports
- Non-buffered, synchronous buffered, and asynchronous buffered output modes
- Buffered outputs use stage and queue areas
- Output plugins support different buffering modes
- Chunk keys can use time, tag, or record fields
- https://docs.fluentd.org/buffer
Supports
- Buffer consists of chunks
- Chunks move from stage to queue before transfer
- Retry wait grows exponentially by default
- Buffer plugins are selected according to system requirements
- https://docs.fluentd.org/configuration/buffer-section
Supports
- Buffer section belongs under a supported output match
- Core includes file and memory buffer plugins
- File buffer is more durable for general workloads
- Chunk keys group events by tag, time, or record field
- Flush, retry, queue-limit, and overflow settings shape outage behavior
- https://docs.fluentd.org/output/forward
Supports
- Forward output sends events to other Fluentd nodes
- Forward output supports load balancing and automatic failover
- Forward output supports at-most-once and at-least-once semantics
- Forward transport supports authentication and TLS settings
- https://docs.fluentd.org/input/forward
Supports
- Forward input receives event streams from Fluentd instances and client libraries
- Forward input uses TCP and supports TLS and authentication
- Forward protocol uses MessagePack internally
- https://docs.fluentd.org/deployment/failure-scenarios
Supports
- Memory-buffered data is lost when the Fluentd process stops
- File-buffered data persists across a normal process restart
- File buffers still fail on disk corruption, I/O failure, or full disk
- Destination outages cause retries until buffer limits are reached
- https://docs.fluentd.org/deployment/high-availability
Supports
- Forwarder and aggregator network roles
- Multiple aggregators provide active and standby destinations
- Forwarders and aggregators can buffer on disk
- Asynchronous delivery has loss and duplication tradeoffs
- https://docs.fluentd.org/deployment/plugin-management
Supports
- fluent-gem installs Fluentd plugins
- External plugins can introduce incompatible dependencies
- Gemfile isolation pins the plugin versions loaded by Fluentd
- Native-extension plugins may need reinstalling after a Ruby upgrade
- https://docs.fluentd.org/input/monitor_agent
Supports
- Core monitor-agent input exposes internal metrics through an HTTP API
- Stable plugin identifiers improve metric attribution
- Metrics include queue size, queued bytes, retries, errors, and slow flushes
- Debug output can expose sensitive configuration values
- https://docs.fluentd.org/monitoring-fluentd/overview
Supports
- Fluentd can expose internal metrics for external monitoring
- Process and port monitoring complement plugin metrics
- https://docs.fluentd.org/container-deployment/kubernetes
Supports
- A DaemonSet can run a Fluentd pod on each selected Kubernetes node
- Node-local placement enables container-log collection
- Log storage destination is a separate design decision
- https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit
Supports
- Fluent Bit is a lightweight telemetry agent in the Fluent ecosystem
- Fluent Bit can collect and forward data to Fluentd
- Fluentd provides a larger Ruby plugin ecosystem and aggregator role
- https://docs.fluentd.org/installation/install-fluent-package
Supports
- Official fluent-package distributions exist for supported operating systems
- Installation choices include packages and platform installers
- https://github.com/sindresorhus/awesome
Supports
- Required starting index used to discover a relevant awesome list
- https://github.com/awesome-foss/awesome-sysadmin
Supports
- Awesome Sysadmin is a curated open-source system-administration resource list
- Its Log Management section includes Fluentd, Loki, rsyslog, and GoAccess
- Loki, rsyslog, and GoAccess were selected as relevant non-duplicate ecosystem destinations
- https://grafana.com/docs/loki/latest/
Supports
- Loki is an open-source logging stack
- Loki indexes log-label metadata and stores compressed log chunks
- Documentation covers deployment modes, configuration, LogQL, and sending logs
- https://docs.rsyslog.com/doc/
Supports
- rsyslog documentation includes installation and first-configuration guidance
- Beginner material covers inputs, rulesets, actions, order, and remote forwarding
- Reference material covers queues and output modules
- https://goaccess.io/man
Supports
- GoAccess is a real-time web-log analyzer and interactive viewer
- GoAccess can render terminal, HTML, JSON, and CSV reports
- GoAccess parses web-server log formats and can read live input