This changelog is intended for community Beat developers. It covers the major
breaking changes to the internal APIs in the official Beats and changes related
to developing a Beat like code generators or fields.yml
. Only the major
changes will be covered in this changelog that are expected to affect community
developers. Each breaking change added here should have an explanation on how
other Beats should be migrated.
Note: This changelog was only started after the 6.3 release.
-
Compare event by event in
testadata
framework to avoid sorting problems 13747
-
For "metricbeat style" generated custom beats, the mage target
GoTestIntegration
has changed toGoIntegTest
andGoTestUnit
has changed toGoUnitTest
. 13341
-
Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. 8543
-
Introduce beat.OutputChooses publisher mode. 12996
-
Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. 12996
-
Add support to close beat.Client via beat.CloseRef (a subset of context.Context). 13031
-
Add checks for types and formats used in fields definitions in
fields.yml
files. 13188 -
Makefile included in generator copies files from beats repository using
git archive
instead of cp. 13193
-
Add new option
IgnoreAllErrors
tolibbeat.common.schema
for skipping fields that failed while converting. 12089
-
Move Fields from package libbeat/common to libbeat/mapping. 11198
-
Metricset generator generates beta modules by default now. 10657
-
The
beat.Event
accessor methods now support@metadata
keys. 10761 -
Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. 10921
-
Support for Logger in the Metricset base instance. 11106
-
Filebeat modules can now use ingest pipelines in YAML format. 11209
-
Prometheus helper for metricbeat contains now
Namespace
field forprometheus.MetricsMappings
11424 -
Update Jinja2 version to 2.10.1. 11817
-
Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. 11777,elastic#12065
-
Update urllib3 version to 1.24.2 11930
-
Add libbeat/common/cleanup package. 12134
-
Only Load minimal template if no fields are provided. 12103
-
Add new option
IgnoreAllErrors
tolibbeat.common.schema
for skipping fields that failed while converting. 12089 -
Deprecate setup cmds for
template
andilm-policy
. Add new setup cmd forindex-management
. 12132
The list below covers the major changes between 7.0.0-rc2 and 7.0 only.
-
Added support for using PYTHON_EXE to control what Python interpreter is used by
make
andmage
. Example:export PYTHON_EXE=python2.7
. 11212
-
Introduce processing.Support to instance.Setting. This allows Beats to fully modify the event processing. 10801
-
Allow multiple object type configurations per field. 9772
-
Move agent metadata addition to a processor. 9952
-
Add (*common.Config).Has and (*common.Config).Remove. 10363
-
Introduce ILM and IndexManagement support to beat.Settings. 10347
-
Generating index pattern on demand instead of shipping them in the packages. 10478
The list below covers the major changes between 6.3.0 and 7.0.0-alpha2 only.
-
The beat.Pipeline is now passed to cfgfile.RunnerFactory. Beats using libbeat for module reloading or autodiscovery need to be adapted. 7017
-
Moving of TLS helper functions and structs from
output/tls
totlscommon
. 7054 -
Port fields.yml collector to Golang 6911
-
Dashboards under _meta/kibana are expected to be decoded. See elastic#7224 for a conversion script. 7265
-
Constructor
(github.com/elastic/beats/libbeat/output/codec/json).New
expects a newescapeHTML
parameter. 7445 -
Packaging has been refactored and updates are required. See the PR for migration details. 7388
-
make fields
has been modified to use Mage (https://magefile.org/) in an effort to make the building a Beat more cross-platform friendly (e.g. Windows). This requires that your Beat has a magefile.go with a fields target. TheFIELDS_FILE_PATH
make variable is no longer used because the value is specified in magefile.go. 7670 -
Outputs must implement String. 6404
-
Renamed
-beat-name
CLI option used inkibana_index_pattern.go
to-beat
for consistency with other scripts indev-tools/cmd
. 8615 -
Systemd unit file template used on Linux packaging now includes environment variables to ease flag overriding. One of them includes the
-e
flag, making beats log to stderr by default on systemd uses. 8942 -
Removed dashboards and index patterns generation for Kibana 5. 8927
-
Move generator packages of Filebeat from
scripts/generator
togenerator
. 9147
-
Libbeat provides a global registry for beats developer that allow to register and retrieve plugin. 7392
-
Added more options to control required and optional fields in schema.Apply(), error returned is a plain nil if no error happened 7335
-
Packaging on MacOS now produces a .dmg file containing an installer (.pkg) and uninstaller for the Beat. 7481
-
Added mage targets
goTestUnit
andgoTestIntegration
for executing 'go test'. This captures the log to a file, summarizes the result, produces a coverage profile (.cov), and produces an HTML coverage report. Seemage -h goTestUnit
. 7766 -
Beats packaging now build non-oss binaries from code located in the x-pack folder. 7783
-
New function
AddTagsWithKey
is added, socommon.MapStr
can be enriched with tags with an arbitrary key. 7991 -
Move filebeat/reader to libbeat/reader 8206
-
Libbeat provides a new function
cmd.GenRootCmdWithSettings
that should be preferred over deprecated functionscmd.GenRootCmd
,cmd.GenRootCmdWithRunFlags
, andcmd.GenRootCmdWithIndexPrefixWithRunFlags
. 7850 -
Set current year in generator templates. 8396
-
You can now override default settings of libbeat by using instance.Settings. 8449
-
Add
-space-id
option toexport_dashboards.go
script to support Kibana Spaces 7942 -
Add
-name
option toasset.go
script to explicitly name the asset rather than using its filename. 8693 -
Add
-out
option tokibana_index_pattern.go
to control the output dir to make it possible to write the generated output tobuild/kibana
instead of_meta/kibana.generated
(but the output dir remains unchanged at this point). 8615 -
Add
module_fields.go
for generatedfields.go
files for modules. 8615 -
Add
mage.GenerateModuleReferenceConfig
for generating reference config files that include configuration sections from the module directory. 8615 -
Add
mage.GenerateFieldsGo
for generating fields.go files. 8615 -
Add
mage.KibanaDashboards
for collecting Kibana dashboards and generating index patterns. 8615 -
Allow to disable config resolver using the
Settings.DisableConfigResolver
field when initializing libbeat. 8769 -
Add
mage.AddPlatforms
to allow to specify dependent platforms when building a beat. 8889 -
Add
cfgwarn.CheckRemoved6xSetting(s)
to display a warning for options removed in 7.0. 8909 -
Add docker image building to
mage.Package
. 8898 -
Simplified exporting of dashboards. 7730
-
Update Beats to use go 1.11.2 8746
-
Allow/Merge fields.yml overrides 9188
-
Filesets can now define multiple ingest pipelines, with the first one considered as the entry point pipeline. 8914
-
Add
group_measurements_by_instance
option to windows perfmon metricset. 8688 -
Bump ECS version to 1.10.0. 25734