Generic HTTP
Overview
The Generic HTTP output is currently a technology preview. The design and implementation are less mature than stable features and subject to change.
The Generic HTTP output can be used to send records to an HTTP endpoint such as the http_endpoint
input of Elastic's Filebeat, or the http
input of Elastic's Logstash.
The following is a example configuration for Elastic Filebeat:
filebeat.inputs:
- type: http_endpoint
enabled: true
listen_address: 0.0.0.0
listen_port: 8888
processors:
- convert:
fields:
- {from: "json", type: "string"}
ignore_missing: true
fail_on_error: false
- decode_json_fields:
fields: ["json"]
process_array: true
target: ""
overwrite_keys: true
add_error_key: false
- drop_fields:
fields: ["json"]
ignore_missing: false
processors:
- drop_fields:
fields:
- agent
- ecs
- host
- input
The ElastiFlow Generic HTTP Output supports sending data using the ECS Schema. The global processor section in the above config is necessary to remove the fields automatically added by Filebeat. These would otherwise conflict with the fields ElastiFlow is emitting. This must be done in a global processor as (from the Filebeat source code) "Builtin fields can be modified using global processors
, and fields
only." This works because as of 6.0.0
ElastiFlow has transitioned to flattened field names. Dropping the Filebeat-emitted parent fields in a global processor does not affect the equivalent flattened field names.
The following is a example pipeline for Elastic Logstash:
input {
http {
host => "0.0.0.0"
port => 8888
ecs_compatibility => "disabled"
}
}
filter {
mutate {
remove_field => ["host", "event", "headers"]
}
}
EF_OUTPUT_GENERIC_HTTP_ENABLE
Specifies whether the Generic HTTP output is enabled.
- Valid Values
true
,false
- Default
false
EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE
Specifies whether the data will be sent using Elastic Common Schema (ECS).
- Valid Values
true
,false
- Default
false
EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE
The maximum time, in milliseconds, to wait for a batch of records to fill before being sent to the HTTP Endpoint.
- Default
2000
EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES
The maximum size, in bytes, for a batch of records being sent to the HTTP Endpoint.
- Default
8388608
EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE
Determines the timestamp source to be used to set the @timestamp
field. Usually end
would be the best setting. However, in the case of poorly behaving or misconfigured devices, collect
may be the better option. For this reason the default value is collect
as it best handles a variety of scenarios.
- Valid Values
start
- Use the timestamp fromflow.start.timestamp
. The flow start time indicated in the flow.end
- Use the timestamp fromflow.end.timestamp
. The flow end time (or last reported time).export
- Use the timestamp fromflow.export.timestamp
. The time from the flow record header.collect
- Use the timestamp fromflow.collect.timestamp
. The time that the collector processed the flow record.
- Default
collect
EF_OUTPUT_GENERIC_HTTP_ADDRESSES
This setting specifies the HTTP servers to which the output should connect. It is a comma-separated list of HTTP servers, including port number.
Do NOT include http://
or https://
in the provided value. TLS communications is enabled/disabled using EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE
.
- Default
127.0.0.1:8888
EF_OUTPUT_GENERIC_HTTP_USERNAME
The username to use when connecting to the HTTP endpoint.
- Default
- ``
EF_OUTPUT_GENERIC_HTTP_PASSWORD
The password to use when connecting to the HTTP endpoint.
- Default
- ``
EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE
This setting is used to enable/disable TLS connections to the HTTP server.
- Valid Values
true
,false
- Default
false
EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION
This setting is used to enable/disable TLS verification of the HTTP server to which the output is attempting to connect.
- Valid Values
true
,false
- Default
false
EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH
The path to the Certificate Authority (CA) certificate to use for verification of the HTTP server to which the output is attempting to connect.
- Default
''
EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS
This setting allows for a comma-separated list of fields that are to be removed from all records.
Fields are dropped after any output specific fields have been added and after any schema conversion. This means that you should use the field names as you see them in the user interface.
- Valid Values
- any field names related to the enabled schema, comma-separated
- Example
flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
- Default
''