Generic HTTP

The Generic HTTP output is currently a technology preview. The design and implementation are less mature than stable features and subject to change.

The Generic HTTP output can be used to send records to an HTTP endpoint such as the http_endpoint input of Elastic's Filebeat, or the http input of Elastic's Logstash.

The following is an example configuration for Elastic Filebeat:

filebeat.inputs:
  - type: http_endpoint
    enabled: true
    listen_address: 0.0.0.0
    listen_port: 8888
    processors:
      - convert:
          fields:
            - {from: "json", type: "string"}
          ignore_missing: true
          fail_on_error: false
      - decode_json_fields:
          fields: ["json"]
          process_array: true
          target: ""
          overwrite_keys: true
          add_error_key: false
      - drop_fields:
          fields: ["json"]
          ignore_missing: false

processors:
  - drop_fields:
      fields:
        - agent
        - ecs
        - host
        - input

The ElastiFlow Generic HTTP Output supports sending data using the ECS Schema. The global processor section in the above config is necessary to remove the fields automatically added by Filebeat. These would otherwise conflict with the fields ElastiFlow is emitting. This must be done in a global processor as (from the Filebeat source code) "Builtin fields can be modified using global processors, and fields only." This works because as of 6.0.0 ElastiFlow has transitioned to flattened field names. Dropping the Filebeat-emitted parent fields in a global processor does not affect the equivalent flattened field names.

The following is an example pipeline for Elastic Logstash:

input {
  http {
    host => "0.0.0.0"
    port => 8888
    ecs_compatibility => "disabled"
  }
}

filter {
  mutate {
    remove_field => ["host", "event", "headers"]
  }
}

EF_OUTPUT_GENERIC_HTTP_ENABLE

Specifies whether the Generic HTTP output is enabled.

• Valid Values

  • true, false

• Default

  • false

EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE

Specifies whether the data will be sent using Elastic Common Schema (ECS).

• Valid Values

  • true, false

• Default

  • false

EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE

The maximum time, in milliseconds, to wait for a batch of records to fill before being sent to the HTTP Endpoint.

• Default

  • 2000

EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES

The maximum size, in bytes, for a batch of records being sent to the HTTP Endpoint.

• Default

  • 8388608

EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE

Determines the timestamp source to be used to set the @timestamp field. Usually end would be the best setting. However, in the case of poorly behaving or misconfigured devices, collect may be the better option. For this reason the default value is collect as it best handles a variety of scenarios.

• Valid Values

  • start - Use the timestamp from flow.start.timestamp. The flow start time indicated in the flow.

  • end - Use the timestamp from flow.end.timestamp. The flow end time (or last reported time).

  • export - Use the timestamp from flow.export.timestamp. The time from the flow record header.

  • collect - Use the timestamp from flow.collect.timestamp. The time that the collector processed the flow record.

• Default

  • collect

EF_OUTPUT_GENERIC_HTTP_ADDRESSES

This setting specifies the HTTP servers to which the output should connect. It is a comma-separated list of HTTP servers, including port number.

Do NOT include http:// or https:// in the provided value. TLS communications is enabled/disabled using EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE.

• Default

  • 127.0.0.1:8888

EF_OUTPUT_GENERIC_HTTP_USERNAME

The username to use when connecting to the HTTP endpoint.

• Default

  • ``

EF_OUTPUT_GENERIC_HTTP_PASSWORD

The password to use when connecting to the HTTP endpoint.

• Default

  • ``

EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE

This setting is used to enable/disable TLS connections to the HTTP server.

• Valid Values

  • true, false

• Default

  • false

EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION

This setting is used to enable/disable TLS verification of the HTTP server to which the output is attempting to connect.

• Valid Values

  • true, false

• Default

  • false

EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH

The path to the Certificate Authority (CA) certificate to use for verification of the HTTP server to which the output is attempting to connect.

• Default

  • ''

EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS

This setting allows for a comma-separated list of fields that are to be removed from all records.

Fields are dropped after any output specific fields have been added and after any schema conversion. This means that you should use the field names as you see them in the user interface.

• Valid Values

  • any field names related to the enabled schema, comma-separated

• Example

  • flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num

• Default

  • ''

EF_OUTPUT_GENERIC_HTTP_ALLOWED_RECORD_TYPES

This setting allows for a comma-separated list of record types that the output will send will emit. If left empty, all types will be allowed by default.

• Valid Values

  • as_path_hop, flow_option, flow, ifa_hop, telemetry, metric, log

• Default

  • 'as_path_hop,flow_option,flow,ifa_hop,telemetry,metric,log'