AWS Flow Logs (S3)

NetObserv Flow can collect AWS VPC Flow Logs stored in S3.

NOTICE

If no S3 bucket is involved because Firehose is exporting over HTTP, you will need to use AWS Flow Logs (Firehose HTTP) settings instead. If, however, flow logs are being exported into an S3 bucket by any means (with or without AWS Firehose involved), then these are the right settings to use.

When NetObserv Flow processes flow logs from the specified S3 bucket, it defaults to looking for a folder called “AWSLogs”. Every time NetObserv processes flow logs, it moves them to another folder called “elastiflow-processed”. This behavior is essential to ensure that the collector can easily identify flow logs which are already processed vs new flow logs that need to be processed still. The schema of the logs remains unchanged; only their location within the S3 bucket is modified.

If Firehose is being used to export flow logs into S3, then there are additional settings required. This is because Firehose does not include the field names in the exported data, therefore NetObserv must know to process the exported flow logs slightly differently.

EF_AWS_VPC_FLOW_LOG_S3_ENABLE

• Valid Values

  • true, false

• Default

  • false

EF_AWS_VPC_FLOW_LOG_S3_BUCKET

The S3 bucket from which to fetch AWS VPC Flow Logs.

• Default

  • ''

EF_AWS_VPC_FLOW_LOG_S3_PREFIX

• Default

  • AWSLogs

EF_AWS_VPC_FLOW_LOG_S3_POOL_SIZE

Specifies the number of concurrent workers to start. Increasing the number of workers will allow the collector to better handle a processing vpc flow logs in s3.

• Default

  • number of license units

EF_AWS_VPC_FLOW_LOG_S3_TLS_ENABLE

This setting is used to enable/disable TLS connections to AWS S3.

• Valid Values

  • true, false

• Default

  • false

EF_AWS_VPC_FLOW_LOG_S3_TLS_SKIP_VERIFICATION

This setting is used to enable/disable TLS verification of the AWS S3 endpoint to which the input is attempting to connect.

• Valid Values

  • true, false

• Default

  • false

EF_AWS_VPC_FLOW_LOG_S3_TLS_CA_CERT_FILEPATH

The path to the Certificate Authority (CA) certificate to use for verification of the AWS S3 endpoint to which the input is attempting to connect.

• Default

  • ''

EF_AWS_VPC_FLOW_LOG_S3_TLS_MIN_VERSION

This setting is used to set the TLS minimum version

• Valid Values

  • 1.2, 1.3

• Default

  • 1.2

EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_ENABLE

This setting is used to declare an S3 bucket receives data from Amazon Firehose instead of directly from VPCs.

• Valid Values

  • true, false

• Default

  • false

EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_LOG_FORMAT

This setting is used when the input is receiving data from Amazon Firehose. It specifies the format of the logs. Each key must be wrapped in a format that looks like ${key} and must be a valid key according to AWS log formats.

You do not need to set this unless you are using Firehose to export flow logs to S3, and if you customized the exported fields to be anything other than the default v1 fields.

• Default

  • ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}

AWS_REGION

The AWS SDK compatible environment variable that specifies the AWS Region to send the request to.

AWS_ACCESS_KEY_ID

Specifies an AWS access key associated with an IAM user or role.

AWS_SECRET_ACCESS_KEY

Specifies the secret key associated with the access key. This is essentially the "password" for the access key.