Skip to main content
Version: 7.1

AWS VPC Flow Logs (S3)

Overview​

NetObserv Flow can collect AWS VPC Flow Logs which are stored in S3. All fields are supported from VPC Flow Log versions 2 thru 5.

When ElastiFlow processes logs from the specified S3 bucket, the input moves them to the elastiflow-processed folder. This behavior is essential to ensure that the collector can easily identify and exclude already processed logs on subsequent queries, thereby preventing the reprocessing of the same logs. The schema of the logs remains unchanged; only their location within the S3 bucket is modified.

EF_AWS_VPC_FLOW_LOG_S3_ENABLE​

  • Valid Values
    • true, false
  • Default
    • false

EF_AWS_VPC_FLOW_LOG_S3_BUCKET​

The S3 bucket from which to fetch AWS VPC Flow Logs.

  • Default
    • ''

EF_AWS_VPC_FLOW_LOG_S3_PREFIX​

  • Default
    • AWSLogs

EF_AWS_VPC_FLOW_LOG_S3_POOL_SIZE​

Specifies the number of concurrent workers to start. Increasing the number of workers will allow the collector to better handle a processing vpc flow logs in s3.

  • Default
    • number of license units

EF_AWS_VPC_FLOW_LOG_S3_TLS_ENABLE​

This setting is used to enable/disable TLS connections to AWS S3.

  • Valid Values
    • true, false
  • Default
    • false

EF_AWS_VPC_FLOW_LOG_S3_TLS_SKIP_VERIFICATION​

This setting is used to enable/disable TLS verification of the AWS S3 endpoint to which the input is attempting to connect.

  • Valid Values
    • true, false
  • Default
    • false

EF_AWS_VPC_FLOW_LOG_S3_TLS_CA_CERT_FILEPATH​

The path to the Certificate Authority (CA) certificate to use for verification of the AWS S3 endpoint to which the input is attempting to connect.

  • Default
    • ''

EF_AWS_VPC_FLOW_LOG_S3_TLS_MIN_VERSION​

This setting is used to set the TLS minimum version

  • Valid Values
    • 1.2, 1.3
  • Default
    • 1.2

EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_ENABLE​

This setting is used to declare an S3 bucket receives data from Amazon Firehose instead of directly from VPCs.

  • Valid Values
    • true, false
  • Default
    • false

EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_LOG_FORMAT​

This setting is used when the input is receiving data from Amazon Firehose. It specifies the format of the logs. Each key must be wrapped in a format that looks like ${key} and must be a valid key according to AWS log formats.

  • Default
    • ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}

AWS_REGION​

The AWS SDK compatible environment variable that specifies the AWS Region to send the request to.

AWS_ACCESS_KEY_ID​

Specifies an AWS access key associated with an IAM user or role.

AWS_SECRET_ACCESS_KEY​

Specifies the secret key associated with the access key. This is essentially the "password" for the access key.