AWS VPC Flow Logs (S3)
Overview
NetObserv Flow can collect AWS VPC Flow Logs which are stored in S3. All fields are supported from VPC Flow Log versions 2 thru 5.
When ElastiFlow processes logs from the specified S3 bucket, the input moves them to the elastiflow-processed
folder. This behavior is essential to ensure that the collector can easily identify and exclude already processed logs on subsequent queries, thereby preventing the reprocessing of the same logs. The schema of the logs remains unchanged; only their location within the S3 bucket is modified.
EF_AWS_VPC_FLOW_LOG_S3_ENABLE
- Valid Values
true
,false
- Default
false
EF_AWS_VPC_FLOW_LOG_S3_BUCKET
The S3 bucket from which to fetch AWS VPC Flow Logs.
- Default
''
EF_AWS_VPC_FLOW_LOG_S3_PREFIX
- Default
AWSLogs
EF_AWS_VPC_FLOW_LOG_S3_POOL_SIZE
Specifies the number of concurrent workers to start. Increasing the number of workers will allow the collector to better handle a processing vpc flow logs in s3.
- Default
number of license units
EF_AWS_VPC_FLOW_LOG_S3_TLS_ENABLE
This setting is used to enable/disable TLS connections to AWS S3.
- Valid Values
true
,false
- Default
false
EF_AWS_VPC_FLOW_LOG_S3_TLS_SKIP_VERIFICATION
This setting is used to enable/disable TLS verification of the AWS S3 endpoint to which the input is attempting to connect.
- Valid Values
true
,false
- Default
false
EF_AWS_VPC_FLOW_LOG_S3_TLS_CA_CERT_FILEPATH
The path to the Certificate Authority (CA) certificate to use for verification of the AWS S3 endpoint to which the input is attempting to connect.
- Default
''
EF_AWS_VPC_FLOW_LOG_S3_TLS_MIN_VERSION
This setting is used to set the TLS minimum version
- Valid Values
1.2
,1.3
- Default
1.2
EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_ENABLE
This setting is used to declare an S3 bucket receives data from Amazon Firehose instead of directly from VPCs.
- Valid Values
true
,false
- Default
false
EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_LOG_FORMAT
This setting is used when the input is receiving data from Amazon Firehose. It specifies the format of the logs. Each key must be wrapped in a format that looks like ${key}
and must be a valid key according to AWS log formats.
- Default
${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
AWS_REGION
The AWS SDK compatible environment variable that specifies the AWS Region to send the request to.
AWS_ACCESS_KEY_ID
Specifies an AWS access key associated with an IAM user or role.
AWS_SECRET_ACCESS_KEY
Specifies the secret key associated with the access key. This is essentially the "password" for the access key.