ElastiFlow Flow Enrichment How To

NetIntel App Identification (NetObserv Flow)

NetIntel App identification identifies SaaS applications using ElastiFlow’s own database.

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml if using ElastiFlow in Docker.

1. Ensure you have a Premium, Trial Edition, or higher account ID and ElastiFlow flow license key. You can request a trial from here.

2. Add your account ID and license key to your /etc/elastiflow/flowcoll.yml or docker compose yml.

EF_LICENSE_ACCEPTED: 'true'
EF_ACCOUNT_ID: '#########################'
EF_FLOW_LICENSE_KEY: '##################################################’

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml: Note: NetIntel is enabled by default. EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE: 'true'

NetIntel Threat Detection (NetObserv Flow)

NetIntel Threat Detection can add threat information to any flow records containing IP addresses that NetIntel cloud service has detected as being malicious.

Instructions:

Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml. —

1. Ensure you have a Premium or Trial Edition or higher account ID and ElastiFlow flow license key

2. Add your account ID and license key to your /etc/elastiflow/flowcoll.yml or docker compose yml.

EF_LICENSE_ACCEPTED: 'true'
EF_ACCOUNT_ID: '#########################'
EF_FLOW_LICENSE_KEY: '##################################################’

1. Ensure that the following key / value pair is present. Note: NetIntel is enabled by default. EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE: 'true'

NetIntel Threat Detection with Mitre Mapping + Scoring (NetObserv Flow)

Instructions:

Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml. —

1. Ensure you have a Premium license account ID and ElastiFlow flow license key

2. Add your account ID and license key to your /etc/elastiflow/flowcoll.yml or docker compose yml.

EF_LICENSE_ACCEPTED: 'true'
EF_ACCOUNT_ID: '#########################'
EF_FLOW_LICENSE_KEY: '##################################################’

1. Ensure that the following key / value pair is present. Note: NetIntel is enabled by default.

EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE: 'true'

Hostnames (NetObserv Flow)

This section describes how you can enable the enrichment of flow records with the hostnames of IP addresses in the flow records.

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

• To enable IP address enrichment with DNS:

  EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE: "true"

• Specify a DNS server for DNS lookups

  EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_IP

• Set DNS server timeouts

  EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_TIMEOUT

• Exclude / include IPs from DNS lookups

  EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_PATH: '/etc/elastiflow/hostname/incl_excl.yml'
  EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_REFRESH_RATE: [Specify refresh rate here]

• Specify your own IP-to-DNS hostname mappings

  EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_PATH: '/etc/elastiflow/hostname/user_defined.yml'

  Change refresh rate EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_REFRESH_RATE

• Enable / disable resolving public / private IP spaces

  EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PRIVATE
  EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PUBLIC

AS Name, AS Numbers (ASNs), and Geolocation

Enrich flow records with with AS name, ASNs, and geolocations

Instructions:

Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml. —

1. Create a free account here: https://www.maxmind.com/en/geolite2/signup

2. After verifying your email address and signing in, click “My account”, and then click “Download databases”

3. Download Maxmind Geolite databases to your ElastiFlow host and store them here:

   /etc/elastiflow/maxmind/GeoLite2-ASN.mmdb

   /etc/elastiflow/maxmind/GeoLite2-City.mmdb

   Tip: You easily download and copy the database files you can use the following command snippets on your ElastiFlow server. Be sure to replace “YOUR_MAXMIND_LICENSE_KEY” with your MaxMind license key. Here is how you get your license key: After signing in, click “my account”, then my license keys. Then generate a new license key. Note the license key for use below.

sudo wget -O ./GeoLite2-ASN.tar.gz "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=YOUR_MAXMIND_LICENSE_KEY&suffix=tar.gz"
sudo wget -O ./GeoLite2-City.tar.gz  "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOUR_MAXMIND_LICENSE_KEY&suffix=tar.gz"
sudo tar -xvzf GeoLite2-ASN.tar.gz --strip-components 1 -C /etc/elastiflow/maxmind/
sudo tar -xvzf GeoLite2-City.tar.gz  --strip-components 1 -C /etc/elastiflow/maxmind/

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_ENABLE: "true"
   EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_ENABLE: "true"

Geolocation of private / RFC1918 IP space (NetObserv Flow)

Instructions:

Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml. —

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: "true"
   EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: "/etc/elastiflow/metadata/ipaddrs.yml"

2. Edit /etc/elastiflow/metadata/ipaddrs.yml, adding the enrichment information you’d like to add to flow records.

Example ipaddrs.yml content:

192.168.0.0/16:
  metadata:
    .geo.loc.coord: 48.167106,11.486918
    .geo.city.name: Munich
    .geo.country.code: DE
    .geo.country.name: Germany
    .geo.tz.name: Europe/Berlin

Application Identities (NetObserv Flow)

Note: ElastiFlow fully supports application identification arriving in flow records from your infrastructure.

Option 1: NetIntel AppID - Flow collector automatically identifies SaaS applications

Please see this.

Option 2: Define custom applications and servers

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_APP_IPPORT_ENABLE: "true"
   EF_PROCESSOR_ENRICH_APP_IPPORT_PATH: "/etc/elastiflow/app/ipport.yml"

2. Edit /etc/elastiflow/app/ipport.yml, adding the enrichment information you’d like to add to flow records.

   Example:

192.168.2.22-192.168.2.23:
  123:
    name: "NTP Infoblox"
    category: "Core Services"
    subcategory: "NTP"
  67:
    name: "DHCP Infoblox"
    category: "Core Services"
    subcategory: "DHCP"
  53:
    name: "DNS Infoblox"
    category: "Core Services"
    subcategory: "DNS"
  443:
    name: "Infoblox DDI"
    category: "Core Services"
    subcategory: "DDI"
192.168.2.251-192.168.2.252:
  443:
    name: "VMware ESXi"
    category: "Hypervisor"
    subcategory: "VMware"
    metadata:
      .location: "roundhill-base-esxi"

Option 3: Enable app identifications when devices do not send option records.

This is required for some devices from the following vendors that are sending app IDs but not send option records that provide a mapping of app names to app IDs: Cisco, Fortinet, Velocloud, Versa, and Viptela

Instructions:

—

Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_APP_ID_ENABLE: "true"
   EF_PROCESSOR_ENRICH_APP_ID_PATH: '/etc/elastiflow/app/appid.yml'

2. Edit /etc/elastiflow/app/appid.yml, adding the IPs and corresponding vendors of your devices.

User-defined Metadata (NetObserv Flow)

You can add virtually any kind of textual data to your flow records easily.

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: "true"
   EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: '/etc/elastiflow/metadata/ipaddrs.yml'

2. Add your enrichment information to /etc/elastiflow/metadata/ipaddrs.yml.

When enriching IPs and IP address ranges, precede the field you are enriching with a dot if you would like enrichment to be added to each side of the conversation. For example: “.src.sec.zone.name” will enrich both flow.src.sec.zone.name and flow.dst.sec.zone.name. If you don't precede src.sec.zone.name with a dot, the metadata gets added to the flow record at the top level.

The following are the fields that will have enrichment applied to them: "flow.src", "flow.dst", "flow.client", "flow.server", "flow.next_hop", "bgp.next_hop", "tunnel.src", "tunnel.dst", "encap.src", "encap.dst", "host", "system", “host.ip”

Network interface Names (NetObserv Flow)

Add user defined metadata to network interfaces.

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

1. Ensure that the following key / value pairs are present in /etc/elastiflow/flowcoll.yml or docker compose yml:

   EF_PROCESSOR_ENRICH_NETIF_METADATA_ENABLE: "true"
   EF_PROCESSOR_ENRICH_NETIF_METADATA_USERDEF_PATH: '/etc/elastiflow/metadata/netifs.yml'

2. Add your enrichment information to /etc/elastiflow/metadata/netifs.yml.

Network Interface Names and Descriptions from SNMP (NetObserv Flow)

Enrich flow records with proper interface names by polling SNMP

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/flowcoll.yml or docker compose yml.

Option 1: If you have one SNMP configuration for all your devices sending flow data to ElastiFlow, configure the following key / value pairs where appropriate:

EF_PROCESSOR_ENRICH_NETIF_SNMP_COMMUNITIES: public
EF_PROCESSOR_ENRICH_NETIF_SNMP_ENABLE: "false"
EF_PROCESSOR_ENRICH_NETIF_SNMP_PORT: 161
EF_PROCESSOR_ENRICH_NETIF_SNMP_RETRIES: 1
EF_PROCESSOR_ENRICH_NETIF_SNMP_TIMEOUT: 2
EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_AUTHENTICATION_PASSPHRASE: ""
EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_AUTHENTICATION_PROTOCOL: noauth
EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_PRIVACY_PASSPHRASE: ""
EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_PRIVACY_PROTOCOL: nopriv
EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_USERNAME: ""
EF_PROCESSOR_ENRICH_NETIF_SNMP_VERSION: 2

Option 2: If you have more than one SNMP configuration in your network, OR you wish to poll SNMP on a different interface than is sending flow data, configure the following key / value pairs where appropriate:

EF_PROCESSOR_ENRICH_NETIF_SNMP_ACCESS_ENABLE: "true"
EF_PROCESSOR_ENRICH_NETIF_SNMP_ACCESS_PATH: /etc/elastiflow/settings/snmp_access.yml
EF_PROCESSOR_ENRICH_NETIF_SNMP_ACCESS_REFRESH_RATE: 15

User-defined Metadata (NetObserv SNMP)

https://docs.elastiflow.com/docs/snmpcoll/enrich_ip_udm#user-defined-metadata-enrichment

Hostnames (NetObserv SNMP)

This section describes how you can enable the enrichment of flow records with the hostnames of IP addresses in the flow records.

Instructions: — Any keys that begin with “EF_” are configured in /etc/elastiflow/snmpcoll.yml or docker compose yml.

• Enable hostname enrichment

  EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE: "true"

• Specify a DNS server for DNS lookups

  • EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_IP

• Set DNS server timeouts

  • EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_TIMEOUT

• Exclude / include IPs from DNS lookups

  • "EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_PATH: '/etc/elastiflow/hostname/incl_excl.yml'

  • Change refresh rate for include / exclude definition file EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_REFRESH_RATE

• Specify your own IP-to-DNS hostname mappings

  • "EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_PATH: '/etc/elastiflow/hostname/user_defined.yml

  • Change refresh rate EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_REFRESH_RATE

• Enable / disable resolving public / private IP spaces

  • EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PRIVATE

  • EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PUBLIC

SNMP Troubleshooting