Auto-Discovery of Devices

Device auto-discovery is currently a technology preview. The design and implementation are less mature than stable features and subject to change.

Overview

ElastiFlow can automatically discover SNMP-enabled devices. To do this, you will install and use a command-line tool called netobserv.

You will need to have NetObserv SNMP 7.12.0 (or later) already installed. That will have the latest version of the SNMP definition files. If upgrading to 7.12.0, make sure you follow the manual upgrade steps for SNMP definition files.

You can install netobserv CLI tool via our deb or rpm packages.

Assuming you use default settings for NetObserv SNMP, here's how you run auto-discovery:

1. Define a "networks" input file at /etc/elastiflow/snmp/autodiscover/networks.yml, and specify the network ranges you want to scan.

   1. You can do this by copying the example file found at /etc/elastiflow/snmp/autodiscover/networks.yml.EXAMPLE and then editing it.

   2. Details are found below on what the file format looks like.

2. Run /usr/share/elastiflow/bin/netobserv autodiscover snmp.

   1. There are also additional flags to customize behavior if you want.

3. Trigger the apply-definitions API to reload the device YAML config for NetObserv SNMP.

Known Issues

Routers can have indirect duplicates sometimes. If a router does not have a non-local loopback interface, as of 7.12.0 release, the device YAML output will likely have all separate addresses of that router listed as separate device entries.

How Device Auto-Discovery Works

Device auto-discovery will:

1. Receive a list of IP ranges you want to scan for devices.

2. Ping sweep all addresses in the ranges you provided to find what devices exist.

3. Use SNMP to get the sysOID value (this specifies what device type it is).

4. Figure out what device_group name NetObserv associates with that sysOID. In other words, NetObserv ships with prebuilt lists of OIDs for various device types, and this step will find that prebuilt list for the given device type.

5. Output YAML file(s) that NetObserv SNMP can read to know what devices to poll. You have to tell NetObserv SNMP to reload config.

Concerning routers: auto-discovery will query every router for its administrative loopback IP address and use that IP address instead. Whichever loopback interface appears first in the list will be the loopback IP used in auto-discovery.

You can safely rerun auto-discovery as many times as you like. If a defined 'group' of network blocks was already scanned, it will not re-scan them. See the "Output" section for more details, including how to force a re-scan.

Devices found in earlier network groups will be excluded from subsequent groups to avoid duplicate definitions.

The command will create the output directory if it doesn't exist. Default location is /etc/elastiflow/snmp/devices

If the auto-discovery process is taking too long or too much bandwidth, you can experiment with the --concurrency flag to balance speed and network load.

Auto-discovery will ignore these 'local' address ranges:

• 127.0.0.0/8

• 169.254.0.0/16

• fe80::1/10

• ::1/128

Installing `netobserv` CLI Tool

You can download and install the netobserv CLI tool on Linux machines via our deb or rpm packages.

deb:

wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv_7.24.0_linux_amd64.deb
sudo apt install ./netobserv_7.24.0_linux_amd64.deb

rpm:

wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv-7.24.0-1.x86_64.rpm
sudo yum install -y netobserv-7.24.0-1.x86_64.rpm

If upgrading from a previously installed rpm, run the following:

sudo rpm -Uhv netobserv-7.24.0-1.x86_64.rpm

Running Device Auto-Discovery

You will need to have NetObserv SNMP 7.12.0 (or later) already installed. That will have the latest version of the SNMP definition files. If upgrading to 7.12.0, make sure you follow the manual upgrade steps for SNMP definition files.

1. Creating the "Networks" File Input

You must tell auto-discovery which network ranges to scan. This is done via a YAML file. This file will contain:

• SNMP credentials to use for authentication

• Connection parameters (port, timeout, retries, etc.)

• Network ranges to scan

Example networks.yml:

# Note: the key name "public", in this example, can be replaced with any string.
# The keys here are arbitrary strings you want to use to identify each 'set'
# of network ranges to scan.
# When auto-discovery is done, it will create new YAML files using this name.
# In this example, it will create a file called "public.yml"
# (the default location would be /etc/elastiflow/snmp/devices/public.yml)
public:
  credentials:
    - version: "1"
      community: public
    - version: "2c"
      community: public
  port: 161
  poll_interval: 60
  timeout: 2000
  retries: 2
  exponential_timeout: false
  networks:
    - 10.101.2.0/24
    - 10.101.3.0/24
    - 10.101.254.0/24

secure_network:
  credentials:
    version: "3"
    username: admin
    authentication_protocol: SHA
    authentication_passphrase: authpass
    privacy_protocol: AES
    privacy_passphrase: privpass
  networks:
    - 192.168.1.0/24

Details about this file format:

Option	Description
credentials	List of SNMP credential configurations to try when connecting to devices
port	SNMP port to connect to (default: 161)
poll_interval	Interval in seconds between SNMP polls (default: 60)
timeout	Timeout in milliseconds for SNMP requests (default: 3000)
retries	Number of retries for failed SNMP requests (default: 2)
exponential_timeout	Whether to use exponential backoff for retries (default: false)
networks	List of network ranges to scan. Either in CIDR notation (e.g., 10.0.0.0/24) or hyphenated rang (e.g., 10.0.200.0-10.0.203.0). You can also list individual IPs.

2. Running Auto-Discovery Command

If you use all the default values, you can just run netobserv autodiscover snmp to do auto-discovery. Otherwise, use one of the below flags to configure how it runs.

Command Line Interface

netobserv autodiscover snmp [flags]

Flag	Default	Description
--networks -n	/etc/elastiflow/snmp/autodiscover/networks.yml	YAML file containing the list of network groups to discover devices from.
--output -o	/etc/elastiflow/snmp/devices	Output directory where discovered devices will be stored.
--mappings -m	/etc/elastiflow/snmp/autodiscover/sysoids	YAML file containing sysObjectID to device group mappings. NetObserv (starting at 7.12.0) ships with mappings.
--concurrency -c	100	Number of concurrent workers to use for device discovery.
--timeout -t	3000	Default timeout in milliseconds for each SNMP request (can be overridden in the networks file).
--retries -r	2	Default number of retries for each SNMP request (can be overridden in the networks file).
--poll-interval -p	60	Default interval in seconds between each SNMP poll when running NetObserv SNMP (can be overridden in the networks file).
--verbose -v	false	Enable verbose output (debug logging).
--overwrite		Normally, auto-discovery will not rescan devices for network groups it already scanned. This flag will rescan, and overwrite output files.

Output

When auto-discovery finishes, you can run ls /etc/elastiflow/snmp/devices to see additional YAML files created. Those files will have devices defined, which NetObserv SNMP will automatically load when you tell it to reapply definitions (next step).

Here's an example output file for the example input file from above:

# /etc/elastiflow/snmp/devices/public.yml
router1_10.101.2.1:
  ip: 10.101.2.1
  port: 161
  version: 2c
  communities:
    - public
  device_groups:
    - generic

switch1_10.101.3.5:
  ip: 10.101.3.5
  port: 161
  version: 2c
  communities:
    - public
  device_groups:
    - cisco_cat_2960 # This was derived from the sysOID of the device.

# /etc/elastiflow/snmp/devices/secure_network.yml
re-router:
  ip: 192.168.1.1
  port: 161
  version: 3
  v3_credentials:
    - username: admin
      authentication_protocol: SHA
      authentication_passphrase: authpass
      privacy_protocol: AES
      privacy_passphrase: privpass
  device_groups:
    - generic

If an output file already exists, it is preserved and unchanged. Auto-discovery will skip scanning any addresses in that network 'group.'

You can use the --overwrite feature to forget perviously scanned results and rescan and recreate output again.

Explaining SysObjectID Mappings

The --mapping (-m) flag is optional. When using this flag, you can provide a YAML file that maps SNMP sysObjectIDs to device group names. This helps categorize discovered devices based on their system object identifiers.

If the -s flag is not provided or if a device's sysObjectID does not match any mapping in the file, the device definition will use the "generic" device group by default.

ElastiFlow's comprehensive set of sysObjectID to device group mappings is both publicly available for download and shipped with NetObserv SNMP product.

Example sysoid-mappings.yml:

.1.3.6.1.4.1.9.1.12: 'cisco_7000' # Cisco 7000 CISCO7010
.1.3.6.1.4.1.9.1.13: 'cisco_2500' # Cisco 2500 CISCO2500
.1.3.6.1.4.1.9.1.14: 'cisco_4000' # Cisco 4000 CISCO4500

Each entry maps a sysObjectID (OID) to a device group name. When a device is discovered with a matching sysObjectID, it will be associated with the specified device group.

3. Telling NetObserv SNMP to Re-Read Device Files

Trigger the apply-definitions API to reload device YAML configurations.

POST localhost:8080/api/v1/snmp/apply-definitions should work, assuming all default settings for NetObserv SNMP.