Metadata Enrichment (Addresses) API

Introduction to the API

NetObserv Flow and NetObserv SNMP provide an API for managing user-defined metadata (UDM) entries. See docs for this feature in NetObserv Flow and NetObserv SNMP. The API allows for creating, reading, updating, and deleting user-defined metadata entries (UDM entries).

The API documented here can be accessed via REST, gRPC, or connectrpc.

Configuring the API

To enable this API, set the following configuration values:

• EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: true

• EF_PROCESSOR_ENRICH_IPADDR_METADATA_API_ENABLE: true

• EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: /etc/elastiflow/metadata/ipaddrs.yml

  • this setting can be configured to any location. The only requirement is that it is set to something.

See additional configuration documentation for NetObserv Flow or NetObserv SNMP

API Common Configuration

This API adheres to the same common config that our other APIs use. This includes:

• optionally configuring basic auth to be required

• optionally configuring TLS

• optionally configuring a different port

See API Reference Overview for more details of what you can configure.

API Context

API Replaces YAML File Usage

Using this API will replace manually updating the UDM YAML file.

When you use this API, it will update the YAML file to represent latest state. In exchange, the product will not notice any updates to the YAML file, nor will it re-read that file.

Using any of the create or update calls (e.g. CreateUdm) will delete all comments in the YAML file, and probably change the order of stanzas in the file. The resulting file will be just the data in the file, not the comments or other formatting. So if you plan to make updates via the API, make sure there's no important information in the comments.

API Concurrency and Error Response

All RPCs are safe to call concurrently. The "read" RPCs will not block normal functioning of the collector, but the "write" RPCs will (very briefly), while the database is being updated and saved to disk.

Write operations are atomic and include updating the YAML file. If a write operation fails for any reason (returns an error), then neither the internal database nor the external YAML file were updated.

Keys are given as strings and parsed as IP addresses, CIDRs (IP prefixes), or IP ranges (a "from" IP and a "to" IP). In the API, if any key cannot be parsed, the operation will abort and return an error listing the problematic key.

Warning about Caching

NetObserv will cache any UDM entries in memory. Even if you make modifications via this API, it will not take immediatetly take effect because of caching. The default length for this cache is two hours.

API Methods

CountUdms POST /api/v1/enrich/ipaddr/udm.v1.UdmService/CountUdms

Return the number of items in the DB.

The below examples assume you start with this set of UDM entries:

# Example ipaddrs.yml file
192.0.2.0/24:
  internal: true
192.0.2.192/26:
  name: atlanta_guest_wifi
  vlan: 1001
  tags:
    - wifi
    - dhcp
  metadata:
    dhcp.pool.name: atlanta_guest_wifi
    .site.id: atlanta
192.0.2.194-192.0.2.198:
  metadata:
    .site.bldg.id: hq
    .site.floor.id: 2
    .site.rack.id: 1
192.0.2.194:
  metadata:
    device.type.name: wifi_ap

Example

# using curl
curl --json "{}" http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/CountUdms

# using grpcurl
grpcurl -plaintext localhost:8080 udm.v1.UdmService/CountUdms

Output

{
  "count": 4
}

ListUdms POST /api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms

Return the records matching the given key. Non-string metadata values (e.g., numbers) are returned as strings.

Detailed Logic

If no key is given, it returns the entire DB.

If exact option is true, the key is matched exactly.

If exact option is false, all DB entries matching the requested key are returned, using the following rules:

• key is a key as listed in the ipaddrs.yml file

  If key is not specified, then exact is ignored and all Udms will be returned.

• If the key is:

  • An IP address

    • That IP will be returned if it exists.

    • Any CIDR that contains the given IP will be returned.

    • Any Range that contains the given IP will be returned.

  • A CIDR

    • Any IP within the given CIDR will be returned.

    • Any CIDR within the given CIDR will be returned.

    • Any Range within the given CIDR will be returned. That is, if the first and last IP in the range is in the given CIDR.

  • A Range

    • Any IP within the given Range will be returned.

    • Any CIDR within the given Range will be returned. That is, the first and last IP in the CIDR are in the given Range. E.g., the first and last IPs in 192.168.1.0/24 are 192.168.1.0 and 192.168.1.255.

    • Any Range within the given Range will be returned. That is, for a given range R1, a range R2 in the database will be returned if R1.From <= R2.From AND R2.To <= R1.To.

As mentioned above, ListUdms is safe to call concurrently; if you have multiple keys to query, you can safely call ListUdms concurrently for each key.

The below examples assume you start with this set of UDM entries:

# Example ipaddrs.yml file
192.0.2.0/24:
  internal: true
192.0.2.192/26:
  name: atlanta_guest_wifi
  vlan: 1001
  tags:
    - wifi
    - dhcp
  metadata:
    dhcp.pool.name: atlanta_guest_wifi
    .site.id: atlanta
192.0.2.194-192.0.2.198:
  metadata:
    .site.bldg.id: hq
    .site.floor.id: 2
    .site.rack.id: 1
192.0.2.194:
  metadata:
    device.type.name: wifi_ap

Example: List all UDM Entries

# using curl
curl --json "{}" http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms

# using grpcurl
grpcurl -plaintext localhost:8080 udm.v1.UdmService/ListUdms

Output

{
  "udms": [
    {
      "key": "192.0.2.194-192.0.2.198",
      "metadata": {
        ".site.bldg.id": "hq",
        ".site.floor.id": "2",
        ".site.rack.id": "1"
      }
    },
    {
      "key": "192.0.2.194",
      "metadata": {
        "device.type.name": "wifi_ap"
      }
    },
    {
      "key": "192.0.2.0/24",
      "internal": true
    },
    {
      "key": "192.0.2.192/26",
      "name": "atlanta_guest_wifi",
      "tags": [
        "wifi",
        "dhcp"
      ],
      "vlan": "1001",
      "metadata": {
        ".site.id": "atlanta",
        "dhcp.pool.name": "atlanta_guest_wifi"
      }
    }
  ]
}

Example: Search UDM Entries

# using curl
curl --json '{"key":"192.0.2.200", "exact":false}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms

# using grpcurl
grpcurl -plaintext -d '{"key":"192.0.2.200", "exact":false}' localhost:8080 udm.v1.UdmService/ListUdms

Output

{
  "udms": [
    {
      "key": "192.0.2.192/26",
      "name": "atlanta_guest_wifi",
      "tags": [
        "wifi",
        "dhcp"
      ],
      "vlan": "1001",
      "metadata": {
        ".site.id": "atlanta",
        "dhcp.pool.name": "atlanta_guest_wifi"
      }
    },
    {
      "key": "192.0.2.0/24",
      "internal": true
    }
  ]
}

ListUdmKeys POST /api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys

Return keys matching the given key. This API is logically equivalent to calling ListUdms on the given key, and then returning only the keys of the returned Udms. If no key is given, returns all keys.

The rules for searching work the same as ListUdm Method

The below examples assume you start with this set of UDM entries:

# Example ipaddrs.yml file
192.0.2.0/24:
  internal: true
192.0.2.192/26:
  name: atlanta_guest_wifi
  vlan: 1001
  tags:
    - wifi
    - dhcp
  metadata:
    dhcp.pool.name: atlanta_guest_wifi
    .site.id: atlanta
192.0.2.194-192.0.2.198:
  metadata:
    .site.bldg.id: hq
    .site.floor.id: 2
    .site.rack.id: 1
192.0.2.194:
  metadata:
    device.type.name: wifi_ap

List All Keys

# using curl
curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys

# using grpcurl
grpcurl '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys

Output

{
  "keys": [
    "192.0.2.0/24",
    "192.0.2.192/26",
    "192.0.2.194-192.0.2.198",
    "192.0.2.194"
  ]
}

List a specific key

# using curl
curl --json '{"key":"192.0.2.194", "exact":true}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys

# using grpcurl
grpcurl -d '{"key":"192.0.2.194", "exact":true}' localhost:8080 udm.v1.UdmService/ListUdmKeys

Output

{
  "keys": [
    "192.0.2.194"
  ]
}

List all keys matching a given key

# using curl
curl --json '{"key":"192.0.2.200", "exact":false}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys

# using grpcurl
grpcurl -d '{"key":"192.0.2.200", "exact":false}' localhost:8080 udm.v1.UdmService/ListUdmKeys

Output

{
  "keys": [
    # Both of these CIDRs contain the given IP.
    "192.0.2.0/24",
    "192.0.2.192/26"
  ]
}

CreateUdm POST /api/v1/enrich/ipaddr/udm.v1.UdmService/CreateUdm

Create a new UDM entry. Return an error if the given key already exists.

# using curl
curl --json '{"udm": { "key": "192.0.3.0/24", "name": "my_udm", "vlan": 2, "tags":["tag1", "tag2"]}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/CreateUdm

# using grpcurl
grpcurl -d '{"udm": { "key": "192.0.3.0/24", "name": "my_udm", "vlan": 2, "tags":["tag1", "tag2"]}}' localhost:8080 udm.v1.UdmService/CreateUdm

Output

{
  "key": "192.0.3.0/24",
  "name": "my_udm",
  "tags": [
    "tag1",
    "tag2"
  ],
  "vlan": "2"
}

BatchCreateUdms POST /api/v1/enrich/ipaddr/udm.v1.UdmService/BatchCreateUdms

Create the given set of Udms and return them. Return an error if any of the given Udms already exist.

Creates are atomic, and include saving the updates to the yml file.

# using curl
curl --json '{"requests":[ 
{"udm": { "key": "192.0.4.0/24", "name": "my_udm2", "vlan": 4, "tags":["tag3", "tag4"]}},
{"udm": { "key": "192.0.5.0/24", "name": "my_udm3", "vlan": 5, "tags":["tag5", "tag6"]}}
]}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchCreateUdms

# using grpcurl
grpcurl -d '{"requests":[
{"udm": { "key": "192.0.4.0/24", "name": "my_udm2", "vlan": 4, "tags":["tag3", "tag4"]}},
{"udm": { "key": "192.0.5.0/24", "name": "my_udm3", "vlan": 5, "tags":["tag5", "tag6"]}}
]}' localhost:8080 udm.v1.UdmService/BatchCreateUdms

Output

{
  "udms": [
    {
      "key": "192.0.4.0/24",
      "name": "my_udm2",
      "tags": [
        "tag3",
        "tag4"
      ],
      "vlan": "4"
    },
    {
      "key": "192.0.5.0/24",
      "name": "my_udm3",
      "tags": [
        "tag5",
        "tag6"
      ],
      "vlan": "5"
    }
  ]
}

The status of the DB after running the above example commands:

curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys
{
  "keys": [
    "192.0.2.0/24",
    "192.0.2.192/26",
    "192.0.3.0/24",
    "192.0.4.0/24",
    "192.0.5.0/24",
    "192.0.2.194-192.0.2.198",
    "192.0.2.194"
  ]
}

DeleteUdm POST /api/v1/enrich/ipaddr/udm.v1.UdmService/DeleteUdm

Deletes the given Udm; the key must match exactly

Deletes are atomic and include updating the yml file.

Example

# using curl
curl --json '{"key": "192.0.3.0/24"}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/DeleteUdm

# using grpcurl
grpcurl -d '{"key": "192.0.3.0/24"}' localhost:8080 udm.v1.UdmService/DeleteUdm

Output

{}

Result

curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys
{
  "keys": [
    "192.0.2.0/24",
    "192.0.2.192/26",
    # note 192.0.3.0/24 is gone
    "192.0.4.0/24",
    "192.0.5.0/24",
    "192.0.2.194-192.0.2.198",
    "192.0.2.194"
  ]
}

UpdateUdm POST /api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm

Update the given Udm with the given data. The UpdateMask field inside the request specifies which fields to update. If the UpdateMask is empty, all fields are updated, including those that were not specified in the update call (which will be set to their "zero value"). (See the example below for more details.)

Return the updated Udm or an error if the given key does not exist.

Example: Without an update_mask

Assuming this starting content

{
  "udms": [
    {
      "key": "192.0.4.0/24",
      "name": "my_udm2",
      "tags": [
        "tag3",
        "tag4"
      ],
      "vlan": "4"
    }
  ]
}

# using curl
curl --json '{"udm":{"key": "192.0.4.0/24", "vlan": 5}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm

# using grpcurl
grpcurl '{"udm":{"key": "192.0.4.0/24", "vlan": 5}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm

Output

{
  "key": "192.0.4.0/24",
  "vlan": "5"
}

This command didn't give an update_mask, so all the fields not specified in the udm object were cleared.

With an update_mask

assuming

{
  "udms": [
    {
      "key": "192.0.5.0/24",
      "name": "my_udm3",
      "tags": [
        "tag5",
        "tag6"
      ],
      "vlan": "5"
    }
  ]
}

# Only update vlan and tags fields

# using curl
curl --json '{"udm":{"key": "192.0.5.0/24", "vlan": 6, "tags":["tag7"]}, "update_mask": "vlan,tags"}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm

# using grpcurl
# Note the different format of the update_mask field for grpcurl
grpcurl -d '{"udm":{"key": "192.0.5.0/24", "vlan": 6, "tags":["tag7"]}, "update_mask": {"paths":["vlan","tags"]}}' localhost:8080 udm.v1.UdmService/UpdateUdm

Output

{
  "key": "192.0.5.0/24",
  "name": "my_udm3", // name not changed
  "tags": [          // all tags replaced
    "tag7"
  ],
  "vlan": "6"        // vlan updated
}

This command did give an update_mask, so only the vlan and tags fields were updated.

BatchUpdateUdms POST /api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms

Update the given Udms with the given data. The UpdateMask field inside the request specifies which fields to update. If UpdateMask is empty, all fields are updated, including those that were not specified in the update call (which will be set to their "zero value"). (See the example below for more details.)

Return the updated Udms or an error if any of the given keys do not exist.

Updates are atomic and include saving the updates to the yml file.

Implementation note: BatchUpdateUdms accepts a single BatchUpdatUdmsRequest struct containing a slice of UpdateUdmRequest structs, and a single UpdateMask field, which is used for the entire batch of updates. Each UpdateUdmRequest struct also has its own UpdateMask, which is ignored.

Without an update_mask

# using curl
curl --json '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "new_name"}},
{"udm": { "key": "192.0.2.194", "name": "new_name2"}} ]}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms

# using grpcurl
% grpcurl -d '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "new_name"}},
{"udm": { "key": "192.0.2.194", "name": "new_name2"}} ]}' localhost:8080 udm.v1.UdmService/BatchUpdateUdms

Output

{
  "udms": [
    {
      "key": "192.0.2.0/24",
      "name": "new_name"
    },
    {
      "key": "192.0.2.194",
      "name": "new_name2"
    }
  ]
}

As above, since no update_mask was specified, all fields not mentioned were cleared.

With an update_mask

• Update only the tags and metadata fields.

• Even though vlan is listed in the first UDM entry, it is not updated.

• Even though name is listed in both UDMs, it is not updated.

• Only the top-level update_mask field is valid. Even though the first request says "update_mask":"name", this is ignored.

### Update only the tags and metadata fields; inner "update_mask":"name" is ignored

# using curl
curl --json '{"requests":[
    {"udm": { "key": "192.0.2.0/24", "name": "not_changed", "vlan":2, "tags":["tag8"]}, "update_mask": "name"},
    {"udm": { "key": "192.0.2.194", "name": "also_not_changed", "metadata":{"key":"value"}}}
],
"update_mask": "tags,metadata" }' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms

# using grpcurl
% grpcurl -d '{"requests":[
    {"udm": { "key": "192.0.2.0/24", "name": "not_changed", "vlan":2, "tags":["tag8"]}, "update_mask": {"paths":["name"]}},
    {"udm": { "key": "192.0.2.194", "name": "also_not_changed", "metadata":{"key":"value"}}}
],
"update_mask": {"paths": ["tags","metadata"]} }' localhost:8080 udm.v1.UdmService/BatchUpdateUdms

Output

{
  "udms": [
    {
      "key": "192.0.2.0/24",
      "name": "new_name",
      "tags": [
        "tag8"
      ]
    },
    {
      "key": "192.0.2.194",
      "name": "new_name2",
      "metadata": {
        "key": "value"
      }
    }
  ]
}

note

In a BatchUpdateUdms request, only the top-level update_mask field is used; the update_mask field in each request object is ignored.

GRPC Interface Details

We expose all the exact same methods via gRPC in addition to HTTP/REST. The gRPC protocol is a different protocol than HTTP/REST. You do not need to use the gRPC interface if you do not want to.

For those that are interested, below are some additional details regarding the gRPC interface to this API.

Inspecting the Schema With grpcurl

You can use grpcurl command to inspect and navigate the grpc schema.

% grpcurl -plaintext localhost:8080 list
udm.v1.UdmService

% grpcurl -plaintext localhost:8080 describe
udm.v1.UdmService is a service:
service UdmService {
  rpc BatchCreateUdms ( .udm.v1.BatchCreateUdmsRequest ) returns ( .udm.v1.BatchCreateUdmsResponse );
  rpc BatchUpdateUdms ( .udm.v1.BatchUpdateUdmsRequest ) returns ( .udm.v1.BatchUpdateUdmsResponse );
  rpc CountUdms ( .google.protobuf.Empty ) returns ( .udm.v1.CountUdmsResponse );
  rpc CreateUdm ( .udm.v1.CreateUdmRequest ) returns ( .udm.v1.Udm );
  rpc DeleteUdm ( .udm.v1.DeleteUdmRequest ) returns ( .google.protobuf.Empty );
  rpc ListUdmKeys ( .udm.v1.ListUdmKeysRequest ) returns ( .udm.v1.ListUdmKeysResponse );
  rpc ListUdms ( .udm.v1.ListUdmsRequest ) returns ( .udm.v1.ListUdmsResponse );
  rpc UpdateUdm ( .udm.v1.UpdateUdmRequest ) returns ( .udm.v1.Udm );
}

% grpcurl -plaintext localhost:8080 describe .udm.v1.BatchCreateUdmsRequest
udm.v1.BatchCreateUdmsRequest is a message:
message BatchCreateUdmsRequest {
  repeated .udm.v1.CreateUdmRequest requests = 1;
}

% grpcurl -plaintext localhost:8080 describe .udm.v1.CreateUdmRequest
udm.v1.CreateUdmRequest is a message:
message CreateUdmRequest {
  .udm.v1.Udm udm = 1;
}

% grpcurl -plaintext localhost:8080 describe .udm.v1.Udm
udm.v1.Udm is a message:
message Udm {
  string key = 1;
  string name = 2;
  repeated string tags = 3;
  uint64 vlan = 4;
  bool internal = 5;
  map<string, string> metadata = 6;
}

See the grpcurl documentation for more details on using the gRPC reflection API.

Proto file

syntax = "proto3";

package udm.v1;

import "google/protobuf/empty.proto";
import "google/protobuf/field_mask.proto";

// Udm represents an IP enrichment entry.
message Udm {
  string key = 1; // The key field will be populated from the yaml key when returning Udms.
  string name = 2;
  repeated string tags = 3;
  uint64 vlan = 4;
  bool internal = 5;
  map<string, string> metadata = 6;
}

// CreateUdmRequest is the request for the CreateUdm RPC.
message CreateUdmRequest {
  Udm udm = 1;
}

// BatchCreateUdmsRequest is the request for the BatchCreateUdms RPC.
message BatchCreateUdmsRequest {
  repeated CreateUdmRequest requests = 1;
}

// BatchCreateUdmsResponse is the response for the BatchCreateUdms RPC.
message BatchCreateUdmsResponse {
  // Udms created
  repeated Udm udms = 1;
}

// UpdateUdmRequest is the request for the UpdateUdm RPC.
message UpdateUdmRequest {
  Udm udm = 1;
  optional google.protobuf.FieldMask update_mask = 2;
}

// BatchUpdateUdmsRequest is the request for the BatchUpdateUdms RPC.
message BatchUpdateUdmsRequest {
  repeated UpdateUdmRequest requests = 1;
  // update_mask fields in requests list are ignored
  optional google.protobuf.FieldMask update_mask = 2;
}

message google.protobuf.FieldMask {
  repeated string paths = 1;
}

// BatchUpdateUdmsResponse is the response for the BatchUpdateUdms RPC.
message BatchUpdateUdmsResponse {
  repeated Udm udms = 1;
}

// ListUdmsRequest is the request for the ListUdms RPC.
message ListUdmsRequest {
  string key = 1;
  bool exact = 2;
}

// ListUdmsResponse is the response for the ListUdms RPC.
message ListUdmsResponse {
  repeated Udm udms = 1;
}

// CountUdmsResponse is the response for the CountUdms RPC.
message CountUdmsResponse {
  int32 count = 1;
}

// ListUdmKeysRequest is the request for the ListUdmKeys RPC.
message ListUdmKeysRequest {
  string key = 1;
  bool exact = 2;
}

// ListUdmKeysResponse is the response for the ListUdmKeys RPC.
message ListUdmKeysResponse {
  repeated string keys = 1;
}

// DeleteUdmRequest is the request for the DeleteUdm RPC.
message DeleteUdmRequest {
  string key = 1;
}