# Metadata Enrichment (Addresses) API ## Introduction to the API NetObserv Flow and NetObserv SNMP provide an API for managing user-defined metadata (UDM) entries. See docs for this feature in [NetObserv Flow](/flowcoll/configuration/enrichment-options/ip-address-enrichment/enrich_ip_udm.md) and [NetObserv SNMP](/snmpcoll/configuration/enrichment-options/enrich_ip_udm/enrich_ip_udm.md). The API allows for creating, reading, updating, and deleting user-defined metadata entries (UDM entries). The API documented here can be accessed via REST, gRPC, or connectrpc. ### Configuring the API To enable this API, set the following configuration values: * `EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: true` * `EF_PROCESSOR_ENRICH_IPADDR_METADATA_API_ENABLE: true` * `EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: /etc/elastiflow/metadata/ipaddrs.yml` * this setting can be configured to any location. The only requirement is that it is set to something. See additional configuration documentation for [NetObserv Flow](/flowcoll/configuration/enrichment-options/ip-address-enrichment/enrich_ip_udm/enrich_ip_udm.md) or NetObserv SNMP ### API Common Configuration This API adheres to the same common config that our other APIs use. This includes: * optionally configuring basic auth to be required * optionally configuring TLS * optionally configuring a different port See [API Reference Overview](/flowcoll/overview.md) for more details of what you can configure. ## API Context ### API Replaces YAML File Usage Using this API will replace manually updating the UDM YAML file. When you use this API, it will update the YAML file to represent latest state. In exchange, the product will not notice any updates to the YAML file, nor will it re-read that file. Using any of the create or update calls (e.g. `CreateUdm`) will delete all comments in the YAML file, and probably change the order of stanzas in the file. The resulting file will be just the data in the file, not the comments or other formatting. So if you plan to make updates via the API, make sure there's no important information in the comments. ### API Concurrency and Error Response All RPCs are safe to call concurrently. The "read" RPCs will not block normal functioning of the collector, but the "write" RPCs will (very briefly), while the database is being updated and saved to disk. Write operations are atomic and include updating the YAML file. If a write operation fails for any reason (returns an error), then neither the internal database nor the external YAML file were updated. Keys are given as strings and parsed as IP addresses, CIDRs (IP prefixes), or IP ranges (a "from" IP and a "to" IP). In the API, if any key cannot be parsed, the operation will abort and return an error listing the problematic key. ### Warning about Caching NetObserv will cache any UDM entries in memory. Even if you make modifications via this API, it will not take immediatetly take effect because of caching. The default length for this cache is two hours. ## API Methods ### CountUdms `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/CountUdms` Return the number of items in the DB.
The below examples assume you start with this set of UDM entries: ``` # Example ipaddrs.yml file 192.0.2.0/24: internal: true 192.0.2.192/26: name: atlanta_guest_wifi vlan: 1001 tags: - wifi - dhcp metadata: dhcp.pool.name: atlanta_guest_wifi .site.id: atlanta 192.0.2.194-192.0.2.198: metadata: .site.bldg.id: hq .site.floor.id: 2 .site.rack.id: 1 192.0.2.194: metadata: device.type.name: wifi_ap ```
**Example** ``` # using curl curl --json "{}" http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/CountUdms # using grpcurl grpcurl -plaintext localhost:8080 udm.v1.UdmService/CountUdms ``` **Output** ``` { "count": 4 } ``` ### ListUdms `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms` Return the records matching the given key. Non-string metadata values (e.g., numbers) are returned as strings. **Detailed Logic** If no key is given, it returns the entire DB. If `exact` option is true, the key is matched exactly. If `exact` option is false, all DB entries matching the requested key are returned, using the following rules: * `key` is a key as listed in the `ipaddrs.yml` file If `key` is not specified, then `exact` is ignored and all Udms will be returned. * If the key is: * An IP address * That IP will be returned if it exists. * Any CIDR that contains the given IP will be returned. * Any Range that contains the given IP will be returned. * A CIDR * Any IP within the given CIDR will be returned. * Any CIDR within the given CIDR will be returned. * Any Range within the given CIDR will be returned. That is, if the first and last IP in the range is in the given CIDR. * A Range * Any IP within the given Range will be returned. * Any CIDR within the given Range will be returned. That is, the first and last IP in the CIDR are in the given Range. E.g., the first and last IPs in 192.168.1.0/24 are 192.168.1.0 and 192.168.1.255. * Any Range within the given Range will be returned. That is, for a given range R1, a range R2 in the database will be returned if R1.From <= R2.From AND R2.To <= R1.To. As mentioned above, `ListUdms` is safe to call concurrently; if you have multiple keys to query, you can safely call `ListUdms` concurrently for each key.
The below examples assume you start with this set of UDM entries: ``` # Example ipaddrs.yml file 192.0.2.0/24: internal: true 192.0.2.192/26: name: atlanta_guest_wifi vlan: 1001 tags: - wifi - dhcp metadata: dhcp.pool.name: atlanta_guest_wifi .site.id: atlanta 192.0.2.194-192.0.2.198: metadata: .site.bldg.id: hq .site.floor.id: 2 .site.rack.id: 1 192.0.2.194: metadata: device.type.name: wifi_ap ```
**Example: List all UDM Entries** ``` # using curl curl --json "{}" http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms # using grpcurl grpcurl -plaintext localhost:8080 udm.v1.UdmService/ListUdms ``` **Output** ``` { "udms": [ { "key": "192.0.2.194-192.0.2.198", "metadata": { ".site.bldg.id": "hq", ".site.floor.id": "2", ".site.rack.id": "1" } }, { "key": "192.0.2.194", "metadata": { "device.type.name": "wifi_ap" } }, { "key": "192.0.2.0/24", "internal": true }, { "key": "192.0.2.192/26", "name": "atlanta_guest_wifi", "tags": [ "wifi", "dhcp" ], "vlan": "1001", "metadata": { ".site.id": "atlanta", "dhcp.pool.name": "atlanta_guest_wifi" } } ] } ``` **Example: Search UDM Entries** ``` # using curl curl --json '{"key":"192.0.2.200", "exact":false}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdms # using grpcurl grpcurl -plaintext -d '{"key":"192.0.2.200", "exact":false}' localhost:8080 udm.v1.UdmService/ListUdms ``` **Output** ``` { "udms": [ { "key": "192.0.2.192/26", "name": "atlanta_guest_wifi", "tags": [ "wifi", "dhcp" ], "vlan": "1001", "metadata": { ".site.id": "atlanta", "dhcp.pool.name": "atlanta_guest_wifi" } }, { "key": "192.0.2.0/24", "internal": true } ] } ``` ### ListUdmKeys `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys` Return keys matching the given key. This API is logically equivalent to calling `ListUdms` on the given key, and then returning only the keys of the returned Udms. If no key is given, returns all keys. The rules for searching work the same as [ListUdm Method](http://localhost:3000/docs/api_ref/enrich_ip_udm_api#detailed-logic)
The below examples assume you start with this set of UDM entries: ``` # Example ipaddrs.yml file 192.0.2.0/24: internal: true 192.0.2.192/26: name: atlanta_guest_wifi vlan: 1001 tags: - wifi - dhcp metadata: dhcp.pool.name: atlanta_guest_wifi .site.id: atlanta 192.0.2.194-192.0.2.198: metadata: .site.bldg.id: hq .site.floor.id: 2 .site.rack.id: 1 192.0.2.194: metadata: device.type.name: wifi_ap ```
**List All Keys** ``` # using curl curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys # using grpcurl grpcurl '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys ``` **Output** ``` { "keys": [ "192.0.2.0/24", "192.0.2.192/26", "192.0.2.194-192.0.2.198", "192.0.2.194" ] } ``` **List a specific key** ``` # using curl curl --json '{"key":"192.0.2.194", "exact":true}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys # using grpcurl grpcurl -d '{"key":"192.0.2.194", "exact":true}' localhost:8080 udm.v1.UdmService/ListUdmKeys ``` **Output** ``` { "keys": [ "192.0.2.194" ] } ``` **List all keys matching a given key** ``` # using curl curl --json '{"key":"192.0.2.200", "exact":false}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys # using grpcurl grpcurl -d '{"key":"192.0.2.200", "exact":false}' localhost:8080 udm.v1.UdmService/ListUdmKeys ``` **Output** ``` { "keys": [ # Both of these CIDRs contain the given IP. "192.0.2.0/24", "192.0.2.192/26" ] } ``` ### CreateUdm `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/CreateUdm` Create a new UDM entry. Return an error if the given key already exists. ``` # using curl curl --json '{"udm": { "key": "192.0.3.0/24", "name": "my_udm", "vlan": 2, "tags":["tag1", "tag2"]}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/CreateUdm # using grpcurl grpcurl -d '{"udm": { "key": "192.0.3.0/24", "name": "my_udm", "vlan": 2, "tags":["tag1", "tag2"]}}' localhost:8080 udm.v1.UdmService/CreateUdm ``` **Output** ``` { "key": "192.0.3.0/24", "name": "my_udm", "tags": [ "tag1", "tag2" ], "vlan": "2" } ``` ### BatchCreateUdms `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/BatchCreateUdms` Create the given set of Udms and return them. Return an error if any of the given Udms already exist. Creates are atomic, and include saving the updates to the yml file. ``` # using curl curl --json '{"requests":[ {"udm": { "key": "192.0.4.0/24", "name": "my_udm2", "vlan": 4, "tags":["tag3", "tag4"]}}, {"udm": { "key": "192.0.5.0/24", "name": "my_udm3", "vlan": 5, "tags":["tag5", "tag6"]}} ]}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchCreateUdms # using grpcurl grpcurl -d '{"requests":[ {"udm": { "key": "192.0.4.0/24", "name": "my_udm2", "vlan": 4, "tags":["tag3", "tag4"]}}, {"udm": { "key": "192.0.5.0/24", "name": "my_udm3", "vlan": 5, "tags":["tag5", "tag6"]}} ]}' localhost:8080 udm.v1.UdmService/BatchCreateUdms ``` **Output** ``` { "udms": [ { "key": "192.0.4.0/24", "name": "my_udm2", "tags": [ "tag3", "tag4" ], "vlan": "4" }, { "key": "192.0.5.0/24", "name": "my_udm3", "tags": [ "tag5", "tag6" ], "vlan": "5" } ] } ``` The status of the DB after running the above example commands: ``` curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys { "keys": [ "192.0.2.0/24", "192.0.2.192/26", "192.0.3.0/24", "192.0.4.0/24", "192.0.5.0/24", "192.0.2.194-192.0.2.198", "192.0.2.194" ] } ``` ### DeleteUdm `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/DeleteUdm` Deletes the given Udm; the key must match exactly Deletes are atomic and include updating the yml file. **Example** ``` # using curl curl --json '{"key": "192.0.3.0/24"}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/DeleteUdm # using grpcurl grpcurl -d '{"key": "192.0.3.0/24"}' localhost:8080 udm.v1.UdmService/DeleteUdm ``` **Output** ``` {} ``` Result ``` curl --json '{}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/ListUdmKeys { "keys": [ "192.0.2.0/24", "192.0.2.192/26", # note 192.0.3.0/24 is gone "192.0.4.0/24", "192.0.5.0/24", "192.0.2.194-192.0.2.198", "192.0.2.194" ] } ``` ### UpdateUdm `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm` Update the given Udm with the given data. The `UpdateMask` field inside the request specifies which fields to update. If the `UpdateMask` is empty, all fields are updated, including those that were not specified in the update call (which will be set to their "zero value"). (See the example below for more details.) Return the updated Udm or an error if the given key does not exist. **Example: Without an `update_mask`** Assuming this starting content ``` { "udms": [ { "key": "192.0.4.0/24", "name": "my_udm2", "tags": [ "tag3", "tag4" ], "vlan": "4" } ] } ``` ``` # using curl curl --json '{"udm":{"key": "192.0.4.0/24", "vlan": 5}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm # using grpcurl grpcurl '{"udm":{"key": "192.0.4.0/24", "vlan": 5}}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm ``` **Output** ``` { "key": "192.0.4.0/24", "vlan": "5" } ``` This command didn't give an `update_mask`, so all the fields not specified in the `udm` object were cleared. **With an `update_mask`** assuming ``` { "udms": [ { "key": "192.0.5.0/24", "name": "my_udm3", "tags": [ "tag5", "tag6" ], "vlan": "5" } ] } ``` ``` # Only update vlan and tags fields # using curl curl --json '{"udm":{"key": "192.0.5.0/24", "vlan": 6, "tags":["tag7"]}, "update_mask": "vlan,tags"}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/UpdateUdm # using grpcurl # Note the different format of the update_mask field for grpcurl grpcurl -d '{"udm":{"key": "192.0.5.0/24", "vlan": 6, "tags":["tag7"]}, "update_mask": {"paths":["vlan","tags"]}}' localhost:8080 udm.v1.UdmService/UpdateUdm ``` **Output** ``` { "key": "192.0.5.0/24", "name": "my_udm3", // name not changed "tags": [ // all tags replaced "tag7" ], "vlan": "6" // vlan updated } ``` This command did give an `update_mask`, so only the `vlan` and `tags` fields were updated. ### BatchUpdateUdms `POST /api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms` Update the given Udms with the given data. The `UpdateMask` field inside the request specifies which fields to update. If `UpdateMask` is empty, all fields are updated, including those that were not specified in the update call (which will be set to their "zero value"). (See the example below for more details.) Return the updated Udms or an error if any of the given keys do not exist. Updates are atomic and include saving the updates to the yml file. Implementation note: `BatchUpdateUdms` accepts a single `BatchUpdatUdmsRequest` struct containing a slice of `UpdateUdmRequest` structs, and a single `UpdateMask` field, which is used for the entire batch of updates. Each `UpdateUdmRequest` struct also has its own `UpdateMask`, which is ignored. **Without an `update_mask`** ``` # using curl curl --json '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "new_name"}}, {"udm": { "key": "192.0.2.194", "name": "new_name2"}} ]}' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms # using grpcurl % grpcurl -d '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "new_name"}}, {"udm": { "key": "192.0.2.194", "name": "new_name2"}} ]}' localhost:8080 udm.v1.UdmService/BatchUpdateUdms ``` **Output** ``` { "udms": [ { "key": "192.0.2.0/24", "name": "new_name" }, { "key": "192.0.2.194", "name": "new_name2" } ] } ``` As above, since no `update_mask` was specified, all fields not mentioned were cleared. **With an `update_mask`** * Update only the `tags` and `metadata` fields. * Even though `vlan` is listed in the first UDM entry, it is not updated. * Even though `name` is listed in both UDMs, it is not updated. * Only the top-level `update_mask` field is valid. Even though the first request says `"update_mask":"name"`, this is ignored. ``` ### Update only the tags and metadata fields; inner "update_mask":"name" is ignored # using curl curl --json '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "not_changed", "vlan":2, "tags":["tag8"]}, "update_mask": "name"}, {"udm": { "key": "192.0.2.194", "name": "also_not_changed", "metadata":{"key":"value"}}} ], "update_mask": "tags,metadata" }' http://localhost:8080/api/v1/enrich/ipaddr/udm.v1.UdmService/BatchUpdateUdms # using grpcurl % grpcurl -d '{"requests":[ {"udm": { "key": "192.0.2.0/24", "name": "not_changed", "vlan":2, "tags":["tag8"]}, "update_mask": {"paths":["name"]}}, {"udm": { "key": "192.0.2.194", "name": "also_not_changed", "metadata":{"key":"value"}}} ], "update_mask": {"paths": ["tags","metadata"]} }' localhost:8080 udm.v1.UdmService/BatchUpdateUdms ``` **Output** ``` { "udms": [ { "key": "192.0.2.0/24", "name": "new_name", "tags": [ "tag8" ] }, { "key": "192.0.2.194", "name": "new_name2", "metadata": { "key": "value" } } ] } ``` note In a `BatchUpdateUdms` request, only the top-level `update_mask` field is used; the `update_mask` field in each request object is ignored. ## GRPC Interface Details We expose all the exact same methods via gRPC in addition to HTTP/REST. The [gRPC protocol](https://grpc.io/) is a different protocol than HTTP/REST. You do not need to use the gRPC interface if you do not want to. For those that are interested, below are some additional details regarding the gRPC interface to this API. #### Inspecting the Schema With grpcurl You can use `grpcurl` command to inspect and navigate the grpc schema. ``` % grpcurl -plaintext localhost:8080 list udm.v1.UdmService % grpcurl -plaintext localhost:8080 describe udm.v1.UdmService is a service: service UdmService { rpc BatchCreateUdms ( .udm.v1.BatchCreateUdmsRequest ) returns ( .udm.v1.BatchCreateUdmsResponse ); rpc BatchUpdateUdms ( .udm.v1.BatchUpdateUdmsRequest ) returns ( .udm.v1.BatchUpdateUdmsResponse ); rpc CountUdms ( .google.protobuf.Empty ) returns ( .udm.v1.CountUdmsResponse ); rpc CreateUdm ( .udm.v1.CreateUdmRequest ) returns ( .udm.v1.Udm ); rpc DeleteUdm ( .udm.v1.DeleteUdmRequest ) returns ( .google.protobuf.Empty ); rpc ListUdmKeys ( .udm.v1.ListUdmKeysRequest ) returns ( .udm.v1.ListUdmKeysResponse ); rpc ListUdms ( .udm.v1.ListUdmsRequest ) returns ( .udm.v1.ListUdmsResponse ); rpc UpdateUdm ( .udm.v1.UpdateUdmRequest ) returns ( .udm.v1.Udm ); } % grpcurl -plaintext localhost:8080 describe .udm.v1.BatchCreateUdmsRequest udm.v1.BatchCreateUdmsRequest is a message: message BatchCreateUdmsRequest { repeated .udm.v1.CreateUdmRequest requests = 1; } % grpcurl -plaintext localhost:8080 describe .udm.v1.CreateUdmRequest udm.v1.CreateUdmRequest is a message: message CreateUdmRequest { .udm.v1.Udm udm = 1; } % grpcurl -plaintext localhost:8080 describe .udm.v1.Udm udm.v1.Udm is a message: message Udm { string key = 1; string name = 2; repeated string tags = 3; uint64 vlan = 4; bool internal = 5; map metadata = 6; } ``` See the [grpcurl documentation](https://github.com/fullstorydev/grpcurl) for more details on using the gRPC reflection API. #### Proto file ``` syntax = "proto3"; package udm.v1; import "google/protobuf/empty.proto"; import "google/protobuf/field_mask.proto"; // Udm represents an IP enrichment entry. message Udm { string key = 1; // The key field will be populated from the yaml key when returning Udms. string name = 2; repeated string tags = 3; uint64 vlan = 4; bool internal = 5; map metadata = 6; } // CreateUdmRequest is the request for the CreateUdm RPC. message CreateUdmRequest { Udm udm = 1; } // BatchCreateUdmsRequest is the request for the BatchCreateUdms RPC. message BatchCreateUdmsRequest { repeated CreateUdmRequest requests = 1; } // BatchCreateUdmsResponse is the response for the BatchCreateUdms RPC. message BatchCreateUdmsResponse { // Udms created repeated Udm udms = 1; } // UpdateUdmRequest is the request for the UpdateUdm RPC. message UpdateUdmRequest { Udm udm = 1; optional google.protobuf.FieldMask update_mask = 2; } // BatchUpdateUdmsRequest is the request for the BatchUpdateUdms RPC. message BatchUpdateUdmsRequest { repeated UpdateUdmRequest requests = 1; // update_mask fields in requests list are ignored optional google.protobuf.FieldMask update_mask = 2; } message google.protobuf.FieldMask { repeated string paths = 1; } // BatchUpdateUdmsResponse is the response for the BatchUpdateUdms RPC. message BatchUpdateUdmsResponse { repeated Udm udms = 1; } // ListUdmsRequest is the request for the ListUdms RPC. message ListUdmsRequest { string key = 1; bool exact = 2; } // ListUdmsResponse is the response for the ListUdms RPC. message ListUdmsResponse { repeated Udm udms = 1; } // CountUdmsResponse is the response for the CountUdms RPC. message CountUdmsResponse { int32 count = 1; } // ListUdmKeysRequest is the request for the ListUdmKeys RPC. message ListUdmKeysRequest { string key = 1; bool exact = 2; } // ListUdmKeysResponse is the response for the ListUdmKeys RPC. message ListUdmKeysResponse { repeated string keys = 1; } // DeleteUdmRequest is the request for the DeleteUdm RPC. message DeleteUdmRequest { string key = 1; } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.elastiflow.com/flowcoll/overview/api-based-metadata-enrichment.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.