> For the complete documentation index, see [llms.txt](https://docs.elastiflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.elastiflow.com/additional-resources-reference-articles/faq/flow_queue_90_full.md). # NetObserv Flow Queues 90% Full The NetObserv Flow’s log reports, `processor to output writer` or `UDP Server to Flow Decoder` are 90% full. ### BACKGROUND There are three queues of importance in ElastiFlow processing, two within ElastiFlow itself and one in the Linux kernel: 1. Output record queue 1. After ElastiFlow processing, waiting to be sent to the configured output 2. Can be monitored via the `record_queue_util` metric from the metrics API 2. Input packet queue 1. Retrieved by ElastiFlow from the Linux UDP receive buffer, and awaiting processing 2. Can be monitored via the `udp_server_packet_queue_util` metric from the metrics API 3. Linux UDP receive buffer 1. Packets received by the Linux UDP socket and awaiting the ElastiFlow application to pick them up 2. Queue depth and UDP receive buffer errors can be checked with the Linux command `ss -uman` As these queues get full, the following happens: * If the output queue fills, processing of records will slow to match the pace of the output * If packets are coming into ElastiFlow faster than they are being processed, the packet queue will fill. No further packets will be retrieved from the Linux UDP receive buffer * If packets are being received on the UDP socket faster than ElastiFlow retrieves them, the UDP receive buffer will fill. Once the buffer has reached its configured maximum size, any further packets will be dropped ### SYMPTOM NetObserv Flow’s log reports one or both of the following messages: ```json {"level":"info","ts":"2023-08-07T08:08:14.301Z","logger":"flowcoll","caller":"flowprocessor/metrics.go:118","msg":"flow processor to output writer is 90% full. This is normal when the collector is starting. If it persists for hours, it may indicate that you are at your license threshold or your system is under-resourced."} ``` ```json {"level":"info","ts":"2023-08-07T08:08:34.264Z","logger":"flowcoll","caller":"server/metrics.go:125","msg":"UDP Server to Flow Decoder is 90% full. This is normal when the collector is starting. If it persists for hours, it may indicate that you are at your license threshold or your system is under-resourced."} ``` These logs might also be accompanied by `throttler` logs: ```log 2023-06-28T21:20:21.821Z warn throttle/restricted_throttle.go:105 [throttler]: start burst 2023-06-28T21:20:41.822Z warn throttle/restricted_throttle.go:111 [throttler]: stop burst 2023-06-28T21:20:41.822Z warn throttle/restricted_throttle.go:117 [throttler]: start recovery 2023-06-28T21:50:42.142Z warn throttle/restricted_throttle.go:123 [throttler]: stop recovery ``` ### PROBLEM It is typical for these messages to occur when the collector first starts, as various internal processes may not yet be fully initialized. However, if the messages persist after the first few minutes, one of the following issues may exist: * ONLY `flow processor to output writer` - This indicates that the system to which data is being output lacks sufficient performance to ingest records at the rate being sent by the collector. This may be due to: * Insufficient CPU, memory, disk space, or excessive disk latency in the output system * Insufficient network bandwidth between the collector and output system might also cause the problem. (also see the NOTE below) * BOTH `UDP Server to Flow Decoder` and `flow processor to output writer` - This is a further progression of the previous condition. The resulting back pressure from the slow downstream system is now likely causing data to be lost. * ONLY `UDP Server to Flow Decoder` - The internal decoder/processor workers cannot keep up with the rate of records being received. This can be caused by one of the following conditions: * More records are being received than are allowed by the license. If so, `throttler` messages will also appear in the log. * The collector has insufficient resources, primarily CPU cores, to process the rate of records being received. * The collector has just been started and the caches (for IPs, interfaces, etc.) have yet to be "warmed up" and the related high latency enrichment tasks are limiting throughput. * An external service used in enrichment is slow to respond - for example DNS, SNMP, etc. {% hint style="info" %} \`6.x\` versions prior to \`6.3.4\`, had an issue with automatically scaling the output pool size for OpenSearch and Splunk based on the Licensed Units. Increasing the output pool size manually, via \`EF\_OUTPUT\_OPENSEARCH\_POOL\_SIZE\` or \`EF\_OUTPUT\_SPLUNK\_HEC\_POOL\_SIZE\` respectively, often solved the issue. Upgrading to \`6.3.4\` or later also fixes the issue. {% endhint %} ### SOLUTION The solution varies depending on the indicated issue, as described in the problem section above. * ONLY `flow processor to output writer` - Increase the performance of the system to which records are being sent. If no resource strain is present, tuning of the number of worker threads may be necessary * BOTH `UDP Server to Flow Decoder` and `flow processor to output writer` - Increase the performance of the system to which records are being sent. If no resource strain is present, tuning of the number of worker threads may be necessary * ONLY `UDP Server to Flow Decoder` * If `throttler` messages will also appear in the log, contact to learn about subscription options which will allow you to collector more flow records. * Increase the CPU cores available to the collector. * If the collector has sufficient CPU resources try increasing the processor pool size by setting [EF\_PROCESSOR\_POOL\_SIZE](/flowcoll/configuration/flow-processing.md). This allows great concurrency of high latency enrichment tasks. * If DNS enrichment is enabled and disabling DNS enrichment resolves the problem, the issue may be with DNS resolution time. Verify DNS server performance and DNS query performance. If lowering DNS timeout alleviates the symptoms, the issue would point to DNS server performance and not collector performance. ### REFERENCE * [EF\_PROCESSOR\_POOL\_SIZE](/flowcoll/configuration/flow-processing.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.elastiflow.com/additional-resources-reference-articles/faq/flow_queue_90_full.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.