Problem with Cluster message in Kibana

QUESTION

Why is the message "Problem with cluster" popping up on some dashboards?

ANSWER

When loading a dashboard in Kibana, you may encounter a pop-up with a message about response time or timeouts. If this happens, click the ‘View details’ link to gather more information.

Using the Inspector Window

The ‘View details’ link opens the Inspector window, which contains three detailed menus. These menus display useful information, including:

• Clusters and shards: Provides details about response times.

• Request: Shows the JSON content of the Elasticsearch request.

• Response: Displays response-related data.

For example, if the Clusters and shards menu shows a response time of 1928ms, this does not appear to be related to the default Elasticsearch query timeout of 30 seconds (30000ms). Instead, it may be related to value suggestion settings.

Inspecting the Request

In the Request menu, you can examine the JSON request. At the top of the JSON, you will find timeout-related settings. For example:

POST /elastiflow-flow-codex-*/_async_search?batched_reduce_size=64&ccs_minimize_roundtrips=true&wait_for_completion_timeout=200ms&keep_on_completion=false&keep_alive=60000ms&ignore_unavailable=true&preference=1737018105014
{
  "timeout": "1000ms",
  "terminate_after": 100000,
  ...
}

In this case:

• Timeout is set to 1000ms.

• Terminate_after is set to 100000.

Inspecting the Response

In the Response menu, look for values such as:

{
  "id": "Fko4RElDc3o0UjZlRnlBTU1veG1HZ3cgenVmUWI2SUlTYVdXbTd0OXRiaFU1UTo2MDI3NTQzMzU=",
  "rawResponse": {
    "took": 1928,
    "timed_out": true,
    ...
}

Here, you can see:

• The took value (1928ms), which matches the response time reported earlier.

• The timed_out field is set to true, indicating a timeout occurred.

Understanding Value Suggestion Timeouts

Kibana uses autocomplete.valueSuggestions to populate dropdown options in filters. By default:

• Timeout for value suggestions is 1000ms.

• Terminate_after is set to 100000.

These values are unrelated to the 30-second Elasticsearch query timeout but can cause issues if the response time exceeds the value suggestion timeout.

Adjusting Timeout Settings

To resolve this issue, you can increase the timeout for value suggestions. Edit the kibana.yml configuration file and add the following settings:

unifiedSearch.autocomplete.valueSuggestions.timeout: 4000
unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000

In this example:

• The timeout is increased to 4000ms, which is twice the response time (1928ms) observed in the example. You can adjust this value based on your environment.

• The terminate_after value remains at 100000.

NOTE: Older versions of Kibana my use different syntax, such as data instead of unifiedSearch. Check your version documentation.

Applying the Changes

1. Save the changes to /etc/kibana/kibana.yml.

2. Restart the Kibana service to apply the new settings:

3. sudo systemctl restart kibana

After applying these adjustments, monitor your dashboards to ensure the issue is resolved. Adjust the timeout further if needed based on observed response times.