When using a managed Kibana dashboard on AWS Elasticsearch, it can take a lot of the heavy lifting and complexity out of setting up your own Kibana dashboard for log aggregation, however you may find that one day your dashboard stops showing results and displays a “☹ No Results Found” error for all your Kibana dashboard insights.
A good place to start troubleshooting this issue with your dashboard is within the AWS dashboard for the ElasticSearch service. When on the management page for your elasticsearch cluster, you will be able to use the “Cluster Health” tab to investigate the health of your cluster.
In our example, our luster health is showing a green status for all sections of the cluster, except for “Cluster writes status (ClusterIndexWritesBlocked)” which is showing a Red status, indicating an issue. This error generally indicates an issue with one of the following:
- Storage Space
- JVM Memory Pressure
- CPU Usage (high usage)
Another thing we can notice from this screen (in the above diagram) we can see that “Total free storage space” is showing 0 space across the cluster. As mentioned above, storage space can be a cause of writes blocked issues.
To verify the above storage issue, we can use the “Instance Health” tab, to dig deeper. On this menu we can verify the following:
- JVM Memory Pressure/Usage is not high (not causing our issue)
- CPU Utilization is not high (not causing our issue)
- Free Space on our data instances is showing “0” free. (confirming storage space may be our issue)
In order to resolve our storage space issue, we have 2 options depending on your requirements such as data retention or costings;
Option 1 – Increase Volume Size
Increase the volume size of our elasticsearch cluster, this can be a short-term fix as the storage may continue to fill until the issue re-occurs. It will also slightly increase the costs of the cluster as more storage will be provisioned. To increase the storage size;
Navigate to your elasticsearch domain within the AWS portal and click “Configure Cluster”
In the Configure Cluster page, scroll down to “Storage Configuration”
Increase the amount of storage you have provisioned for your cluster, once you have updated this value, scroll to the bottom and click on “Submit” to apply your changes. It may take some time for the changes to apply.
Option 2 – Clear Old Search Data
Clear old search data that is no longer required. This can be done selectively and also can be configured to automatically cycle data to avoid your storage filling up (using scripts or the curator function). To clear out old data within your Kibana dashboard;
Navigate to your Kibana cli console, on our version, this can be found in the developer section of our Kibana management window.
First we need to run a command to list all of the data sets our instances have stored:
This will list all of the current indices that are being used and the amount of storage space each one is using. For our example, I can see there are some loaded for “July 2019”, I do not need any information from this early in the year and will delete all July results.
The next command we will issue will clear all July indices from the cluster. To do this, we could delete values one by one by using the full name of the indices such as:
DELETE cwl-2019-07-14 DELETE cwl-2019-07-25
The above can be selective but can take a while to remove all values from a month or year. In order to remove all of July’s indices we can use the wildcard character “*”. For example, to delete all of July’s results I am going to use:
We could even delete all data from 2019 by using:
Click the play button next to the command console once you have entered your data matching pattern to delete data. Once complete, you should see “acknowledged : true” in the results window to the right.
If we re-run our command to check what indices are in the cluster, we can now see that all of the data for July is no longer shown in the list. This has now been removed.
We can check our storage space from the Kibana console using the command:
The above shows that there is now free space on the cluster instances, we can also navigate to our AWS dashboard to check that not only free space is showing in the dashboard but that “Cluster Index Writes Status” (ClusterIndexWritesBlocked) is no longer showing red.
Our Instance Health is showing similar to our Kibana command output that there is free space now available:
Looking at our Cluster Health, we can see that the free space metric has now jumped up (while amount of searchable data has dropped) and that our Cluster Index Writes status has begun to show green.
For the real check, if we navigate to our Kibana dashboard, we can see that all our data graphs and search results are now working correctly.