Uploaded image for project: 'Ibexa IBX'
  1. Ibexa IBX
  2. IBX-2015

Elasticsearch - amount of shards grows quickly to the top limit

    XMLWordPrintable

Details

    • [4.3] - Sprint 5
    • Ibexa Content

    Description

      Steps to reproduce:

      1. Install clean installation of Ibexa DXP v3.3.
      2. Use Elasticsearch as a search engine (https://doc.ibexa.co/en/3.3/guide/search/elastic/#step-1-download-and-install-elasticsearch).
      3. After reindexing the content access http://localhost:9200/_cat/shards.
      4. Observe the result.
      5. Add some languages, content types, contents in different parts of the content tree.
      6. Observe the shards under http://localhost:9200/_cat/shards.

      Result:
      There are plenty of shards with a really low amount of space taken, including UNASSIGNED shards which are treated as standard shards when reaching the shard limit in a single node.

      Having rather a medium project with multiple languages can quickly reach shard limit (set to 1000 as default) to the maximum which would prevent indexing any further content without any error.

      If we debug it deeper during content indexing we will see the following errors returned by Elasticsearch API:
      Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open. During the indexing, the user is not informed about this and he can be unaware that some contents were not indexed.

      Before version 7 of Elasticsearch, the default number of shards per index was set to 5. This limit has been lowered to 1. This shows that one should keep a smaller amount of shards due to performance reasons (a higher number of shards means much higher memory consumption when compared to the smaller number of shards of much larger size). Increasing the shards limit is not a way to go according to elastic's good practices.

      Expected result:
      The number of shards should be kept to minimum.

      Designs

        Attachments

          Activity

            People

              Unassigned Unassigned
              bartlomiej.wajda@ibexa.co Bartłomiej Wajda
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: