Uploaded image for project: 'Ibexa IBX'
  1. Ibexa IBX
  2. IBX-719

Allow SearchService to iterate over large resultsets

    XMLWordPrintable

Details

    • Icon: Story Story
    • Resolution: Done
    • Icon: Medium Medium
    • 4.0.0-alpha1, 3.3.7
    • None
    • Open Source
    • None
    • [4.0] - Sprint 14

    Description

      While working on Migration Bundle, we discovered that current SearchService API does not allow iterating over resultset, i.e. forces loading all rows returned from Doctrine query into PHP's memory.

      Doctrine already allows iterating over each row one at a time and it is specifically used for batch processing of entities: https://www.doctrine-project.org/projects/doctrine-orm/en/2.7/reference/batch-processing.html#iterating-results

      This will allow operations that require iterating over whole database contents to become faster, since it would limit the number of SQL queries that are required. This is true regardless of whether the query in question would make use of indices. In fact, it is a lot more impactful in cases where sequential table scan is needed by database to present results.

      In cases like Migration Bundle it will prevent query collisions from occuring: long running migration generation commands run into risk of some records being omitted or duplicated depending on mid-operation database changes.

      Designs

        Attachments

          Activity

            People

              Unassigned Unassigned
              pawel.niedzielski@ibexa.co Paweł Niedzielski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: