POC: Export records using CQL


Content:


PURPOSE:

MDEXP-281 - Getting issue details... STATUS

Explore approaches for export inventory instances by using a file with .cql extension with CQL query content.

OPTIONS:

There are two options, how we can use CQL query to export records:

  1.  Use the CQL to  fetch all the record`s that match a query from inventory;
  2.  Use the CQL to fetch all the record`s UUIDs, that match a query from inventory;
  • Use the CQL to  fetch all the records that match a query from inventory:

Advantages:

  1. We don`t need to store a file with UUIDs while using this approach;
  2. Reduced calls to inventory;

Disadvantages:

  1. There is no way to show to the user the correct progress bar because for now, inventory returns not proper total records count.
  2. This approach requires a lot of changes in code and the whole architecture of the data-export module;
  • Use the CQL to fetch all records UUIDs, that match a query from inventory:

  Advantages:

  1. This approach requires minimal changes in code;
  2. While using this option, we will use the existing logic for calculating progress;

Disadvantages:

      1. Compared with existing logic, we will have to make 1 additional call to instance-bulk/ids endpoint to fetch record`s UUIDs;

CQL query sanitizing in mod-data-export

We are using the public API of mod-inventory-storage, in my opinion, we should not do any work at our side to sanitize the CQL query. However, if the user will send '*' as a query, the process of getting all instance`s UUIDs for the second option could be very long, it will hit the performance. In such a case, we can change the instance-bulk/ids endpoint in mod-inventory-storage, and add a query limit and offset, then collect the responses by using them.

CONCLUSION:

After analyzing both approaches, I came to the conclusion that it is better to use the second option.