PERF-212 - Getting issue details... STATUS
AsyncMigration API
An asynchronous mechanism has been added to migrate large amounts of data. Additional API has been added to the mod-inventory storage (link).
For now, this API supports "publicationPeriodMigration" and "itemShelvingOrderMigration", but it can be extended to support another kind of migrations very simply.
To retrieve a list of available migrations call GET /inventory-storage/migrations
Response
{
"asyncMigrations": [
{
"migrations": [
"publicationPeriodMigration"
],
"affectedEntities": [
"INSTANCE"
]
},
{
"migrations": [
"itemShelvingOrderMigration"
],
"affectedEntities": [
"ITEM"
]
}
],
"totalRecords": 2
}
To start migration use POST /inventory-storage/migrations/jobs
Request
{"migrations": ["itemShelvingOrderMigration"]}
To start several migrations in the scope of one job it’s possible to run a job with several migrations. In this case, each migration should have its own counter, but status and progress will be tracked on job level.
Request
{"migrations": ["itemShelvingOrderMigration", "publicationPeriodMigration"]}
To check status or cancel job use GET | DELETE /inventory-storage/migrations/jobs/{id}
GET Response
{
"id": "aa5a11b4-88c9-4372-82fb-7089b8b37c0f",
"migrations": [
"itemShelvingOrderMigration"
],
"affectedEntities": [
"ITEM"
],
"published": [
{
"migrationName": "itemShelvingOrderMigration",
"count": 20
}
],
"processed": [
{
"migrationName": "itemShelvingOrderMigration",
"count": 20
}
],
"jobStatus": "Completed",
"submittedDate": "2022-03-21T11:01:24.890+00:00",
"finishedDate": "2022-03-21T11:01:25.311+00:00"
}
Performance tests results for 1 node
Environment setup
Env: https://falcon.ci.folio.org/
Number of records: 1_000_000
Mod-Inventory-Storage setup: 1 node, resources:
limits: cpu: 192m memory: 864Mi requests: cpu: 128m memory: 536Mi
Results
{
"id": "8202b79d-11eb-44ce-aefa-f4489a20d3fb",
"name": "publicationPeriodMigration",
"affectedEntities": [
"INSTANCE"
],
"published": 1000000,
"processed": 1000000,
"jobStatus": "Completed",
"submittedDate": "2022-02-18T16:47:25.082+00:00",
"finishedDate": "2022-02-18T19:34:22.092+00:00"
}
Result: 2 hours, 47 minutes and 17 seconds
Performance tests results for 2 nodes
Environment setup
Env: https://falcon.ci.folio.org/
Number of records: 1_000_000
Mod-Inventory-Storage setup: 2 nodes, resources:
limits: cpu: 192m memory: 864Mi requests: cpu: 128m memory: 536Mi
Results
{
"id": "cb210eba-4fa3-4d59-8cda-9c15640e4138",
"name": "publicationPeriodMigration",
"affectedEntities": [
"INSTANCE"
],
"published": 1000000,
"processed": 1000000,
"jobStatus": "Completed",
"submittedDate": "2022-02-18T20:01:27.062+00:00",
"finishedDate": "2022-02-18T21:29:54.003+00:00"
}
3 Comments
Craig McNally
Craig McNally
Given that this async migration process is currently unique to this module, it's probably worth documenting how this works in the README, or at least linking out to supporting documentation from the README.
Oleksii Kuzminov
Craig McNallyThanks for the feedback. "Published" means the number of entries that have been successfully published to the Kafka topic. This document can be added to the readme, but as I understand it, this is not a common case for mod-inv-storage, but the API documentation is in place. If you think we need to add additional information to the readme file or rename something, please let me know or create a JIRA ticket.