DATA Import Marc Authorities Testing - June 14, 2022
UIDATIMP-1128 - Getting issue details... STATUS
modules versions
- Inventory Module (mod-inventory-18.1.6)
- Inventory Storage Module (mod-inventory-storage-23.0.2)
- Inventory Update Module (mod-inventory-update-2.0.2)
- Source Record Manager Module (mod-source-record-manager-3.3.8)
- Source Record Storage Module (mod-source-record-storage-5.3.2)
- Data Import Module (mod-data-import-2.4.2)
- Data Import Converter Storage (mod-data-import-converter-storage-1.13.3)
| file | duration | time to create 1 record (sec) |
|---|---|---|
| 5K | 00:02:54.122 | 0.035 |
| 10K | 00:04:36.96 | 0.028 |
| 25K | 00:10:56.179 | 0.026 |
| 50K | 00:20:51.024 | 0.025 |
maximum number of record may be created in 30 min is 75000, without background activities (150000 in one hour)
and with maximum background activities is about only 190 records (CICO 20 users and DI on three tetants same time)
I guess the most efficient is data import authorities - long files, about 50K, since the performance is maximum and the probability of an error during download is still small.
multitenant test
test performed with file 25K all tetants run at the same time with background activities CICO 20 USERS
| fs09000000 | fs07000001 | fs07000002 |
|---|---|---|
| 00:43:06.658 | 00:43:51.385 | 00:45:18.735 |
time to create 1 record 0.11 sec
DI with background CICO one tenant
average response time CICO + DATA Import Marc Authorities 20users 1 tenant
| Check-In Controller | 1.610 |
| Check-Out Controller | 1.867 |
CICO without background activities
| Check-In Controller | 0.836 |
| Check-Out Controller | 1.022 |
Multitenant Authority Data Import> 2 kafka partitions
| file | fs09000000 | fs07000001 | fs07000002 |
|---|---|---|---|
| 5K | 00:07:36.579 | 00:07:39.64 | 00:08:10.303 |
| 10K | 00:14:50.992 | 00:12:56.94 | 00:14:21.905 |
| 25K | 00:40:39.063 | 00:42:00.941 | 00:42:21.044 |
| 50K | 01:37:59.688 | 01:38:11.591 | 01:14:52.07 |
one tenant > 2 kafka partitions
| file | duration |
|---|---|
| 5K | 00:02:03.027 |
| 10K | 00:02:52.886 |
| 25K | 00:11:18.37 |
| 50K | 00:25:24.663 |
one tenant > 1 kafka partition
| 5K | 00:02:37.534 |
| 10K | 00:04:09.538 |
| 25K | 00:10:11.571 |
| 50K |