Combined test for data export worker v2.0.3- (Nolana) 2022-12-01

Owned by Olga Kondratenko
Last updated: Dec 13, 2022
5 min read


Overview

Per PERF-329 - Getting issue details... STATUS the following tests were performed using the data export workflow (bulk edits, eHoldings, circulation log, and edifacts) to determine what load can the exports run successfully together 

Infrastructure

PTF -environment 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 AWS region (comparing to 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region for Nolana 2.1.0-SNAPSHOT.85)
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-data-export-spring1.5.021256204818445121536 (1236 in MG)
mod-data-export-worker2.0.3621024307226005122048 (1536 in MG)
mod-users19.0.012128 (258 in MG)1024896128768
mod-notes4.0.0121281024896128m322 (768m in MG)
mod-inventory19.0.112102428802592512m1814m

mod-inventory-storage

25.0.112102422081952512m1440m
okapi4.14.713102416841440512m922m
mod-orders12.5.012102420481440512m896m
mod-orders-storage13.4.0121281024896128m768m
mod-circulation23.3.01210241024896128m768m
mod-circulation-storage15.0.012102415361440512m896m
mod-agreements5.4.01212815921488512m968m
nginx-okapinginx-okapi:2022.03.02121281024896--

MG- Morning Glory release

Summary

Test report for combined mod-data-export-worker functionality 2022-12-01. 

  •  24 different concurrent jobs can be performed successfully in about 19 minutes (time to process the longest one - 2 jobs -10k item records).
  • Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules.
  • CPU - nginx-okapi, nod-configuration, and mod-orders-storage is spiking up to 250% at the start of tests. For mod-users-180%, mod-notes-130%, and mod-agreements-200%. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
  • For all tests - RDS CPU utilization did not exceed 77%.

In general, Nolana is more stable compared to the Morning Glory release. All were tested with a high load. For PERF-335 - Getting issue details... STATUS , it can be reproduced only if the load on mod-data-export-worker is high and has a solution UIBULKED-182 - Getting issue details... STATUS for Nolana release.

Results

Test Runs (initial assumption of the data sets for combo testing)

Test Run

EdifacteHoldings

Bulk-edit

user barcodes

Bulk edit

item barcodes

Bulk edit

Holdings HRIDs

BursarCirculation logMemory Behavior (Trend)CommentTime to process Results

#1

10 jobs 1K orders each7 jobs 9,194 titles each1 job -2500 records1 job -10k records 1 job -10k recordsDNR35K records
memory usage increases for mod-data-export-worker during the test from 38% up to 58%. No leak suspect was found in the heap dump. Stable for all other modules.		21 jobs totalAbout 18 min (time to process the longest one - job -10k item records)21 successful. 2 eHolding jobs were scheduled for 1 min 45 sec.

#2

10 jobs 1K orders each7 jobs 9,194 titles each2 jobs -2500 records each2 jobs -10k records each2 jobs -10k records eachDNR35K records24 jobs totalAbout 19 min (time to process the longest one - 2 jobs -10k item records)

24 successful, 2 bulk edits scheduled with
PERF-335 - Getting issue details... STATUS , 2 eHoldings were scheduled for 2 min and 5 min.

#3

10 jobs 1K orders each7 jobs 9,194 titles each2 jobs -2500 records each2 jobs -10k records each2 jobs -10k records eachDNR35K records24 jobs totalAbout 19 min (time to process the longest one - 2 jobs -10k item records)

24 successful, 2 bulk edits scheduled with
PERF-335 - Getting issue details... STATUS for up to 2 min, 2 eHoldings were scheduled up to 2 min.

#4

1 job -2500 records1 job -10k records 1 job -10k records 


About 18 min (time to process the longest one - job -10k item records)all successful. 1 eHolding and 2 Edifact jobs were scheduled for 1 min 45 sec.

Test results

Test Run

Edifact

Time to process

v2.0.3 

Time to process Nolana's snapshotResultseHoldings

Time to process 

v2.0.3

Time to process Nolana's snapshotResults

Bulk-edit

user barcodes

Time to process 

v2.0.3

Time to process Nolana's snapshotResults

Bulk edit

item barcodes

Time to process 

v2.0.3

Time to process Nolana's snapshotResultsComment
#110 jobs 1K orders each1 m 20 s1 min 40 ssuccess7 jobs 9,194 titles each4 m 55s±10 minsuccess1 job -2500 records±2 min3 min 45 secsuccess1 job -10k records17 min 30 sec26-27 minsuccess


all successful, bulk edits scheduled with
PERF-335 - Getting issue details... STATUS for up to 2 min, eHoldings were scheduled for up to 5 min.

#210 jobs 1K orders each2 m 20 s1 min 50 ssuccess7 jobs 9,194 titles each5 m 35 s±12 minsuccess2 jobs -2500 records each2min 20 sec2 min 10 sec for a successful job

success

2 jobs -10k records each±19 min28 min 30 secsuccess
#310 jobs 1K orders each2 m 5 s1 min 37 ssuccess7 jobs 9,194 titles each5 m 31 s±12 minsuccess2 jobs -2500 records each2min 20 sec2 min 50 sec for a successful job

success

2 jobs -10k records each±19 min28 min 30 secsuccess
#4







1 job -2500 records2 min 15 sec3 min 45 secsuccess1 job -10k records18 min26-27 minsuccess

Test Run

Bursar

Time to process 

v2.0.3

Time to process Nolana's snapshotResults

Circulation log

Time to process

v2.0.3 

Results

Bulk edit

Holdings HRIDs

Time to process 

v2.0.3

Time to process Nolana's snapshotResultsComment
#1DNRDNR
DNR35K records5 m 15ssuccess1 job -10k records±7 min-success
#2DNRDNR
DNR35K records5 m 38 ssuccess2 jobs -10k records each±8 min-

success


#3DNRDNR
DNR35K records5 m 3 ssuccess2 jobs -10k records each±8 min-

success


#4






1 job -10k records7 min 30 sec-success

 * - average job processing time for the number of records specified ( upload identifiers + editing)


Resource Usage






RDS CPU utilization 







Test Run #1

Memory Trends of combined test:

  • Memory trend: memory usage increases for mod-data-export-worker during the test from 38% up to 49%. No leak suspect was found in the heap dump. Stable for all other modules.

CPU usage of combined test:



Instance CPU usage of combined test:


RDS CPU usage of combined test:



Additional information:

For 12 concurrent bulk edit jobs (4- 10k Items records, 4 - 2,5k users records, 4- 10k holdings records) all items and holdings jobs FAILED with response 500 or response 502

Example of error details:

[500 Internal Server Error] during [GET] to [http://inventory/items?query=barcode%3D%3D%22H79118086%22&limit=1] [InventoryClient#getItemByQuery(String,long)]: [Holdings request (id==(332c88e0-e712-4cec-a3a4-f7a620c4f2e7)) failed 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
] (InternalServerError)
[502 Bad Gateway] during [GET] to [http://inventory/items?query=barcode%3D%3D%22H97502733%22&limit=1] [InventoryClient#getItemByQuery(String,long)]: [<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
] (BadGateway)
ncp3/mod-inventory/e52839606c944c82b93fc1154d7a3a8b
13:37:01 [] [] [] [] ERROR Items                Failed to retrieve bound-with parts, status code: 502


13:37:01 [] [] [] [] INFO  ?                    108095/bulk-edit;025851/instance-storage RES 502 2937us mod-inventory-storage-25.0.1 http://mod-inventory-storage-b.ncp3.folio-eis.us-east-1:8051/mod-inventory-storage/instance-storage/instances/35793b16-c6a1-48f7-9c2e-32670133dbbf