Folio Development Teams
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

IN PROGRESS

Overview


In the scope of PERF-524 it's needed to run tests to answer questions: 

  • Can the current System accommodate an average load? Load model described here NLA load model investigation and creation.
  • What happened at peak times when all workflows are running at once? 
  • Typical KPIs:
    • Service CPU
    • Service Memory
    • DB CPU
    • DB Memory
  • Response times
  • Duration of long workflows
  • Recommendations to improve on scaling up/out modules to accommodate peak times

Summary


Recommendations & Jiras (Optional)

Jiras


Test Runs & Results

Test #

# configuration

Test durationcomments
1All workflows started at the same time1 hourRes 500 ERROR during Data Import process
2Data import and data export started with 15 min delay1 hourRes 500 ERROR during Data Import process
3Data import and data export started with 1 min delay1 hourRes 500 ERROR during Data Import process
4Data import started with 1 min delay and data export started with 20 min delay35 minRes 500 ERROR during Data Import process
5Same as Test# 4 Jenkins configuration with increased DB_MAXPOOLSIZE = 200 for mod_srs and mod_srm30 minRes 500 ERROR during Data Import process. Data import duration decreased 2 times.
6Test without Data import1 hourNO errors

 Test results from 1st test run (1st, 2nd and 3rd test run results are similar):

Test #

Workflow name 

Total time it takes to complete workflow

Comment


Avg With DI95th pct with DIAvg no DI95th pct no DI
1Checkin

1.5911.054
2Checkout

1.9481.650
3View invoices

0.7630.913
4Create invoices

1.1741.370
5Edit invoices

1.5811.897
6Delete invoices

0.8040.927
7Approving Invoices

1.4531.940
8View Authority records

0.2890.381
9View MARC tag table

0.9871.284
10View holdings records

1.5261.922
11View Bib

0.8411.168
12View patron records

0.5660.883
13Delete patron records

0.6381.070
14Update patron records

1.0431.625
15Create patron records

1.0981.261
16View Ledger

0.0500.088
17Create ledger

0.6160.761
18Edit ledger

0.0540.085
19Delete a ledger

0.0460.080
20Export bib "Default instances export job profile"

--
21Export holdings "Default holdings export job profile"

--
22Export authority records "Default authority export job profile"

--

DI "DISC HRID match"

--

DI "DS LA edeposit records update"

--

DI "DISC New edeposit records"

--

DI "DISC New NON edeposit records"

--

View item records

1.2891.649

update item records

0.9981.250

delete item records

0.9271.099

Monitoring Pick Slips and Requests GET /circulation/requests

0.3590.480

Monitoring Pick Slips and Requests GET /circulation/pick-slips/

0.1120.256

Monitoring Pick Slips and Requests

0.3030.303

Users loan renewal

1.4671.661

Item-level requests

0.6690.973

View vendor records

0.7131.165

Edit vendor records

5.1996.190

Create vendor records

1.0641.200

Delete vendor records

0.4120.522

Create purchase orders

1.6251.733

View purchase orders

1.2051.435

Edit purchase orders

2.0762.984

Delete purchase orders

1.4321.830

Retrieving instances and holdings

0.0350.073

Edit MARC tag table

3.4244.257

Fiscal close - end of FY rollover





Blacklight: View an inventory record JMeter script



0.8211.042

Blacklight: Create a Request JMeter script

1.1221.404

Blacklight: Create a View Patron record JMeter script



0.0730.110

Memory Utilization

Memory utilization:

  • mod-source-record-manager
  • mod-source-record-storage
  • mod-inventory-storage


All other modules behave stable during Data Import.

*This test was performed after a run of 2 sets of the same jobs (1k, 5k, 10k, 22.7k, 50k records twice)



Service CPU Utilization 

*On chart below - each little spike corresponds to each DI job performed. 

**Some of spikes is shorter than the others - because of differences in number of records imported.

**Test #1 has higher CPU usage because it has background activities (CICO 5 users + DI )



Most CPU-consuming modules: 

  • mod-quick-marc - 79%
  • mod-source-record-storage - 74%
  • mod-inventory - 69%
  • mod-source-record-manager - 67%
  • others - usage less than 30%


Instance CPU Utilization


RDS CPU Utilization 

Predictable that each DI job is consuming a lot of DB CPU (each spike here corresponds to each DI job).

Approximately DB CPU usage is ± 96%

Appendix

Infrastructure

PTF -environment ncp3 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
  • Kafka topics partitioning: 
    • DI_RAW_RECORDS_CHUNK_READ -2 
    • DI_RAW_RECORDS_CHUNK_PARSED -2
    • DI_PARSED_RECORDS_CHUNK_SAVED -2
    • DI_SRS_MARC_AUTHORITY_RECORD_CREATED -2
    • DI_COMPLETED -2


Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-data-import2.7.181256204818445121292
mod-di-converter-storage2.0.2521281024896128768
mod-source-record-storage5.6.52421024409636885123076
mod-source-record-manager3.6.21421024409636885123076
mod-inventory-storage26.0.01021024220819523841440
mod-inventory20.0.4821024288025925121814

Methodology/Approach

To test Baseline DI and DI with CICO 5 concurrent users the JMeter scripts were used.

Multitenant testing

  • test 1-5: testing DI on each tenant consecutively (5 jobs from 3 tenants = 15 test runs)
  • test 6-8: testing DI jobs from two tenants simultaneously with 1 min ramp-up.
  • test 9: testing DI jobs from 3 tenants simultaneously with 1 min ramp-up.






  • No labels