Skip to end of metadata
Go to start of metadata

















Overview

  1. In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - PERF-225 - Getting issue details... STATUS

We tested with 1, 20 virtual users against 10 million Inventory records. 

       Test plan:

  1. Run reindex for tenant A.
  2. Run test with n users for tenant B


Backend:

  • mod-search-1.6.4
  • mod-inventory-18.1.4
  • mod-inventory-storage-23.0.2
  • okapi-4.13.0

Frontend:

  • folio_inventory-9.0.1

Environment:

  • 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
  • 80 FOLIO back-end modules deployed in 166 ECS services
  • 3 okapi ECS services
  • 7 m5.xlarge  EC2 instances
  • writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

  • authority key search
  • boolean search
  • contributors search
  • filter search
  • keyword search
  • subjects search
  • title search
  • wildcard search
  • wildcard prefix search


mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

Query1 user - Baseline (avg)1 user - Under the reindexing load (avg)20 user - Baseline (avg)20 user - Under the reindexing load (avg)
Regular336 ms448 ms728 ms0.850 ms
Wildcard1.42 seconds1.63 seconds2.32 seconds2.71 seconds (2 failed requests for  /search/instances: title all *1randomword)

Show total requests processed increases as we increase the number of users concurrently.


1 user - Baseline (avg) ms1 user - Under the load (avg) ms20 user - Under the load (avg) ms20 user - Under the load (avg) ms
Total requests processed in 60 minutes test run1071580516924459420

Comparison with Kiwi


1 user1 user reindexing20 users20 users reindexing

Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)
Regular64833656944831907287009850
Wildcard queries660268561159418569811160832
Wildcard prefix queries134411740115692657174243624220094212



1 user1 user reindexing20 users20 users reindexing

PercentageTimes better than KiwiPercentageTimes better than KiwiPercentageTimes better than KiwiPercentageTimes better than Kiwi
Regular Delta48.15%121.27%177.18%487.87%8
Wildcard queries Delta59.39%271.66%383.32%592.54%13
Wildcard prefix queries Delta87.05%777.03%479.20%480.86%5

CPU
MemoryNumber of Tasks

KiwiLotusKiwiLotusKiwiLotus
mod-search256256536144021
okapi25612853654433
mod-authtoken12812836032222
mod-permissions12812853653722
Elastic SearchNo differences

Test Runs

1 User without any background reindexing

  • runtime 60 minutes


RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence2330.0650.2500.2640.2980.5200.7210.9730.3020.520
fs09000000GET /search/authorities keyword <> randomword2330.0650.2460.2610.2890.5480.8101.7710.3120.548
fs09000000GET /search/authorities keyword = *random sentence*2330.0650.5670.7100.9621.6812.2402.5820.8741.681
fs09000000GET /search/authorities keyword = random sentence2330.0650.0170.0210.0230.0340.0480.0740.0230.034
fs09000000GET /search/authorities keyword = randomword2330.0650.0140.0240.0560.1350.2040.4490.0440.135
fs09000000GET /search/authorities keyword == random sentence2330.0650.0170.0200.0220.0350.0550.1010.0220.033
fs09000000GET /search/authorities keyword == randomword2330.0650.0150.0220.0340.0890.1640.2410.0340.089
fs09000000GET /search/authorities keyword all *randomword*2330.0650.3020.5540.8241.1981.6752.0130.6441.198
fs09000000GET /search/authorities keyword all random sentence2330.0650.0170.0210.0250.0400.0760.2300.0250.040
fs09000000GET /search/authorities keyword all randomword2330.0650.0250.0470.0890.1580.2700.3790.0670.158
fs09000000GET /search/authorities keyword any random sentence2330.0650.0170.0920.1290.2000.3150.3540.0980.199
fs09000000GET /search/authorities keyword any randomword2330.0650.0140.0280.0620.1450.1970.2280.0460.145
fs09000000GET /search/instances: contributors <> 2random words2330.0650.2550.2740.2960.4220.7521.5990.3020.420
fs09000000GET /search/instances: contributors = *randomword2330.0650.4490.4810.5700.9431.3571.7900.5620.942
fs09000000GET /search/instances: contributors = randomword*2330.0650.0130.0410.1600.2190.2420.2760.0840.218
fs09000000GET /search/instances: contributors == 2random words2330.0650.0130.0200.0260.0400.0690.0780.0230.039
fs09000000GET /search/instances: contributors all 2random words2330.0650.0140.0170.0240.0380.1360.2110.0230.038
fs09000000GET /search/instances: contributors all randomword2330.0650.0140.0890.1790.2400.3640.4410.1120.238
fs09000000GET /search/instances: contributors any 2random words2330.0650.0170.1520.1960.2400.3900.6220.1390.239
fs09000000GET /search/instances: keyword <> 1word2330.0650.2580.2820.3240.4890.7091.2190.3230.488
fs09000000GET /search/instances: keyword <> random 2words2330.0650.2590.2810.3300.5700.7250.8030.3260.569
fs09000000GET /search/instances: keyword = 1word*2330.0650.0180.1700.2070.2740.4560.5910.1590.273
fs09000000GET /search/instances: keyword = random 2words2330.0650.0160.0510.0820.1530.2990.3130.0630.153
fs09000000GET /search/instances: keyword == random 2words2330.0650.0160.0280.0410.0820.1220.3700.0370.082
fs09000000GET /search/instances: keyword all random 2words2330.0650.0180.0590.0960.1980.3970.4340.0800.197
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang2330.0650.0200.0750.1210.1940.2890.4620.0930.194
fs09000000GET /search/instances: keyword any random 2words2330.0650.0490.2030.2370.3430.5991.2010.2200.342
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")2330.0650.0430.0570.1720.2370.3770.4530.1050.236
fs09000000GET /search/instances: languages == lang2330.0650.0430.0540.1480.1850.3580.4610.0940.184
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"2330.0650.0470.0700.2130.2440.4940.6180.1270.243
fs09000000GET /search/instances: subjects <> 1randomword2330.0650.0150.1830.2190.2590.3570.6970.1690.258
fs09000000GET /search/instances: subjects = (randomword OR randomword)2330.0650.0230.1730.2160.2580.4070.4530.1670.257
fs09000000GET /search/instances: subjects = *1randomword2330.0650.6470.7430.8721.3272.3032.9160.8501.325
fs09000000GET /search/instances: subjects = 1randomword*2330.0650.0130.1160.1580.2100.2700.2890.1040.209
fs09000000GET /search/instances: subjects == 1randomword2330.0650.0140.0540.1090.2010.2450.3900.0810.199
fs09000000GET /search/instances: subjects all 1randomword2330.0650.0150.0720.1600.2290.3040.3530.0990.227
fs09000000GET /search/instances: subjects all randomword AND title all randomword2330.0650.0150.0490.0730.1300.2450.3750.0590.130
fs09000000GET /search/instances: subjects all randomword NOT title all randomword2330.0650.0150.1870.2360.3340.5140.7450.1820.333
fs09000000GET /search/instances: subjects all randomword OR title all randomword2330.0650.0260.2130.2560.3250.4610.5280.2140.324
fs09000000GET /search/instances: subjects any 2random words2330.0650.0160.1320.1760.2450.4590.8020.1410.243
fs09000000GET /search/instances: title <> 3 word sentence2320.0650.2600.2820.3140.4990.6920.7510.3200.498
fs09000000GET /search/instances: title == 3 word sentence2320.0650.0170.0310.0500.1100.1940.2300.0430.110
fs09000000GET /search/instances: title all *1randomword2320.0654.6025.8118.43811.23414.75015.8336.88611.234
fs09000000GET /search/instances: title all 1randomword*2320.0650.0170.0850.1740.2360.3800.6220.1170.234
fs09000000GET /search/instances: title all 3 word sentence2330.0650.0200.0660.1150.1880.2640.2980.0840.188
fs09000000GET /search/instances: title any 3 word sentence2330.0650.1160.3130.6901.7472.4292.8330.5851.745



Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.0840.0%
fs09000000GET /search/instances: keyword = 1word*0.1590.0%
fs09000000GET /search/instances: subjects = 1randomword*0.1040.0%
fs09000000GET /search/instances: title all 1randomword*0.1170.0%

0.2676
Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*0.8740.0%
fs09000000GET /search/authorities keyword all *randomword*0.6440.0%
fs09000000GET /search/instances: contributors = *randomword0.5620.0%
fs09000000GET /search/authorities keyword all *randomword*0.6440.0%
fs09000000GET /search/instances: subjects = *1randomword0.850.0%
fs09000000GET /search/instances: title all *1randomword6.8860.0%

1.74
  • Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

  • Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.


  • Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

CPU and Memory Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

mod-search memory is stable around 40% throughtout the test run


1 User with background reindexing

  • runtime 60 minutes


Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18% 

RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence1750.0490.0880.2210.3390.5570.6851.1040.2630.557
fs09000000GET /search/authorities keyword <> randomword1750.0490.0870.1580.2890.5050.5921.0690.2160.505
fs09000000GET /search/authorities keyword = *random sentence*1750.0490.5700.9171.2031.7082.3373.7681.0131.708
fs09000000GET /search/authorities keyword = random sentence1750.0490.0170.0210.0280.0460.0670.1060.0260.046
fs09000000GET /search/authorities keyword = randomword1750.0490.0150.0280.0540.1400.1860.3420.0460.140
fs09000000GET /search/authorities keyword == random sentence1750.0490.0170.0210.0290.0680.1670.2100.0290.068
fs09000000GET /search/authorities keyword == randomword1750.0490.0150.0260.0400.1500.2290.2710.0420.149
fs09000000GET /search/authorities keyword all *randomword*1750.0490.3110.6570.8391.2691.4903.5620.7201.269
fs09000000GET /search/authorities keyword all random sentence1750.0490.0170.0260.0350.0730.1370.1980.0330.073
fs09000000GET /search/authorities keyword all randomword1750.0490.0250.0470.1200.1950.2790.6520.0830.195
fs09000000GET /search/authorities keyword any random sentence1750.0490.0180.1450.1900.3420.4080.4250.1490.342
fs09000000GET /search/authorities keyword any randomword1750.0490.0140.0300.0900.2080.2440.3870.0620.208
fs09000000GET /search/instances: contributors <> 2random words1750.0490.1440.3130.3870.6680.7730.9690.3390.667
fs09000000GET /search/instances: contributors = *randomword1750.0490.4600.7390.9401.4452.0642.2450.8301.444
fs09000000GET /search/instances: contributors = randomword*1750.0490.0140.0770.2130.3130.3740.4070.1190.312
fs09000000GET /search/instances: contributors == 2random words1750.0490.0140.0270.0360.0610.1090.2080.0320.061
fs09000000GET /search/instances: contributors all 2random words1750.0490.0140.0200.0280.0480.0580.0720.0250.048
fs09000000GET /search/instances: contributors all randomword1750.0490.0150.1430.2570.3250.4210.6070.1590.325
fs09000000GET /search/instances: contributors any 2random words1750.0490.0190.2050.2720.3810.4370.5930.1980.379
fs09000000GET /search/instances: keyword <> 1word1750.0490.1440.3240.4290.6530.7691.0730.3520.652
fs09000000GET /search/instances: keyword <> random 2words1750.0490.1490.3160.4180.6901.0371.1770.3650.688
fs09000000GET /search/instances: keyword = 1word*1750.0490.0180.2280.2670.3770.5870.7990.2170.376
fs09000000GET /search/instances: keyword = random 2words1750.0490.0170.0720.1230.1980.2710.4210.0870.198
fs09000000GET /search/instances: keyword == random 2words1750.0490.0170.0320.0740.1780.3030.3450.0600.178
fs09000000GET /search/instances: keyword all random 2words1750.0490.0190.1060.1560.2710.3600.4350.1250.271
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang1750.0490.0480.1690.2260.3060.4920.8070.1810.306
fs09000000GET /search/instances: keyword any random 2words1750.0490.0540.2820.3720.5600.7961.0310.3240.559
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")1750.0490.0450.0810.1830.2840.4080.4410.1250.283
fs09000000GET /search/instances: languages == lang1750.0490.0440.0810.1640.2990.3790.5190.1200.298
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"1750.0490.0490.1080.2340.4230.6150.6760.1670.422
fs09000000GET /search/instances: subjects <> 1randomword1750.0490.0160.2490.2810.3800.4480.5550.2280.379
fs09000000GET /search/instances: subjects = (randomword OR randomword)1750.0490.0240.2480.2870.3630.4640.6820.2420.362
fs09000000GET /search/instances: subjects = *1randomword1750.0490.6931.0711.3051.8492.3682.8551.1561.848
fs09000000GET /search/instances: subjects = 1randomword*1750.0490.0140.1690.2180.2830.3080.5760.1440.281
fs09000000GET /search/instances: subjects == 1randomword1750.0490.0150.0760.1590.2880.3660.5650.1110.287
fs09000000GET /search/instances: subjects all 1randomword1750.0490.0140.1120.2120.2970.3971.2430.1420.296
fs09000000GET /search/instances: subjects all randomword AND title all randomword1750.0490.0230.0880.1220.2000.2970.5110.1000.200
fs09000000GET /search/instances: subjects all randomword NOT title all randomword1750.0490.0230.2790.3410.4450.5570.7090.2650.444
fs09000000GET /search/instances: subjects all randomword OR title all randomword1750.0490.0680.3100.3580.5210.7280.8360.3260.520
fs09000000GET /search/instances: subjects any 2random words1750.0490.0200.2060.2620.3450.4950.5970.2010.344
fs09000000GET /search/instances: title <> 3 word sentence1750.0490.1520.3470.4580.7711.1311.1800.3970.770
fs09000000GET /search/instances: title == 3 word sentence1750.0490.0170.0370.1200.2040.3590.4130.0790.204
fs09000000GET /search/instances: title all *1randomword1750.0495.0068.10311.27318.09322.11422.7489.56818.093
fs09000000GET /search/instances: title all 1randomword*1750.0490.0170.1130.2450.3910.5040.6390.1590.390
fs09000000GET /search/instances: title all 3 word sentence1750.0490.0390.1330.1780.2740.3170.7770.1480.274
fs09000000GET /search/instances: title any 3 word sentence1750.0490.1750.4730.9952.8223.7204.9220.8602.821
Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.1190.0%
fs09000000GET /search/instances: keyword = 1word*0.2170.0%
fs09000000GET /search/instances: subjects = 1randomword*0.1440.0%
fs09000000GET /search/instances: title all 1randomword*0.1590.0%

0.159


Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*1.0130.0%
fs09000000GET /search/authorities keyword all *randomword*0.7200.0%
fs09000000GET /search/instances: contributors = *randomword0.8300.0%
fs09000000GET /search/instances: subjects = *1randomword1.1560.0%
fs09000000GET /search/instances: title all *1randomword9.5680.0%

2.657
  • Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.


  • Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing


  • Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing


http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850


CPU and Memory Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.


mod-search memory is stable at around 65% throughout the test run and also during background reindexing


20 Users without any background reindexing

  • runtime 60 minutes


RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence15080.4200.1490.7231.0421.7812.5896.3430.8501.781
fs09000000GET /search/authorities keyword <> randomword15110.4200.1560.7131.0301.8402.8875.7960.8511.836
fs09000000GET /search/authorities keyword = *random sentence*15080.4200.8831.6712.0482.8474.6878.3261.7882.847
fs09000000GET /search/authorities keyword = random sentence15080.4200.0310.4600.7171.3762.21910.2180.5961.376
fs09000000GET /search/authorities keyword = randomword15110.4200.0260.4490.7111.4062.5946.0430.5931.404
fs09000000GET /search/authorities keyword == random sentence15080.4200.0680.4470.6931.2782.1535.4110.5581.278
fs09000000GET /search/authorities keyword == randomword15100.4200.0160.4720.7341.5612.5777.3780.6141.561
fs09000000GET /search/authorities keyword all *randomword*15100.4190.5451.3591.7432.5453.7226.9971.4682.545
fs09000000GET /search/authorities keyword all random sentence15090.4190.0530.4710.7221.3612.4915.3850.5911.361
fs09000000GET /search/authorities keyword all randomword15120.4200.0210.4860.7711.6032.6485.1970.6321.603
fs09000000GET /search/authorities keyword any random sentence15080.4200.0510.5070.7671.4642.6145.4630.6361.464
fs09000000GET /search/authorities keyword any randomword15120.4200.0190.4850.7591.5462.5594.9320.6131.546
fs09000000GET /search/instances: contributors <> 2random words15060.4200.2290.8451.2022.0953.86610.6960.9982.094
fs09000000GET /search/instances: contributors = *randomword15060.4200.6021.3601.7282.5263.4114.8991.4722.526
fs09000000GET /search/instances: contributors = randomword*15060.4200.0640.5080.7781.5482.3196.5980.6421.547
fs09000000GET /search/instances: contributors == 2random words15060.4200.0650.4590.7021.4012.3736.6190.5881.401
fs09000000GET /search/instances: contributors all 2random words15060.4200.0580.4720.7491.4242.6185.9750.6071.424
fs09000000GET /search/instances: contributors all randomword15080.4200.0420.5500.8131.5122.4606.0070.6701.512
fs09000000GET /search/instances: contributors any 2random words15060.4200.0900.5900.8601.5582.39511.6320.7261.556
fs09000000GET /search/instances: keyword <> 1word15030.4190.2190.8901.2192.0472.91810.9881.0122.046
fs09000000GET /search/instances: keyword <> random 2words15040.4190.2150.8521.1901.9242.8716.3200.9751.922
fs09000000GET /search/instances: keyword = 1word*15030.4190.0650.5980.8671.5582.5874.8830.7281.557
fs09000000GET /search/instances: keyword = random 2words15040.4190.0770.4720.7141.4362.5534.6210.6011.434
fs09000000GET /search/instances: keyword == random 2words15040.4190.0790.4890.7711.4172.2546.3590.6141.417
fs09000000GET /search/instances: keyword all random 2words15050.4190.0570.5260.7701.4532.7309.9470.6491.453
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang15060.4200.0660.4920.7401.4972.1714.6920.6191.484
fs09000000GET /search/instances: keyword any random 2words15040.4190.1880.6610.9411.5952.5207.0170.7901.594
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")15050.4190.1030.6120.9031.6852.5728.2750.7461.681
fs09000000GET /search/instances: languages == lang15050.4190.1060.5840.8761.5302.4307.0040.7121.528
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"15060.4190.0990.6380.9291.7272.7917.6070.7781.720
fs09000000GET /search/instances: subjects <> 1randomword14980.4180.0820.5890.8671.6653.14714.8820.7441.664
fs09000000GET /search/instances: subjects = (randomword OR randomword)15080.4200.1070.5910.8611.5352.4985.1270.7191.534
fs09000000GET /search/instances: subjects = *1randomword15000.4180.8271.7962.2113.1604.4727.1001.9153.160
fs09000000GET /search/instances: subjects = 1randomword*15020.4190.0520.5440.8041.4952.2697.5140.6601.495
fs09000000GET /search/instances: subjects == 1randomword14990.4180.0570.5540.8401.6962.6764.5950.6941.695
fs09000000GET /search/instances: subjects all 1randomword15020.4190.0700.5470.8001.4162.3674.6320.6581.415
fs09000000GET /search/instances: subjects all randomword AND title all randomword15080.4200.0680.4890.7671.4562.20311.4350.6191.456
fs09000000GET /search/instances: subjects all randomword NOT title all randomword15080.4200.0960.6190.9161.6532.4306.5270.7481.649
fs09000000GET /search/instances: subjects all randomword OR title all randomword15080.4200.1520.6220.8961.6522.6547.7500.7581.650
fs09000000GET /search/instances: subjects any 2random words14990.4180.0880.5820.8731.5592.6568.0470.7241.557
fs09000000GET /search/instances: title <> 3 word sentence14970.4180.2660.8881.2292.0404.1837.9221.0282.038
fs09000000GET /search/instances: title == 3 word sentence14970.4180.0790.4990.7821.5512.4378.3890.6401.551
fs09000000GET /search/instances: title all *1randomword14970.4175.25310.62713.45519.91422.20128.92411.47819.914
fs09000000GET /search/instances: title all 1randomword*14970.4180.0640.5990.9071.7063.1547.8310.7641.704
fs09000000GET /search/instances: title all 3 word sentence14980.4180.0720.4820.7561.5042.5886.3530.6301.504
fs09000000GET /search/instances: title any 3 word sentence14980.4180.2701.0081.7053.7555.00914.3451.3963.754


Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.6420.0%
fs09000000GET /search/instances: keyword = 1word*0.7280.0%
fs09000000GET /search/instances: subjects = 1randomword*0.6600.0%
fs09000000GET /search/instances: title all 1randomword*0.7640.0%

0.698




Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*1.7880.0%
fs09000000GET /search/authorities keyword all *randomword*1.4680.0%
fs09000000GET /search/instances: contributors = *randomword1.4720.0%
fs09000000GET /search/instances: subjects = *1randomword1.9150.0%
fs09000000GET /search/instances: title all *1randomword11.4780.0%

3.624


  • Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~28 seconds For all other queries it is closer to 10 seconds or less.


  • Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.


  • Response time distribution

Out of ~69K requests, 31K requests had a response time > 700 milliseconds All these 31K requests were wildcard queries


  • CPU and Memory Utilization

mod-search's CPU was reasonable at around 30% throughout the test run which is 20% more than 1 user. 


Memory is stable at around 65% throughout the test run


20 Users with background reindexing

  • runtime 60 minutes
RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence12970.3610.2020.8231.1131.9552.6363.5440.9361.955
fs09000000GET /search/authorities keyword <> randomword12990.3610.1100.7941.1362.1233.4879.5010.9682.123
fs09000000GET /search/authorities keyword = *random sentence*12950.3600.9351.9352.4003.3034.11311.9182.0453.303
fs09000000GET /search/authorities keyword = random sentence12950.3600.1020.5510.8151.5232.52310.9370.6801.523
fs09000000GET /search/authorities keyword = randomword13000.3610.0160.5510.8271.6792.68012.5880.7071.679
fs09000000GET /search/authorities keyword == random sentence12970.3610.0940.5650.8691.6262.5348.2890.7111.626
fs09000000GET /search/authorities keyword == randomword12980.3610.0210.5560.8391.6082.5828.3490.7011.608
fs09000000GET /search/authorities keyword all *randomword*12980.3610.5641.5201.9973.0834.29010.0921.6913.083
fs09000000GET /search/authorities keyword all random sentence12970.3610.1050.5490.8291.6572.6558.9730.6881.657
fs09000000GET /search/authorities keyword all randomword13000.3610.0170.5640.8301.6172.8249.5270.7141.617
fs09000000GET /search/authorities keyword any random sentence12970.3610.1470.6040.8681.7632.8189.4300.7511.763
fs09000000GET /search/authorities keyword any randomword13000.3610.0180.5670.8481.6453.0128.0220.7211.645
fs09000000GET /search/instances: contributors <> 2random words12920.3610.2460.9351.2732.1853.2197.7191.0722.184
fs09000000GET /search/instances: contributors = *randomword12930.3600.6721.6112.0002.9194.00513.5281.7642.919
fs09000000GET /search/instances: contributors = randomword*12930.3600.0940.6150.9481.8133.1788.8550.7911.813
fs09000000GET /search/instances: contributors == 2random words12920.3610.1180.5420.8091.5672.5359.1010.6751.567
fs09000000GET /search/instances: contributors all 2random words12920.3610.0920.5660.8131.6002.30913.3260.6851.600
fs09000000GET /search/instances: contributors all randomword12930.3600.1390.6650.9841.8892.99711.8200.8291.887
fs09000000GET /search/instances: contributors any 2random words12920.3610.1520.6920.9771.8423.0318.1690.8441.841
fs09000000GET /search/instances: keyword <> 1word12860.3590.2800.9691.3122.1503.0665.6731.1002.149
fs09000000GET /search/instances: keyword <> random 2words12870.3600.3030.9611.3032.1393.00612.2401.1062.138
fs09000000GET /search/instances: keyword = 1word*12870.3600.1680.7381.0341.9082.8729.9410.8851.907
fs09000000GET /search/instances: keyword = random 2words12890.3600.1150.5720.8841.6392.5495.7100.7141.639
fs09000000GET /search/instances: keyword == random 2words12870.3600.1310.5700.8701.7592.5728.3730.7251.759
fs09000000GET /search/instances: keyword all random 2words12900.3600.1170.6060.9161.7322.67811.0650.7721.731
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang12920.3610.1080.5920.8751.6332.5488.5740.7331.633
fs09000000GET /search/instances: keyword any random 2words12890.3600.2690.7751.1131.9602.8388.8730.9311.959
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")12920.3610.1880.7411.0351.8212.85811.8660.8681.819
fs09000000GET /search/instances: languages == lang12920.3610.1750.6860.9831.8272.7655.4470.8231.826
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"12920.3610.1640.7781.0891.9273.10812.4750.9221.926
fs09000000GET /search/instances: subjects <> 1randomword12850.3600.1480.6970.9841.7282.6189.4520.8391.727
fs09000000GET /search/instances: subjects = (randomword OR randomword)12940.3600.2000.7351.0321.8523.15211.8120.8921.847
fs09000000GET /search/instances: subjects = *1randomword12850.3600.9232.1212.5573.7135.2818.4622.2523.712
fs09000000GET /search/instances: subjects = 1randomword*12850.3600.1300.6410.9451.6682.9459.1140.7871.667
fs09000000GET /search/instances: subjects == 1randomword12850.3600.1070.6120.9181.7252.91911.9330.7751.724
fs09000000GET /search/instances: subjects all 1randomword12860.3590.1520.6680.9841.8152.78712.3400.8151.813
fs09000000GET /search/instances: subjects all randomword AND title all randomword12940.3600.1180.6050.8731.7272.7146.7240.7391.727
fs09000000GET /search/instances: subjects all randomword NOT title all randomword12940.3600.1230.7381.0641.8522.8765.6050.8821.851
fs09000000GET /search/instances: subjects all randomword OR title all randomword12940.3600.2060.7471.0511.9863.53911.7220.9241.985
fs09000000GET /search/instances: subjects any 2random words12850.3600.1310.6910.9671.8392.7289.5870.8391.837
fs09000000GET /search/instances: title <> 3 word sentence12850.3600.2660.9591.3132.1033.43916.4931.1322.102
fs09000000GET /search/instances: title == 3 word sentence12850.3600.0780.5930.8901.7964.45316.1460.8201.796
fs09000000GET /search/instances: title all *1randomword12840.3595.48112.40315.78922.02225.26933.00313.30922.022
fs09000000GET /search/instances: title all 1randomword*12840.3600.1200.6820.9811.9883.63212.4760.8651.987
fs09000000GET /search/instances: title all 3 word sentence12850.3600.0970.5900.8641.7663.47512.1520.7791.766
fs09000000GET /search/instances: title any 3 word sentence12850.3600.3261.1091.9424.3736.04112.5491.6124.371



Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.7910.0%
fs09000000GET /search/instances: keyword = 1word*0.8850.0%
fs09000000GET /search/instances: subjects = 1randomword*0.7870.0%
fs09000000GET /search/instances: title all 1randomword*0.8650.0%

0.832




Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*2.0450.0%
fs09000000GET /search/authorities keyword all *randomword*1.6910.0%
fs09000000GET /search/instances: contributors = *randomword1.7640.0%
fs09000000GET /search/instances: subjects = *1randomword2.2520.0%
fs09000000GET /search/instances: title all *1randomword13.3090.01% (2 failed requests)

4.212


  • Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~30 seconds For all other queries it is closer to 10 seconds or less.


  • Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.


  • Response time distribution

Out of ~69K requests, 32K requests had a response time > 700 milliseconds All these 32K requests were wildcard queries


  • CPU and Memory Utilization

CPU spikes linearly 40% as we start background reindexing. When we start the JMeter script run, the CPU continues to move upwards and reaches a max of 70%. Okapi also consumes resources but it is less than 40% which reasonable.


Memory is stable at around 65% throughout the test run and also during background reindexing 


Longevity 24 hours test


Looking at CPU, Memory and Graphana dashboard, the longevity test for mod-search looks promising. mod-search is stable. Http request for wildcard prefix failed which is expected fs09000000GET /search/instances: title all *1randomword_GET_500

Also, not related to mod-search, there was an issue with the carrier io environment but other than the test looks good.


CPU and Memory Utilization

mod-search is around 40% which is relatively stable. CPU spiked to 40% from 30% after starting background ES reindexing.


Memory also looks stable at around 61%

JIRA tickets created as part of this investigation

  1. MSEARCH-331 - Getting issue details... STATUS  
  2. MSEARCH-337 - Getting issue details... STATUS

Additional information

  • Kafka topics related to mod-search

{EnvID}.{tenant}.inventory.instance
{EnvID}.{tenant}.inventory.item
{EnvID}.{tenant}.inventory.holding-record
{EnvID}.{tenant}.inventory.bound-with (in new versions of mod-search\mod-inventory)







  • No labels