Status

DONE

StakeholdersAll BE developers
Outcome


Created date

 

Owner

NOTICE

This decision has been migrated to the Technical Council's Decision Log as part of a consolidation effort.  See:  DR-000025 - Karate API Integration Tests Implementation Guidelines


Background

Some bugs can only be discovered by integrating modules together. It is challenging to identify these on an ongoing basis.

Some integration tests were already implemented. They use Postman collections. They are simple to write but harder to maintain.

It can be challenging to express more complex scenarios. Many of which were contributed by ThunderJet team.

ThunderJet teams has considered a range of tools and recommend Karate tool.

No alternative  proposals has been submitted  therefore Karate framework is preferred method for creating API integration tests in 2020.

API Integration Test Goal

  1. Automated regression
  2. Cross module interaction 
  3. Validate Producer/Consumer relationship
  4. Permissions
  5. No mock objects

QA Goals for 2021

Priority

Action Item

R1

R2

R3

1

Move testing activities to teams scratch environments

50%

100%


2

Build API Integration Test Coverage (Karate framework)

35%

50%

75%

3

Implement UI Integration Test (New BigTest) - 109 test cases

40%

100%


4

Implement manual sprint regression using testing community

20%

50%

100%

5

Implement UAT events using SIG resources

20%

50%

100%

6

PO to complete specs for performance acceptance test

100%



7

Implement performance acceptance test


50%

100%

8

Extend BugFest time frame (Approved)




Scope 

  1. Identify most important business flows
  2. Add test case to cover a bug condition
  3. Don't write tests for all possible conditions

Please note that the intent is not to replace rest assured/junit with karate but instead to get the benefits of both approaches

Create tests for new features

  1. Review if new feature has important business workflows and include test creation for these workflows into estimates
  2. Features are all ranked and it should determine if tests have to be created 
  3. SM need to drive NFR and make sure that Karate tests are scheduled into Sprints 

Test plan creation and estimation

  1. Before the end of R1 teams have to analyze existing API and identify important business flows  and created stubs for them.
  2. Create Jira tickets for implementation of this tests.
  3. Create an estimate for each Jira ticket
  4. Evaluate each workflow based on downstream dependencies

Blocked by:

FAT-145 - Getting issue details... STATUS

FAT-146 - Getting issue details... STATUS

Coverage calculation, reporting

  1. Coverage defined as % of implemented test cases to total test case count.
  2. Coverage will be calculated and reported by Jenkins pipeline

Test execution before and after PR

  1. Part of acceptance criteria - execute Katate tests before PR
  2. Deploy module in Scratch env. and run applicable tests
  3. Module changes and test changes are merged at the same time
  4. Test will be run automatically after PR in Jenkins pipeline the next day:
    1. https://jenkins-aws.indexdata.com/job/FOLIO_Reference_Builds/job/folio-api-tests-karate/

    2. https://jenkins-aws.indexdata.com/job/FOLIO_Reference_Builds/job/folio-api-tests-postman/

These pipelines are currently broken -  FOLIO-2982 - Getting issue details... STATUS

Changes to DOD

  1. Every team should adjust DOD to require API Integration Tests 

Depreciation of Postman tests

  1. Mostly affects Spitfire and Thunderjet
  2. Continue to use Postman tests if team doesn't have desire to switch
  3. Passing tests are required  by acceptance criteria
  4. Postman test pipeline must be:
    1. operational 
    2. have no failures
  5. Defer decision when to deprecate these tests. 

How to evaluate for success

Implementation Tracking Page:  Karate API Integration Testing

Proposed Action Items

  • Each team should change definition of done to make API integration tests required
  • All new BE features must include Karate tests. 
  • R1 Goal: Each team has to create a test plan for existing API interfaces
  • R1 Goal: Each team should provide estimates for the test plan. Estimates will be presented at the Capacity Planning meeting so that this work can be officially scheduled into sprints.



 

Helpful Info


Tech Leads Meeting notes

2020-04-29 Meeting notes

Karate Proposal
2020-05-27 Meeting notes Karate Integration Tests PoC - Q&A
2020-08-05 Meeting notesImplementation of Karate tests 

                                

GitHub repo and documentation: https://github.com/folio-org/folio-integration-tests

Karate resources are at the bottom of README.md

Karate API Integration Testing  implementation tracking page










  • No labels

6 Comments

  1. Are Karate tests required for software libraries like https://github.com/folio-org/folio-isbn-util that don't depend on any other FOLIO code?

  2. Are Karate tests required for modules like https://github.com/folio-org/mod-courses that depend on a database only but do not depend on any other FOLIO module?

  3. It would help me to follow this decision if there was a clear summary of the decision somewhere, e.g. all teams must write API integration tests using Karate for all new endpoints

  4. How can a team execute API Integration tests prior to a draft pull request ( https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests#draft-pull-requests ) or a regular pull request that is opened for several developers to collaborate on new code?

    1. By deploying modules in a scratch dev environment.

      1. At that time the code is not yet written because the pull request is opened before any code is written or before the code is complete. How can the scratch environment run code that doesn't exist or isn't complete?