Page History

WOLFcon early bird registration open until June 18th. Call for proposals is open until June 5th; aiming to have an agenda ready before the early bird registration closes. Draft proposals are okay; they can be revised later.

FOLIO Council election results should be coming out soon after the Community Council meeting on Monday.

Data Import: Current State to Desired State

Background documents:

ARLEF Data Import Report, 2023-04-12

EBSCO's Report to the ARLEF Report, 2023-04-24

Report on Data Import, by Corrie Hutchinson, 2023-05-05

EBSCO's Update on Data Import Troubleshooting, 2023-05-23

From Data Import/MM:

• Data import error logs that need to be better
• Data Import live performance times

Desired outcomes:

• Shared understanding of severe shortcomings of data import
• Shared understanding of next steps for addressing data import
• Shared understanding of timeline for improvement
• Determination of how data import remediation will be prioritized

This comes out of various documents and the meeting in Stanford, and includes two reports from EBSCO.

Problem statement: we don't have a reliable and performant solution for loading records to support daily workflows and monthly/quarterly workflows. (Leaving aside system migrations for the moment.) Record loads of less than 1000 records can timeout. Libraries are unable to manage basic workflows such as loading electronic records. FOLIO users must plan to work off-hours to complete import jobs.

Slide 6 describes in-progress efforts across these topics: architecture, infrastructure, development, and product management.

"Large" loads are about 100,000 records, and there need to be plans to address loads of that size and much smaller sizes. Also noting that some types of loads are more performant than other types, so it isn't necessarily about the number of records. "Chunking" can also allow for smaller loads to run interspersed with a larger load. (Single record imports jump the queue right now.)

In addition to the documentation prepared, interviews with current users of FOLIO help make the issues more understandable. (There is a governance question of who "customer" or user is, and if it is inclusive of all hosting providers and those not using a hosting provider.)

Performance is one issue, as is the capabilities of the logic that is a part of data import.

Questions from the meeting include how bugs are prioritized and whether there can be a dashboard of work that is happening on Data Import. (A dashboard is in the works.) Libraries need to be able to know when the functionality will be implemented so they can plan for their local workflows, including when functionality is deemed out-of-scope.

Short term: complete work in progress and address critical production needs (continue performance improvements and reliability/scalability improvements). Collaborate with other hosting providers and self-hosted institutions. Provide realistic performance benchmarks. Define an approach for addressing failed records and logging issues.

Mid-term: review the data import roadmap; continue architecture and infrastructure improvements; development to address failed records and logging issues.

Longer-term (12 months): continue architecture, infrastructure, and development improvements

Proposal for release changes: extend the Poppy release to the current Q-release date (November 2023). There is functionality (other than data import) that is not ready that is driving this decision.

PPT - Data_Import_Improvements.pptx

Proposal for combined Poppy/Quesnelia release

See notes from Release Management Stakeholders