2023-05-08 Reporting SIG Meeting notes

Date

08 May 2023

Attendees

Present?	Name	Organization	Present?	Name	Organization
x	Arthur Aguilera	University of Colorado, Boulder	x	Eliana Lima	Fenway Library Organization
x	Sharon Beltaine	Cornell University		Eric Luhrs	Lehigh University
	Erin Block	University of Colorado, Boulder	x	Linda Miller	Cornell University
	Nancy Bolduc	Cornell University		Nassib Nassar	Index Data
x	Shannon Burke	Texas A&M	x	Elena O'Malley	Emerson
	Suzette Caneda	Stanford University	x	Tod Olson	University of Chicago
	Lloyd Chittenden	Marmot	x	Jean Pajerek	Cornell University
	Tim Dannay	Mount Holyoke College	x	Kimberly Pamplin	Texas A&M University
x	Axel Doerrer	University Mainz	x	Scott Perry	University of Chicago
	Shelley Doljack	Stanford University		Natalya Pikulik	Cornell University
	Stefan Dombek	Leipzig University		Bob Scheier	Holy Cross
x	Jennifer Eustis	U. Massachusetts Amherst / Five College	x	Vandana Shah	Cornell University
	Lynne Fors	Wellesley College	x	Kimberly Smith	Middle Tennessee State University
	Lisa Furubotten	Texas A&M		Linnea Shieh	Stanford University
	Alissa Hafele	Stanford University		Clare Spitzer	Stanford University
x	Kara Hart	Wellesley College		Amelia Sutton	U. Massachusetts
	Corrie Hutchinson	Index Data		Simona Tabacaru	Texas A&M
	Jamie Jesanis	Wellesley College		Huey-Ning Tan	Stanford University
	Jeanette Kalchik	Stanford University		Vitus Tang	Stanford University
	Kevin Kishimoto	Stanford University		Irina Trapido	Stanford University
	Ingolf Kuss	hbz		Catherine Tuohy	Emmanuel College
	Alexander Lao	Stanford University		Kevin Walker	The University of Alabama
	Joanne Leary	Cornell University	x	Angela Zoss	Duke University

Discussion Items

Item

Who

Notes

Attendance & Notes

Sharon

Attendance & Notes

Today's attendance-taker: Linda (or substitute)
Today's note-takers: Team Leads for project updates

Announcements /
Reminders

Sharon

Upcoming meeting topics

May 18: SIG Onboarding Part 2 (Development Work) - tentative

Combining MM and RA/UM subgroups

Current plan is to meet monthly on 2nd Tuesdays at 1pm Eastern
First meeting of new subgroup: tomorrow (Tuesday, May 9) at 1pm Eastern
- Zoom: https://duke.zoom.us/j/92324148267?pwd=V3lCSGJBSm5mYm4xamVFNFJqOGhzZz09
Will be talking about how the group we be organized moving forward
Would love to see new people join!

Reporting Implementation Table

Please continue to fill out details in Reporting Implementation by Institution

Any new members?

Welcome/introductions

How to find our latest recordings

Recording details available on our Reporting SIG Meetings and Notes page and Query Development Meetings page

(Always) Recruiting New Query Developers

The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

Demonstrations of ldpmarc

Jennifer, Angela

https://librarydataplatform.org/ (LDP1, Metadb, LDLite)
https://librarydataplatform.org/resources/
https://github.com/5-C-Folio/LDlite-Queries
At the Five Colleges (5C), they have LDLite on a server and a systems librarian runs it
Have customized to get ERM
Everything except for inventory is downloaded once per night; inventory and SRS are once a week over the weekend
LDLite is not real-time, just a snapshot
LDLite transformations are similar to Metadb, so the data structure looks like Metadb - two tables, one with the original JSON data, one with the data transformed into a flat tables
In LDLite, the data transformation drills into the arrays and objects, so many of the derived tables we use in Metadb are not useful
marctab table is the result of the ldpmarc transformation, converting MARC to a tabular data format
(marc_records_lb is parsed JSON of SRS MARC Bib)
marctab doesn't have state (old or active), so need to go to the original MARC JSON for that (records table)
What kind of queries does 5C run?
- looking for orphan instances (instances without holdings or items attached); data import creates a lot of orphans, something is wrong; these orphans still show up in discovery, which is confusing for users
queries look like LDP1 and Metadb queries - a lot of left joins
how are you connecting using DBeaver?
- the python script is run by an administrator from a server, and then it pushes data into a shared PostgreSQL database, and most people connect to that directly through DBeaver
Is LDLite similar to LDP that you run the whole database refresh once a night?
- Our systems librarian set it up to run an update overnight for everything except inventory and marc SRS; those are big and take a long time, and if they hang it will grind FOLIO to a halt
What scale does this seem to be suitable for?
- Not great at large scale
- People get confused that it's not current data, and people who don't know SQL still can't use it
- with billing, especially hard because can't even really get real time data
- 5C actually doesn't pull everything
- Even with LDLite separate database, will have some queries that hang up the reporting database, will need to kill it
- Are the database tables indexed like LDP1 and Metadb?
Inventory and SRS often have the same thing; might be better to use data from inventory when possible

Recurring Items (Updated weekly, but not always discussed in meeting)

Item

Who

Notes

Review of In-Progress Projects (Recurring)

Build a directory of extracted and derived tables
- update coming (April 24)
Build Metadb query cookbook-style documentation
- has been opened and sent to the development teams for planning/prioritization
- haven't settled on a style exactly, but have some examples
Build Training Program for FOLIO Data Model
- ACQ/ERM drafts have been presented and will be updated
Port LDP1 derived tables to Metadb
- possibly done??
Recruit from new institutions not represented
- recommend that the development teams reach out to SIGs for an update, invite new members

Review the release notes for FOLIO Analytics, LDP1, LDLite, LDP Reporting App, ldpmarc, Metadb Projects (Recurring)

LDP1 release notes
Metadb release notes (via tags)
LDLite release notes (via tags)
ldpmarc release notes
LDP Reporting App release notes (mod-ldp, ui-ldp) and JIRA issues (mod-ldp, ui-ldp)
FOLIO Analytics release notes

Updates and Query Demonstrations from Various Reporting Related Groups and Efforts Projects (Recurring)

Community & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries. Each week, these groups will share reports/queries with the Reporting SIG. Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

Meetings have become more of a lab session, working through specific problems
Contact Angela if you would like to join these meetings; second Tuesdays at 1pm Eastern
Context
- Meeting notes: https://docs.google.com/document/d/1UnzG64tl917LOH2FtWhCEPlSOsnJWKL-0eu88Ouo1DU/edit
- Current status of RA/UM issues: https://github.com/folio-org/folio-analytics/issues?q=is%3Aissue+is%3Aopen+RA%2FUM
- Status of LDP1 → Metadb querying porting: https://docs.google.com/spreadsheets/d/1efLc6QMJyGuXyQM26pZbti-B6wFhJuZt7s7FJbhdXYo/edit#gid=0

MM Working Group

We are now holdings joint meetings with RA/UM. Our recurring meeting time has changed to the 2nd Tuesday from 1-2pm ET.
Zoom URL: https://duke.zoom.us/j/92324148267?pwd=V3lCSGJBSm5mYm4xamVFNFJqOGhzZz09
Meeting ID: 923 2414 8267
Passcode: 358968

ERM Working Group

Current topics
- Reporting for FOLIO Apps
  - Open Access
  - eUsage
  - eHoldings
Meetings are bi-weekly on tuesdays 11am ET alternating with ACQ Working Group
- Next meeting will be at 9th, May
- Contact Stefan Dombek if you would like to get a calendar invitation

ACQ Working Group

group is working with ERM group on ACQ-ERM FOLIO Data Model Training
group is working on ACQ-related (Acquisitions) derived tables for the next Folio-Analytics release
- see ACQ Derived Tables to Convert from LDP1 to Metadb
for latest query development updates, see RM Prototype and Query Development Status
Meetings are biweekly on Tuesdays 11am-noon ET; contact Sharon Beltaine if you would like a calendar invitation
Meeting Recordings are here: FOLIO ERM-RM Report Dev Teams

Reporting SIG Documentation Subgroup

Morning Glory documentation is live on https://docs.folio.org/docs/
Nolana documentation is in review
Orchid documentation will be in progress soon, and plans are underway to include beta-level documentation for Metadb
Additional Context
- The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

no updates currently
new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
New organizational structure for External Statistics reports
- external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
- these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
- institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
- each reporting development team will take responsibility for the queries in their area for the external statistics clusters

D-A-CH Working Group (D-Reporting)

Discussion on draft statement: "Hosting and Support"
- The members of D-Reporting need more time to review the document and provide feedback
Ongoing topics
- Onboarding training
- DBS statistics
  - JIRA-Issues, Rpt-Clusters
  - Gap analysis to statistic codes, user counts etc.
  - Identifying functions that we need for statistics in Germany but are not yet implemented in FOLIO
Meetings are currently held by appointment
- Last meeting: 8th, May
- Next meeting will be at 15th, May (1 pm, CEST)
- Contact Stefan Dombek if you would like to get a calendar invitation

Product Council

(December 8 meeting) PC shared a report of the SIG check-ins
would love for PC to check-in yearly

For all recent work on FOLIO Reporting SQL development:

https://github.com/folio-org/folio-analytics/commits/main