2018-07-23 Reporting SIG notes

Today's notetakers: Ingolf Kuss

Last week's notetaker: Sharon Beltaine and Holly Mistlebauer

Sharon will report on the status of the data warehouse project.

Technical Council has been asked to work on data warehouse design, along with developers. The proposal is in Jira. Jesse Koennecke (to be PC chair) is taking a look at it. TC is not appointing resources. Tod has discussed this topic (data warehouse design) with Harry.

Tod: Reference data warehouse implementation (PoC). Will we include resources from outside FOLIO, like counter statistics ? What about licensing of the data ?

Some tickets will be aligned with Epic UXPORD-330 (Analytics and Audit Data Logging for External Reporting). How many tickets will be aligned with that ? What is left if we "peel the onion" ? What type of data quantities are we looking for ? How long will we keep the data (consider Cloud vs. on-site deployment) ? Need to limit the scope, right now it's wide open (Harry).

Sharon: Many institutions use multiple reporting tools. Do we need to test all of those ? Maybe we will start with one (Harry), then prioritize the list. The reporting tool analysis and building the reports in the data warehouse environment will be in the responsibility of the Reporting SIG.

Harry: There will be many issues, once the PoC has been set up. We must set priorities for them.

Sharon: Having an SQL-like structure in the PoC data warehouse will open new possibilities (to build the reports).

Harry: Tableau and Kabana don't require SQL table format.

Tod: There is a lot of expertise in emebbed SQL. Do we assume working with a different technique ?

Harry: SQL systems tend to be very limited in performance.

Sharon: Tableau is an expensive product. Need to test with an environment that is accessible by many institutions. Need a test dataset to load into the environment.

Tod: How do we provision access to the data ? We should start with a common tool.

Kevin: We are working with a series of canned reports. We use Access et.al.We need a "generic" way to access the data, like an API. Could push the data from there into an SQL database.

Sharon: The data lake gives you the full flexibility, working on unstructured data. We need to document that.

Harry: What is the "generic" sort of structure that we are going to agree upon ? We can define that structure. Could be No-SQL. We should decide and document that.

Sharon: We have non-functional requirements (from previous discussions) and need to agree upon functional requirements. These could be built upon the notes from a discussion between Tod and Harry: Notes toward a reference implementation . Need to have a more refinded set of functional requirements.

Holly: We need a lead technical person. Need not be full-time.

Tod: Use comments and suggesting on that document  Notes toward a reference implementation rather than editing it.

Look at data mining tools. Feature of PostgreSQL to index data needs to be looked at.

Please review the Discuss post created by Cate describing a proposal to support the auto-generation of Location code and Location display name:  https://discuss.folio.org/t/auto-generating-location-codes-and-location-display-format/1964

• feel free to make comments

We will take a look at the current collection of reporting issues in the Folio JIRA Issue Tracking System:

-see new wiki page on using Reporting JIRA Issue Filters

-What tasks require following up?

-Are user stories correct and complete?

-When will Reporting SIG members assign themselves to external reports?

-what label is best for reports that will be generated in the data warehouse (non in app reports), e.g., dwreport?

• looking at REP-issues (Reporting JIRA project)
  The developers will also be us, the people of the Reporting SIG.
  In the project are issues for: evaluating tools, tools to build the data warehouse, tools to build up the reports, tools to build up spreadsheets (CSV). Issue will be labeled "tools", "data warehouse", "appreport" (for in-app reports) etc.
• What label to use for the data warehouse reports ? "dw-report"
• need more descriptions for REP-2; this should come out of the gap analysis.
• open UXPROD issues labeled as "appreports". Those are the in-app reports. They are in different functional areas. Assignees should work with the relevant SIG, but also with the Reporting SIG. Become a Watcher of those tickets you are interested in. Use the Reporting SIG spreadsheet to tie up your issues with the JIRA tickets. UXPROD tickets can be confusing because they have not been used only for Epics, but also for Features.
• Reporting JIRA Issue filters: There are 6 shared filters. Two links have been provided for each type of issues, one to see the open issues, the other one to see the closed issues.

(Reminder)

Important Notes to Reporting SIG members:

• If your report is identified as a "Yes - In-App Report": Please drop a copy in the appropriate functional area folder in the Library Stats folder in the Reporting SIG Google Drive folder and link them to the "Link to Sample" column in the Reporting SIG Master Spreadsheet, if you haven' yet. It really helps Product Owners to have a report to view to understand what is needed for a given in app report.
• Please keep track of all the reports you added to the spreadsheet as Product Owners may add questions and remarks to the spreadsheet.
• If you do not agree with a certain report being identified as "No In-App report" please turn to Holly.

Review and update Topics for Future Reporting SIG Meetings

Topics for next meeting:

• Data warehouse functional requirements
• Consortia SIG wants to talk to us about Reporting.
• Penn Libraries Metridoc solution