2022-08-10 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Aug 10, 2022
6 min read

Recordings are posted Here (2022+) and Here (pre-2022)                   Slack channel for Q&A, discussion between meetings

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Ann-Marie Breaux (Deactivated) Jennifer Eustis Monica Arnold Lynne Fors Christie Thomas Taylor Smith Jeanette Kalchik Jamie Jesanis (Unlicensed) Jenn Colt 

Morning Glory

Agenda topics:

  • MM SIG presentation last week on Data Import and Morning Glory/Nolana
  • Deleting import logs via API:
    • Available as of Morning Glory
    • There was a Slack question recently about being able to delete Import logs via API as of Morning Glory. We have added some documentation in the Data Import tips and tricks. Please see this page: Data Import Logs, in the last section: Deleting Import Logs via API.
    • Questions:
      • The log data can be queried and compiled via the LDP; what happens with that when 1) the log is marked for deletion (via the UI or API) and then 2) completely deleted in the next 24-hour sweep?
      • Can the log info still be queried via API when it has been marked for deletion but not yet deleted?
      • Jenn: Cornell started playing with log info via API, but stopped until they go live on Lotus
      • NOTE: This has nothing to do with the instance "mark for deletion" discussion
  • MARC Modifications and Field protections
    • MG bug: MODDICORE-248 - Getting issue details... STATUS
    • Modify scenarios – describe real use cases, for  use  in  developer  analysis  and  test  cases
      • https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/673f7ba3-27d2-4bf4-935f-f358f46583fe?sort=name
      • At beginning of job
        • Single record imports: remove junk fields from OCLC, clean up ocm/ocn/on fields, remove 029, some 9xx fields, add some data to allow for creating holdings/items (Jennifer E)
        • Assign a collection ID to a vendor file; correct some MARC errors via DI instead of MarcEdit (Jenn C)
        • MARC Bib for print record and want to add a URL; match on HRID, add the URL, update the record with this collection name - edge case; then update action for now (Jeanette)
      • At end of job
        • Remove 9xx fields after data has been harvested for use in holdings or item (Jennifer)
        • Remove 856s after it has been mapped into a holdings or item (Jennifer)
      • After match
        • Match to instance, after no match for instance, then create instance, then modify to add 856 proxy for school A, then create holdings for school A (Jennifer E)
        • Then another job to match to instance that was just created, and after match, modify to add proxy for school B, then create holdings for school B, then match to same instance, modify for school C, create holdings for school C (Jennifer E) - right now all of this is in separate jobs, after modifications in MarcEdit, then removing 856 from Bib at the end, so 6 total jobs; would be great to reduce to 2, and then perhaps 1 eventually
        • After match for a particular school, edit 856 to assign correct proxy, then update holdings (Jennifer)
        • Bulk edits - find location of X and change to Y (Christie) - not a modification scenario, if using something other than location for match
        • Match on location and then delete the location from 9xx data after it was used for the match (Christie) - can do at the end of job
        • Existing SRS MARC Bib with JSTOR  (Lisa) - talk about when we go into details for MARC Updates
      • At beginning and end of job
        • Yes
      • At beginning, after match, and at end of job?
        • Potentially yes
    • Any scenario where a job would have a Modify action without a Create or Update?
      • Example: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/5564537e-9bda-426f-a755-de5ae16d7ebd?sort=name
        • Maybe in a bulk edit type scenario, where you are wanting to identify records and then change them, without exporting and re-importing, but we have the current export/reimport workaround, and potentially bulk-edit without exporting in the future
        • What would be the expected outcome for a job set up like this? Should we allow modify actions without a subsequent create or update for SRS MARC Bib and/or Instance? NO - would break the current vision of data import (see 
        • Confirm that you can have Modify and Update MARC Bib actions in the same job profile, under the same match; otherwise add bug (2 actions on the same kind of record in a row) (Jenn C)
        • And start making a list of invalid job profile structures so that we can begin to block them ro make them useable
          • Modify with no Create or Update MARC Bib or Authority
          • Match on Instance ID, then Modify/Update MARC Bib (currently have to match on MARC ID before a MARC action)
        • Also causes weird log summary results when same record is visited twice (multi instead of created or updated); multi should only show when multiple (holdings or items) are created/updated with multiple actions, which cannot currently happen via DI
          • Make sure there is a bug for the current weird log results
          • Revisit what the log info should look like when multiple holdings/items acted upon via a single incoming MARC Bib
    • Would it be valid to say that field protections and overriding field protections only invoke when there is one of these Update actions in a job profile?
      • Update MARC Bib (protect and override)
      • Update Instance (protect only, if override need an Update MARC Bib action)
      • Update MARC Authority (protect only, in previous discussion, decided that overrides are not needed for MARC Authority, only Bibs)
      • Not for update Holdings, Item actions (no MARC record underneath)
      • Not for update MARC Holdings action (field protections not currently allowed in FOLIO, per previous discussion)
    • Misc
      • Make sure there are Jiras for on-demand cleanup scripts to 
        • Identify and remove HRIDs stored as 035s (which then cleans up the related instance identifiers derived from the 035)
        • Identify and clean up 035s with various versions of the OCLC prefix (end up with one 035 with only the (OCoLC) prefix (which then cleans up the related instance identifiers derived from the 035)
  • Lotus HF3 Hot fixes for Data Import:
    • View list here
    • MODSOURCE-509 - Getting issue details... STATUS : working except one small issue; when an incoming MARC Bib has an HRID in 001, an 035 will be created; documenting as known issue in Lotus; attempting to correct in Morning Glory
    • MODDICORE-272 - Getting issue details... STATUS : working but would be good to have additional testing
    • MODSOURMAN-841 - Getting issue details... STATUS : working but would be good to have additional testing
  • Morning Glory Bug fixes for Data Import: 
    • View complete list here
    • View unclosed bugs here
      • Some issues have different solutions for Lotus HF versus MG Bugfix versus Nolana, e.g. MODSOURCE-509 - Getting issue details... STATUS versus MODSOURCE-528 - Getting issue details... STATUS , or MODSOURMAN-841 - Getting issue details... STATUS versus MODSOURMAN-843 - Getting issue details... STATUS ; fixed Lotus first, now MG in progress
      • MODDICORE-275 - Getting issue details... STATUS : tested on Snapshot, and seems to be working properly; would be good to have someone else test it on Snapshot and confirm, before we release it to MG BF
      • MODSOURMAN-840 - Getting issue details... STATUS : finishing the fix; files where some records have 999 ff's and some don't were giving us the most trouble
      • MODINV-709 - Getting issue details... STATUS :
        • Separate profiles for 1) match by POL and update Inventory records and 2) match by VRN and update Inventory records work properly
        • Aiming to fix in Nolana; use separate profiles as a workaround for MG

Upcoming meetings:

  • 17 August 2022
    • Order field mapping profile questions
    • UIDATIMP-294 - Getting issue details... STATUS
      • Go over POL limit override, Create separate POs based on MARC value 
      • POL Subscription fields: included as part of one-time order in Orders app; include in field mapping?
      • Overall screen - any questions?
  • 24 August 2022
    • Combined meeting with quickMARC Subgroup, to go over MARC Authority and MARC Bibliographic headings linking (only manually for now, not via DI)

Chat:

From Christie Thomas (she/her) to Everyone 01:40 PM
I think it would be an update and then the 856 would be the only field field impacted, correct?

From Jenn Colt to Everyone 01:40 PM
it's update, but update has repeatable field problems

From Christie Thomas (she/her) to Everyone 01:41 PM
Right, Jenn. We have not attempted to use  update in that what.
in that way ..

From Jenn Colt to Everyone 01:41 PM
i tried, it was not good
lol

From Christie Thomas (she/her) to Everyone 01:52 PM
it does not work.
that sounds like the job that lisa mccoll was describing earlier.

From jeanette kalchik to Everyone 01:59 PM
Thank you. Still trying to get some of the logistics of what can do and cannot do.

From Jennifer Eustis (she/her) to Everyone 02:00 PM
I think it gets stuck with the ones I tested

From Jenn Colt to Everyone 02:01 PM
have to go…

From Christie Thomas (she/her) to Everyone 02:02 PM
yeah, the log information is really confusing.

From jeanette kalchik to Everyone 02:08 PM
Thank you so much!