2022-4-19 Bulk Edit Working Group Meeting Notes

• Magda: OK looking at the Scrum Board. There's a lot of progress.
  

• Magda: I updated the agenda in the last two minutes, because I talked with the developers and they mentioned some behavior on uploading the identifiers especially when the file of identifiers is large, it may take some time and incorrectly display the matched records. We would like to add progress before the preview of matched records screen populates while making sure we have gathered all the records, but without constantly polling which causes some problems. The proposed solution would be a similar progress bar to what is displayed while bulk edit is in progress (displayed below).

• Erin: Okay. So just to give it more time to handle the larger files?
• Magda: Yes. If you are uploading a larger data set, the user will have some visual cue  That there is something happening and it’s not just hanging.
• Erin: That makes sense to me. I don't know if anybody else has any particular feedback. I see Leeda is giving a thumbs up. Yeah. As long as we're consistent with the behavior of the other parts of the app, that makes sense to me.
• Magda: Okay, thank you. That would make developers also very happy because we've been struggling with different issues around this area. And I would like to have some solution that is acceptable to the user. And I think it will help once we start moving with larger data sets. the other part on my abdomen that I'm sorry, I skipped the development status and a scrum board.
• Magda: The action button in snapshot is not displaying due to a data polling issue.

• Magda: I would like to talk about the export manager.

• Magda: This is kind of a by-product of the work that we have done on bulk edit because we use the same modules that are being used for data export.

• Magda: The files used in bulk edit are being logged in the export manager. So for example, when you click on the bulk edit identifiers file, you get the list of the identifiers that were submitted.

• Magda: So if I opened the file it is matched records for the user barcodes that were submitted before.

• Magda: This is a byproduct of what we have done. And we could manage it for additional functionality.

• Erin: My question would be when you are using bulk edit you do not have to have access to export manager to do the functions that need to be done.

• Magda: This is valid and a very good point. This is a separate set of permissions that are not included in the bulk edit permissions but it will let you access those files.
  

• Erin: Does this represent a permission issue or a security issue? People can get downloadable copies of edited files, including user information. When maybe we don't want that to happen.

• Bob: And I'm not sure I'm following. It seemed like what you were trying to show is duplicative of what you would get just within the app itself.

• Magda: Yes. But then you are getting historical data. You will be able to go to the job that was previously executed with a link to the file.

• Bob: So it is not totally necessary for the user to go here, but someone who had the necessary permissions could come here and see the history.

• Magda: Exactly. This is not required but rather additional functionality that we will have because we are using the common component.

• Magda: What I was thinking is we can also utilize the export log which will provide a little bit more information about, what happened, what the job was, who executed and if something went wrong, what was wrong?
  

• Magda: One more thing I wanted to show. This is an example of the useful history when you have failed queries.  It shows us what was wrong. This was really nice when I was executing this yesterday. But, if we want to use it, then we need to come up with ideas of how we are going to filter the result. Or do you think we should just hide them because of what Erin said about security issues?

• Erin: My question is would people have access to export managers who would not have access to the user's app or to other places and could potentially end up seeing information like addresses and identifiers that an institution would not want them to see.
  
• Magda: I will double-check this Erin because this is a serious security breach, but we definitely don't want to introduce.
  
• Lloyd:  I wanted to check. What you're demonstrating now is functionality for Lotus, is that correct? And it is changing to an in-app?
• Magda: No. Bulk edit is not part of Lotus. It has been pushed back to Morning Glory. By the end of Morning Glory we will have:
  • Users - CSV approach
  • Items - adding an in-app approach limited to changing temporary and permanent locations, and some of the item statuses.

  • We will have those two parallel tracks that we will be working on combining in future releases.

• Magda: In Nolana, we will have the CSV approach for items and the in-app approach for users plus holdings, et cetera. The reason when you are doing it is that there is value in the CSV approach. It gives users a lot of flexibility that we will not be able to provide with the in-app approach at any time soon.
• Magda: However, users will need to know what they are getting into as they are able to modify each field in the record using a CSV approach. So this will be divided into separate permission sets.
• Lloyd: So you'll be running both systems in parallel Indefinitely because there are some things you're not going to be able to implement in-app? Eventually, is everything going to be an-app?
• Magda: No. I think there will be two different approaches and we will support both of them indefinitely unless there is no interest in that interest in the community. But we do have here that the user wants to have this flexibility that the CSV approach provides.

• Sara: So we have export manager, data export, and in acquisitions we're talking a lot about export and getting data out. There's the whole voucher exporting and in agreements, there's an export function. Is there any discussion about using one app for export? This came to mind with Aaron's question about who will see what.

• Magda: So this is a good question. And this is a little bit outside the scope of bulk edit functionality.  This is really confusing for everyone, including me because I did not know export manager was being built while we already had data export. Ane the complexity of the problem is the fact that each uses a different technology. Export manager is using Spring, and data export uses  DB, the FOLIO approach. There is a plan to combine them at some point. But for this to happen, we will need to move data export to Spring, which is happening on the backend in this release. There will be no difference for a user in data export. But it put us in the position where we can start talking about combining data export with export manager. I know that orders is using export manager. All EDIFACT exports are being done using export manager. I cannot talk about other apps, but I will say that almost everyone is heading toward export manager. So our next step will be to combine data export with export manager.

• Sara: So then looping back, how will we control what we see and can access here if we're all going to end up here.

• Magda: This is a good question. And to be honest, I'm not sure it was raised by others. I will need to reach out to peers from other areas about how they are handling that because I don't think that it's being handled in any way, and I'm fully aware that for user it is extremely sensitive. So we will need to address it before the end of the release of Morning Glory. This will be driven by the permissions, right? If you don't have permissions for the users app, you don't see them here at all. And as I said, I am not able to present good examples. But, we may come back to this. I'm very thankful for the feedback making me aware of the security compromise of personal data.
  

Bulk edit - rollback changes

• Magda: So we've been thinking of adding the rollback changes functionality. That would happen in our CSV approach when a user uploades a file and then wants to rollback the changes. We can do this because we have those files stored on the server and we can go back to them.

• 
  

  Magda: But the question I have is how long should we allow users to follback changes? So for example, you did the changes today and in two weeks you notice there's a problem. Do you really want to have this functionality available for two weeks or indefinitely? Or would you limit this to a day or two?

• Erin: How would you get to the list of the stuff that you did if you needed to roll it back?

•  Magda: You have the local copy of the file that you upload to make the changes. So the process would be:
  

  1. You uploaded the file with the changes.

  2. The changes are committed in the database, but then you realize it is not what you expected there is a serious problem.

  3. So you go to the file that you get when you matched records.

  4. You click rollback changes and it will go back to the file state before you made the changes.

• Erin: So you go to the export manager to do that?

• Magda: No. We are, we are in the bulk edit. You will not see a list of the files here. We have the files on the server. We don't have them in the UI in our initial implementation.

• Erin: So, I upload the file and FOLIO recognizes this file and presents the rollback option?

• Bob: My mind went right to how Google keeps a history of changes where you can select a restore point. Not sure this makes sense here as it does in a CMS. Also, it would be a setting where you would set how far back in time you would want to keep files. This would allow each institution to make this a local decision. Some institutions may like the idea of keeping them longer than others.

• Magda: So. you are suggesting having a review of the files, right?

• Bob: Yeah, a list of the files in a pane where you can select to roll back to that point in time.
  
• Erin: The second question there is the one that makes me go, I don't think you should be able to do it for that long because I don't want to overwrite changes that have happened since my bulk edit.
• Magda: Exactly. So this is the second question on my screen.
• Bob: It will keep track of all changes that happened.
• Magda: We don't keep track on that level. In the user records, for example. We have versioning in inventory, but not on the user records. So we cannot keep track of all the changes.
• Magda: So you commit changes and then somebody else makes the changes to the record. Then you realize that your bulk edit was not the right one and you want to go back. So you are retrieving your original file. You're uploading and committing changes and overwriting everything that happened between your original bulk edit and when you do the rollback.
  

• Lloyd: I wonder if maybe it shouldn't be set so that you can only roll back the single most recent change to that record type that might solve that problem of changes made by other users.

• Erin: From my perspective, I don't want bulk edit to change anything if somebody else has changed it since then. Because I don't have the context to know what happened to that record, even if it was just an individual change. Um, and I don't want to screw somebody else up.

• Sara: I think this is actually really dangerous. Unless it's an immediate thing where I hit the button and then need to immediately undo it, I don't think I should be able to do this. Other than that I should have to go through the process of downloading the data again to avoid changing any other changes that were done in the interim.

• Magda: This is very good feedback. So did I understand correctly Sarah, that you said, the rollback could make things even more dangerous. You would prefer that if some unfortunate change occurs, you have the list of identifiers and then you do the whole update again, is this correct?

• Sara: This is what I would be for. As long as I can get at that pile again, my identifiers have been saved somewhere, right? Then I reversed my change based on the change made and did not overwrite any further changes.

• Mark: How difficult would it be to flag any records that have been changed in the interim when you go to rollback records and give the option of changing the ones that have not changed and then I can make the decision whether this rollback is too expensive? I think you can just check the dates to see if the record has changed.
  

• Magda: I think in early meetings, we stated it's better not to do something If this introduces more problems than solutions. So I may start to work rollback feature and then we will discuss all the best approaches and for now just hold rolling back changes.

• Lloyd: Yeah. I, I just wanted to agree with Sarah. I think that makes a lot of sense instead of a rollback feature you could have the ability to grab all the records you changed in some previous edit. It would keep all the bulk edits you've done for some period of time and you could pull those together then perform another bulk edit based on that.

• 
  

  Bob: There is no locking of records to handle two users working on the same file in bulk edit, or in another app while you are doing a bulk edit? Correct?
  

• Magda. Correct. At this point, there is no locking of the records. We will need to have a more discussion about what would be the ideal solution for reverting and rolling back the changes. So I will come back with more follow-up questions like this is definitely not something simple. I thought that we could include this in Morning Glory as a simple user story. But from the feedback I'm getting, we will probably introduce more problems than help. So let's wait and work on some more complex approaches. I think we are all in agreement we need to talk about this more. And for now, if something goes wrong, we need to go through the whole process of updating the records again.