2021-06-30 Meeting Notes

Jakub Skoczen

How data gets updated across modules. Status updates:

Data sync brought up in Tech Leads several times over last year. App Interaction has joined in recent meetings. Conclusion is for Khalilah Gambrell to pull together a working group to address the issues. Will involve people from several groups.

See also Data synchronization | dependencies across apps.

Raman Auramau has been working on a technical proposal for data synchronization using Kafka. If the above group is formed in a timely fashion, then this will be the logical audience for the proposal.

Questions:

1. When will Kafka be available for general use?
   1. Timeline is not yet certain, will need agreement on conventions for use. Practically, that would be a least 2022. Unclear how this will be resourced.
2. How does this relate to mod-pubsub ?
   1. Can't answer until we see design, however we expect them to co-exist.
   2. Though at a practical level, it seems teams are moving away and towards Kafka direct.
   3. TC needs to provide guidance.

Brainstorming/next steps (e.g a PoC for redrawing module boundaries or having a specific team/group of people do some further investigation, etc)

Topic has been discussed in various venues, though no conclusion.

Problem seems to be the number of containers required for deployment and the dependencies between them. These combine for a very complex deployment, and only increases as we add components. Complexity seems to be increasing from release to release.

One proposal was to get a group together to do a PoC and redefine module boundaries. Another proposal was to reconsider the microservices architecture.

Questions for this group: Do we see this as a problem? Is it more important than other issues?

As a problem, this affects everybody: developers, operations, hosting providers. Release retrospectives always turn this up as an issue.

Much talk about combining some storage modules to reduce dependencies and reduce duplicated data. ("Distributed ball of mud" - spampell)

Can we flip this and not describe the problems/solutions (which seems to have caused previous discussion to collapse) and describe what we want FOLIO to look like?

Proposition: an interested party should be able to install a fully functional "FOLIO LSP" on commodity hardware for testing and development.

With current distribution method, there's a parallel to be drawn with Linux distributions. The applications have versions which go with releases, and we'd like to be able to update applications without having to upgrade the entire distribution.

Notion of optional dependencies: currently FOLIO is basically an all-or-none proposition. There is no practical way to install and use just modules X, Y, Z. We have the possibility of defining optional dependencies, but teams do not seem to be using it. It would allow for modules to fail gracefully. Think that teams are not doing this because they are not being asked to, and the people expressing needs do not think like that. How would be demonstrate this?

If we can redraw boundaries to have fewer dependencies, that's a win. But right now the way we evaluate dependencies, all dependencies are hard dependencies.

Proposed Target:

1. Apps should be able to be run independently, (previous draft: larger, loosely coupled applications that can be installed independently, e.g. run acquisitions without inventory.)
2. You can run a fully-functional FOLIO on commodity hardware.

Target needs to be refined, there are product questions here. What is the unit of use? Is the LSP a whole app, or is it smaller apps? What is it that the community needs, where the want to get to?

Actions:

1. Can we find resources to define those apps that should be able to run independently? (Product Council?)
   
   1. Ask PC for product/application boundaries
   2. Ask for resources to PoC, which would include not just dev teams, but POs and Business Analysts and testing teams