Date and time
|Kafka Topics per Tenant Collection|
This is an RFC (request for comment) submitted to the Technical Council (TC).
Background: If you host Kafka as a Cloud Service, there can be quite high charges which increase per so-called Kafka Topic. A Kafka Topic can be thought of as a queue of messages which a module subscribes to. If each tenant hosts ca. 70 modules, which have one Topic per module, an you host many tenants – which is typically a scenario for hosting providers or self-hosting library unions (German Verbundzentralen) – the number of Topics and thus the hosting costs for Kafka do quickly increase.
To mediate this, this RFC proposes Topics per so-called tenant collection. Summary of the RFC "With this RFC, either Kafka topics are created for a collection of tenants or Kafka topics for each tenant can be configured for each deployed module instance. Using a collection of tenants will reduce the number of Kafka topics and subsequently the number of partitions."
Ingolf brings this to the attention of SysOps SIG. For SysOps, some parts of the proposal are of interest:
"For a module instance the
See also the sections "Migration To Tenant Collection Topics" and "Migration to New Module Version" in that document:
Feedback on the Pull Request by the TC: rfcs/0000-kafka-topics-per-tenant.md at kafka_topics_per_tenant · folio-org/rfcs (github.com)
The RFC is near the end of the review stage.
|Folio Notebooks / other Tools used by members of this group|
Jeremy is authoring a repository folio-notebooks, jermnelson/folio-notebooks: Examples of Using Jupyter Notebooks with the FOLIO LSP (github.com)
It is run in a Python virtual environment.
Jeremy is using Apache Airflow for data migration. System librarians can actively participate in the migration process: What is Airflow? — Airflow Documentation (apache.org)
Ingolf is using Catmandu for data manipulation (pre-migration): Catmandu (librecat.org) . MARC data can be "fixed" (converted, transformed, added, deleted...) on a field-by-field basis.
Jason is using Camunda instead.
Ingolf mentions Metafacture – Welcome , a tools which is similar to Catmandu and has been developed by his colleagues at hbz.
FSE FOLIO Migration Tools: About the tools — FSE FOLIO Migration Tools documentation (folio-migration-tools.readthedocs.io)
A comprehensive toolset for migration tasks to FOLIO:
FOLIO Migration Tools currently covers the following data sets:
There is a group which has a FOLIO Slack channel and holds regular meetings: #fse_folio_migration_tools .
This group has been led by Theodor Tolstoy until last week. Group is now being maintained by Brooks Travis.
Of this group, Jeremy and Ingolf use FSE migration tools.
|Reporting Hosting LDP/Metadb||A&M is self-hosting LDP. Power-users have access to the database via DBeaver. The don't write to the LDP directly. They have access via Microsoft Remote Desktop (a windows 22+ server ).|
|Not discussed today / Topics for next meeting:|
|Status of Integrations|
- Type your task here, using "@" to assign to a user and "//" to select a due date