JIRA Feature | UXPROD-2696 |
Architect | Mikhail Fokanov |
Development Team | Firebird |
Dev Lead | Slava Khandramai |
PO | Stephanie Buck |
Page status | Draft |
Module involved | Dev Lead | Review |
---|---|---|
Inventory-storage | ||
Remote Storage | ||
Dematic Edge | Viachaslau Khandramai |
High-level architecture
A remote storage location is a physical location where books and other physical items are stored.
Note: Accession queue, accession table and remote location settings are 3 tables in Postgres database (in the corresponding schema for tenant).
Remote-storage related business logic
Inventory-storage module
- The module should send messages (hereinafter update notifications) on every update/create/delete to the corresponding Kafka topic (e.g. instance-updates, holding-updates, item-updates)
Remote-Storage module
- The module should be used for storing and retrieving all remote-storages settings
- The module should be used to set remote-storage for the location from UI. This info should be stored in Locations table in remote-storage DB schema
- There module has access to the Accession queue table in the same DB schema
- The module subscribes for holding/items update notifications. If permanent or temporary location is changed from the one which is not marked as Remote (e.g. doesn't exist in Locations table) to the one which is marked as remote and corresponding remote-storage accession setting is set to "Folio-initiated", this item/holding is added to the Accession queue
- The module should provide API for adapter modules, namely:
- The REST method to retrieve items for accession list by: flag isAccessioned, remote-storage name, date, limit.
- The REST method to set flag isAccessioned to true
- The REST method to retrieve configuration of the remote storage (e.g. name, URL, etc.)
Note: In order to increase performance remote-storage settings and remote-storage locations could be cached. If there is one instance of module the cache eviction can be simply managed (as all updates are done through this module). If there are several instances of module cache eviction can be managed by some some eviction policy or the cache could be made distributed.
Requesting a Remote Storage Item for Circulation
Remote-storage module should listen to events in PubSub, which are provided for audit functionality.
- If there is a page request for this item and item’s effective location (retrieved from inventory-storage and cached) is remote - circulation Request is sent to retrieval requests queue. Retrieval requests queue is database table like the accession queue database table. The same pooling from dematic-edge module mechanism should be used for it.
- Remote Storage response indicates whether an item is successfully requested or rejected (with rejection details).
- If item was successfully requested, item is deleted from the retrieval request queue.
- If an item is failed to be requested because of the server error, FOLIO keeps the item in the queue.
- If an item is rejected because of a logical reason (item not found, item is locked, item is out of the container etc) then the item is deleted from the queue and error is reported.
- Edge-dematic module should provide API for check-in items in its service point (e.g. remote service point)
Dematic edge module
- Module will have connection to remote-storage module API
- The module should send items to Dematic (for Dematic Staging director) based on schedule or time interval, which is configured in remote-storage settings
- The module should response with the list of items (for Dematic EMS)
Dematic edge module deployment details
Dematic StagingDirector connection should be established from the dematic edge Folio module. Therefore Dematic edge module needs to know the name of all tenants, which has StagingDirector connection. In order to provide it before the deployment the list of tenant names (e.g. ids) should be put to AWS parameters store. The tenant names list separated by comma (e.g. diku, someothertenantname) should be stored in AWS param store (like it is put for API_KEYs) in the variable with key: stagingDirector_tenants.
API endpoints of remote-storage module
Method | URL | Permissions | Description |
---|---|---|---|
POST | /remote-storage/configurations | remote-storage.configurations.item.post | Creates a remote storage configuration |
GET | /remote-storage/configurations | remote-storage.configurations.collection.get | Retrieves all remote storage configurations |
GET | /remote-storage/configurations/{configurationId} | remote-storage.configurations.item.get | Retrieves a remote storage configuration by id |
PUT | /remote-storage/configurations | remote-storage.configurations.item.put | Updates a remote storage configuration |
DELETE | /remote-storage/configurations/{configurationId} | remote-storage.configurations.item.delete | Deletes a remote storage configuration by id |
API provides the following URLs for working with mappings between Folio locations and remote storage configurations:
Method | URL | Permissions | Description |
---|---|---|---|
POST | /remote-storage/mappings | remote-storage.mappings.item.post | Creates new or updates an existing location mapping |
DELETE | /remote-storage/mappings/{folioLocationId} | remote-storage.mappings.item.delete | Deletes location mapping by Folio location id |
GET | /remote-storage/mappings | remote-storage.mappings.collection.get | Get list of all remote storage mappings |
GET | /remote-storage/mappings/{folioLocationId} | remote-storage.mappings.item.get | Get remote storage mapping for location id |
API to provide access to accession queue
Method | URL | Permissions | Description |
---|---|---|---|
POST | /remote-storage/accession/{remoteStorage} | remote-storage.accession.item.post | Method to set accessioned to current date for the list of items, in order to mark the items in accession queue for the specified remote-storage configuration as accessioned. The list of items is provided in { "itemBarCodes" : ["barCode1","barCode2", ...] } |
GET | /remote-storage/accession/{remoteStorage} | remote-storage.accession.item.get | Retrieves entities from the accession queue as list. Parameters: remoteStorage - remote-storage configuration isAccessioned - boolean - optional query parameter (default false), is used to get only entities, that hasn't been accessioned (e.g. findByRemoteStorageAndAccessionedNotNull();) limit - optional query parameter |
Dematic edge endpoints *
Method | URL | Description | Example of response |
---|---|---|---|
POST | /edge-dematic/lookupNewAsrItems/{remoteStorage} | Method to set accessioned to current date for the list of items, in order to mark the items in accession queue for the specified remote-storage configuration as accessioned. The list of items is provided in { "itemBarCodes" : ["barCode1","barCode2", ...] } | |
GET | /edge-dematic/lookupNewAsrItems/{remoteStorage} | Retrieves entities from the accession queue as list. Parameters: remoteStorage - remote-storage configuration id. The response should be in XML, e.g.: | <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> |
GET | /edge-dematic/lookupAsrRequests/{remoteStorage} | Empty response (if there is no items for accession):
|
Note: * Dematic StagingDirector does not use an API and connect to the Folio system via TCP socket.
10 Comments
Marc Johnson
I think this proposal includes at least two architecturally significant decisions:
I'm aware that the first of those has been included in a separate proposal specifically to address challenges with data import performance and reliability. My understanding was that was currently considered a special case.
Mikhail Fokanov
Direct use of Kafka from modules (rather than via pub-sub)
Yes, this is one of the decisions. Taras Spashchenko implemented as a POC for SRS and it appeared to be more stable (fault-tolerant) and more direct. We discussed with DM and Taras Spashchenko, that it will be merged to master in the nearest future. It is safer to work directly with Kafka without REST. If we need conformation from someone, that Kafka can be used directly here please let me know.
Dependencies other than PostgreSQL database storage for storage modules
Do you mean the Kafka dependency? Yes there will be such dependency, but it is not a dependency for storage but for interaction.
Marc Johnson
From my recollection of my last conversation with Vince Bareau the decision to directly used Kafka was considered a special case for data import at the moment, as the architectural decision to generally allow modules direct access hadn't been decided.
Vince Bareau Taras Spashchenko Am I misrepresenting our conversations or has the architectural decision been made to generally allow modules direct access to Kafka?
Yes, I am referring to the decision for a storage module to have a dependency other that PostgreSQL for storage. My understanding is that intent of the storage modules was to be small in scope so they could be easily replaced and that they wouldn't have other dependencies. There is every chance I am mis-recollecting that intent.
Marc Johnson
In the diagram, there appears to be a bi-directional interaction between the remote storage module and the dematic edge module.
FOLIO does not currently support cyclic-dependencies. Is there something special about edge modules that allows them to avoid this constraint?
Mikhail Fokanov
If there is a constraint "not to have bi-directional interactions", we can simplify things and get rid of push of items for remote-storage accession and retain only pulling of items from the accession queue, but for Staging Director do it with small time interval (e.g. 5 seconds). I think it will be good simplification. I'll ask about its feasibility in channel.
Marc Johnson
Okapi will not allow cyclic dependencies within the modules enabled for a tenant.
Given that one of these modules is an edge module, I don't know if it is registered with Okapi. If it isn't, how will the internal module know how to locate the edge module to make requests to it?
Craig McNally
This whole bi-directional edge API issue is avoidable. See my comments below.
Marc Johnson
Are you suggesting that outbound connections to the remote storage system go via an internal module, yet inbound ones go via an edge module?
Craig McNally
Yes, exactly. The purpose of the edge API is to provide access to FOLIO APIs w/o using the password/credential auth flow. They're intended to be one-way. FWIW we have other internal modules which call out to external services already - these don't use edge APIs, Two examples of this I can think of off the top of my head are:
Also note that many, including myself consider the edge APIs external to FOLIO, so if you have an internal module making calls to an edge API, you're essentially doing the same thing (making a call to somewhere outside of FOLIO from an internal module)
Craig McNally
Why must the push go through the edge? I think the pattern we've followed in most of the edge APIs is to keep them very simple/thin. Any required business logic goes into a separate backend module and the edge module does little more than provide API key auth and invocation of the backend module which does the heavy lifting.
The model for external ordering systems for example has one edge API (edge-orders), and (will eventually have) multiple backend modules (one per vendor). Right now we only have mod-gobi, but there are plans for others, e.g. mod-harrassowitz, mod-ebsconet, etc. I think this model might be worth consideration here. A feature has been added fairly recently that allows this routing to be setup via configuration, so you don't need to write any code to add routes for additional backend modules. See Edge-orders: Configuration Based Routing. This could probably be generalized in edge-common and used here too.