Page tree
Skip to end of metadata
Go to start of metadata
JIRA FeatureUXPROD-2696
ArchitectMikhail Fokanov
Development TeamFirebird
Dev LeadSlava Khandramai
PO

Stephanie Buck

Page statusDraft
Module involvedDev LeadReview
Inventory-storage
Remote Storage



Dematic Edge

Viachaslau Khandramai
or Dematic external team



High-level architecture

A remote storage location is a physical location where books and other physical items are stored.



Note: Accession queue, accession table and remote location settings are 3 tables in Postgres database (in the corresponding schema for tenant).

Remote-storage related business logic

Inventory-storage module

  • The module should send messages (hereinafter update notifications) on every update/create/delete to the corresponding Kafka topic (e.g. instance-updates, holding-updates, item-updates)

Remote-Storage module

  • The module should be used for storing and retrieving all remote-storages settings
  • The module should be used to set remote-storage for the location from UI. This info should be stored in Locations table in remote-storage DB schema
  • There module has access to the Accession queue table in the same DB schema
  • The module subscribes for holding/items update notifications. If permanent or temporary location is changed from the one which is not marked as Remote (e.g. doesn't exist in Locations table) to the one which is marked as remote and corresponding remote-storage accession setting is set to "Folio-initiated", this item/holding is added to the Accession queue

  • The module should provide API for adapter modules, namely:
    • The REST method to retrieve items for accession list by: flag isAccessioned, remote-storage name, date, limit.
    • The REST method to set flag isAccessioned to true
    • The REST method to retrieve configuration of the remote storage (e.g. name, URL, etc.)

Note: In order to increase performance remote-storage settings and remote-storage locations could be cached. If there is one instance of module the cache eviction can be simply managed (as all updates are done through this module). If there are several instances of module cache eviction can be managed by some some eviction policy or the cache could be made distributed.

Requesting a Remote Storage Item for Circulation

Remote-storage module should listen to events in PubSub, which are provided for audit functionality.

  1. If there is a page request for this item and item’s effective location (retrieved from inventory-storage and cached) is remote - circulation Request is sent to retrieval requests queue. Retrieval requests queue is database table like the accession queue database table. The same pooling from dematic-edge module mechanism should be used for it.
  2. Remote Storage response indicates whether an item is successfully requested or rejected (with rejection details).
    1. If item was successfully requested, item is deleted from the retrieval request queue. 
    2. If an item is failed to be requested because of the server error, FOLIO keeps the item in the queue.
    3. If an item is rejected because of a logical reason (item not found, item is locked, item is out of the container etc) then the item is deleted from the queue and error is reported. 
  3. Edge-dematic module should provide API for check-in items in its service point (e.g. remote service point)

Dematic edge module 

  • Module will have connection to remote-storage module API
  • The module should send items to Dematic (for Dematic Staging director) based on schedule or time interval, which is configured in remote-storage settings
  • The module should response with the list of items (for Dematic EMS)


Dematic edge module deployment details

Dematic StagingDirector connection should be established from the dematic edge Folio module. Therefore Dematic edge module needs to know the name of all tenants, which has StagingDirector connection. In order to provide it before the deployment the list of tenant names (e.g. ids) should be put to AWS parameters store. The tenant names list separated by comma (e.g. diku, someothertenantname) should be stored in AWS param store (like it is put for API_KEYs) in the variable with key: stagingDirector_tenants.

API endpoints of remote-storage module

MethodURLPermissionsDescription
POST/remote-storage/configurationsremote-storage.configurations.item.postCreates a remote storage configuration
GET/remote-storage/configurationsremote-storage.configurations.collection.getRetrieves all remote storage configurations
GET/remote-storage/configurations/{configurationId}remote-storage.configurations.item.getRetrieves a remote storage configuration by id
PUT/remote-storage/configurationsremote-storage.configurations.item.putUpdates a remote storage configuration
DELETE/remote-storage/configurations/{configurationId}remote-storage.configurations.item.deleteDeletes a remote storage configuration by id

API provides the following URLs for working with mappings between Folio locations and remote storage configurations:

MethodURLPermissionsDescription
POST/remote-storage/mappingsremote-storage.mappings.item.postCreates new or updates an existing location mapping
DELETE/remote-storage/mappings/{folioLocationId}remote-storage.mappings.item.deleteDeletes location mapping by Folio location id
GET/remote-storage/mappingsremote-storage.mappings.collection.getGet list of all remote storage mappings
GET/remote-storage/mappings/{folioLocationId}remote-storage.mappings.item.getGet remote storage mapping for location id

API to provide access to accession queue

MethodURLPermissionsDescription
POST/remote-storage/accession/{remoteStorage}remote-storage.accession.item.post

Method to set accessioned to current date for the list of items, in order to mark the items in accession queue for the specified remote-storage configuration as accessioned. The list of items is provided in { "itemBarCodes" :  ["barCode1","barCode2", ...] }

GET/remote-storage/accession/{remoteStorage}remote-storage.accession.item.get

Retrieves entities from the accession queue as list. Parameters:

remoteStorage - remote-storage configuration

isAccessioned - boolean -  optional query parameter (default false), is used to get only entities, that hasn't been accessioned (e.g. findByRemoteStorageAndAccessionedNotNull();)

limit - optional query parameter

Dematic edge endpoints *

MethodURLDescriptionExample of response
POST/edge-dematic/lookupNewAsrItems/{remoteStorage}

Method to set accessioned to current date for the list of items, in order to mark the items in accession queue for the specified remote-storage configuration as accessioned. The list of items is provided in { "itemBarCodes" :  ["barCode1","barCode2", ...] }


GET/edge-dematic/lookupNewAsrItems/{remoteStorage}

Retrieves entities from the accession queue as list. Parameters: remoteStorage - remote-storage configuration id. 

The response should be in XML, e.g.:


<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<asrItems>

<asrItems>

<asrItem>

<itemNumber>82703197</itemNumber> <-- barcode

<title>Philologia antiqua.</title>

<author> </author> <-- primary contributer

<callNumber> PA1.A1P45 </callNumber> <-- effective call number

</asrItem>

</asrItems>

</asrItems>
GET/edge-dematic/lookupAsrRequests/{remoteStorage}


Empty response (if there is no items for accession):
<?xml version="1.0"encoding="UTF-8" standalone="yes"?>

<asrResponse>

<code>011</code>

<message>Currently no request found</message>

</asrResponse>

Response with items:
<?xml version="1.0"encoding="UTF-8" standalone="yes"?>

<asrRequests>

<asrRequests>

<asrRequest>

<holdId>354167</holdId>

<itemBarcode>82703197</itemBarcode>

<title>Philologia antiqua</title>

<author></author>

<callNumber>PA1.A1P45</callNumber>

<patronBarcode>7137806</patronBarcode>

<patronName>Bottorff,David</patronName>

<requestDate>2020-07-29 00:00:00.0</requestDate>

<pickupLocation>JRLMAIN</pickupLocation>

<requestStatus>1</requestStatus>

<requestNote></requestNote>

</asrRequest>

</asrRequests>

</asrRequests>


Note: * Dematic StagingDirector does not use an API and connect to the Folio system via TCP socket.

  • No labels

10 Comments

  1. I think this proposal includes at least two architecturally significant decisions:

    • Direct use of Kafka from modules (rather than via pub-sub)
    • Dependencies other than PostgreSQL database storage for storage modules

    I'm aware that the first of those has been included in a separate proposal specifically to address challenges with data import performance and reliability. My understanding was that was currently considered a special case.

    1. Direct use of Kafka from modules (rather than via pub-sub)

      Yes, this is one of the decisions. Taras Spashchenko implemented as a POC for SRS and it appeared to be more stable (fault-tolerant) and more direct. We discussed with DM and Taras Spashchenko, that it will be merged to master in the nearest future. It is safer to work directly with Kafka without REST. If we need conformation from someone, that Kafka can be used directly here please let me know.

      Dependencies other than PostgreSQL database storage for storage modules

      Do you mean the Kafka dependency? Yes there will be such dependency, but it is not a dependency for storage but for interaction.

      1. Yes, this is one of the decisions. Taras Spashchenko implemented as a POC for SRS and it appeared to be more stable (fault-tolerant) and more direct.

        From my recollection of my last conversation with Vince Bareau the decision to directly used Kafka was considered a special case for data import at the moment, as the architectural decision to generally allow modules direct access hadn't been decided.

        Vince Bareau Taras Spashchenko Am I misrepresenting our conversations or has the architectural decision been made to generally allow modules direct access to Kafka?


        Do you mean the Kafka dependency? Yes there will be such dependency, but it is not a dependency forstoragebut forinteraction.

        Yes, I am referring to the decision for a storage module to have a dependency other that PostgreSQL for storage. My understanding is that intent of the storage modules was to be small in scope so they could be easily replaced and that they wouldn't have other dependencies. There is every chance I am mis-recollecting that intent.

  2. In the diagram, there appears to be a bi-directional interaction between the remote storage module and the dematic edge module.

    FOLIO does not currently support cyclic-dependencies. Is there something special about edge modules that allows them to avoid this constraint?

    1. If there is a constraint "not to have bi-directional interactions", we can simplify things and get rid of push of items for remote-storage accession and retain only pulling of items from the accession queue, but for Staging Director do it with small time interval (e.g. 5 seconds). I think it will be good simplification. I'll ask about its feasibility in channel.

      1. If there is a constraint "not to have bi-directional interactions"

        Okapi will not allow cyclic dependencies within the modules enabled for a tenant.

        Given that one of these modules is an edge module, I don't know if it is registered with Okapi. If it isn't, how will the internal module know how to locate the edge module to make requests to it?

        1. This whole bi-directional edge API issue is avoidable.  See my comments below.

          1. Are you suggesting that outbound connections to the remote storage system go via an internal module, yet inbound ones go via an edge module?

            1. Yes, exactly.  The purpose of the edge API is to provide access to FOLIO APIs w/o using the password/credential auth flow.  They're intended to be one-way.  FWIW we have other internal modules which call out to external services already - these don't use edge APIs,  Two examples of this I can think of off the top of my head are:

              • mod-kb-ebsco-java which calls the EBSCO knowledge base. 
              • At least one of the acquisitions (finance?) modules use the JavaMoney API which under the hood calls out to external services to periodically retrieve currency exchange rates.  

              Also note that many, including myself consider the edge APIs external to FOLIO, so if you have an internal module making calls to an edge API, you're essentially doing the same thing (making a call to somewhere outside of FOLIO from an internal module)

  3. Why must the push go through the edge?  I think the pattern we've followed in most of the edge APIs is to keep them very simple/thin.  Any required business logic goes into a separate backend module and the edge module does little more than provide API key auth and invocation of the backend module which does the heavy lifting.  

    The model for external ordering systems for example has one edge API (edge-orders), and (will eventually have) multiple backend modules (one per vendor).  Right now we only have mod-gobi, but there are plans for others, e.g. mod-harrassowitz, mod-ebsconet, etc.  I think this model might be worth consideration here.  A feature has been added fairly recently that allows this routing to be setup via configuration, so you don't need to write any code to add routes for additional backend modules.  See Edge-orders: Configuration Based Routing.  This could probably be generalized in edge-common and used here too.