Platform agnostic object storage for Exports

Owned by Kruthi Vuppala
Last updated: Nov 21, 2022 by Craig McNally
Status

DONE

Stakeholders
Outcome
Created date

 

Owner

Kruthi Vuppala

NOTICE

This decision has been migrated to the Technical Council's Decision Log as part of a consolidation effort.  See: DR-000005 - Platform agnostic object storage for Exports   

PURPOSE:

MDEXP-19 - Getting issue details... STATUS

To come up with a platform agnostic storage solution for storing the export files generated by the export process

THINGS TO CONSIDER:

  • There is already process in place for storing the generated files to AWS S3, so it is better to find a solution that requires minimal changes
  • The solution must be able to support cloud as well as on prem storage, since FOLIO can be expected to be hosted on various platforms 
  • Easy to retrieve the files for download, currently the module uses AWS S3 presigned URLs, to stream files directly for download, instead of stressing the server or adding load on the UI 
  • The file sizes can vary from few KB to GB, based on the number of UUIDs being exported, the solution should be able to support varied file sizes


OPTIONS 

MINIO

POC: https://github.com/KVupp/folio-export-aws/blob/master/src/main/java/org/folio/folio_export_aws/MinIo.java

Pros:

  • Can be setup with multiple backend storage solutions: Acts as a gateway for  storage solutions like AWS S3, Azure blob store, HDFS etc. Or can be set up on a local persistent storage volume 
  • Provides an S3 compatible API
  • Minimal code changes required to current code with ability to switch between solutions easily with command line parameters
  • Hardware agnostic and works on a variety of physical and virtual/container environments. 
  • A UI and CLI are also available for accessing the files

                            

Cons:

  • Doesn't seem to support credential chain like aws sdk, so need to send key and access key to module via parameters
  • Doesn’t have multi file upload like AWS S3, but we can upload files one by one

Reference: https://docs.min.io/


OPENIO

Pros: 

  • S3 compatible API

Cons:

  • No API  support to fetch presigned URL


Reference: https://www.openio.io/


DATABASES

Relation or Non relational databases have an overhead of storing static files either  as large files or when stored in chunks.

Costs are higher compared to any other Object stores, and backing them up becomes more difficult as the file sizes grow

Though database can be logical because all the data is present in a single area , but the disadvantages that come with it overshadow the advantages

Specifics related to postgresDB identified in a different spike are documented here: (https://folio-org.atlassian.net/wiki/x/OQ4V)


Other options Considered:

  • CEPH (https://ceph.io/) - This combines object/file and block storage, and has different daemons for monitoring, storage and gateways. Ceph seems like a good option for more complex and large object stores. But in case of smaller stores like data export, this option is more complex
  • S3 Proxy: https://github.com/gaul/s3proxy - This is another option that can be used, but there is not enough documentation and from a maintainability perspective did not pursue further.



DECISION

MINIO with it's advantages seems like the go to option with it's easy setup and minimal code changes required for data-export