This decision has been migrated to the Technical Council's Decision Log as part of a consolidation effort. See: DR-000005 - Platform agnostic object storage for Exports
- MDEXP-19Getting issue details... STATUS
To come up with a platform agnostic storage solution for storing the export files generated by the export process
THINGS TO CONSIDER:
- There is already process in place for storing the generated files to AWS S3, so it is better to find a solution that requires minimal changes
- The solution must be able to support cloud as well as on prem storage, since FOLIO can be expected to be hosted on various platforms
- Easy to retrieve the files for download, currently the module uses AWS S3 presigned URLs, to stream files directly for download, instead of stressing the server or adding load on the UI
- The file sizes can vary from few KB to GB, based on the number of UUIDs being exported, the solution should be able to support varied file sizes
- Can be setup with multiple backend storage solutions: Acts as a gateway for storage solutions like AWS S3, Azure blob store, HDFS etc. Or can be set up on a local persistent storage volume
- Provides an S3 compatible API
- Minimal code changes required to current code with ability to switch between solutions easily with command line parameters
- Hardware agnostic and works on a variety of physical and virtual/container environments.
- A UI and CLI are also available for accessing the files
- Doesn't seem to support credential chain like aws sdk, so need to send key and access key to module via parameters
- Doesn’t have multi file upload like AWS S3, but we can upload files one by one
- S3 compatible API
- No API support to fetch presigned URL
Relation or Non relational databases have an overhead of storing static files either as large files or when stored in chunks.
Costs are higher compared to any other Object stores, and backing them up becomes more difficult as the file sizes grow
Though database can be logical because all the data is present in a single area , but the disadvantages that come with it overshadow the advantages
Specifics related to postgresDB identified in a different spike are documented here: (https://wiki.folio.org/x/ewYuAg)
Other options Considered:
- CEPH (https://ceph.io/) - This combines object/file and block storage, and has different daemons for monitoring, storage and gateways. Ceph seems like a good option for more complex and large object stores. But in case of smaller stores like data export, this option is more complex
- S3 Proxy: https://github.com/gaul/s3proxy - This is another option that can be used, but there is not enough documentation and from a maintainability perspective did not pursue further.
MINIO with it's advantages seems like the go to option with it's easy setup and minimal code changes required for data-export
Given the decision to use MINIO, I have a few questions.
Will MINIO be set up in the shared hosted environments, in order for this solution to be tested by folks? If not, how will it be tested?
As I understand it, MINIO is a separate infrastructure service.