Published: 20 May 2023
Known Uses
In the Dutch Energy sector we have EDSN (Energy Data Services Netherlands) as central data platform for market-facilitating. Cooperation and communication between market parties (suppliers, grid operators, metering companies) is facilitated via the central infrastructure of EDSN. There are use cases in which central data needs to be shared from the source with the decentral market party sinks (for instance for reporting purposes). This is achieved via de Data Transfer Resource Pattern, technically implemented by the S3 cloud storage protocol (the so-called S3 buckets). Such cloud storage is a technique that allows for the exchange of large data sets, low implementation costs, availability, durability, scalability, decoupling and it has a broad adoption in the industry (most parties are capable of connecting to cloud storage).
The pattern variant currently in place is Relay Resource: Every market party sink has a unique bucket where only the required semi-finished data products are delivered to. Housekeeping rules (like retention periods) are mutually agreed upon, and based on business processes. Replay and deletion are responsibilities of the data sink. The shared resource address is agreed upon in advance. Future plans are to optimize the process and to use push notifications (event driven) to inform clients proactively in order to enable faster processing of data and discoverability (which links nicely to the Link Lookup Resource Pattern as described in the book 😊!)
Further known uses from public cloud providers are:
- Amazon S3 REST API: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html
- Google Transfer API: https://cloud.google.com/storage-transfer/docs/reference/rest
- There are many other Cloud Storage services: OneDrive, SharePoint, Azure, etc.
Discussion Input
There are of course security and privacy risks when enterprises store and share files via a Web-flavored shared repository (like in the cloud). However, there are ways to avoid these risks, and the advantage of having all your data in one convenient place that can be accessed at any time from any place is worth investigating as long as the implementation is compliant with government regulations, security and privacy rules, and other legal obligations.
Read the complete pattern on api-patterns.org