CloudEvents in the Dutch Energy Sector

In this article, authors Mark Dirksma (EDSN) and Ton Donker (Enexis) describe the minimal set of CloudEvents context attributes required to define the metadata of an event in the Dutch (NL) energy sector. This minimal set is an energy sector-specific adaptation of the the (Dutch) NL GOV profile on CloudEvents. The NL GOV profile for CloudEvents, in turn, profiles the international CloudEvents specification to standardize the exchange of information about events, specifically applicable to the Dutch public sector. CloudEvents is an open specification for event-driven architecture, providing standards for how event models should be structured.

This minimal set was determined through intensive collaboration between the Dutch energy sector-wide API Strategy Working Group and EDSN. It prescribes the set of context attributes that an event in the energy sector must contain, making CloudEvents part of the Dutch energy sector event-driven API guidelines.

CloudEvents

CloudEvents is a vendor neutral specification for defining the format of event data. The CloudEvents standard was initiated by the Cloud Native Computing Foundation (CNCF) in March 2018. Since its start, CNCF CloudEvents has become an essential, globally standardized tool for developers working with cloud-native, serverless, and event-driven applications. Originating from a collaboration of over 40 companies, such as Microsoft, Google, SAP, Solace, IBM, Oracle, Azure, and Amazon, CloudEvents establishes a core set of metadata attributes and specifies how these attributes are mapped to protocol messages and data encodings. The goal of CloudEvents is to replace the various methods of working with event-driven systems with a standardized approach to promote interoperability.

A ‘less is more’ approach has been chosen, deliberately limiting the standard to a small set of descriptive metadata about events. This enables producers to publish events in a standardized way, intermediaries to handle events in a standardized manner, and, for instance, issue standardized notifications to consumers. CloudEvents is an international standardized event metadata model in the inforrmation technology industry, with bindings to all major messaging protocols (such as MQTT, Kafka, AMQP, HTTP, WebSockets) and encodings (including Avro, JSON, XML, ProtoBuf).

Picture source: Post | Feed | LinkedIn

CloudEvents-NL profile

In 2022, VNG Realisatie conducted the Notification Services Project on behalf of the Ministry of the Interior. The goal of the project was to standardize the automated notification of applications within the Dutch government and to enable the government to work more effectively and frequently in an event-driven manner. One of the products of the Notification Services Project is the NL GOV profile for CloudEvents (aka CloudEvents-NL profile). This NL profile builds on the CloudEvents specification and includes guidelines for using the CloudEvents specification within the Dutch government. It has been agreed how government organizations should use certain attributes with metadata information. The following attribute types have been identified:

context attributes: attributes with meta-information specifically intended for purposes such as filtering and routing;
payload attributes: that describe the actual event that took place so that consumers can correctly interpret and use them;
extension attributes: additional context attributes that can be included in an event to provide extra metadata beyond the core set of required attributes;

The CloudEvents-NL profile, based on the CloudEvents specification, prescribes the exclusive use of context attributes for filtering and routing, not payload data. This enables extensive standardization and processing by a multitude of intermediaries. Payload data should preferably be considered by an intermediary as a ‘black box of data’ that needs to be transferred from producer to consumers. Context metadata will be encapsulated in the context attributes. CloudEvents specifies which context attribute must be included as a mandatory metadata element in an event, based on the principle of not imposing more requirements on the parties involved than necessary. The CloudEventsEvents-NL profile is an adaption of the CloudEvents specification and the CloudEvents-NL profiles the CloudEvents specification to standardize the automated exchange of information about events, specifically applicable to the Dutch government.

Dutch Energy Sector

EDA (Event Driven Architecture) is on the rise, and in the Dutch energy sector, it has been adopted as a valuable pattern for optimizing data exchange in the energy information domain. The energy sector is aiming to organize consistent and unambiguous sources of truth, such as connections and meter registers. Changes to registers often require immediate notifications, and an Event Driven Architecture pattern provides an efficient way to quickly deliver these changes to the right parties. Event-Driven Architecture is a design pattern used in applications where it’s essential to respond to events as quickly as possible.

We primarily see its application when a register is updated (due to an event in the external world or a market process) and a market role (supplier, grid operator, metering company) needs to act on that update. Whether it includes both the business and technical event-driven architecture, information-rich (‘fat’) or information-poor (‘thin’), and whether based on eventing (broadcast, directed to all subscribed consumers, without addressing) or based on messaging (narrowcast, one or more specific recipients, thus with addressing), there must be a standard way to communicate the event envelope. The envelope expresses the context metadata of the event and since ‘Context is King‘, it is important to standardize the structure of the envelope and specify the minimal set of required context attributes. This article outlines the context attributes required to be present in all Dutch energy sector events.

Context Attributes

The following example shows a Dutch energy sector event serialized in JSON format (the data payload contains dummy values and is illustrative, not complete):

{ "specversion": "1.0", "id": "e65558c4-2734-44f1-b04e-63923b0ab979", "type": "mdm.meter.updated", "source": "urn:ean13:8716859111111:cmr", "time": "2023-09-22T14:01:54.957124Z", "dataversion": "1.0.1", "datacontenttype": "application/json", "data": { "meterId": "E000900000149000012345", "status": "active", "reading": { "value": "2450.75", "unit": "kWh" }, "lastReadingTimestamp": "2023-12-08T10:30:00Z" } }

The required core attributes of a CloudEvent are present:

• specversion: the version of the CloudEevents specification;
• id: uniquely identifies the event;
• type: identifier of the event type (“what happened?”);
• source: URI identifying the event origin (“where did it happen?”);

In addition to the CloudEvent core attributes, the following attributes are present in the example and are used in energy sector events:

• time: time the event was produced by the source system;
• dataversion: semantic version of the event;
• datacontenttype: describes the content type of the data value;
• data: the event payload;

The image below shows which event attributes are specified by the CloudEvents-NL profile as mandatory and optional, and which are specified as mandatory and optional by the NL energy sector:

Below is a detailed description of the context attributes applicable in the Dutch energy sector:

specversion:

Identifies the version of the CloudEvents specification used by the event. This enables the interpretation of the context. Conformance requirements for event producers mandate that they must use a value of ‘1.0’ when referring to this version of the specification.
Type: String
"specversion": "1.0"

id:

Attribute id contains a unique identifier of the event in the form of (for example) a UUID.
CloudEvents provides a solution to implement idempotency, requiring that the combination of id and source uniquely identifies an event, which can be used as the idempotency key in downstream implementations. Therefore, the combination of id and source must be unique from the producer’s perspective. A consumer can use the combination of id and source to determine if an event has already been processed.
Type: String;
"id": "e65558c4-2734-44f1-b04e-63923b0ab979"

type:

The energy sector-specific application of type deviates from the specification in the CloudEvents-NL profile and prescribes that type should always be a composite attribute of context or domain indication, object, and performed action (in past tense). For example, EDSN will include the corresponding Business Solutions abbreviations in this attribute as context indication. See the example below where mdm stands for meter data management. The polyseme meter is disambiguated by the linguistic boundaries of meter data management. The type of meter is specified through type. In our sector, the Nbility capability model can serve as a guide for identifying bounded contexts or domains. Including the bounded context name in de event name is detailed in various DDD-sources, such as: Schemata | VLINGO XOOM.
Type: String
"type": "mdm.meter.updated"

source:

The attribute source contains a sequential unique identifier of the organization and the source system that publishes the event. This attribute reveals the source of the event (the event producer) and is a refinement of the CloudEvents-NL profile. It prescribes that the source should always be a composite attribute consisting of the organization and the source system. In the CloudEvents-NL profile, this is a recommendation. In the example below, the EAN13 of EDSN (dummy value) is used to identify the producing organization, and ‘cmr’ (central meter register) is specified as the logical source system.
Type: URI-reference
"source": "urn:ean13:8716859111111:cmr"

time:

The attribute time contains the timestamp when the source system produced the event. This guideline is a refinement of the CloudEvents-NL profile and prescribes that time must always be present as a context attribute. In the CloudEvents-NL profile, time is an optional attribute. Note that the time attribute does not contain the timestamp of when the event itself occurred, but rather the time the event was generated or created by the event producer. This explicit specification of the time attribute makes it unnecessary to use the extension attribute recordedtime. Timestamps are crucial in the energy sector to ensure proper indexing and accurate, traceable event recording.
Type: timestamp in UTC according to ISO 8601, in the format: yyyy-mm-ddThh:mm.ffffffZ (2024-05-27T13:15:01.450371Z)
"time": "2023-09-22T14:01:54.957124Z"

dataversion:

The extension attribute dataversion contains the semantic version of the event data represented by the data attribute.
This extension attribute is specific to the energy sector. The CloudEvents-NL profile does not prescribe this attribute, but enables semantic versioning to be included in the type attribute. A dedicated extension attribute, dataversion, indicates the semantic version of the event, making it more explicit for consumers to route and filter events. Furthermore, this attribute follows the semantic versioning indicated in the info.version field in the OAS description of the accompanying synchronous REST API. Alongside the event-based system, synchronous REST APIs will also be available for thin-event scenarios and as an alternative for market roles that are not yet technically equipped to connect to the event-driven systems. If an incompatible change is needed, a new type with an increased major dataversion segment must be introduced. This new type must be written to a separate topic / queue with identical major version, proposing an identical global URI versioning between REST API and event. By introducing the extension attribute dataversion, the versioning scheme of events and the accompanying REST API’s are in sync. This enables a uniform version strategy, increasing the transparency of version information for clients.
Type: String that follows the semantic versioning specification
"dataversion": "1.0.1"

datacontenttype:

The attribute datacontenttype describes how the event data is encoded. Event producers must provide the datacontenttype context attribute to enable event consumers to determine the content type of the event data.
This guideline is a refinement of the CloudEvents-NL profile and mandates that datacontenttype must always be present as a context attribute if event data is present. In the CloudEvents-NL profile, this is a recommendation.
Type: String conform RFC 2046

data:

The attribute data encompasses the event payload. This attribute does not impose any restrictions on the type of this information. It is encoded in a media format specified by the datacontenttype attribute (e.g.: application/json). Note that the data model of the event payload must match the resource model of the corresponding REST API, if it is also offered to consumers alongside the event.
This guideline is a refinement of the CloudEvents-NL profile and mandates that the data attribute must always be present if there is payload data. In the CloudEvents-NL profile, this is a recommendation.

CloudEvents-NL (extension) attributes currently not in use in the Dutch energy sector:

subject: no reason to use this attribute as an additional filter yet. The source attribute is sufficient as a qualifier for any specific event.
dataref: no need for an additional reference to a location where the event payload is stored. In case of an event notification (thin event) de data payload will contain information how to retrieve the referenced payload data.
dataschema: identifies the schema that data adheres to. Description of the data in the message should ideally be included in the complementary AsyncAPI specification or in additional documentation.
sequence: together with sequencetype this extension attribute is not used at this moment since there is a single message consumer per queue. However, once multiple consumers are reading messages from the same queue, the processing order of messages cannot be guaranteed and in that case attribute sequence can be helpful and will be added to the minimal set of context attributes.

Binary versus Structured content mode

CloudEvents offers two content modes for encoding event data in messages: binary mode and structured mode:

In binary mode, the CloudEvent attributes (except for the data and datacontenttype) are mapped to protocol metadata (such as HTTP headers) prefixed with ce-. Only the event data is sent as the message payload.
In structured mode, the entire CloudEvent, including its attributes and data, is serialized into a single format, such as JSON, and sent as the payload of the message. For this mode, the Content-Type header is set to application/cloudevents+json.

The image below shows the example event in both operating modes in an HTTP POST request:

The choice between binary mode and structured mode in CloudEvents is not straightforward. Binary mode excels in efficiency and direct payload consumption by leveraging protocol-specific features. In contrast, structured mode offers a unified, protocol-agnostic representation of events, simplifying the processing and routing of events by intermediaries.

Given that many organizations in the Dutch energy sector still have limited experience with standardized event exchange, we decided to use structured content mode as it is a relatively simple mechanism. The structured content mode seems most suitable for use across organizational boundaries. Additional decision drivers include:

Unified Representation: Encapsulates the entire event, including metadata and data, in a self-contained single format such as JSON, making it easier for consumers to process the event as a whole. The structured encoding has the benefit of being fully self contained.
Ease of Inspection: Facilitates easier inspection, routing, and transformation by intermediaries, as the entire event is in a consistent, parseable format.
Event sourcing and storing: This comprehensive representation of events aligns well with the needs of event sourcing and event stores, where complete and self-describing events are crucial.
Protocol Agnosticism: Decouples the event from the specifics of the underlying transport protocol, enhancing interoperability and flexibility in switching protocols. The structured format proves particularly advantageous in multi-hop routing scenarios. For instance, when an event is forwarded through a series of protocols—HTTP to AMQP, then to an Avro file on disk, and finally back to HTTP — the structured format keeps everything together. The structured approach allows simple forwarding across multiple protocols in various distributed systems used in the Dutch energy sector.

Patterns for API Design

Thin versus Fat Event

CloudEvents advises that event producers should try to keep the size of individual events within reasonable limits, often suggesting a size of no more than 64 KB. However, this is not a mandatory requirement and does not impose a hard limit. CloudEvents emphasizes the importance of keeping events compact:

Generally, CloudEvents publishers SHOULD keep events compact by avoiding embedding large data items into event payloads and rather use the event payload to link to such data items.

This brings us to the discussion of thin versus fat events. Among the different event types, there are two main event patterns: event notification (‘thin’) and event-carried state transfer (‘fat’). The thin event (also known as clickbait event – thank you Jacqui Read! 🙂 ) only informs the consumer that something has happened. The consumer must then fetch the necessary state information from the relevant service. The fat event itself contains all the necessary state information that the consumer needs. This reduces the need for additional queries to other services.

Privacy and security considerations play a crucial role in deciding between event notification and event-carried state transfer (‘privacy and security by design’). Requiring recipients to explicitly request detailed information, prevents the sharing of sensitive data over the messaging infrastructure and ensures that subscribers undergo additional authorization to access the information. Generally, notifications with minimal information (‘thin’ events) are more challenging to implement because they require additional effort from both producers and consumers. However, in scenarios where rich information (‘fat’ events) suffices, the fat event approach is often preferred due to its simpler implementation. In our sector, we favor the fat event approach when it can be implemented easily, responsibly, and in compliance with privacy and security by design principles. Additionally, the concept of data ownership plays a significant role in determining the suitability of thin or fat events.

Designing systems that require flexibility, extensibility, scalability, and security in an ever-evolving domain is challenging. The context and rules that CloudEvents provide are just enough, offering sufficient structure while allowing room for flexibility and extensibility.

One aspect that CloudEvents doesn’t specify is infrastructure or messaging protocols. In our sector, we’ve agreed to use AMQP for asynchronous communication across party boundaries. Various platforms implement the AMQP standard, and they can be either on-premises or cloud-based.

We implement an ‘at-least-once’ delivery strategy, which also means consumers need to ensure idempotency.

To make things a bit more tangible, below are two high-level implementation examples between systems, illustrating the fat event and thin event flows. Please note that, for brevity, we have not included any authentication or authorization steps.

Wrap-up

In this article, we outlined the essential CloudEvents context attributes required for event metadata in the Dutch energy sector. Using the CloudEvents-NL profile as a basis, we described and explained domain-specific applications and extensions of the standard, integrating CloudEvents into the Dutch energy sector’s API guidelines. These guidelines form the foundation for future developments, such as AsyncAPI implementations for publisher-subscriber contracts, and assist in creating a comprehensive event catalog based on standardized context attributes.

Thanks for reading and we value your comments and readership!

Originally published on: CloudEvents in the Dutch Energy Sector | LinkedIn

	Metadata Element: the context attributes hold control metadata primarily and act as filters. Attributes `source` and `datacontenttype` have a strong provenance association. Attributes `specversion`, `dataversion` and `id` qualify as Id Elements as well. Applying this structure pattern and categorizing metadata this way enhances the understanding of each metadata element’s purpose, role, and the information it contains. Feel free to check out one of our Adoption Stories, where we provide feedback from the practitioner community on the Metadata Element pattern.
	Attribute `data` forms a simple Parameter Tree in our example and since nesting is not advisable the context attribute envelope qualifies as an Atomic Parameter List.
	Context Representation: in case of protocol bridging, metadata can get lost in transition, not guaranteeing end-to-end quality of service. The problem that Context Reprentation pattern addresses is: … that it is difficult to ensure that control information control information exchanged between consumer and provider is able to pass each kind of intermediary (including gateways and service buses) in a distributed system successfully but remains unmodified when the underlying protocol switches. Indeed: CloudEvents are also used in distributed systems where an events can go through many hops. Solution to this problem is: Combine and group all Metadata Elements that carry the desired information into a custom representation element in request and/or response messages. Do not transport this single Context Representation in protocol headers, but place it in the message payload. Indeed: regardless of protocol switches, header information in the payload stays the same and reaches the client. The structured content mode can bee seen as application of the Context Representation Pattern to achieve protocol-independence throughout the invocation hierarchy.

Leave a Comment Cancel Reply