Meta Data Element

_{Published: 20 August 2023}

Known Uses

The Metadata Element pattern is widely used in both Dutch government APIs and Dutch Energy sector APIs. HTTP header parameters like X-Request-ID, X-Correlation-ID and ETag (to mention a few) are applied very often.

In the Energy sector, we have introduced additional custom headers:

X-Object-ID (response): alternative to the header Location in case a complete URL should not be returned in the response. X-Object-ID contains the identification element of the created resource, which makes it a known use of the Id Element pattern.
X-Principal-Token (request): contains the unique identification of the calling party (person or system) in the context of a request. The exact interpretation of the field’s content is context-dependent; it could be, for example, an OAuth access token.

Both hold control and provenance metadata. Note that we have prefixed the parameters with X (although not recommended, see for instance here and here) but we still like this simple way of identifying custom headers.

The document “API-strategie” (in Dutch) describes a specific use of metadata information: Metadata Endpoints are provided. In an overall system, APIs do not exist in isolation. When APIs form a chain or collectively realize an interface, they contribute to ecosystem awareness. Therefore, APIs actively contribute to “context awareness” by providing metadata endpoints:

Metadata Endpoint for Application Properties: This endpoint provides detailed information about the application properties such as its name, release information (version number), description, and the standards used, along with their version numbers. This information will allow API consumers to understand the characteristics and specifications of the application they are interacting with (https://.../api/v1/app-info)
Metadata Endpoint for Application Status: This endpoint will provide real-time information about the application status. It could include a simple value like UP/DOWN to indicate whether the application is currently operational or not. This status information is crucial for API consumers to assess the availability and health of the application before making any requests. (https://.../api/v1/app-health)

By making these two Metadata Endpoints available, API providers ensure that consumers have access to essential information in a standardized manner, enabling them to interact with APIs more efficiently and with greater awareness of the underlying applications. Example: https://aandeslagmetdeomgevingswet.nl/publish/pages/168043/omgevingsdocumenten-ow-validatie-v2.json (see “Meta” resources).

In the Dutch government APIs, Metadata Elements like HTTP header Content-CRS and others are used in geospatial contexts. Via de ‘Content-CRS’ (CRS: Coordinate Reference System) header, the client can negotiate on the required content in the response. Request and response may be based on another coordinate reference system. This applies the HTTP-mechanism for content negotiation: https://docs.geostandaarden.nl/api/API-Strategie-mod-geo/#crs-negotiation. This is an example of provenance and control metadata.

Zalando uses the X-Flow-ID: a generic parameter to be passed through service APIs and events and written into log files and traces, comparable with the X-Request-ID and X-Correlation-ID we use in our sector. Again, we see known uses and realizations of provenance and control metadata.

A nice example of provenance metadata is the concept of an identification scheme. A domain object can be uniquely identified by an MRID (Master Resource IDentifier). To determine the (functional) source of a given MRID, each MRID attribute possesses a supplementary attribute called identificationScheme, which classifies the MRID using a URI. By using the identificationScheme and a conversion API, it is also possible to “reclassify” a given MRID, allowing access to additional resource information from another source (where the given resource corresponds to a different MRID value).

Since metadata is as important as the data itself, we observed additional patterns, not included in the book, such as Metadata Resources and even dedicated Metadata APIs. The concept of a metadata resource is used by various companies and standards:

And Metadata APIs:

The Salesforce Metadata API is a specific API provided by Salesforce, a leading cloud-based Customer Relationship Management (CRM) platform. It is designed to allow developers and administrators to manage and retrieve customizations, configurations, and metadata for Salesforce organizations programmatically: https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_intro.htm
LinkedIn Social Metadata API: https://learn.microsoft.com/en-us/linkedin/marketing/integrations/community-management/shares/social-metadata-api?view=li-lms-2023-07&tabs=http

Metadata Resources are special types of Information Holder Resources, and Metadata APIs can be positioned as Backend Integration APIs that expose such Metadata Resources. They often come as Solution-Internal APIs or Community APIs; the above two examples are Public APIs.

Discussion Input

Many RESTful API design books discuss metadata and the various methods of incorporating metadata into request and response messages. Apart from the standard protocol-level headers, metadata can also be included within the payload and reflected in status codes. While knowing how to transmit metadata is vital in API design, understanding the distinct types of metadata, their roles, purposes, and the kind of information each variant of metadata can offer is equally essential. So far, we have not seen an API design book in which metadata is qualified and categorized as is done in “Patterns for API Design”. According to the book, metadata falls into three main categories: control, aggregated and provenance metadata elements. In short this stands for respectively: “technical usage hints” (identifiers, filters, API Keys, hypermedia controls), “statistic and calculations” (page counters, total of results) and “description and provenance” (version number, owner, location information). Categorizing metadata this way adds a better understanding of the purpose and role of the specific metadata element and the information it contains, both for provider and consumer and especially in the case of custom protocol headers! Applying the Metadata Element pattern and its metadata qualifications will certainly add value to the API design and make API usage more accurate and consistent since metadata is as important as the data itself.

An important design discussion is where to store metadata: in protocol headers or in body payload? Example: Pagination in “plain” JSON is supported by HTTP headers with metadata (X-Pagination-Page, X-Pagination-Limit), whereas HAL incorporates the pagination parameters in the payload. The default use of standard headers and header extension is strongly dependent on the chosen protocol and platform. And in case of protocol bridging metadata can get lost in transition, not guaranteeing end-to-end quality of service. We are looking forward to read the pattern Context Representation.

When a Metadata Element is part of the message payload it can be prefixed by an underscore _ to denote its metadata functionality and to distinguish it from non-metadata elements.

Known Uses

Discussion Input

Recommended Reading

Leave a Reply Cancel reply