# Information System — Architectural Explanations

This chapter tracks design decisions, resolved architectural criticisms, and the rationale behind fundamental implementation choices in the Information System (including both the Data Model and the REST Service behaviors). It serves as an architectural reference to understand *why* the system behaves the way it does.

## Abandonment of Facet Sharing Across Resources

In the initial conceptual drafts of the model (prior to its formalization), a single **Facet** instance could be attached to multiple different **Resource** instances (via **ConsistsOf** instances). 
The initial idea was to prevent data duplication. For example, if multiple **Resource** instances shared the same point of contact, they could all point to the exact same **ContactFacet** instance.

**Reflection:** This approach introduced severe operational ambiguity. If a client updated the shared **ContactFacet** instance to reflect a new email address, did they intend to update the contact info for *all* **Resource** instances it is attached to, or did they only want to assign a new contact to the specific **Resource** instance they were operating on? Ideally, the client should have detached the old **Facet** instance and created a new one, but this required strict and error-prone client logic. Updating a shared **Facet** instance could lead to massive, unintended side effects across the infrastructure.

**Decision:** This feature was completely abandoned to ensure predictability and safety. The model now strictly enforces that **a Facet instance is owned by a single Resource instance**. If a contact changes, updating the **Facet** instance (or creating a new one) only affects the specific **Resource** instance it belongs to.

## Architectural Abstraction: Reference Counting vs. Graph Internals

The API contract dictates a "Reference Counting" paradigm for the lifecycle of instances: when an instance is removed from its last Context (i.e., its reference count hits zero), the system automatically Garbage Collects (destroys) it.

**Reflection:** Under the hood, the Resource Registry does not actually use an integer counter; it relies on the internal tracking mechanisms of the underlying database and its partitioned views to determine if an instance is completely orphaned.

**Decision:** From an external API perspective, this internal graph mechanism is strictly an implementation detail. The REST API documentation deliberately abstracts this away and uses the well-known "Reference Counting" terminology. This provides API consumers with an immediately understandable, solid mental model of the lifecycle guarantees (orphan = destroyed), shielding them from the underlying database technologies.

## Hierarchical Graph Union: Pre-Computed Hierarchical Contexts

The API provides a "hierarchical capability" (via `hierarchical=true`) that applies to both reads and massive hierarchical mutations (create, update, privileged Add to Context, Admin Remove from Context). This allows users to query or mutate instances across a parent Context and all its descendants as if they were a single environment, despite Contexts being strictly isolated by default.

**Reflection:** Performing recursive queries or sequentially looping over every single child Context across isolated graph partitions at runtime is computationally expensive. It would create a massive synchronous bottleneck for reads and dangerous timeouts for massive hierarchical mutations (e.g., executing an Admin Remove or a privileged Add to Context down a tree of 100 VREs).

**Decision:** To efficiently support this feature, the Information System entirely sidesteps this O(N) traversal penalty by employing a "write-time materialization" strategy. For every flat Context in the infrastructure, the backend physically creates and maintains a companion **Hierarchical Context** (an explicit super-set partitioned graph).

When an instance is created in a Context, the write operation explicitly injects it into multiple Venn diagrams. To illustrate this, consider the D4Science 3-level hierarchy (`Root` → `VO` → `VRE`). *(Note: This 3-level depth is used strictly as a pragmatic example; the Information System inherently supports an arbitrary N-level Context hierarchy).* When an instance is created in the deepest Context (the VRE), it is physically injected (stamped) **strictly** into two local boundaries:
1. The specific `Flat Context` of the VRE.
2. The `Hierarchical Context` of the VRE.

Crucially, the instance is **NOT** physically stamped with the Context identities of its ancestors (the VO or the Root). Instead, the parent Contexts automatically gain visibility over it because the backend links the VRE's `Hierarchical Context` to the VO's `Hierarchical Context` via **Role Inheritance** within the database's security model.

> **Directionality Clarification (Domain Tree vs Security Inheritance):**
> There are two valid and complementary directions in this architecture:
> 1. **Domain hierarchy direction:** parent Context -> child Context (the business topology).
> 2. **Security inheritance direction:** child hierarchical role -> parent hierarchical user/role membership (the visibility wiring).
>
> This is why the model can look "reversed" at implementation level without any contradiction. The business tree still flows parent -> child, while effective read visibility for ancestors is achieved by linking child-side hierarchical identities into parent-side hierarchical identities.

> **Architectural Note: Pre-computing Hierarchical Contexts for leaf nodes**
> At first glance, creating a `Hierarchical Context` for a VRE might seem redundant, since it currently has no children. However, the IS-Model is generic and **does not enforce a fixed number of hierarchical levels**. 
> The system cannot predict if an administrator will dynamically create a new sub-VRE level underneath. If the backend waited until a new child level was created to dynamically generate and backfill all the parent hierarchical graphs, it would trigger a massively expensive database migration. We also ruled out using a static initialization property to hardcode the "maximum depth", because any future change to that depth would paralyze the infrastructure. Proactively materializing the Hierarchical Context at every single level ensures that the graph is inherently future-proof and structurally consistent, at zero runtime cost.

This write-time investment unlocks massive performance gains for all subsequent hierarchical operations:
- **O(1) Reads:** The Service routes hierarchical requests to this specific partitioned graph, leveraging database-level visibility to execute the query as a single, highly performant operation.
- **Optimized Hierarchical Mutations:** When an authorized role (`IS-M`, `Infra-M`, `IS-Admin`, `Ctx-M`, or `IS-Editor`, according to endpoint-specific rules) executes a multi-Context hierarchical mutation, such as Add to Context with `hierarchical=true` or hierarchical instance synchronization, the system leverages the pre-computed Hierarchical Context definitions to optimize downstream propagation without relying on expensive runtime graph traversals.

### Two Different "Implicit" Behaviors in Point Reads

A recurring API-design question is why point reads on Contexts and point reads on Instances do not use the same visibility-switching style.

**Reflection:** At first glance, both seem to be "single-resource reads", so one might expect symmetric implicit behavior. In reality, they control different dimensions:
1. **Context point reads** (`GET /contexts/{context-uuid}`, `GET /contexts/@current`) target a Context identity that is already explicit (UUID) or token-bound (`@current`). A read by UUID reaches the cemetery only for the roles that can see it, and `@current` never resolves to a deleted Context, but the semantic target of the request does not change.
2. **Instance point reads** (`HEAD`/`GET /instances/{type}/{instance-uuid}`) can either evaluate visibility in the token Context only, or in the entire descendant subtree. This is a true scope expansion of the lookup domain, not just an internal channel selection.

**Decision:** The API adopts a deliberate asymmetry:
1. For Context point reads, role-based visibility channel selection can remain implicit.
2. For Instance point reads, subtree expansion must remain explicit via `hierarchical=true`.

This preserves URI predictability and avoids hidden cross-Context broadening on instance lookups, while still enabling strict discovery->fetch scope symmetry when clients opt into hierarchical discovery.

This underlying mechanism is an implementation detail intentionally hidden behind the abstract REST API.

## Contexts as Record-Level Security (RLS) Groups

To fully understand how "Venn diagrams" are physically implemented, we must look at the database layer. A Context is not a separate database table or a separate graph instance. Instead, the Resource Registry relies entirely on **Record-Level Security (RLS)**.

Under the hood, every single Context translates into a set of database-level identities (typically a `Reader` role/user and a `Writer` role/user). When an instance is "added to a Context," the database engine simply assigns the corresponding `Reader` and `Writer` identities to that specific instance. 
Consequently, when the API Gateway validates a caller's token, it opens a database session using the specific read or write identity tied to that token's Context. The database engine natively filters out any instance that is not assigned to that identity, creating a mathematically perfect, invisible Venn diagram.

```eval_rst
*(Note on Hierarchical Contexts): The "Hierarchical Contexts" mentioned previously are implemented using this exact same mechanism. The backend maintains additional ``HierarchicalReader`` and ``HierarchicalWriter`` identities. Rather than stamping every record with multiple roles, the graph engine executes* :math:`O(1)` *visibility filtering by assigning the child's standard read/write roles to the parent Context's Hierarchical identities (Role Inheritance).*
```

```eval_rst
**The Performance Cost of Context Extraction:**
While injecting records into Contexts using RLS roles enables :math:`O(1)` visibility filtering for standard queries, the *reverse* operation—asking the system "tell me all the Contexts this instance belongs to"—is an expensive computational task. 
To generate the ``contexts`` map for a given instance, the backend must inspect the internal security arrays of the record, extract the raw ``Reader``/``Writer`` identities, and then reverse-lookup those identities against the system's Context cache to derive the human-readable Context Paths. Although a dedicated caching layer mitigates this penalty, the cross-referencing process remains inherently heavy. This is the architectural reason why the ``contexts`` map is omitted from default API responses and must be explicitly requested.
```

**The Scalability Critique (Number of Identities):**
A valid counter-critique is: *Doesn't maintaining all these parallel hierarchical identities bloat the security layer and exhaust the database's internal limits?* 
Assuming a 3-level infrastructure (Root -> VO -> VRE) with ~200 Contexts total, we have `200 Contexts * 2 (Flat/Hierarchical) * 2 (Reader/Writer identities) = ~800 security identities`. 
The underlying database engine easily handles record-level security by linking records to these 800 identities. This is a trivial number for any modern database engine.

**Future-Proofing (Migration to PostgreSQL RLS):**
While the system currently relies on OrientDB, as the framework plans a future migration to PostgreSQL, this architecture maps perfectly to PostgreSQL's native **Row-Level Security (RLS)**. In Postgres, we won't even need to create 800 native database identities. The infrastructure can simply pass the user's authorized Context UUIDs as a session variable (`SET app.visible_contexts = '...'`), and the RLS policy will evaluate array intersections in microseconds (`WHERE context_id = ANY(current_setting('app.visible_contexts')::uuid[])`). The current write-time materialization model perfectly primes the system for this highly efficient Postgres transition.

### Context Re-Parenting, Renaming, and Subtree Migration

Because the hierarchical visibility is entirely based on role inheritance (`ORestricted` roles in OrientDB) rather than statically copying hierarchy paths into every single instance, migrating an entire Context subtree (Context Re-Parenting) becomes an extraordinarily elegant and performant operation.

When an administrator updates the `parent` of a Context via a `PATCH` request, the system does **not** need to traverse and update the hundreds of thousands of instances contained in that Context and its descendants. Instead, the backend simply recalculates the role inheritance tree in the security model (using OrientDB's native role hierarchy). The new parent Context's identity simply "inherits" the roles of the migrated child Context, and the old parent instantly loses them. 

```eval_rst
The underlying instances remain physically untouched because their internal access-control fields still simply point to their local Context roles. The database engine's real-time evaluation of the updated role hierarchy guarantees that the new parent immediately gains :math:`O(1)` visibility over the migrated subtree, while the old parent instantly loses it. This eliminates the need for complex, time-consuming data migration protocols during a structural reorganization of the infrastructure, rendering the critique of "missing migration protocols" completely moot.
```

```eval_rst
> **Administrative Responsibility in Subtree Migrations:**
> Because this operation is :math:`O(1)` and intrinsically "carries over" all instances belonging to the migrated Context and its descendants into the new parent's visibility scope, the System Administrator must carefully evaluate the business impact. Migrating a subtree means moving live data from one hierarchical branch to another. To ensure state consistency during this massive topological shift and prevent accidental "live exposure", the API strictly enforces that **a Context can only be reparented or renamed if its local ``state`` is non-``active``** (e.g., ``suspended``). It is entirely the administrator's responsibility to determine if the new parent *should* inherit visibility over all those existing instances, or if a manual data cleanup (removing specific instances) is required *before* executing the ``PATCH`` on the Context tree. The framework trusts the administrator's authority to make these sweeping topological shifts once the branch is safely locked down.
```

## "System" Environments (Infrastructure Metadata)

The beauty of the underlying database-level filtering (RLS) is that it is universally applied not just to standard instances, but to the framework's internal infrastructure. 
Because native database schema management engines are insufficient to capture the rich, dynamic concepts of the IS-Model, the framework uses the same strategy to store Types, Context representation, and Query Templates representation as system records protected through the same storage/RLS mechanism.

To protect them using the exact same RLS engine without mixing them with user instances, the system utilizes hardcoded **"Service Venn Diagrams" (System Environments)**. These Environments use highly recognizable, reserved UUIDs (where all characters are identical) to distinguish them instantly from predictable, standard Context UUIDs:
- **Contexts Definition Environment:** `ffffffff-ffff-ffff-ffff-ffffffffffff` (Where the representations describing the VREs/VOs themselves are stored).
- **Types/Schema Environment:** `eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee` (Where the definitions of **Resource**, **Facet**, **IsRelatedTo**, **ConsistsOf**, and **`Property`** schemas reside).
- **Query Templates Environment:** `dddddddd-dddd-dddd-dddd-dddddddddddd` (Where stored queries reside).
- **Admin Environment:** `00000000-0000-0000-0000-000000000000` (Depending on the implementation, this DB identity acts as a macro-view bypassing RLS filtering to see everything).

> **Universal Reservation Rule:** It is important to note that **any UUID composed entirely of the same repeating hexadecimal character** (e.g., `1111...11`, `aaaa...aa`) is strictly reserved by the system, even if it is not currently assigned to a specific System Environment. They are reserved globally to ensure availability for future infrastructural needs. If a client attempts to use or inject one of these reserved UUIDs in any standard REST operation (e.g., as a new instance ID or a target Context ID), the system will instantly reject the operation with a `400 Bad Request`.

**System Environment Isolation & Graph Partitions:**
To fully grasp the elegance of this design, one must understand that the backend physically realizes any **Context** (e.g., a standard VRE) and any System Environment (e.g., the **Types/Schema Environment**) through isolated **graph partitions** (implemented via database-level identities like users and roles).

Every single graph partition in the database is strictly bound to a UUID. 
- For **Contexts**, the partition's internal identities are built using the public UUID of the **Context** instance itself (e.g., `ReaderUser-<UUID>`).
- For System Environments, the backend leverages the exact same partitioning logic but permanently binds them to the reserved, repeating-hex UUIDs detailed above (e.g., `ReaderUser-eeeeeeee...`).

The user's authorization token does **not** contain a UUID. The token securely identifies the **Context** path, and the API Gateway / backend maps this path exclusively to a valid operational **Context** UUID. Because this mapping logic mathematically resolves only to operational partitions, **it is impossible for a token to inadvertently bind to a System Environment** (like `eeeeeeee...`). 

> **What about Hierarchical Contexts?**
> A sharp reviewer might ask: *If standard Contexts and System Environments are native partitions, what about the "Hierarchical Contexts" discussed earlier?* 
> They are implemented leveraging the exact same RLS engine. Rather than inventing a completely new UUID or a separate entity, the backend simply creates parallel hierarchical identities appended with the exact same base **Context** UUID (e.g., `HReaderUser-<UUID>`). By managing standard and hierarchical visibility through these parallel role suffixes natively tied to the same UUID, the system uniformly handles standard Contexts, Hierarchical Contexts, and System Environments via the exact same highly optimized DB-level engine.

Furthermore, when the API manages Contexts (e.g., via the `/contexts` collection), this operation serves a dual purpose: it creates/updates the representation of the **Context** stored securely inside the **Contexts Definition Environment**, and simultaneously manages the creation of the underlying **Context** partition (the actual RLS Venn diagram) where user instances will be placed. 
This strict decoupling guarantees that System Environments are entirely invisible to standard data-graph operations (like `GET /instances/{type}?hierarchical=true`) and are safely walled off, manipulated exclusively by the backend logic.

**Rationale for UUID-based System Environments:**
One might wonder why system infrastructure requires UUIDs instead of simple string tags. The system internally models every partition as an "Environment". Using a universal UUID-based approach provides three critical advantages:
1. **Implementation Uniformity:** The underlying security and persistence layers treat a system environment exactly like any standard user VO or VRE. The DB filtering logic remains identical and highly optimized.
2. **Visual Diagnosability:** Using reserved UUIDs made of identical repeating characters (e.g., `ffff...`) makes it instantly recognizable to an administrator that a record belongs to the system infrastructure and not to a user Context.
3. **Security Filtering:** It provides a deterministic way to reject malicious or malformed client requests. If a standard user API call attempts to inject or reference one of these reserved UUIDs, the API Gateway or the service itself can instantly trap and reject the request.

Furthermore, using UUIDs universally guarantees absolute decoupling from mutable properties. For instance, while a Context's `path` (e.g., `/gcube/devsec`) might be renamed during its lifecycle, its UUID remains constant, preserving the Context membership of all instances.

By treating the infrastructure representations as just another set of partitioned records by these reserved UUIDs, the Resource Registry achieves an incredibly elegant, unified security model where everything—from a user's dataset to the core schema definition itself—is protected by the exact same database-level RLS engine.

## Type Evolution and Backwards Compatibility (Static vs Dynamic Schema)

A recurring debate in schema-mixed graph databases is how to handle the evolution of Type definitions. Critics often suggest that the API should support dynamic, online "magic" schema updates (e.g., automatically injecting new mandatory fields or rewriting schemas on the fly via `PATCH /types`).

**Decision:** The architecture deliberately rejects online, dynamic schema mutation via REST in favor of strict, code-driven backwards compatibility. 
Changing a Type definition is not a trivial JSON update; conceptually, it is equivalent to executing an `ALTER TABLE` in a relational database. This operation is exclusively the domain of a Database Administrator. It requires a profound impact assessment on existing instances (the "rows" in the DB), decisions on default values, and evaluating whether to retain old fields as extra fields (leveraging the schema-mixed mode) or completely drop them. 

Furthermore:
1. **Data Migration:** Allowing a Type update via REST would inherently mean passing a complex data migration procedure within the payload to handle existing instances, which contradicts the simple declarative nature of a Type definition.
2. **Backend Limitations:** Some structural changes require offline operations because the underlying backend database simply does not support them online. Similar to relational DBs, certain migrations require dumping the DB, creating the new schema in a fresh instance, and establishing a restore procedure that maps the old dump to the new model.
3. **Client Fragility:** Even if instances could be flawlessly migrated by the backend in lenient mode, an online schema change could silently break existing clients. For example, if an integer field (validated via min/max) is converted into an enumerated string (validated via regex), instances might migrate successfully, but old clients expecting an integer will instantly crash upon deserialization.

Instead, Type evolution is treated as a strict engineering procedure governed by code definitions (the base `information-system-model` and the specific `gcube-model` for D4Science). While the infrastructure allows communities to dynamically define and inject their own custom Types at runtime, Java clients gracefully handle unknown custom Types by automatically instantiating the closest known ancestor (thanks to the supertypes mechanism). When a Type must evolve, the changes are mapped directly into the Java classes using Jackson annotations (`@JsonSetter`, `@JsonIgnore`, `@Deprecated`). This "Copy-on-Write" and field-aliasing approach ensures that the backend can seamlessly deserialize old JSON formats from legacy clients while serving the new format to updated clients. It favors robustness, determinism, and backwards compatibility in production over the theoretical utopia of dynamic schema-less injection.

**Future Directions (Safe Online Evolution):**
In future iterations, the architecture aims to identify specific scenarios where an online update is intrinsically safe (for example, the addition of a new non-mandatory field). Formalizing these safe mutations will carry a dual implication:
1. The server will be enhanced to accept these specific, non-breaking updates via REST.
2. Clients will be contractually required to be resilient against these safe evolutions (e.g., dynamically ignoring unknown fields or handling missing optional fields gracefully without crashing).

## Elimination of `notnull` as an Independent Property Constraint

Earlier iterations of the `PropertyDefinition` schema exposed two ostensibly orthogonal booleans for constraining a property's value: `mandatory` (the property must be present in the payload) and `notnull` (if present, the property's value cannot be explicitly `null`). A property could therefore be `mandatory: false, notnull: true` (optional, but must be a real value if given), or `mandatory: true, notnull: false` (always present, but its value could be `null`).

**Reflection:** Under `PUT`'s full-replace semantics, omitting a key from the body can only mean one thing — the property is absent from the resulting representation — since that is what "replace with exactly this" already requires for anything not mentioned. The only genuine design freedom left is whether an explicit `null` should be treated *differently* from omission, rather than collapsing into the same outcome.

A natural objection at this point: why not simply accept that `PUT` supports its own three distinct states — omitted, explicit `null`, and a real value — independently of whatever `PATCH` does? The reason is not merely that it would disagree with `PATCH`'s own merge-patch convention (RFC 7396, where `null` already means "delete this member"), although that disagreement is a real, additional cost, forcing every client to remember a different meaning for `null` depending on which HTTP method it happens to be using. The deeper reason is that no client ever has a genuine reason to prefer one of the two spellings — omission or explicit `null` — over the other for the same intention ("I have no value for this property"). The two spellings are an accident of what JSON syntax happens to allow, not two different facts a client is ever trying to convey. Whenever a server draws a behavioral distinction that has no corresponding distinction in client intent, that distinction becomes a trap: it fires whenever a client's own serialization path happens to produce one spelling instead of the other, for reasons entirely unrelated to what the client meant (a class deserializing an absent field into a language-level `null`, a form library initializing untouched fields to `null` for internal consistency, and so on) — not because the client deliberately chose it.

Once `null` and omission are treated as the identical client intention, `notnull` on a property that is not `mandatory` becomes permanently unreachable by any client action: an optional property can always be omitted, achieving exactly the outcome an explicit `null` would have achieved, so no request could ever be constructed that violates `notnull` there. The only place it retained any observable effect was paired with `mandatory: true`: an update or creation providing no value for the property could either be rejected (`notnull: true`) or silently persisted with the key retained and its value set to a literal `null` (`notnull: false`) — the same client action producing two different persisted outcomes purely depending on the property's own `mandatory` flag:

<!-- Deliberately tagged `json` although it contains comments and two objects: the highlighter
     cannot parse it and leaves the block uncoloured, which reads better here than the `javascript`
     colouring. The build warning "Could not lex literal_block as json" on this block is expected. -->
```json
// Same request intent — "no value for phone" — two different persisted outcomes,
// depending purely on the property's own `mandatory` flag in the schema:

// phone is NOT mandatory → the key disappears from the instance entirely
{ "name": "Luca", "email": "luca@d4science.org" }

// phone IS mandatory, notnull: false → the key is retained, valued null
{ "name": "Luca", "email": "luca@d4science.org", "phone": null }
```

But even that residual case is internally inconsistent: marking a property `mandatory` declares the information essential, while `notnull: false` simultaneously permits its value to carry no information at all — preserving only the syntactic shell (the key is present) while quietly discarding the substance the constraint was meant to guarantee. The one concrete benefit of retaining a `null`-valued key rather than an absent one — accommodating client deserializers that assume a fixed key set rather than tolerating a missing one — is a narrow implementation convenience for such clients, not a genuine requirement of the domain model.

**Decision:** `notnull` was removed as an independent, separately-settable field on `PropertyDefinition`. `mandatory` alone now carries the full weight of the constraint: a `mandatory` property must always be present with a genuine, non-`null` value (a request providing neither is rejected, or resolved via the property's declared `defaultValue` when it has one); a non-`mandatory` property may be entirely absent, and in a full representation (`POST`, `PUT`) submitting `null` for it has exactly the same effect as omitting it outright — one meaning, one outcome, regardless of the property or how deep inside a nested property value it sits. `PATCH` is the one exception, by its own merge semantics ([RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396)): there an omitted property is left unchanged and `null` removes it.

## The Structural Delta (Controlled Evolution vs Anarchy)

The Open-World Assumption (both its Topological Extensibility and its Schema-Mixed Data Extensibility) allows a degree of flexibility that does not result in an anarchic evolution of the data model; the reality is quite the opposite.

To ensure long-term maintainability, the system strongly types and segments information into small, highly reusable blocks (the **Resource**, **Facet**, **IsRelatedTo**, **ConsistsOf**, and **`Property`** schemas, akin to the LEGO metaphor introduced previously). This enforces strict structural governance: the system dictates exactly how core components connect and what their fundamental properties are. For instance, pairing the exact same **ContactFacet** schema with different **ConsistsOf** subtypes (e.g., **HasOwner** versus **HasCurator**) yields entirely different business semantics from the exact same foundational blocks.

However, within this strict governance, the architecture deliberately leaves a structural "delta" of extensibility—a margin of maneuverability. By allowing the injection of arbitrary property fields or the topological attachment of unforeseen **Facet** instances, the system grants operators the essential flexibility needed to accommodate unpredictable requirements, corner cases, and dynamic telemetry without forcing a formal, system-wide schema migration for every minor deviation.

## Query Template Evolution (Online but Admin-Gated)

Similar to Type evolution, modifying or deleting a stored Query Template carries the risk of silently breaking dependent clients. If a dashboard or a microservice relies on a specific template `name` or parameter signature, altering it will cause runtime crashes.

**Decision:** While Types cannot be modified online via REST because they require complex database schema migrations, Query Templates *can* be updated online because they are stored definitions with no dependent database schema. An update is a versioned upsert: it is accepted only with a `version` higher than the persisted one, advanced as far as the change requires (see [Versioned Upsert Semantics](10-api-query-templates-and-access.html#operational-notes)). Query Template creation, modification, and deletion require an Information System administrative role (`IS-Admin`, `IS-M`, or `Infra-M`). This lets authorized callers maintain Query Templates without granting them Type-management or whole-system maintenance capabilities.

Administrators are expected to perform an impact assessment before altering a template. This is achieved by inspecting the **Accounting** (Analytics) systems of the infrastructure to verify if any clients are still actively invoking the template. Furthermore, to maintain decoupled consistency, Query Templates cannot be renamed via the API; an administrator must explicitly delete the old template and create a new one, ensuring a deliberate transition.

## Sensitive Values Are Queried Through the Caller-Visible Representation

Filtering a query against a stored sensitive value and redacting only the returned representation is not privacy preserving. Even when the field itself is hidden, the presence or absence of a row, result counts, ordering, pagination boundaries, groups, and aggregates can reveal whether the secret satisfied a predicate. Removing the matched row or only the sensitive branch after evaluation has the same defect: the observable shape of the result still depends on the hidden value.

Rejecting every query that references a sensitive field avoids that disclosure but makes sensitivity an authorization error and prevents one shared Query Template from serving callers with different visibility. It also creates a special case that does not generalize cleanly when sensitivity can be attached to arbitrary fields.

**Decision:** every supported query operation is evaluated against the caller-visible logical representation, as if the authorization and privacy transformation had occurred before query evaluation. This privacy view precedes and is independent of response shaping: enrichment and projection controls decide which fields are returned, but do not change the values available to predicates or operators. A caller entitled to a sensitive value queries its stored value. A caller who sees a redacted sentinel queries that sentinel; a real secret therefore cannot match, while the sentinel can. If a field's privacy contract hides the field completely instead of emitting a sentinel, queries treat it as absent. Implementations may optimize this evaluation, but the observable result must be identical to querying the transformed representation and must never depend on a hidden stored value.

For the current system identity fields, `metadata.createdBy` and `metadata.lastUpdateBy` are exposed to non-privileged roles as the string `"[REDACTED]"`, so predicates and every other value-consuming query operation use that string. The same rule applies after Query Template variable substitution. Merely referencing a sensitive field is not a `403 Forbidden` condition; `403` remains reserved for an inaccessible endpoint, scope, operation, or parameter.

This decision is independent of how future field-level sensitivity is declared or how values are protected at rest. Any such mechanism must define the caller-visible representation for each supported data type and preserve this query invariant.

## Abandonment of RESTful Transactions in favor of Atomic Operations

In the early stages of the Information System model, specific design choices like using deeply nested paths were influenced by the need to make transactions more efficient from a locking perspective. However, the requirement to perform complex transactions—often to resolve issues that have now become structural, such as the "Chicken-and-Egg" instantiation problem—introduced significant architectural and integration complexity.

**Reflection:** To provide more powerful and easier-to-use APIs for integrators, the explicit support for formal RESTful transactions within the Information System was entirely removed in favor of simpler, atomic operations and batching patterns. 
The interesting outcome of this process was that the extensive design experience and locking strategies originally developed for IS transactions were generalized and successfully applied to pure REST systems. This led to the subsequent publication of the ReLock transaction model [^frosini-2021]. 

**Decision:** While formal REST transactions (where the client explicitly opens, operates, and commits) were ultimately abandoned within the IS itself to reduce client complexity, every ordinary API mutation is fully atomic and executes within a single, backend-managed database transaction.

This strict atomicity guarantee includes massive multi-Context mutations such as updates with `hierarchical=true` or `syncExistingHierarchy=true`. There are no "split-brain" scenarios, partial commits, or "best-effort" eventual consistency models within one such request: if a mutation encounters a schema violation or an optimistic locking conflict in the 99th descendant Context, the entire operation is rolled back.

`System Restore` is deliberately outside this ordinary mutation contract. It is an offline maintenance procedure that is accepted only against an empty target and recreates the registry through a topologically ordered sequence of writes. The backend does not wrap that complete sequence in one database transaction; an unexpected persistence failure requires the partially populated target to be recreated before retrying. There is no pre-existing registry state to preserve or merge.

## No Bulk Setting of Context Memberships

A single request adding an instance to some Contexts and removing it from others, or replacing its whole set of Contexts, was designed and withdrawn. Its outcome depends on the order of evaluation — additions before removals, removals before additions, or a computed final set — and the orders are not equivalent: when the request also removes the token's own Context, that Context is the source whose visible **Relation** instances drive the propagation of the additions, so removing it first leaves the additions without a well-defined source topology, while adding first copies from a source that is then removed. Rules to pin down every case (forbidding the removal of the token's Context, requiring it in a full replacement) made the operation hard to reason about, and a caller misjudging the scope of a replacement could remove instances across many Contexts at once.

**Decision:** Context membership changes identify one destination or current Context at a time. `Add to Context` and `Remove from Current Context` are each atomic and return the instances they affected. A privileged hierarchical Add still names one destination root and expands from it explicitly; global hierarchical removal remains a separate `/admin` operation because of its destructive semantics.

## Privileged Add to Context Uses the Canonical Membership Endpoint

`IS-Editor` may create a **Resource** instance with `hierarchical=true`, but it cannot execute a
hierarchical Add to Context. The latter triggers an avalanche over topology that may have changed
since the caller last read it, so it requires `IS-Admin`, `IS-M`, or `Infra-M` and supports
`dryRun=true`.

A creation distributes what its payload describes: the caller knows, before sending the request, every instance the operation will write in the descendant Contexts.

A membership change on an existing instance triggers the propagation avalanche of everything related to that instance at the moment the request is executed, including **Relation** instances created by other callers after the caller last read the graph. Applied hierarchically, that avalanche is multiplied across every descendant Context. Its extent therefore cannot be known in advance by the caller, and a request that looked bounded when it was prepared can add, or remove, far more than intended across the subtree. Removal is the more dangerous direction, since removing a **Resource** instance from a Context also drops its **IsRelatedTo** instances there.

**Considerations weighed:**

- **"The avalanche follows propagation constraints someone chose."** It does: an `add=propagate` constraint is a modelling choice, made by whoever knows the model and its consequences. But the caller of a hierarchical membership change is usually not the author of the relations it drags along; they may have been created later, by other services. One author's mistake would become every descendant Context's problem.
- **"A single `Add to Context` is just as unpredictable."** It is, and `dryRun` and `propagationMode=shallow` mitigate it. What the hierarchical form adds is the multiplication across the subtree, and the atomicity turns a single failing descendant into a rollback of the whole operation: the operation becomes both wider and more brittle.
- **"Then hierarchical creation without hierarchical sharing is inconsistent."** The need behind it — push *my* **Resource** instance down a subtree — is met without the avalanche by the hierarchical `PUT`, which carries the **Resource** instance with the structure its payload describes and no **IsRelatedTo** instances (`09-api-instances-and-admin.md`, *Hierarchical Distribution*). Pushing an instance *together with everything attached to it* is the dangerous case, and it stays with the roles entitled to it. The hierarchical `PUT` is not an add at all: it synchronizes the caller's payload over the whole subtree, whether or not the **Resource** instance is already present, and propagation constraints play no part in it, since the caller states explicitly what every descendant must hold. Only where the **Resource** instance is absent does its effect resemble an admin `surgical` add of the **Resource** instance, which however follows the propagation constraints of its **ConsistsOf** instances, while the payload does not; where the **Resource** instance is present, the hierarchical `PUT` also removes whatever the payload omits.

**Decision:** every Add to Context variant uses the canonical membership URI
`PUT /instances/{type}/{instance-uuid}/contexts/{new-context-uuid}`. Role-gated parameters express
the escalation: `propagationMode=surgical`, `hierarchical=true`, and
`bypassHierarchyCheck=true` require `IS-Admin`, `IS-M`, or `Infra-M`. This keeps one client method
and one semantic operation while preserving an explicit opt-in for every high-impact behavior.
Hierarchical removal remains under `/admin`: unlike addition, it can erase membership and related
topology throughout a subtree or the complete system.

## Protocol Transparency in Error Handling

Many modern enterprise frameworks (e.g., Spring Boot) and specifications (like JSON:API) default to returning verbose JSON error payloads that explicitly duplicate HTTP protocol metadata within the body. A typical error payload often looks like this:
```json
{
  "timestamp": "2026-04-16 12:09:26.804 +0200",
  "status": 409,
  "error": "Conflict",
  "message": "Missing mandatory Facet instance...",
  "path": "/instances/EService/1a50a46d..."
}
```

**Decision:** The Resource Registry API explicitly rejects this verbose envelope pattern in favor of a minimalist, ROA-compliant (Resource-Oriented Architecture) approach. When an error occurs, the API returns only the domain-specific business details in the JSON body:
```json
{
  "message": "Missing mandatory Facet instance..."
}
```

This strict adherence to Protocol Transparency is driven by four core ROA principles:
1. **Separation of Concerns (No HTTP Tunneling):** In a pure REST architecture, HTTP is the application protocol, not a dumb transport tunnel. Protocol-level metadata belongs exclusively in the protocol headers. Forcing a client to parse a JSON body to discover that a `409 Conflict` occurred violates this principle.
2. **Single Source of Truth (Preventing Ambiguity):** Duplicating the status code inside the JSON creates a risk of contradiction. If a reverse proxy or API Gateway intercepts a timeout and changes the HTTP header to `504 Gateway Timeout`, but the backend had already serialized `"status": 409` in the JSON body, the client is left with conflicting information. The API ensures the HTTP status line is the sole, authoritative source of truth for the request outcome.
3. **Network Infrastructure Visibility:** Intermediaries like WAFs, Gateways (SmartGears), and proxies (HAProxy) rely on HTTP headers to perform logging, alerting, and routing. They do not deeply inspect JSON payloads. By relying on HTTP status codes and standard headers, error metrics are natively visible to the network infrastructure.
4. **Eliminating Redundancy:** The JSON body should explain the "Why" (the domain-specific reason), not the "What" or "Where" (which the protocol already knows). The HTTP status code provides the "What", and the "Where" (`path`) is inherently known by the client that originated the request (or is formally returned via the standard `Content-Location` HTTP header).

By keeping the error JSON strictly focused on business logic (`message`), the API maintains a clean, unambiguous, and highly protocol-compliant contract.

The `message` reports the first violation found, not every one: validation stops at the first failure. A payload listing every failed constraint was considered and not adopted, since it would require every validation path to collect violations instead of failing fast.

## Over-Regulation vs. Pragmatic Power (The "Straitjacket" Fallacy)

A recurring critique in formal API reviews is the apparent "danger" of giving unprivileged writers (`IS-Editor`) the power to alter critical structural data. For example, critics often point out that allowing an `IS-Editor` to mutate a `PropagationConstraint` on an existing relation via `PATCH` is a massive security hazard, because changing `cascade` to `keep` alters the fundamental destruction logic of the infrastructure. They suggest locking these fields down or gating them behind `IS-Admin` roles.

**Decision:** The architecture actively rejects this hyper-regulation in favor of operational pragmatism. This decision is rooted in the history and operational realities of the infrastructure:

1. **Consistency of Trust:** If an `IS-Editor` is trusted to create an instance and define its `PropagationConstraint` in the first place, they must conceptually possess the authority to correct it if they made a mistake. Restricting the update to an admin creates an artificial bottleneck.
2. **The Price of Automation:** The previous generation of the Information System used a radically simplified model. It was universally loved because it could be learned in five minutes and presented no complex "dangers." However, it catastrophically failed to efficiently represent the real-world complexity of a federated infrastructure, forcing human operators to spend countless hours manually synchronizing states across environments. The current IS intentionally embraces complexity (e.g., the Avalanche Effect) to automate reality. While the power of cascading graph mutations can be intimidating, it is the necessary trade-off for a system that actually models a complex world accurately.
3. **Audience and Environment:** The IS is not a public wiki open to the internet. It is a highly controlled backbone designed to be operated primarily by automated services and expert human operators within a secure infrastructure. Human errors will happen—an operator might accidentally delete a core **Resource** instance, or break an endpoint URL in an **AccessPointFacet** instance, rendering a service unreachable. It is impossible to "baby-proof" a complex infrastructure against every conceivable human error or bad query without turning the system into an unusable straitjacket.
4. **The SDLC Defense:** Automated services mutating the IS are subject to strict Software Development Life Cycles (Dev -> Pre-Prod -> Prod). Bugs in client logic are addressed through staging and testing, not by paralyzing the central registry's API with draconian, micro-managed permission checks on individual structural fields. 

In summary, the Resource Registry is designed as a powerful engine for those who know how to drive it. We accept the theoretical risks of "tampering" to preserve the agility, expressiveness, and automated power required by modern federated infrastructures.

## The Nature of Federation: Modeling vs. Active Publishing

A common misunderstanding regarding the term "Federated Research Infrastructure" (which often triggers theoretical security critiques from external reviewers) is the assumption that external federated entities actively write or "push" their data directly into the central Resource Registry. This leads to theoretical concerns about external actors injecting malicious data or tampering with queries, or internal actors secretly cross-pollinating Contexts (e.g., using `bypassHierarchyCheck`).

**Decision:** The architecture explicitly separates the *act of federating* from the *Information System*. The Resource Registry is strictly the "Database of the Infrastructure". 
Federation does not mean external systems write to the IS. It means the central infrastructure discovers external infrastructure resources (e.g., via Harvesting services) and *models* them centrally within the IS. 

This is where the Facet-based design shines: the central infrastructure can represent a generic third-party computational node as a base **Resource** instance, but attach a highly specialized **Facet** instance to it to capture the unique nuances of that specific federation's integration. 

Because it is the central infrastructure itself (via trusted internal microservices and administrators) that writes to the IS, operations that "break boundaries"—such as an `IS-Manager` injecting a harvested dataset across divergent subtrees via `bypassHierarchyCheck`—are not security vulnerabilities or "unreachable islands of data." They are deliberate, centralized modeling decisions. The IS simply provides the flexible model to represent these resources; the actual policies, security filtering, and usage rules are enforced by the specific services that consume the registry, not by the registry itself. 

## REST API: Collection URIs vs Schema Representations

A common observation regarding the Type Management APIs is the apparent dissonance between the endpoint URI and its JSON payload. The API exposes `PUT /types/{type-name}` to register a new type, yet the JSON payload contains a `type` discriminator such as `"type": "ResourceSchema"`. 

**The Reflection:**
Why not use `PUT /schemas/{type-name}` to perfectly align the URI with the payload representation?

**The Architectural Choice:**
This separation perfectly embodies the principles of REST and Resource-Oriented Architecture (ROA):
- The **URI** (`/types/EService`) identifies the *logical resource* being managed within the domain. The Resource Registry manages a "Type System". Administrators are fundamentally registering and discovering "Types".
- The **JSON Payload** (`{"type": "ResourceSchema"}`) is the *representation* of that resource. It dictates the structural validation rules, properties, and cardinalities of the Type. In modern IT terminology (JSON Schema, OpenAPI Schema), a structural blueprint is universally called a *Schema*.

By keeping the URI as `/types` and the payload representation explicitly as `*Schema` (e.g., `ResourceSchema`, `FacetSchema`), the architecture cleanly resolves the historic cognitive dissonance between "Type as a semantic label" and "Schema as a structural blueprint". The URL conceptually addresses the Type, while the body provides its Schema representation.

## Naming Conventions

The Information System is designed around one naming convention for the names it defines, applied in the model, in the stored data, and in the API alike:

| Identifier | Convention | Examples |
|------------|------------|----------|
| Property and field names (model, stored data, JSON) | lowerCamelCase | `consistsOf`, `isRelatedTo`, `propagationConstraint`, `createdBy`, `previousPaths` |
| Query parameter names | lowerCamelCase | `metadataPolicy`, `membershipScope`, `syncExistingHierarchy`, `bypassHierarchyCheck` |
| Query Template variable names | `` followed by lowerCamelCase | `state`, `$entryName` |
| Enumerated values | lowerCamelCase (single words are therefore all lowercase) | `active`, `subtree`, `propagate`, `cascadeWhenOrphan` |
| Type names | UpperCamelCase (PascalCase) | `ConsistsOf`, `EService`, `SoftwareFacet` |
| Query Template names | lowercase, hyphen-separated words (kebab-case) | `get-metadata-profile-by-name`, `is-monitoring-all-eservice` |

**The Reflection:**
Why lowerCamelCase rather than `snake_case` (`consists_of`, `metadata_policy`), which many data models and public APIs adopt?

```eval_rst
**The Architectural Choice:**
No convention is universally right: both are widespread, and what matters is that the model, the data, and the API never mix them. lowerCamelCase was chosen because:
1. **One convention for fields and parameters:** a client sees JSON field names and query parameter names side by side; they follow the same rule, so a field can be referenced as a parameter or a query key without translation.
2. **One name from model to wire:** a property has the same name in the Type definition, in the stored instance, in the JSON representation, and in a query. No renaming layer exists between them, in the service or in any client.
3. **Distinct from Type names:** Type names are UpperCamelCase, as class names are in object-oriented modelling. The initial letter alone tells a Type (``ConsistsOf``) from a field (``consistsOf``).
4. **Common practice:** it is the convention of large API ecosystems (Kubernetes, Google APIs, Microsoft APIs) and of JSON-based toolchains generally.
5. **Query Template names:** a Query Template is an operation called by name, and its name is the path segment that identifies it (``/query-templates/get-metadata-profile-by-name``). It therefore follows the convention of URL path segments described below, lowercase with hyphen-separated words, which also rules out two names that differ only in capitalization. Names must match ``^[a-z][a-z0-9]*(-[a-z0-9]+)*$``; a create or update with any other name returns ``400 Bad Request``.
```

Clients that prefer another convention map the names in their own serialization layer; the API does not offer alternative spellings.

**Names governed by an external standard:** Two kinds of name follow the convention of the standard that defines them, not the one above:
- **HTTP headers** follow the HTTP conventions, for their names and for their values. Names are hyphen-separated words with initial capitals (`X-Result-Scope`, `X-Resolved-Context-Path`), as the standard headers are (`Content-Type`, `Content-Location`); they are case-insensitive ([RFC 9110 Section 5.1](https://datatracker.ietf.org/doc/html/rfc9110#section-5.1)), so clients must not depend on the capitalization they receive. Values, their tokens, and their parameter keys are lowercase with hyphen-separated words (`state-filtered`, `previous-path`, `ignored-query-parameters`, `requested-value`), as in the standard headers (`Cache-Control: no-cache, max-age=3600`, `Prefer: return=minimal`, `Link` relation types such as `latest-version`) and as the HTTP Structured Field Values format requires for keys ([RFC 8941 Section 3.1.2](https://datatracker.ietf.org/doc/html/rfc8941#section-3.1.2)). Query parameter values are not header values: they are API data and follow the lowerCamelCase convention above.
- **Fixed URL path segments** are lowercase, with words separated by hyphens (`/query-templates`, `/raw-query`, `/contexts`). URL paths are case-sensitive ([RFC 3986 Section 6.2.2.1](https://datatracker.ietf.org/doc/html/rfc3986#section-6.2.2.1)), so an all-lowercase form leaves no room for capitalization mistakes, and the hyphen is the common word separator in URLs. Path *variables* are data, not names defined by the API: a `{type-name}` segment carries the Type name as it is (`/instances/EService`), and Type names follow the class-name convention of the table above.

## REST API: Canonical Identifiers for System Objects (Name vs UUID)

For system-level objects like Types and Query Templates, the exact, case-sensitive `name` is the stable public identity. Their JSON representations also include an `id` UUID, yet the REST API strictly addresses them by `name` (e.g., `/types/{type-name}`, `/query-templates/{name}`). The API intentionally forbids using the UUID in the URL path.

**The Reflection:**
Why not allow addressing these resources by both their name and their UUID for maximum flexibility (e.g., accepting both `GET /types/EService` and `GET /types/1234-5678`)?

**The Architectural Choice:**
Restricting the path strictly to the `name` is a foundational REST best practice driven by several factors:
1. **Preventing URL Aliasing (Cache Reliability):** If the API allowed both paths, HTTP caches (like reverse proxies, CDNs, or browser caches) would treat them as two distinct objects. If a template were updated via its name, a cache hit on its UUID path would return stale data. Forcing a single "Canonical URI" eliminates aliasing and guarantees cache reliability.
2. **Domain Language over Database Artifacts:** Types and Query Templates are configuration entities designed to be authored and referenced by human developers and code strings (e.g., `execute("my-query")`, or `type="EService"` in a JSON payload). The UUID is merely an internal artifact required by the graph database engine for referential integrity. Exposing the UUID as a routing key would unnecessarily pollute the external API contract with internal implementation details.
3. **HATEOAS Predictability:** When the server generates standard hypermedia links (like `Location` or `Content-Location` headers), it must deterministically choose a canonical URI. By enforcing the `name` as the sole addressable key, HATEOAS references remain natively predictable and readable for clients.

Consequently, while the server honors or generates the representation `id` UUID during creation and may validate a submitted UUID as a later consistency assertion, URL routing and public identity rely exclusively and rigidly on the readable `name`.

## REST API: Flat Addressability vs Nested ROA Paths

Because the IS-Model enforces a strict logical hierarchy where a **Facet** instance is entirely bound to a **Resource** instance (a strict composition, as **Facet** instances cannot be shared across different **Resource** instances), there is a strong temptation to map this hierarchy directly into the REST API.

**The Topological Dilemma:**
From a strict Resource-Oriented Architecture (ROA) perspective, one might expect deeply nested API paths to reflect the business-domain subordination:
`/instances/{resource-type}/{res-uuid}/consistsof/{co-uuid}/facet-type/{facet-uuid}`

**Decision:** The Resource Registry deliberately rejects deeply nested ROA paths in favor of a universal **"Flat Addressability"** pattern: `/instances/{type}/{uuid}` for *every* instance in the system, regardless of whether it is a **Resource** instance, a **Facet** instance, or a **Relation** instance.
This design choice is an elegant realization of the "Graph IS" nature for three critical reasons:
1. **O(1) Direct Addressability:** In a Graph Database, every graph element possesses a globally unique UUID. Forcing a client to construct a fragile URI with 3 distinct UUIDs simply to update a **Facet** instance is redundant.
2. **Developer Experience (DX):** Deeply nested URIs become an integration nightmare. If a client needs to make a surgical update to a single **StateFacet** instance, forcing them to retrieve the parent **Resource** instance and **ConsistsOf** instance UUIDs first severely degrades the developer experience and inflates payload complexity.
3. **Topological Honesty:** While semantically subordinated in the business domain, a **Facet** instance is topologically an independent graph instance. Addressing it directly via `/instances/{facet-type}/{facet-uuid}` perfectly honors its topological independence.

By keeping the API surface entirely flat, the framework delegates the enforcement of the logical hierarchy to the graph engine's native constraints (via validation and propagation rules), maintaining a clean, minimal, and highly predictable REST contract.

## REST API: Mandatory Client-Supplied UUIDs for Structural Idempotency

A recurring design question is why the API requires the client to generate and submit a UUID for every nested **Facet** and **Relation** instance in a `PUT` payload, rather than letting the server generate identifiers for newly-created elements — a pattern common in many REST APIs.

**The Reflection:** [RFC 9110 Section 9.2.2](https://datatracker.ietf.org/doc/html/rfc9110#section-9.2.2) requires that repeating a `PUT` produce the same effect as a single request. If the server were allowed to auto-generate an identifier whenever a nested instance omitted one, a network retry of the exact same payload — submitted because the client never received the response to its first, already-successful attempt — would be indistinguishable from a second, independent request to create another instance: the server has no way to tell "same intent, resubmitted" from "new intent". Protecting this instead with a server-side response cache keyed by a client-generated token (the `Idempotency-Key` pattern used by several APIs, see the [IETF Idempotency-Key draft](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key)) only ever protects the exact-retry path: it makes one specific request safe to repeat, but the underlying `PUT` itself is still not actually idempotent, and a second, legitimately distinct request that happens to omit a nested identifier could still silently duplicate or misidentify a nested instance.

**The Architectural Choice:** Every instance submitted in a `PUT` payload — the root **Resource** and every nested **Facet**/**Relation** alike — must always carry a UUID chosen by the client, whether the instance already exists (update in place) or is being created for the first time (a UUID that does not yet exist). Within a `PUT`, no instance identifier is ever server-generated. This makes `PUT` genuinely idempotent by construction: repeating the exact same payload, for any reason, converges on the same graph state, with no caching layer, no header, and no opt-in required.

**What It Does Not Cover:** Idempotency protects against *duplication*, not against *clobbering*. If a `PUT` succeeds, its response is lost, another client then updates the same resource, and the first client retries, the retry re-applies its full payload and silently overwrites the intervening change. An `Idempotency-Key` would not close this gap either: the retry would return a stale response and leave the first client unaware. The lost-update problem is a separate failure mode, addressed by optimistic concurrency control (`ETag`/`If-Match`), evaluated on its own.

**The Complement: `POST` for Creation Without Identifiers:** Generating a UUID for every nested **Facet** and **Relation** is a real cost for a client that has no interest in their identifiers. `POST /instances/{type}` removes it: a genuinely non-idempotent Create ([RFC 9110 Section 9.3.3](https://datatracker.ietf.org/doc/html/rfc9110#section-9.3.3)), to which the idempotency requirement motivating the choice above does not apply. An `id` is optional on every instance of the payload, root and nested alike: when absent, the server generates one, so two identical requests create two distinct graphs; when present, it must be globally unused because `POST` always creates and never updates. Making the `id` optional at every level, rather than on the root alone, keeps the case of a nested **Facet** whose identifier the client must choose in advance for external correlation. The two verbs thus split the two needs instead of weakening either: `PUT` for a retry-safe create-or-update, `POST` for a creation that does not care about identifiers. The contract is in [`09-api-instances-and-admin.md`](09-api-instances-and-admin.md).

## The `/admin` Segregation for Whole-System and Destructive Operations

The `/admin` prefix is reserved for whole-system maintenance, backend diagnostics, and destructive
global operations: System Dump/Restore, raw queries, and hierarchical removal.

Add to Context remains on its canonical membership URI even for cross-branch or hierarchical
variants. These are constructive Information System operations needed by authorized callers;
their risk is controlled through narrow role gates, explicit parameters, and `dryRun`, rather than
by forcing the same client to switch to a second URI for the same membership resource.

Keeping the remaining highly privileged operations behind a dedicated URI path makes it easier to
enforce strict network policies at the API Gateway or WAF level.

## Scope Expansion: Context Hierarchy vs Type Polymorphism

The Information System Model is designed around two orthogonal dimensions of "scoping" or "grouping":
1. **Contexts (Hierarchical Boundaries):** Contexts form a tree of logical environments (Venn diagrams).
2. **Types (Hereditary Boundaries):** Types form a rooted multiple-inheritance Directed Acyclic Graph (DAG).

When querying or mutating the API, a client often needs to expand their operational scope across these boundaries. The API provides two structurally equivalent parameters to traverse these dimensions:
- `hierarchical=true` traverses the Context tree downwards (including descendants).
- `polymorphic=true` traverses the Type DAG downwards (including subtypes), eliminating duplicates when a subtype is reachable through multiple inheritance paths.

While conceptually analogous (both expand scope through descendant relationships), their default behaviors and execution permissions are intentionally inverted due to their operational nature:

- **`hierarchical` defaults to `false`**: Contexts represent strict operational boundaries (tenants). By default, operations must be strictly confined to the requested boundary to prevent accidental cross-tenant data leakage. Expanding the scope is a powerful action (a sort of cross-Context transaction that simplifies interactions by replacing *N* calls with 1) and therefore requires an explicit opt-in (`hierarchical=true`).
- **`polymorphic` defaults to `true`**: Types represent data schemas. In object-oriented paradigms, querying for a base Type (e.g., `Service`) inherently implies an interest in all its concrete specializations (e.g., `EService`, `VirtualService`). Thus, polymorphic expansion is the natural default, and restricting it requires an explicit opt-out (`polymorphic=false`). Additionally, because an instance belongs to a single concrete Type at creation, Type expansion is strictly a **read-only** concept, whereas Context expansion applies to both reads and writes.

This inversion of defaults is not a contradiction, but a pragmatic alignment with the distinct domain semantics of tenancy versus object-orientation.

### Local vs. Subtree Operational Roles

To prevent role overloading and adhere strictly to the principle of least privilege, the system separates operational writers by mutation scope. Role assignment policy is deployment-specific and does not alter this API contract:
- **`IS-Contributor` (Local Writer):** Allows write operations exclusively within the current token's Context. It completely denies cross-Context sharing (`Add to Context`) and hierarchical writes. Hierarchical reads remain permitted, enabling subtree visibility.
- **`IS-Editor` (Subtree Publisher):** Grants operational authority over the current Context and its entire subtree, allowing instances to be pushed or synchronized downwards to child Contexts.
*(Note: Pure read-only access is delegated to the `IS-Member` Local Reader role, which shares the same subtree read visibility as the Contributor but possesses no write privileges).*

### Subtree Authority vs. Administrative Override

A frequent architectural debate concerns the limits of the `IS-Editor` role within its competence subtree. If an `IS-Editor` of a parent Context has jurisdiction over its descendants, what prevents them from inadvertently corrupting a child Context that an administrator has intentionally `suspended` for maintenance?

The architecture resolves this by strictly differentiating **Subtree Operational Authority** from **Administrative Override**:

1. **Subtree Operational Authority (`IS-Editor`):** An `IS-Editor` possesses genuine jurisdiction over their competence subtree. They can read hierarchically (`hierarchical=true`) to gain situational awareness, and they can push resources into descendant Contexts (via `PUT ?hierarchical=true`, `Add to Context`, or `syncExistingHierarchy`). However, they **cannot override Context state locks**: an `Add to Context` into a descendant Context that is `suspended` or `created` is rejected with a `403 Forbidden`, while a hierarchical write (`PUT ?hierarchical=true` or `syncExistingHierarchy`) leaves those Contexts untouched and reports them with `X-Result-Scope: state-filtered`, as hierarchical reads do.
2. **Administrative Override (`IS-Admin`, `Ctx-M`, or higher):** Administrative instance operations may run in `created` or `suspended` Contexts within the role's jurisdiction. `Ctx-M` applies this override only in its managed subtree; `IS-Admin`, `IS-M`, and `Infra-M` apply it according to their universal Information System authority. This override does not imply that `IS-Admin` can change Context lifecycle.

By enforcing this dichotomy, the system guarantees that operational daemons (like an API Gateway automatically pushing `EService` instances into child VREs as an `IS-Editor`) can operate seamlessly without requiring elevated privileges, while simultaneously ensuring that administrative maintenance locks remain inviolable against operational traffic.

## Membership Representation Uses an Independent Scope Axis

Instance selection and Context-membership representation answer different questions and therefore
use independent parameters. `hierarchical=true` expands which instances an operation selects or
mutates from the token Context into its descendant subtree. It does not decide which Context
memberships are shown for each selected instance.

`contextsPolicy=none|root|all` controls where `contexts` maps appear in the returned representation:
nowhere, on top-level response elements only, or on top-level and nested elements. Whenever such a
map is present, `membershipScope=subtree|global` independently controls its membership boundary.
`subtree` is the default and uses the token Context plus descendants; a Root Context token therefore
covers the complete tree. `global` ignores the token Context for membership visibility and is
restricted to the universal roles `IS-M`, `Infra-M`, and `IS-Admin`.

This separation deliberately permits requests such as
`hierarchical=true&contextsPolicy=root&membershipScope=global`: select instances from one subtree,
attach a membership map only to each top-level result, and populate that map with every global
membership. The explicit `subtree` boundary is already the requested result, so omitting external
memberships emits no `X-Result-Scope` token.

## Authentication Ecosystem Abstraction (D4Science Implementation)

The core Resource Registry API is deliberately agnostic to the authentication protocol (e.g., OIDC, SAML, api-keys) or the specific Identity Provider. It strictly expects an upstream component to validate the request and populate a `ThreadLocal` context with the user's identity and evaluated roles (both global and Context-bound). This separation guarantees that the Information System can be deployed in any environment (e.g., using Auth0, Okta, or Entra ID) by simply implementing a compatible token-parsing filter.

**The D4Science Deployment:**
Within the D4Science infrastructure, this upstream validation is performed by the **SmartGears API Gateway**, leveraging the `gcube-secrets` library to abstract multiple authentication paradigms seamlessly:

1. **OIDC (Keycloak):** 
   When a client authenticates via the standard Keycloak infrastructure, SmartGears intercepts the OpenID Connect JWT. It maps trusted realm roles to universal roles (`IS-Manager`, `Infrastructure-Manager`, and `IS-Admin`) and Context resource-access roles to Context-bound roles (`Context-Manager`, `IS-Editor`, `IS-Contributor`, and `IS-Member`).

2. **gCube Authorization Service:**
   For backward compatibility, the gateway also supports resolving `api-key` tokens against the gCube Authorization Service. When a request arrives with an `api-key`, `gcube-secrets` resolves it against the service, which directly returns the caller's identity and all associated roles (both global and context-specific).

**The Architectural Win:**
To the Resource Registry, it is entirely irrelevant whether an `Infrastructure-Manager` role was asserted by parsing a Keycloak JWT realm claim, or dynamically retrieved by resolving an `api-key` against the gCube Authorization Service. The `gcube-secrets` library elegantly normalizes both paradigms into the exact same `ThreadLocal` identity representation. This abstraction shields the Resource Registry from the complexities of token lifecycles, role merging, and Identity Provider migrations, embodying a robust, decoupled microservice architecture.
