National Information Exchange Model — High-Level Version Architecture
Version 3.0alpha1
September 3, 2014
NIEM Technical Architecture Committee (NTAC)
Contents
Abstract

This document specifies the National Information Exchange Model (NIEM) version architecture. This specification identifies the processes, artifacts, responsibilities and how they interact to produce new releases of and updates to the NIEM model. This draft establishes a regular schedule for major and minor releases per the new Release Optimization Strategy (approved by NIEM ESC June 2014).

Status

This document defines the design for versioning that evolved from the collaborative work of the NIEM Technical Architecture Committee (NTAC), and subsequent work of the NIEM Business Architecture Committee (NBAC). The Release Optimization Strategy is a recommendation to the Executive Steering Council (ESC) to allow a regular annual NIEM release as needed. In June 2014 the ESC approved the recommendation. [NIEM High-Level Version Architecture v3.0] supersedes the previous [NIEM High-Level Version Architecture v1.0] dated 31 July 2008.

This specification is a product of the NIEM Program Management Office (PMO).

Email comments on this specification to niem-comments@lists.gatech.edu.

1. Introduction

This document describes the National Information Exchange Model (NIEM) version architecture at a high level. It describes the approach and the major artifacts involved, without specifying them at a detailed level. This document is intended to frame further discussion and specification efforts, leading to a precise specification in the future.

The version architecture discusses issues that have inherently conflicting tradeoffs. It seeks a balance of concepts drawn from previous NTAC discussions and ideas. Major tradeoffs include:

1.1. Scope

The HLVA describes how NIEM governance updates the data components and schema documents that comprise a NIEM release. This document is a high-level, abstract form of the eventual specification. It describes, at a high level, topics that will be specified by a complete version architecture.

This document assumes that the reader is familiar with NIEM and its artifacts. Readers needing additional background on NIEM should refer to documents on the NIEM Web site at http://reference.niem.gov/.

1.2. Background

The NIEM is represented as a set of XML Schema documents that define data components. Implementers of information exchange package documentation (IEPD) use these data components to define information exchange packages (IEP) that will be transmitted in an implementation. The goal of NIEM is to define common data components that are highly reusable, to improve the economy of implementing information exchanges, to increase comprehension of those exchanges, and to reduce errors in exchanges.

The NIEM Technical Architecture Committee (NTAC) and the NIEM Business Architecture Committee (NBAC) govern NIEM architecture and content respectively. The NBAC and NTAC may appoint standing subcommittees and transitory tiger teams who take responsibility for specific issues. NTAC and NBAC make technical recommendations to the NIEM Program Management Office (PMO). The PMO is the operational body that directs and executes NIEM daily operations, development, maintenance, and public relations. The Executive Steering Council (ESC) is the senior executive level group that reviews and approves major NIEM directions and activities.

[Definition: NIEM governance]

The aggregation of the Executive Steering Council (ESC), the NIEM Program Management Office (PMO), the NIEM Technical Architecture Committee (NTAC), and the NIEM Business Architecture Committee (NBAC).

The NIEM content model is intentionally partitioned into specialized domains under NIEM core.

[Definition: NIEM core]

The content of a NIEM release that is common or universal to all (or almost all) domains.

[Definition: NIEM domain]

A partition of a NIEM release that represents specialized data requirements for a particular community of interest or line of business. This term may also refer to the governance group for a domain.

A NIEM domain is sometimes associated with or managed by an organization independent of NIEM, and may have its own rules, procedures, schedule, and priorities. The data components that make up NIEM are published as a set of NIEM releases. Each NIEM release is composed of a set of schema documents that include a NIEM core schema document, special infrastructure schema documents (structures.xsd and appinfo.xsd), multiple domain schema documents, and code table schema documents. Each domain or code table schema document has a body of domain representatives that is responsible for its content and maintains its own timeline for updates and publication. For this document, the term domain applies to both domain and code table working groups.

For more information on what is included in a NIEM core, what domains and code table schema documents are defined, and what they contain, see http://release.niem.gov.

2. Objectives

The NIEM High Level Version Architecture (HLVA) described by this document is meant to achieve the following objectives:

2.1. Not Objectives

The following are not objectives of the NIEM High Level Version Architecture (HLVA):

3. Overview

The HLVA provides a framework for changes to be performed, in a systematic way, on a set of artifacts. These are outlined in subparts of this section. The activities that lead to changes in the schema documents and other artifacts are driven by a desire to improve the model. Improvements include correcting errors, meeting previously unforeseen requirements, and adapting to new needs and new uses of the model. One concern is how we define improvement to the model. A common term for improving systems is harmonization, a term which we adopt for this effort.

[Definition: harmonization]

Harmonization (of a schema document or set of schema documents) is the process of modifying a schema document set in an incremental fashion for the purpose of improving the quality of the schema document set with respect to some criteria. For example, a schema document set may be harmonized to reduce semantic overlap of its components. A schema document set may also be harmonized to ensure uniformity of the vocabulary used in definitions of its components.

Harmonization may be conducted within a single schema document, or may be conducted across multiple schema documents of a schema document set.

The version architecture does not anticipate that NIEM will ever be completely harmonized (a theoretically perfect state for a schema document set, in which all components perfectly satisfy all criteria). In practice, a fully harmonized state is never reached, and the goal of harmonization is incremental improvement, rather than perfection.

This section outlines the basic structure of the architecture. Many details are in later sections.

3.1. Classes of Information

There are several classes of information that are employed by the version architecture. They are defined as follows:

[Definition: model package description]

As defined by [NIEM MPD Specification 3.0]:

A model package description is a ZIP file (See [PKZIP]) that:

  • includes a set of logically cohesive W3C XML Schema documents and other supporting files that represent one or more reusable or implementable XML information models, and
  • has an MPD class of http://reference.niem.gov/niem/specification/model-package-description/3.0/#MPD, and
  • adheres to all the rules in [NIEM MPD Specification 3.0] for the model package description conformance target.

This term may be abbreviated MPD.

[Definition: NIEM release]

An MPD that is a set of schema documents published by the NIEM PMO. Each schema document defines data components for use in information exchanges. A release should:

  • be easy to use.
  • define each concept in a coherent way.
  • be independent of other releases, although any given schema document may occur in multiple releases.
  • be of high quality, and has been vetted by NIEM governance.

A NIEM release can be any one of three types:

[Definition: major release]

A NIEM release in which the NIEM core reference schema document has changed since the previous releases. The first integer of the version number indicates the major release series; for example, NIEM versions 1.0, 2.0, and 3.0 are different major releases.

[Definition: minor release]

A NIEM release in which the NIEM core has not changed from previous releases in the series, but at least one or more domain reference schema documents have changed. A second integer greater than zero in the version number indicates a minor release (for example, NIEM version 2.1). Note also that major version 2.0 and minor version 2.1 are in the same series (i.e., series 2) and contain the same NIEM core schema document.

[Definition: micro release]

A NIEM release in which neither the NIEM core nor the domain reference schema documents have changed from the previous major or minor release, but one or more new reference schema documents have been added (without impact to domain or core schemas). A third integer greater than zero in the version number indicates a micro release (for example, NIEM version 2.1.1; note that this release does not currently exist).

As a general rule, a published NIEM release is never revoked or removed from the release area. However, updates to a NIEM release may be published.

[Definition: published update]

A published update is a schema document or set of schema documents that is issued by a NIEM domain or NIEM governance body. An update may be new content or may be an update of content that was previously included in a NIEM release. A published update may define new versions of content from NIEM release or other published content. The issuing body vets each update before publication, but the update is not subject to review by other NIEM bodies. A published update must be conformant, but otherwise it has fewer constraints on quality than does a NIEM release.

There are currently two classes of published updates in NIEM:

[Definition: domain update]

An MPD that adds or modifies components in its own schema documents associated with a NIEM release. A domain update is published to the publication area by the NIEM domain owning the schema document that the domain update modifies.

[Definition: core update]

An MPD that adds or supplements components in a NIEM core within a NIEM release. A core update is published to the publication area by the NIEM PMO (at the recommendation of NBAC and NTAC). A core update is additive or supplemental in nature; it does NOT replace or modify components in a published NIEM core.

Through review and usage of published releases, potentially new or missing requirements, flaws, or other concerns will be identified in NIEM. These issues are recorded for review, discussion, and action (as needed) by NIEM governance.

[Definition: NIEM issue]

Users of NIEM schema documents and components will find problems that should be addressed. These will include erroneous syntax or definitions, inconsistencies between different parts of the model, and new requirements that are not yet satisfied. A NIEM issue is a record of one of these problems. An issue’s record may include discussion of the problem, possible solutions, and changes to the model required to resolve the problem.

3.2. Data Stores

There are several data stores in the architecture. General terms are used here, to avoid introducing any requirements or restrictions associated with an existing tool, such as a registry, repository, or specific Web software such as Bugzilla or Sharepoint. This description seeks only to describe requirements and functionality for these data stores.

[Definition: release area]

A network location at which NIEM releases are stored and made publicly available. The release area is http://release.niem.gov/.

[Definition: publication area]

A network location at which core updates, domain updates (and code table updates), and other artifacts are stored and made publicly available. The publication area is http://publication.niem.gov/.

[Definition: issue tracking area]

A tool or network location that supports registration, modification, and collaborative discussion of issues pertaining to NIEM schema documents, concepts, data components, and other artifacts. (This may be or may include the NCCT - NIEM Configuration and Control Tool.)

[Definition: collaboration area]

A tool or network location that supports cooperative editing of artifacts, as well as discussion by domains and other NIEM working groups. Each domain or working group receives its own partition within the collaboration area. Partitions are private; access to each is restricted to members of the appropriate working group.

3.3. Activities

There are five major activities involved in managing versions of NIEM. domain representatives, NBAC, and NTAC engage in these processes as they improve and extend NIEM.

[Definition: domain update process]

A change process that results in a domain update. A domain may publish updates to its schema documents to the publication area. A domain update proposes updates to future NIEM release.

[Definition: core update process]

A change process that results in a core update. NBAC may publish updates to a NIEM core to the publication area. A core update is always additive or supplemental in nature; it CANNOT replace or modify a NIEM core.

[Definition: domain reconciliation]

A process through which NBAC resolves conflicts between domain updates. This process results in a reconciled, coherent schema document set that is published as a minor release.

[Definition: cross-domain harmonization]

A process through which NBAC initiates tiger teams and working groups to resolve inconsistencies, overlaps, and other semantic issues between the domains, and between the domains and core. The results of these efforts will go into the issue tracking area, to be included in the next appropriate major or minor release.

[Definition: core synchronization]

The results of cross-domain harmonization are merged into a new release of the NIEM core namespace, with domains synchronized to the new NIEM core. Together, these form a major release.

3.4. A Walkthrough of Versioning

The NIEM versioning process contains several activities, performed by various parties. This section walks through these activities, and highlights the inputs and outputs of each. Each activity and artifact is discussed in more detail in later sections.

Figure 3-1: Activity flow for the NIEM versioning processes.

The following is a brief high-level walkthrough of versioning, and describes the processes diagrammed in Figure 3-1, Activity flow for the NIEM versioning processes., above. The items numbers are keyed to the diagram.

  1. NIEM releases and domain updates are available for use in IEPDs and other use.
  2. Through use and analysis of NIEM releases and published content, domain users will identify problems and new requirements for domain and core content. Users include developers of IEPDs, as well as implementers and users of exchanges.
  3. These problems and requirements will be entered and tracked as issues in the issue tracking area.
  4. NIEM domains use issues as the basis for incremental improvements, extensions, and proposed changes to NIEM releases. This process is called the domain update process. Domains may work together via a partition of the collaboration area.
  5. Domains publish their domain updates in three ways:

  6. Periodically, the NBAC reconciles domain updates into a new minor release. The goals of this process are:

    • to resolve the conflicts and inconsistencies that are created by new and modified domain content
    • to produce a coherent set of reference schema documents for the domains
    • to perform some incremental harmonization on that schema document set. This process is called domain reconciliation, and is scheduled annually. In this process, the NBAC examines issues, published updates, and releases, and reconciles changes to domains into a single coherent domain schema document set. The NBAC ensures that all domains reference each other in a consistent way.
  7. The result of this reconciliation process is published as the next minor release. This release is coherent. This release has a version identifier. This release is called a NIEM minor release, and it contains no modification to the NIEM core namespace. It may contain some additional core content (in a new namespace), but will not modify the NIEM core namespace. This is published annually, as the result of domain reconciliation. A minor release has a two-part numeric version identifier that does not end in 0, such as 4.2. For those issues that are not resolved via domain reconciliation, issues in the issue tracking area are updated, created, and closed, as appropriate.
  8. At any time, asynchronously with other processes, NBAC or NTAC may initiate a tiger team to conduct cross-domain harmonization. A tiger team is a small working group of subject matter experts (SMEs) formed to solve a specific set of problems. A tiger team exists only for a short time; it is disbanded once it has served its purpose.
  9. A tiger team that is formed by the NBAC or NTAC will solve some set of problems with the NIEM model. For example, it may resolve a set of major harmonization issues between the domains and core. Or, it may address a new requirement that requires thought and experience. Or, it may identify, within the domains, components with semantic overlap, and will pull them up into core for common definition and use. This process is called cross-domain harmonization.
  10. The results of tiger teams will be made available in a partition of the publication area. This partition may hold schema documents that contain additions to and extensions of the NIEM core namespace. Each schema document is published with a unique namespace, distinct from the NIEM core namespace. The domains may use these schema documents in domain updates, and IEPDs may use these schema documents, as well. The contents of these partitions of the publication area will be considered during the domain reconciliation process (step 6, above), and a supporting schema document containing additional core content may be published with the periodic minor release.
  11. Every third year, the results of cross-domain harmonization and domain reconciliation are brought together into a new NIEM major release (instead of a minor release). This process is called core synchronization. The NIEM core namespace is updated with help from resolved issues in the issue tracking area, as well as the published updates from the domains and tiger teams. Domain definitions that were dependent on the previous NIEM core are harmonized with the updated NIEM core namespace.
  12. Core synchronization results in a new NIEM major release, with an updated NIEM core namespace. A major release has a two-part numeric version identifier that ends in 0, such as 4.0.
4. Activities

This section contains further discussion and details of the major activities of the NIEM version architecture.

4.1. Domain Reconciliation

Once a NIEM release is published, updates and requests for changes accumulate. Users post issues (in the issue tracking area) describing problems with and unmet requirements of NIEM schema documents.

In response to these issues, domains publish domain updates to their namespaces. They identify and define new content that they require. They define and update code tables that they require. These updates are published in the domain partitions of the publication area.

The NIEM Business Architecture Committee (NBAC) reconciles these requests with the latest published release, in order to produce a coherent NIEM schema document set. This reconciliation will result in a schema document set that is easily used by IEPD developers.

The NBAC makes decisions on how to accommodate changes. In the publication area, domains publish updates prior to NBAC approval, ensuring autonomy of the domains. The NBAC determines how best to migrate these updates into the next minor release. The NBAC conducts domain reconciliation annually.

This reconciliation process will consist of the following stages:

  1. Gather and prepare input to the reconciliation process.

    This input will come from open issues in the issue tracking area, and from domain partitions of the publication area. The content of the publication area will be evaluated for impacts. Reports will be prepared that may include:

    • Reports describing the impacts of proposed changes on other domain namespaces.
    • Reports assembling the comments that other domains have made about proposed changes.
    • Reports containing technical metrics that quantify the

      • consistency of harmonization on the input schema document set, and
      • impacts of proposed changes.
    • NTAC recommendations on technical methods and techniques for handling domain requests.
    • Conformance [NIEM NDR] and quality checks [NIEM Quality Assurance]. Conformance and quality checks are performed prior to publication of schema documents to the publication area, but additional reports on quality and conformance may be made available to the NBAC for reconciliation.
  2. Reconcile proposed domain updates.

Starting with the previous NIEM release, the NBAC will review and apply the changes that were requested by the domains. For each proposed domain update, approve the update, or modify it as required to maintain high quality across all NIEM schema documents.

When a change is applied, make modifications in other domains, as required, to keep the schema document set coherent. Where domain updates introduce integrity issues, modify the schema document set to eliminate those problems. Propagate domain updates through impacted namespaces, as appropriate.

Note that there is no requirement or implication that any harmonization will be done by tools, or automatically in any way. The goal of this architecture is to ensure that the responsible party has the authority, schedule, goal, and ability to do what is needed to ensure that domain changes result in a coherent set of domain schema documents that can easily be used in IEPDs.

4.1.1. NBAC Decisions

Through the process of domain reconciliation, the NBAC will resolve many issues of cross-domain conflict. This may include changes to one domain that affect another domain, or overlaps of content between domains. Through the reconciliation process, the NBAC will make adjustments as-needed to ensure that domain updates do not harm the quality of the NIEM release as a whole.

During its periodic domain reconciliation, the NBAC may conduct additional harmonization. Issues addressed may be low hanging fruit, such as particularly easy or notable problems. Or, the NBAC may try to resolve one or more aspects of harmonization across the NIEM model, such as vocabulary consistency. During reconciliation, it is likely that time will be short, and the level of effort will be limited. So, during reconciliation, extremely broad changes would be avoided, as would changes that required significant consideration by subject matter experts. Such issues would be handled by tiger teams and small groups in a separate process referred to as cross-domain harmonization.

The NBAC, as the designated body for harmonization, may make any decision that it deems prudent to integrate domain updates with the previous NIEM release. This may include the modification of dependent namespaces to accommodate domain changes, the creation of new elements for indirect (substitution) methods, and the modification of definitions and component names.

The NBAC may also choose to reject, in whole or in part, any domain update. Of course, the NBAC should have valid reasons for such action. However, the NBAC is given the responsibility of acting in the best interest of the NIEM community, rather than in the interest of a single domain. Note that no change the NBAC makes will affect any content that is published by the domains in the publication area, nor will any content from the release area be affected. That content stands as it is, without change. Barring a valid objection against a domain update, the NBAC would adopt the update for the next version of the NIEM domain schema document, and would propagate that change into dependent schema documents, as required. This may require that the NBAC exercise judgment.

4.1.2. Coherence

This document uses the term coherence to describe whether components of a schema document set refer to each other in a consistent manner:

[Definition: coherence]

A schema document set is coherent when it has the following properties:

  • It does not refer to a schema document outside the set, and
  • It does not include two different versions of the same component in an incompatible way.

The coherence of a NIEM distribution is best described by showing a schema document set that is not coherent. In this schema document set, a Justice 4.1 version is based on Justice 4.0, with a new element (NewElement) added to ArrestType. The latest schema documents are outlined in the rounded boxes below. The schema document set that is composed of these schema documents is not coherent.

Figure 4-1: A schema document set that is not coherent.

Why is this not coherent? An IEPD would not be able to use the newest schema documents (Screening and Justice-4.1) together as designed. Instead, the IEPD would require a substantial amount of work to make these two schema documents work together. For example, the developer of an IEPD might duplicate NewElement within its extension namespace. Or, it might create an extension of the 4.0 ArrestType that uses the 4.1 NewElement. Such changes would be required for each incompatibility between versions. In an environment where many domains are updating frequently, this presents IEPD developers with a substantial problem.

However, the NBAC could reconcile the request from the domain (adding NewElement) with other requests from other domains. It could then produce the reconciled schema document set, which would look like the following:

Figure 4-2: A coherent schema document set.

This schema document set could be easily used by an IEPD developer. It would not require IEPD developers to develop custom extensions to fix problems in the schema document set. The new Screening namespace (1.1) is integrated with the new Justice namespace. The final version architecture specification will address the structure of version identifiers in the domain schema documents. Specifically, it may be desirable for version identifiers for bumped namespaces to be easily distinguished from version identifiers for domain-initiated changes. The specific rules that apply to version identifiers of schema documents will be defined by the version architecture specification.

4.2. NIEM Micro Releases

When a domain has created its updated content, it may determine that the content should be released as a coherent NIEM release. This will be verified by tools that check conformance and coherence of the resulting schema document set. Once these checks are complete, a domain may elect to publish its update directly as a NIEM micro release in the release area, rather than as a published update in the publication area.

Schemas published in this manner may be more durable than schema documents published to the publication area. That is, schema documents so published may work better with more versions of NIEM. NIEM releases will not refer to schema documents in the publication area; schema documents from the publication area are reconciled and modified before their release, and they receive new target namespace URIs. Schemas published directly to releases avoid reconciliation. They also will not have to be modified as they go from the publication area to the release area, as they avoid the first step. A prime candidate, and simple example, of this method is a code table. By defining a code element such that it is substitutable for an element from a NIEM release, multiple versions of a schema document set may be usable at the same time. A new version of such a code table may be released as part of a NIEM micro release without requiring rework of other schema documents. NIEM micro releases require tools for validation and verification. They also require a commitment to frequent NIEM releases, with limited vetting.

4.3. Cross-Domain Harmonization

Larger and deeper issues of semantic consistency and modeling will be addressed by tiger teams and small groups of subject matter experts, as determined by the NBAC. The tasks undertaken by these teams will, in many cases, not require the entire NBAC to weigh in. The use of such teams will enable parallel harmonization efforts. This will result in greater scalability, as the number of domains increase.

The products of cross-domain harmonization will include:

The additions to NIEM core that appear within the publication area may be immediately used by IEPDs, and domains may base published updates on those schema documents.

4.4. Core Synchronization

Periodically, these products will be synchronized with domain content, to produce a new major release. A major release will contain a new NIEM core namespace. A major release provides an opportunity to execute destructive changes, such as component deletions. At the time of a major release, other major changes to the NIEM release schema document set may be considered by the NBAC, such as changes to structures and appinfo, which are schema documents that are part of the NIEM infrastructure. For more information on structures and appinfo, see the [NIEM NDR] and other documents on the NIEM Web site at http://reference.niem.gov/.

NIEM core synchronization will occur every third year. However, a NIEM core synchronization may be held back for a longer period if changes to be released do not meet certain thresholds in terms of volume or severity. The NBAC must balance the need for regular scheduled releases with the set of components that must be harmonized across domains for inclusion into the core.

In between updates to NIEM core, the NBAC may add core content via additional components in additional schema documents in a minor release of NIEM, without updating the NIEM core schema document. These would be additive changes only, and would not modify the original NIEM core schema document in any way.

5. NIEM Releases

The NBAC domain reconciliation activity results in a NIEM minor release.

5.1. Properties of a NIEM Release

A NIEM release has the following properties:

5.2. Status of a NIEM Release

There are two types of harmonization/reconciliation:

  1. between the domains (domain reconciliation), and
  2. with NIEM core and the domains (core synchronization)

This provides an opportunity to distinguish between the subsequent releases. Reconciliation of domains results in a minor release, while reconciliation of NIEM core results in a major release.

5.3. Version Identifiers for NIEM Releases

Domain-synchronized NIEM release are given minor release version numbers. NIEM core-synchronized NIEM release are given major release version numbers. Direct releases due to domain updates are given micro release numbers.

When core synchronization occurs, the results are published as a NIEM major release. These releases will be given identifiers such as 2.0, 3.0, and 4.0. These releases will yield a NIEM core namespace that has the same version identifier as the NIEM release.

When domain reconciliation occurs, without update of the NIEM core namespace, the results are published as a NIEM minor release. These releases have indentifiers such as 2.1, 2.2, 2.3, 3.1, 3.2, and 3.3. When a domain update process occurs without reconciliation, the domain may request from NBAC a new release that incorporates that update, and the results are published as a NIEM micro release. These releases have identifiers such as 2.1.1, 2.1.2, 2.2.1, and 2.2.2.

For NIEM release identifiers, major release have two integers (e.g., 2.0). The major integer (2), and the minor integer (.0). The minor release following 2.0 is 2.1. The micro release following 2.1 is 2.1.1. There is no version 2.1.0.

For NIEM release identifiers, the value following 2.9 is 2.10. Each number (major, minor, or micro) is a field storing an integer, not a single digit.

The version identifiers for released domain schema documents (in the release area) and published schema documents (in the publication area) are not described by this HLVA. Constraints on these identifiers have not yet been determined, and will be specified fully in the final version architecture specification.

6. Data Stores

The version architecture defines several areas in which artifacts of NIEM are stored. Each may be implemented using new or existing tools. The goal of this document is to describe the areas, what information they contain, and what functionality is needed to support activities that use these areas.

The specific software and tools that make up these data stores are not specified by this document. Such specifications will be detailed by later documents.

6.1. The Release Area

All releases of NIEM are published to the release area.

All schema documents published to the release area are persistent, and are available for immediate use by IEPD developers and other users.

A single schema document may be a part of multiple releases. For example, a single release of the NIEM core namespace will be a part of one or more minor releases. The release area will provide metadata to make clear exactly which schema documents are associated with which releases. This metadata is referred to as the release manifest. The format of this metadata is yet to be determined. The release area is http://release.niem.gov/.

6.2. The Collaboration Area

The collaboration area supports cooperative editing of artifacts, as well as discussion by domains and other NIEM working groups. Each domain or working group receives its own partition within the collaboration area.

These partitions are private, as access to each partition is restricted to the members of the appropriate working group.

The precise mechanics and content of the collaboration area are to be determined.

6.3. The Publication Area

The publication area is a well-known location for domains and other working groups to publish schema documents and other artifacts for NIEM. This has two major purposes:

  1. It allows published content to be immediately used.
  2. It provides a concrete method for changes to be expressed for domain reconciliation and core synchronization.

All content of the publication area is persistent. That is, it never goes away once it is published. Also, it is never modified. Revised versions of published content may also be published, however, no artifact is removed or revised. This ensures that all content of the publication area may be freely used by any development effort, without fear of it changing or disappearing. All content of the publication area is available for use by IEPD developers, as well as developers of other schema documents and applications.

6.3.1. Partitions of the Publication Area

Each domain and working group controls a partition of the publication area. This partition is the publication site for products of the working group. The domain may place any number of schema documents into its publication area.

6.3.2. Use of the Publication Area

Schema documents within the publication area may provide additional content, or describe other changes to a NIEM release. Here is how the publication area is used:

  • After a NIEM release is published, it becomes available for use by IEPDs. Through use and analysis of that NIEM release, domain users will identify problems and new requirements for domain content and core content. These will be conveyed via the issue tracking area.
  • The domain, under its own governance, determines what changes it believes should be made to the latest NIEM distribution. It expresses these changes by publishing schema documents and supporting metadata in its publication area partition. Requests by the domain for changes to other domains and core will be made through the issue tracking area and the NCCT.
  • Schema documents published to the publication area are immediately available for use by IEPDs. Both domain IEPDs and cross-domain IEPDs may use any schema documents in the publication area, as they require.
  • The schema documents, supporting metadata, and issues (in the issue tracking area) describe changes the domain makes to its domain and requests for changes to core content. The requested changes are considered by the NBAC, and are reconciled with the last release, and with changes requested by other domains.
  • The result of the NBAC reconciliation is published as the next NIEM release.

Schemas in the domain partition of the publication area may extend components within a NIEM release, instead of superseding namespaces within NIEM release.

The schema documents in NIEM 2.0 contained schema documents that superseded the schema documents in NIEM 1.0. Schema documents in the domain publication areas are not required to supersede older schema documents in this manner. Instead, schema documents in the publication area may extend schema documents in previous NIEM distributions. This may make domain updates easier to use by IEPDs. For example, suppose domain JXDM wishes to add an element ArrestSubject to the type ArrestType that occurred in JXDM-4.0. It may publish a schema document in its publication area that extends the JXDM 4.0 ArrestType. This extension may then add ArrestSubject to the ArrestType. A substitutable element may be made to provide for the new ArrestType to be used in IEPDs.

Extensions made in the manner above may be easier for IEPDs to use than replacement components. When a domain publishes an update as an extension of the latest release, that update may be easier for an IEPD to use than if it was published as a replacement of the latest release.

The specific method used in a domain update is left for the domain to determine. Some domains may choose to replace components, while others may extend components. It is expected that best practices will be determined over time. Best practices may vary from domain to domain, as some domains may prefer to use common extensions, while others may prefer that each IEPD do its own integration work.

6.3.3. Namespaces for Content of the Publication Area

Each schema document that is posted to the publication area will have a namespace that meets several criteria.

  • It is unique to the posted schema document. There will not be two schema documents posted to the publication area that have the same namespace URI.
  • It clearly identifies that it is a namespace from the publication area.
  • It clearly identifies the domain or working group that is responsible for it.
  • It clearly identifies a version of the schema document target namespace. An example is http://publication.niem.gov/niem/domains/cyfs/2.1/, where the namespace is similar to that in a NIEM release, but with publication instead of release.
6.3.4. Additional Requirements of the Publication Area

Each schema document in the publication area has a target namespace that is distinct from that of all other schema documents. There will not be two different schema documents in the publication area with the same namespace, nor will there be a schema document in the publication area with the same namespace as a schema document in a NIEM release.

Each schema document in the publication area is persistent. Like the schema documents in NIEM releases, once published, the schema documents will not go away. These schema documents will remain available for use by IEPDs.

Schema documents in the publication area will be accompanied by change logs, which give comprehensible descriptions of the changes that were made to schema documents. This is discussed in Section 7.2, Change Log, below.

6.4. The Issue Tracking Area

NIEM currently conducts NIEM issue tracking using the NIEM Configuration Control Tool (NCCT). This is an issue tracking system with forums for the NTAC, NBAC, and domains. This document uses the more general terminology issue tracking area rather than referring directly to the NCCT, as whether the version architecture will use the NCCT in its current form is yet to be determined.

The purpose of the issue tracking area is to capture and track requirements and problems from end users, developers, domains, and other people and bodies associated with NIEM. Although issues come from users, whether or not users will directly interact with the issue tracking area is yet to be determined. Issues may be entered by a helpdesk, in order to ensure that duplicates are not entered, and that issues are specified fully and clearly, so that additional work is not created downstream.

The issue tracking area may include a conventional issue tracking system as well as forums/bulletin boards for discussing issues. Each domain and working group may be given a partition of the issue tracking area, as needed. These groups will have control over their issues, and may dispatch them and resolve them as desired.

To be determined is whether a domain may also have issue tracking systems and methods outside the NIEM-provided issue tracking area. This will reduce transparency, and may impair cooperation between working groups. It may also increase error when moving issues from one body to another.

One group may delegate an issue to another group, when the alternate group is better able to address the issue.

7. Additional Artifacts
7.1. Release Manifest
[Definition: release manifest]

An artifact that lists many schema documents, each of which may be a part of multiple releases. A release manifest expresses

The mechanics and format of the release manifest are to be determined. There may be a single manifest for all contents of the release area. There may also be an individual manifest for each release.

7.2. Change Log
[Definition: change log]

An artifact that describes the changes applied to an MPD since its previous version.

The publication area will contain domain updates. These updates are provided as XML Schema documents, and will be accompanied by a change log.

A change log is able to express a single big change that is composed of a set of small changes. For example, the renaming of an element may include changes to many types that used that element.

Without a change log, changes made between versions of a schema document would not include an expression of what the changes to the schema document might mean. Although the revised schema document may be fully formed and well defined, it may be unclear what impact the change might have on other schema documents.

For example, upon renaming an element (from an old name to a new name), the change is represented in XML Schema by the original element (with the old name) not appearing in the revised schema document, and a new element (with the new name) appearing in the revised schema document.

A type in another domain may be impacted by this change. If the type used the original element with the old name, it would be unclear what changes should be made to the type to accommodate the renaming.

In order to describe changes to schema documents with enough detail to enable reconciliation, schema document definitions are accompanied by metadata that describe changes. This metadata will include a list of changes, without restating the details that occur in the schema documents that are in the publication area.

The mechanics and format of the change log are to be determined. The format will be in XML, and may include XPath expressions for relating change log entries to updated schema documents. The change log XML file may describe the following about each change made by a domain:

Users will be able to audit change logs against schema document updates to look for

Descriptions of the changes made are provided to the NBAC during reconciliation, and would help them to determine the contents of the next NIEM release.

7.3. Namespace URI Directive

Through the NBAC's domain reconciliation process, it may be determined that some namespaces must be adjusted to accommodate changes to other namespaces. That is, although the domain has no new updates for its namespace, the NBAC must change the domain namespace to accommodate changes in other domains. In this situation, we say that the domain namespace was bumped.

[Definition: bump]

A process whereby a domain schema document requires a new target namespace URI as a result of one or more changes made to another NIEM domain schema document that impact the content of the schema document to be bumped.

The version architecture does not allow the application of changes directly to a published namespace. A changed schema document must be given a new target namespace URI. So, when a domain namespace is bumped, it must be given a new namespace URI.

[Definition: namespace URI directive]

An artifact that specifies the new target namespace URI for a new schema document resulting from the need to bump it.

The exact mechanics of how a namespace URI directive is published is to be determined. One option is for it to be an XML file that is published in the publication area, within the domain's partition.

There are several goals for the namespace URI directive:

The namespace URI directive allows the domain to express the structure of its target namespace URIs, and the sequence of version identifiers. The domain will be able to specify how namespace URIs are incremented when the namespace is updated or bumped.

8. General Release Timelines

The NIEM version architecture is designed to be agile, accommodate regular change, and respond efficiently to stakeholder requirements. To do so, NIEM maintains the ability to execute predictable, repeatable, and consistent release cycles, while also allowing its domains to publish updates independently (on their own timelines). At a high level, this section describes the timelines, coordination, and options for major and minor release cycles.

[Definition: release cycle]

A period of time during which appropriate groups of NIEM governance and domain representatives engage in the activities that result in a major or minor NIEM release.

NIEM will publish an annual release in the same quarter of each year. Within each three-year period, the NIEM program will execute two consecutive minor release cycles, followed by a major release cycle.

A release date will be scheduled (internally established by the PMO) for the beginning of the target quarter. The release date may shift later in the quarter if necessary. This adds a degree of flexibility and safety to the schedule to accommodate additional pre-release phases (e.g., Beta2 or beta3) or other unforeseen events, while still maintaining consistency and predictability. NIEM PMO will communicate anticipated scheduling details to the NIEM Community for planning.

If required to meet schedule, a release cycle may be time-boxed or content-scoped (or both).

For each release cycle, the PMO will assign a release coordinator selected from the major stakeholders. The release coordinator will monitor and report progress to PMO, NTAC, NBAC, and domains, resolve issues of schedule and scope, and coordinate participating entities (NIEM governance, domains, release manager, etc.).

Ordinarily NIEM will not issue a minor release for a previous major version series (such as version 2.0). However, a request by a domain (or other entity) to issue a minor release for such (e.g., a request for a Version 2.2) will be jointly considered by PMO, NBAC, and NTAC. If approved, it will be executed between annual release cycles (since the scope of change is expected to be small).

8.1. Release Cycle Stages

A typical NIEM release cycle is comprised of three basic stages of development, each of which may require several iterations. These are defined as follows:

[Definition: alpha]

The initial stage of a release cycle. Primary activities focus on integrating architectural improvements (if any) as well as domain and core content changes previously prepared as domain and core updates since the last release. Other activities include reconciliation, core synchronization (when appropriate), appropriate types of harmonization, refactoring/refinement of content as needed, application of resolved issues, anditerative NIEM governance reviews and feedback. The most significant types, volume, and scale of changes occur during alpha stage.

[Definition: beta]

The midterm stage of a release cycle. At the beginning of this stage the developing release has stabilized to the degree that remaining changes are minor — major adjustments for architecture have been completed; no significant new content may be integrated; release schemas are in a state that will allow user beta testing (i.e., the release is expected to be NIEM-conformant). Changes are still allowed; however, these are much smaller in magnitude and complexity and are generally only corrections and adjustments. Examples of beta stage changes include adjustments to data component structure; movement of properties; renaming and redefining; addition or refinement of keyword, usage, and example metadata (associated with definitions); addition of a very small number new components (essentially inadvertent omissions or additions needed to adjust structure), and occasionally very minor corrections to architecture. If beta testing, reviews, and feedback discover major flaws that require significant changes, then the release may revert back to the alpha stage.

[Definition: release candidate]

The final stage of a release cycle. A release candidate should be identical to, and therefore is a potential final production (i.e., operational) release. Only extremely minor corrections to a release candidate are allowed. These include minor corrections to character strings, names, definitions, namespaces, etc. At this stage absolutely NO architectural or significant content changes are allowed. Any such changes will require regenerating the release as a beta product. Activities during this stage are are essentially only testing, review, feedback, and possibly very minor refinements. As this is the final review, diligence must ensure quality and conformance before publication. If testing, reviews, and feedback discover flaws that require significant changes, then the release may revert back to the beta, or even alpha stage. Very minor refinements to a release candidate will require regenerating it as a new release candidate.

In the past, NIEM has often referred to a so-called pre-alpha stage. In fact, this is not a formal stage of a NIEM release cycle.

[Definition: pre-alpha]

Any time period during which the NIEM program is NOT executing a release cycle, i.e., the time period after the last release cycle ends to before the next begins. Pre-alpha activities may include (but are not limited to): production of domain and core updates; identification and resolution of issues, identification of new requirements, refactorization of NIEM objects, and essentially any other activity that can be performed in a preparatory manner before a release cycle begins, and recorded into the issue tracking area and subsequently applied in the next appropriate release cycle.

Pre-alpha activities significantly reduce the surge activity often perceived or occurring with a release cycle. The need for a significant spike in resources during a NIEM release cycle is generally a myth; most often it is the result of postponement or procrastination of the regular pre-alpha activities that should always be common practice in NIEM governance.

8.2. Release Cycle Exceptions and Options

Based on history, two minor releases and one major release in a three-year period (each release cycle executed annually) will generally accommodate the expected demand for future model changes. So, by default the appropriate release will always be planned each year without the need for prior approvals.

There is no desire to force unnecessary releases. Therefore, as circumstances demand, the NTAC, NBAC, domains, and the PMO may jointly evaluate conditions and elect to adjust the schedule as needed. Special options include:

8.3. Outline of the Three-Year Cycle

Figure 8-1, The NIEM Three Year Cycle., below, illustrates a typical NIEM three-year period. The following notes explain the activity tracks within the diagram:

Figure 8-1: The NIEM Three Year Cycle.

Although face-to-face governance meetings are extremely valuable for release planning, coordination, harmonization, and issue resolution, they are not absolutely necessary. In the event that face-to-face meetings cannot be held, teleconferences, collaboration tools, and other means will be sufficient to execute a release.

9. Summary

Under the NIEM version architecture, domains are free to publish updates on their own timelines, without being required to wait for NIEM governance or other domains to act. The architecture ensures that all such updates are available for use in IEPDs or other uses of NIEM schema documents. The architecture ensures that NIEM releases will be available to developers at regularly scheduled intervals, and that those releases will be easy to use, and free of structural inconsistency.

The version architecture also ensures that domain changes are available in a concrete form that may be incorporated into the next NIEM release. It also ensures that NIEM releases are scheduled such that they may incorporate recent domain changes. NIEM governance has authority to make required modifications in order to ensure consistency between the domains and core.

This document has described the NIEM version architecture at a high level. The high level description will be expanded and refined into a specification. The specification will detail the processes, activities and artifacts that are required to maintain high-quality and transparent change management for NIEM.

Appendix A. Acronyms and Abbreviations
Acronym / AbbreviationLiteral or Definition
ESC Executive Steering Council
HLVA High Level Version Architecture
IEP Information Exchange Package
IEPD Information Exchange Package Documentation
MPD Model Package Description
NDR Naming and Design Rules
NIEM National Information Exchange Model
NBAC NIEM Business Architecture Committee
NCCT NIEM Configuration Control Tool
NTAC NIEM Technical Architecture Committee
PMO Program Management Office
SSGT Schema Subset Generation Tool
URI Uniform Resource Identifier
W3C World Wide Web Consortium
XML Extensible Markup Language
XSD XML Schema Definition

 

Appendix B. References

[NIEM Conformance]: NIEM Conformance, Version 3.0, NIEM Technical Architecture Committee (NTAC), 15 August 2014. Available from http://reference.niem.gov/niem/specification/conformance/3.0/.

[NIEM Conformance Target Attribute Specification]: NIEM Conformance Target Attribute Specification, Version 1.0, NIEM Technical Architecture Committee (NTAC), 31 July 2014. Available from http://reference.niem.gov/niem/specification/conformance-target-attribute/1.0/.

[NIEM Domain Update Specification]: NIEM Domain Update Specification, Version 1.0, NIEM Technical Architecture Committee (NTAC), 5 November 2010. Available from http://reference.niem.gov/niem/specification/domain-update/1.0/.

[NIEM High-Level Tool Architecture]: NIEM High-Level Tool Architecture, Version 1.1, NIEM Technical Architecture Committee, 1 December 2008. Available from http://reference.niem.gov/niem/specification/high-level-tool-architecture/1.1/.

[NIEM High-Level Version Architecture v1.0]: NIEM High Level Version Architecture (HLVA), Version 1.0, NIEM Technical Architecture Committee, 31 July 2008. Available from http://reference.niem.gov/niem/specification/high-level-version-architecture/1.0/.

[NIEM High-Level Version Architecture v3.0]: NIEM High Level Version Architecture (HLVA), Version 3.0, NIEM Technical Architecture Committee, 3 September 2014. Available from http://reference.niem.gov/niem/specification/high-level-version-architecture/3.0/.

[NIEM Implementation Guide]: NIEM Implementation Guide, NIEM Program Management Office. Available from https://www.niem.gov/aboutniem/grant-funding/Pages/implementation-guide.aspx.

[NIEM MPD Specification 3.0]: NIEM Model Package Description (MPD) Specification, Version 3.0, NIEM Technical Architecture Committee (NTAC), 15 August 2014. Available from http://reference.niem.gov/niem/specification/model-package-description/3.0/.

[NIEM MPD Toolkit]: NIEM Model Package Description Toolkit, Version 3.0, NIEM Technical Architecture Committee (NTAC), 15 August 2014. Available from http://referencce.niem.gov/niem/resource/mpd/3.0/.

[NIEM NDR]: NIEM Naming and Design Rules (NDR), Version 3.0, NIEM Technical Architecture Committee (NTAC), 31 July 2014. Available from http://reference.niem.gov/niem/specification/naming-and-design-rules/3.0/.

[NIEM Quality Assurance]: NIEM Quality Assurance Strategy and Plan, Version 1.0, NIEM Technical Architecture Committee (NTAC), 20 May 2008. Available from http://reference.niem.gov/niem/guidance/quality-assurance-strategy-and-plan/1.0/.

[NIEM User Guide]: NIEM User Guide, Volume 1, U.S. Department of Justice, Office of Justice Programs, (date unknown). Available from http://reference.niem.gov/niem/guidance/user-guide/vol1/.

[PKZIP]: APPNOTE.TXT - .ZIP File Format Specification, Version: 6.3.2, Revised: 28 September 2007, Copyright (c) 1989 - 2007 PKWare Inc. Available from http://www.pkware.com/documents/casestudies/APPNOTE.TXT.

[RFC 2119 Key Words]: Bradner, S., Key words for use in RFCs to Indicate Requirement Levels, IETF RFC 2119, March 1997. Available from http://www.ietf.org/rfc/rfc2119.txt.

[RFC 3986 URI]: Berners-Lee, T., et al., Uniform Resource Identifier (URI): Generic Syntax, Request for Comment 3986, Network Working Group, January 2005. Available from http://tools.ietf.org/html/rfc3986.

[W3-XML]: Extensible Markup Language (XML), Version 1.0, Fifth Edition, W3C Recommendation 26 November 2008. Available from http://www.w3.org/TR/2008/REC-xml-20081126/.

[W3-XML-InfoSet]: XML Information Set, Second Edition, W3C Recommendation 4 February 2004. Available from http://www.w3.org/TR/2004/REC-xml-infoset-20040204/.

[W3-XML-Namespaces]: Namespaces in XML, Second Edition, World Wide Web Consortium 16 August 2006. Available from http://www.w3.org/TR/2006/REC-xml-names-20060816/.

[W3C XML Schema Datatypes]: XML Schema Part 2: Datatypes, Second Edition, W3C Recommendation 28 October 2004. Available from http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/.

[W3C XML Schema Structures]: XML Schema Part 1: Structures, Second Edition, W3C Recommendation 28 October 2004. Available from http://www.w3.org/TR/2004/REC-xmlschema-1-20041028/.

Appendix C. Index of Definitions