path: root/src/lib/yang/yang.dox

// Copyright (C) 2018-2022 Internet Systems Consortium, Inc. ("ISC")
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at http://mozilla.org/MPL/2.0/.

/**
 @page libyang libkea-yang - Kea YANG Utilities Library

The libkea-yang library was developed to handle base YANG operations,
such as retrieving YANG schema and configuration and translating
data between YANG and JSON that is understandable by Kea.

@section yangTranslator Translator between YANG and JSON

An essential concept is the idea of translator. It is a primitive that is able
to convert certain data structure between YANG and JSON. It is envisaged
that more complex translators will use other translators to handle more
complex data structures. For details, see @ref isc::yang::Translator.
It is also envisioned that translators could do the translation automatically by
relying on capabilities of iterating through data, retrieving data type
information and value information from nodes through the libyang and sysrepo
APIs.

Note that although the initial focus is on translation from YANG to JSON (so
Kea can retrieve its configuration), the opposite translation direction -
from JSON to YANG - is also very useful, for at least three reasons. First,
in many cases we can use it in tests to check that conversion back and forth
doesn't lose anything: yang = toYang(toJson(yang)). Secondly, YANG modules
cover two major types of data: configuration and run-time state. While
we're initially focusing on getting the configuration, the run-time state
is something that Kea is expected to provide. Kea uses JSON internally in many
places and that data will have to be exported in YANG format. Thirdly, the
Event::Update (SR_EV_UPDATE prior to sysrepo v2) callback allows mid-flight
configuration changes before data is
committed to the sysrepo datastore. If it ever will be used in the future,
changes applied during this step will most likely come from Kea's current JSON
configuration. As such, JSON to YANG translation will be necessary. One
application for this is reverting stuff that is necessary for Kea - Sysrepo
communication like the unix socket.

All translators take a Session pointer (a structure provided by Sysrepo that
is responsible for maintaining a connection) in constructors and derive from
the basic / base class and recursively from translators for embedded parts.

@c isc::yang::Translator provides several public methods:

  - @c isc::yang::Translator::checkAndGet() is able to retrieve a YANG node that
    can be retrieved through complex logic which is abstracted through a lambda.

  - @c isc::yang::Translator::checkAndGetLeaf() is a convenience wrapper
    over @c isc::yang::Translator::getItem().

  - @c isc::yang::Translator::checkAndGetAndJsonifyLeaf() retrieves elements
    that are strings in the YANG schema, but that require passing through
    @c isc::data::Element::fromJSON() when translating to ElementPtr.

  - @c isc::yang::Translator::checkAndStringifyAndSetLeaf() does the opposite of
    @c isc::yang::Translator::checkAndGetAndJsonifyLeaf(). It takes an arbitrary
    Element and sets it as a YANG string in sysrepo.

  - @c isc::yang::Translator::checkAndSetLeaf() is a convenience wrapper
    over @c isc::yang::Translator::setItem().

  - @c isc::yang::Translator::checkAndSetLeafList() is able to push multiple
    leaf list nodes to the same xpath to form a leaf list.

  - @c isc::yang::Translator::checkAndSetUserContext() is specifically tailored
    for setting user context in sysrepo. It's use is frequent enough to have
    earned its own function.

  - @c isc::yang::Translator::deleteItem() deletes the data node found at
    given xpath.

  - @c isc::yang::Translator::findXPath() retrieves any data node found at any
    xpath. It is computationally intensive. Use sparingly.

  - @c isc::yang::Translator::forAll() iterates over the node found at
    given xpath and all its descendants and calls the given function.

  - @c isc::yang::Translator::getData() retrieves any data node found at any
    xpath. The difference from @c isc::yang::Translator::findXPath() is that it
    does not throw if the data node is not found and instead returns nullopt.

  - @c isc::yang::Translator::getItem() retrieves and translates a leaf
    from YANG to Element.

  - @c isc::yang::Translator::getList() retrieves a list from sysrepo and
    translates it from YANG to Element.

  - @c isc::yang::Translator::getMandatoryLeaf() fetches a leaf that is expected
    to be present in the YANG data node, most of the time a YANG key.

  - @c isc::yang::Translator::schemaNodeExists() checks if an xpath is valid
    from the YANG schema point of view. Not used anywhere, but it's here to
    substitute logic that had been previously removed.

  - @c isc::yang::Translator::setItem() translates a leaf from Element to
    YANG and sets it in sysrepo.

  - @c isc::yang::Translator::setMandatoryLeaf() sets a leaf that is expected
    to be present in the Element node, most of the time a YANG key.

  - @c isc::yang::Translator::translateFromYang(optional<DataNode>, string)
    translates a YANG leaf to an Element node based on YANG type.
    All YANG types are explicitly handled.

  - @c isc::yang::Translator::translateToYang(ConstElementPtr, LeafBaseType)
    translates an Element leaf to a string based on the YANG type.
    All YANG types are explicitly handled.

Some of these methods have a counterpart that have "Diverging" in their name.
They are exceptionally used in the case where YANG xpath and Element map key are
different. This facilitates identifying these diverging nodes.

@section yangTranslatorPool Pool translator

@c isc::yang::TranslatorPool is the standard example of a translator
for a structured value. Its constructor takes a module name: the code
implements some variants to accommodate the module with shared code
moved into a common private routine. When called with an unsupported
module, generic methods of all structure translators throw
@c isc::NotImplemented.

Note pools show two shortcomings in IETF modules:
 - option sets make to track changes nearly impossible: the only easy
  code is to translate the whole configuration.
 - prefix and start - end forms of pool ranges are both mandatory.
  (reported to authors' so should be fixed in the next version).

All structure translators depend on @c isc::yang::Translator and
some of them depend on other structures, for instance
@c isc::yang::TranslatorPool depends on
@c isc::yang::TranslatorOptionDataList which itself, as all list translators,
depends on the corresponding list item translator
@c isc::yang::TranslatorOptionData. This multiple inheritance forms
a graph with the basic and the configuration translators at the two ends.
Multiple inheritance and its diamond issue are handled by C++ with
the virtual inheritance: depending classes must be virtually inherited
and explicitly constructed.

@section yangTranslatorSubnet Subnet translator

The new thing here is the use of adaptors to move timers from subnets
to pools and back.

@section yangAdaptor Adapting JSON configuration

Adaptors are tools which adapt JSON complete or partial configuration
before translation to YANG to ease this translation or after translation
from YANG to follow the Kea syntax, for instance by adding static
components which are not in the module.

Methods provided by adaptors are class methods (i.e. declared static).
Specific adaptors can be derived from the isc::yang::Adaptor base class.

There are a few basic adaptors and per structure adaptors. The second
category of adaptors are divided into:
 - from JSON to YANG adaptors or pre-processing which adapt a JSON
  configuration to make it acceptable by a from JSON to YANG (setXXX)
  translators. For a Kea module this kind of adaptors fill some required
  but missing fields, or only transform a configuration into a canonical
  form. Note for a Kea module and a configuration taken from config-get
  or config-write it likely does nearly nothing but the code must
  handle any hand written configuration so these adaptors are always
  applied.

 - from YANG to JSON adaptors or post-processing which adapt translated
  YANG configuration (by getXXX) to make it acceptable by a Kea server.
  By definition, they are not defined for Kea modules.

@section unitTestsSysrepo Running unit tests with Sysrepo

To run YANG/NETCONF/Sysrepo tests you need to compile Kea with Sysrepo support:

@verbatim
./configure --with-libyang --with-sysrepo --with-libyang-cpp --with-sysrepo-cpp
@endverbatim

For details, see Section "YANG/NETCONF support" in the Kea Administrator
Reference Manual: https://kea.readthedocs.io/en/latest/arm/integrations.html#yang-netconf.

You also need to install YANG modules, so the unit tests are able to
retrieve, add, update and generally interact with the sysrepo information.
There are several Kea modules (*.yang in src/share/yang/modules/), mostly usable
in production, but one called keatest-module is only used in unit tests. To be
able to run unit tests as a non-root user, which is the recommended way, make
sure the sysrepo repository and /dev/shm/sr* are owned by said user. One way to
prevent sporadic chown-ing is to install sysrepo and the Kea modules as
non-root.

To install all the modules, run the following script:

@verbatim
./src/share/yang/modules/utils/reinstall.sh
@endverbatim

If the YANG modules are already installed, they will be upgraded.
To avoid any upgrade hastle, the -u flag may be passed to uninstall the modules
before installing again.

@verbatim
./src/share/yang/modules/utils/reinstall.sh -u
@endverbatim

Alternatively to install each module, issue the following command:

@verbatim
sysrepoctl -i "src/share/yang/modules/${module}.yang"
@endverbatim

To verify that you have the schemas installed, do this:
@verbatim
sysrepoctl -l
@endverbatim

Make sure that keatest-module and other necessary modules are on the list.

As DHCP modules are still being developed, if the revision has been bumped,
reinstalling it will update the module automatically. Otherwise, it can be
useful to uninstall them before reinstalling a more recent version:

@verbatim
sysrepoctl -u <module-name>
@endverbatim

Tests use these modules which you can find in src/share/yang/modules in addition
to keatest-module:
 - ietf-dhcpv6-server
 - kea-ctrl-agent
 - kea-dhcp-ddns
 - kea-dhcp4-server
 - kea-dhcp6-server

Those modules depend on the following modules:
 - ietf-inet-types
 - ietf-yang-types
 - ietf-interfaces
 - kea-types
 - kea-dhcp-types

The following modules are extracted from the IETF DHCPv6 YANG draft:
 - ietf-dhcpv6-client
 - ietf-dhcpv6-options
 - ietf-dhcpv6-relay
 - ietf-dhcpv6-types

All are available in the src/share/yang/modules directory using the
<module-name>[@<revision>].yang syntax for file names.
src/share/yang/modules/utils provides a few utilities for developers:
 - check-revisions.sh which verifies if the revision in the file name
  and in the file content matches
 - check-hashes.sh which detects updates in the file content without
  a revision change using the SHA-256 hash of the to YIN translation.
  Updates hashes automatically if -a is passed to the script.
 - gen-revisions.sh which produces the module / revision table of
  the yang_revisions.h header file.
 - reinstall.sh which installs all the modules.

You can run this tool:

@verbatim
src/lib/yang/pretests/sysrepo_setup_tests
@endverbatim

to verify that your environment is ready. If there is anything
wrong, it will enumerate the problems and will suggest how to solve
them.

@section yangMTConsiderations Multi-Threading Consideration for YANG Utilities

The YANG utilities are not thread-safe. Note as they are used only in a
configuration context it is not a problem, and the libyang and sysrepo libraries
are multi-threaded so their APIs are thread-safe.

*/