diff options
Diffstat (limited to 'comm/third_party/botan/doc/dev_ref')
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/configure.rst | 407 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/contents.rst | 20 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/continuous_integration.rst | 75 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/contributing.rst | 268 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/fuzzing.rst | 91 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/mistakes.rst | 77 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/oids.rst | 43 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/os.rst | 61 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/reading_list.rst | 93 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/release_process.rst | 129 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/test_framework.rst | 314 | ||||
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/todo.rst | 199 |
12 files changed, 1777 insertions, 0 deletions
diff --git a/comm/third_party/botan/doc/dev_ref/configure.rst b/comm/third_party/botan/doc/dev_ref/configure.rst new file mode 100644 index 0000000000..7eb8358edf --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/configure.rst @@ -0,0 +1,407 @@ +Understanding configure.py +============================ + +.. highlight:: none + +Botan's build is handled with a custom Python script, ``configure.py``. +This document tries to explain how configure works. + +.. note:: + You only need to read this if you are modifying the library, + or debugging some problem with your build. For how to use it, + see :ref:`building`. + +Build Structure +-------------------- + +Modules are a group of related source and header files, which can be +individually enabled or disabled at build time. Modules can depend on +other modules; if a dependency is not available then the module itself +is also removed from the list. Examples of modules in the existing +codebase are ``asn1`` and ``x509``, Since ``x509`` depends on (among +other things) ``asn1``, disabling ``asn1`` will also disable ``x509``. + +Most modules define one or more macros, which application code can use +to detect the modules presence or absence. The value of each macro is +a datestamp, in the form YYYYMMDD which indicates the last time this +module changed in a way that would be visible to an application. For +example if a class gains a new function, the datestamp should be +incremented. That allows applications to detect if the new feature is +available. + +What ``configure.py`` does +----------------------------- + +First, all command line options are parsed. + +Then all of the files giving information about target CPUs, compilers, +etc are parsed and sanity checked. + +In ``calculate_cc_min_version`` the compiler version is detected using +the preprocessor. + +Then in ``check_compiler_arch`` the target architecture are detected, again +using the preprocessor. + +Now that the target is identified and options have been parsed, the modules to +include into the artifact are picked, in ``ModulesChooser``. + +In ``create_template_vars``, a dictionary of variables is created which describe +different aspects of the build. These are serialized to +``build/build_config.json``. + +Up until this point no changes have been made on disk. This occurs in +``do_io_for_build``. Build output directories are created, and header files are +linked into ``build/include/botan``. Templates are processed to create the +Makefile, ``build.h`` and other artifacts. + +When Modifying ``configure.py`` +-------------------------------- + +For now, any changes to ``configure.py`` must work under both CPython 2.7 and +CPython 3.x. In a future major release, support for CPython2 will be dropped, +but until then if making modifications verify the code works as expected on +both versions. + +Run ``./src/scripts/ci_build.py lint`` to run Pylint checks after any change. + +Template Language +-------------------- + +Various output files are generated by processing input files using a simple +template language. All input files are stored in ``src/build-data`` and use the +suffix ``.in``. Anything not recognized as a template command is passed through +to the output unmodified. The template elements are: + + * Variable substitution, ``%{variable_name}``. The configure script creates + many variables for various purposes, this allows getting their value within + the output. If a variable is not defined, an error occurs. + + If a variable reference ends with ``|upper``, the value is uppercased before + being inserted into the template output. + + * Iteration, ``%{for variable} block %{endfor}``. This iterates over a list and + repeats the block as many times as it is included. Variables within the block + are expanded. The two template elements ``%{for ...}`` and ``%{endfor}`` must + appear on lines with no text before or after. + + * Conditional inclusion, ``%{if variable} block %{endif}``. If the variable + named is defined and true (in the Python sense of the word; if the variable + is empty or zero it is considered false), then the block will be included and + any variables expanded. As with the for loop syntax, both the start and end + of the conditional must be on their own lines with no additional text. + +Adding a new module +-------------------- + +Create a directory in the appropriate place and create a info.txt file. + +Syntax of ``info.txt`` +------------------------ + +.. warning:: + + The syntax described here is documented to make it easier to use + and understand, but it is not considered part of the public API + contract. That is, the developers are allowed to change the syntax + at any time on the assumption that all users are contained within + the library itself. If that happens this document will be updated. + +Modules and files describing information about the system use the same +parser and have common syntactical elements. + +Comments begin with '#' and continue to end of line. + +There are three main types: maps, lists, and variables. + +A map has a syntax like:: + + <MAP_NAME> + NAME1 -> VALUE1 + NAME2 -> VALUE2 + ... + </MAP_NAME> + +The interpretation of the names and values will depend on the map's name +and what type of file is being parsed. + +A list has similar syntax, it just doesn't have values:: + + <LIST_NAME> + ELEM1 + ELEM2 + ... + </LIST_NAME> + +Lastly there are single value variables like:: + + VAR1 SomeValue + VAR2 "Quotes Can Be Used (And will be stripped out)" + VAR3 42 + +Variables can have string, integer or boolean values. Boolean values +are specified with 'yes' or 'no'. + +Module Syntax +--------------------- + +The ``info.txt`` files have the following elements. Not all are required; a minimal +file for a module with no dependencies might just contain a macro define. + +Lists: + * ``comment`` and ``warning`` provides block-comments which + are displayed to the user at build time. + * ``requires`` is a list of module dependencies. An ``os_features`` can be + specified as a condition for needing the dependency by writing it before + the module name and separated by a ``?``, e.g. ``rtlgenrandom?dyn_load``. + * ``header:internal`` is the list of headers (from the current module) + which are internal-only. + * ``header:public`` is a the list of headers (from the + current module) which should be exported for public use. If neither + ``header:internal`` nor ``header:public`` are used then all headers + in the current directory are assumed public. + + .. note:: If you omit a header from both internal and public lists, it will + be ignored. + + * ``header:external`` is used when naming headers which are included + in the source tree but might be replaced by an external version. This is used + for the PKCS11 headers. + * ``arch`` is a list of architectures this module may be used on. + * ``isa`` lists ISA features which must be enabled to use this module. + Can be proceeded by an ``arch`` name followed by a ``:`` if it is only needed + on a specific architecture, e.g. ``x86_64:ssse3``. + * ``cc`` is a list of compilers which can be used with this module. If the + compiler name is suffixed with a version (like "gcc:5.0") then only compilers + with that minimum version can use the module. + * ``os_features`` is a list of OS features which are required in order to use this + module. Each line can specify one or more features combined with ','. Alternatives + can be specified on additional lines. + +Maps: + * ``defines`` is a map from macros to datestamps. These macros will be defined in + the generated ``build.h``. + * ``libs`` specifies additional libraries which should be linked if this module is + included. It maps from the OS name to a list of libraries (comma seperated). + * ``frameworks`` is a macOS/iOS specific feature which maps from an OS name to + a framework. + +Variables: + * ``load_on`` Can take on values ``never``, ``always``, ``auto``, ``dep`` or ``vendor``. + TODO describe the behavior of these + * ``endian`` Required endian for the module (``any`` (default), ``little``, ``big``) + +An example:: + + # Disable this by default + load_on never + + <isa> + sse2 + </isa> + + <defines> + DEFINE1 -> 20180104 + DEFINE2 -> 20190301 + </defines> + + <comment> + I have eaten + the plums + that were in + the icebox + </comment> + + <warning> + There are no more plums + </warning> + + <header:public> + header1.h + </header:public> + + <header:internal> + header_helper.h + whatever.h + </header:internal> + + <arch> + x86_64 + </arch> + + <cc> + gcc:4.9 # gcc 4.8 doesn't work for <reasons> + clang + </cc> + + # Can work with POSIX+getentropy or Win32 + <os_features> + posix1,getentropy + win32 + </os_features> + + <frameworks> + macos -> FramyMcFramerson + </frameworks> + + <libs> + qnx -> foo,bar,baz + solaris -> socket + </libs> + +Supporting a new CPU type +--------------------------- + +CPU information is stored in ``src/build-data/arch``. + +There is also a file ``src/build-data/detect_arch.cpp`` which is used +for build-time architecture detection using the compiler preprocessor. +Supporting this is optional but recommended. + +Lists: + * ``aliases`` is a list of alternative names for the CPU architecture. + * ``isa_extensions`` is a list of possible ISA extensions that can be used on + this architecture. For example x86-64 has extensions "sse2", "ssse3", + "avx2", "aesni", ... + +Variables: + * ``endian`` if defined should be "little" or "big". This can also be + controlled or overridden at build time. + * ``family`` can specify a family group for several related architecture. + For example both x86_32 and x86_64 use ``family`` of "x86". + * ``wordsize`` is the default wordsize, which controls the size of limbs + in the multi precision integers. If not set, defaults to 32. + +Supporting a new compiler +--------------------------- + +Compiler information is stored in ``src/build-data/cc``. Looking over +those files will probably help understanding, especially the ones for +GCC and Clang which are most complete. + +In addition to the info file, for compilers there is a file +``src/build-data/detect_version.cpp``. The ``configure.py`` script runs the +preprocessor over this file to attempt to detect the compiler +version. Supporting this is not strictly necessary. + +Maps: + * ``binary_link_commands`` gives the command to use to run the linker, + it maps from operating system name to the command to use. It uses + the entry "default" for any OS not otherwise listed. + * ``cpu_flags_no_debug`` unused, will be removed + * ``cpu_flags`` used to emit CPU specific flags, for example LLVM + bitcode target uses ``-emit-llvm`` flag. Rarely needed. + * ``isa_flags`` maps from CPU extensions (like NEON or AES-NI) to + compiler flags which enable that extension. These have the same name + as the ISA flags listed in the architecture files. + * ``lib_flags`` has a single possible entry "debug" which if set maps + to additional flags to pass when building a debug library. + Rarely needed. + * ``mach_abi_linking`` specifies flags to enable when building and + linking on a particular CPU. This is usually flags that modify + ABI. There is a special syntax supported here + "all!os1,arch1,os2,arch2" which allows setting ABI flags which are + used for all but the named operating systems and/or architectures. + * ``sanitizers`` is a map of sanitizers the compiler supports. It must + include "default" which is a list of sanitizers to include by default + when sanitizers are requested. The other keys should map to compiler + flags. + * ``so_link_commands`` maps from operating system to the command to + use to build a shared object. + +Variables: + * ``binary_name`` the default name of the compiler binary. + * ``linker_name`` the name of the linker to use with this compiler. + * ``macro_name`` a macro of the for ``BOTAN_BUILD_COMPILER_IS_XXX`` + will be defined. + * ``output_to_object`` (default "-o") gives the compiler option used to + name the output object. + * ``output_to_exe`` (default "-o") gives the compiler option used to + name the output object. + * ``add_include_dir_option`` (default "-I") gives the compiler option used + to specify an additional include dir. + * ``add_lib_dir_option`` (default "-L") gives the compiler option used + to specify an additional library dir. + * ``add_sysroot_option`` gives the compiler option used to specify the sysroot. + * ``add_lib_option`` (default "-l%s") gives the compiler option to + link in a library. ``%s`` will be replaced with the library name. + * ``add_framework_option`` (default "-framework") gives the compiler option + to add a macOS framework. + * ``preproc_flags`` (default "-E") gives the compiler option used to run + the preprocessor. + * ``compile_flags`` (default "-c") gives the compiler option used to compile a file. + * ``debug_info_flags`` (default "-g") gives the compiler option used to enable debug info. + * ``optimization_flags`` gives the compiler optimization flags to use. + * ``size_optimization_flags`` gives compiler optimization flags to use when + compiling for size. If not set then ``--optimize-for-size`` will use + the default optimization flags. + * ``sanitizer_optimization_flags`` gives compiler optimization flags to use + when building with sanitizers. + * ``coverage_flags`` gives the compiler flags to use when generating coverage + information. + * ``stack_protector_flags`` gives compiler flags to enable stack overflow checking. + * ``shared_flags`` gives compiler flags to use when generation shared libraries. + * ``lang_flags`` gives compiler flags used to enable the required version of C++. + * ``warning_flags`` gives warning flags to enable. + * ``maintainer_warning_flags`` gives extra warning flags to enable during maintainer + mode builds. + * ``visibility_build_flags`` gives compiler flags to control symbol visibility + when generation shared libraries. + * ``visibility_attribute`` gives the attribute to use in the ``BOTAN_DLL`` macro + to specify visibility when generation shared libraries. + * ``ar_command`` gives the command to build static libraries + * ``ar_options`` gives the options to pass to ``ar_command``, if not set here + takes this from the OS specific information. + * ``ar_output_to`` gives the flag to pass to ``ar_command`` to specify where to + output the static library. + * ``werror_flags`` gives the complier flags to treat warnings as errors. + +Supporting a new OS +--------------------------- + +Operating system information is stored in ``src/build-data/os``. + +Lists: + * ``aliases`` is a list of alternative names which will be accepted + * ``target_features`` is a list of target specific OS features. Some of these + are supported by many OSes (for example "posix1") others are specific to + just one or two OSes (such as "getauxval"). Adding a value here causes a new + macro ``BOTAN_TARGET_OS_HAS_XXX`` to be defined at build time. Use + ``configure.py --list-os-features`` to list the currently defined OS + features. + * ``feature_macros`` is a list of macros to define. + +Variables: + * ``ar_command`` gives the command to build static libraries + * ``ar_options`` gives the options to pass to ``ar_command`` + * ``ar_output_to`` gives the flag to pass to ``ar_command`` to specify where to + output the static library. + * ``bin_dir`` (default "bin") specifies where binaries should be installed, + relative to install_root. + * ``cli_exe_name`` (default "botan") specifies the name of the command line utility. + * ``default_compiler`` specifies the default compiler to use for this OS. + * ``doc_dir`` (default "doc") specifies where documentation should be installed, + relative to install_root + * ``header_dir`` (default "include") specifies where include files + should be installed, relative to install_root + * ``install_root`` (default "/usr/local") specifies where to install + by default. + * ``lib_dir`` (default "lib") specifies where library should be installed, + relative to install_root. + * ``lib_prefix`` (default "lib") prefix to add to the library name + * ``library_name`` + * ``man_dir`` specifies where man files should be installed, relative to install_root + * ``obj_suffix`` (default "o") specifies the suffix used for object files + * ``program_suffix`` (default "") specifies the suffix used for executables + * ``shared_lib_symlinks`` (default "yes) specifies if symbolic names should be + created from the base and patch soname to the library name. + * ``soname_pattern_abi`` + * ``soname_pattern_base`` + * ``soname_pattern_patch`` + * ``soname_suffix`` file extension to use for shared library if ``soname_pattern_base`` + is not specified. + * ``static_suffix`` (default "a") file extension to use for static library. + * ``use_stack_protector`` (default "true") specify if by default stack smashing + protections should be enabled. + * ``uses_pkg_config`` (default "yes") specify if by default a pkg-config file + should be created. diff --git a/comm/third_party/botan/doc/dev_ref/contents.rst b/comm/third_party/botan/doc/dev_ref/contents.rst new file mode 100644 index 0000000000..6505762e1f --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/contents.rst @@ -0,0 +1,20 @@ +Developer Reference +===================== + +This section contains information useful to people making +contributions to the library + +.. toctree:: + :maxdepth: 1 + + contributing + configure + test_framework + continuous_integration + fuzzing + release_process + todo + os + oids + reading_list + mistakes diff --git a/comm/third_party/botan/doc/dev_ref/continuous_integration.rst b/comm/third_party/botan/doc/dev_ref/continuous_integration.rst new file mode 100644 index 0000000000..a109301e66 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/continuous_integration.rst @@ -0,0 +1,75 @@ +Continuous Integration and Automated Testing +=============================================== + +CI Build Script +---------------- + +The Travis and AppVeyor builds are orchestrated using a script +``src/scripts/ci_build.py``. This allows one to easily reproduce the CI process +on a local machine. + +Travis CI +----------- + +https://travis-ci.com/github/randombit/botan + +This is the primary CI, and tests the Linux, macOS, and iOS builds. Among other +things it runs tests using valgrind, compilation on various architectures +(currently including ARM, PPC64, and S390x), MinGW build, and a build that +produces the coverage report. + +The Travis configurations is in ``src/scripts/ci/travis.yml``, which executes a +setup script ``src/scripts/ci/setup_travis.sh`` to install needed packages. +Then ``src/scripts/ci_build.py`` is invoked. + +AppVeyor +---------- + +https://ci.appveyor.com/project/randombit/botan + +Runs a build/test cycle using MSVC on Windows. Like Travis it uses +``src/scripts/ci_build.py``. The AppVeyor setup script is in +``src/scripts/ci/setup_appveyor.bat`` + +The AppVeyor build uses `sccache <https://github.com/mozilla/sccache>`_ as a +compiler cache. Since that is not available in the AppVeyor images, the setup +script downloads a release binary from the upstream repository. + +LGTM +--------- + +https://lgtm.com/projects/g/randombit/botan/ + +An automated linter that is integrated with Github. It automatically checks each +incoming PR. It also supports custom queries/alerts, which likely would be worth +investigating but is not something currently in use. + +Coverity +--------- + +https://scan.coverity.com/projects/624 + +An automated source code scanner. Use of Coverity scanner is rate-limited, +sometimes it is very slow to produce a new report, and occasionally the service +goes offline for days or weeks at a time. New reports are kicked off manually by +rebasing branch ``coverity_scan`` against the most recent master and force +pushing it. + +Sonar +------- + +https://sonarcloud.io/dashboard?id=botan + +Sonar scanner is another software quality scanner. Unfortunately a recent update +of their scanner caused it to take over an hour to produce a report which caused +Travis CI timeouts, so it has been disabled. It should be re-enabled to run on +demand in the same way Coverity is. + +OSS-Fuzz +---------- + +https://github.com/google/oss-fuzz/ + +OSS-Fuzz is a distributed fuzzer run by Google. Every night, each library fuzzers +in ``src/fuzzer`` are built and run on many machines, with any findings reported +to the developers via email. diff --git a/comm/third_party/botan/doc/dev_ref/contributing.rst b/comm/third_party/botan/doc/dev_ref/contributing.rst new file mode 100644 index 0000000000..f6effb3f36 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/contributing.rst @@ -0,0 +1,268 @@ + +Notes for New Contributors +=================================== + +Source Code Layout +------------------------------------------------- + +Under ``src`` there are directories + +* ``lib`` is the library itself, more on that below +* ``cli`` is the command line application ``botan`` +* ``tests`` contain what you would expect. Input files go under ``tests/data``. +* ``build-data`` contains files read by the configure script. For + example ``build-data/cc/gcc.txt`` describes various gcc options. +* ``scripts`` contains misc scripts: install, distribution, various + codegen things. Scripts controlling CI go under ``scripts/ci``. +* ``configs`` contains configuration files for emacs, astyle, pylint, etc +* ``python/botan2.py`` is the Python ctypes wrapper + +Library Layout +---------------------------------------- + +* ``base`` defines some high level types +* ``utils`` contains various utility functions and types +* ``codec`` has hex, base64 +* ``block`` contains the block cipher implementations +* ``modes`` contains block cipher modes (CBC, GCM, etc) +* ``stream`` contains the stream ciphers +* ``hash`` contains the hash function implementations +* ``passhash`` contains password hashing algorithms for authentication +* ``kdf`` contains the key derivation functions +* ``mac`` contains the message authentication codes +* ``pbkdf`` contains password hashing algorithms for key derivation +* ``math`` is the big integer math library. It is divided into three parts: + ``mp`` which are the low level algorithms; ``bigint`` which is a C++ wrapper + around ``mp``, and ``numbertheory`` which contains higher level algorithms like + primality testing and exponentiation +* ``pubkey`` contains the public key algorithms +* ``pk_pad`` contains padding schemes for public key algorithms +* ``rng`` contains the random number generators +* ``entropy`` has various entropy sources used by some of the RNGs +* ``asn1`` is the DER encoder/decoder +* ``x509`` is X.509 certificates, PKCS #10 requests, OCSP +* ``tls`` contains the TLS implementation +* ``filters`` is a filter/pipe API for data transforms +* ``compression`` has the compression wrappers (zlib, bzip2, lzma) +* ``ffi`` is the C99 API +* ``prov`` contains bindings to external libraries like OpenSSL and PKCS #11 +* ``misc`` contains odds and ends: format preserving encryption, SRP, threshold + secret sharing, all or nothing transform, and others + +Sending patches +---------------------------------------- + +All contributions should be submitted as pull requests via GitHub +(https://github.com/randombit/botan). If you are planning a large +change email the mailing list or open a discussion ticket on github +before starting out to make sure you are on the right path. And once +you have something written, free to open a [WIP] PR for early review +and comment. + +If possible please sign your git commits using a PGP key. +See https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work for +instructions on how to set this up. + +Depending on what your change is, your PR should probably also include an update +to ``news.rst`` with a note explaining the change. If your change is a +simple bug fix, a one sentence description is perhaps sufficient. If there is an +existing ticket on GitHub with discussion or other information, reference it in +your change note as 'GH #000'. + +Update ``doc/credits.txt`` with your information so people know what you did! + +If you are interested in contributing but don't know where to start check out +``doc/dev_ref/todo.rst`` for some ideas - these are changes we would almost +certainly accept once they've passed code review. + +Also, try building and testing it on whatever hardware you have handy, +especially unusual platforms, or using C++ compilers other than the regularly +tested GCC, Clang, and Visual Studio. + +FFI Additions +---------------- + +If adding a new function declaration to ``ffi.h``, the same PR must +also add the same declaration in the Python binding ``botan2.py``, in +addition the new API functionality must be exposed to Python and a +test written in Python. + +Git Usage +---------------------------------------- + +Do *NOT* merge ``master`` into your topic branch, this creates +needless commits and noise in history. Instead, as needed, rebase your +branch against master (``git rebase -i master``) and force push the +branch to update the PR. If the GitHub PR page does not report any +merge conflicts and nobody asks you to rebase, you don't need to +rebase. + +Try to keep your history clean and use rebase to squash your commits +as needed. If your diff is less than roughly 100 lines, it should +probably be a single commit. Only split commits as needed to help with +review/understanding of the change. + +Python +---------------------------------------- + +Scripts should be in Python whenever possible. + +For configure.py (and helper scripts install.py, cleanup.py and build_docs.py) +the target is stock (no modules outside the standard library) CPython 2.7 plus +latest CPython 3.x. Support for CPython 2.6, PyPy, etc is great when viable (in +the sense of not causing problems for 2.7 or 3.x, and not requiring huge blocks +of version dependent code). As running this program successfully is required for +a working build, making it as portable as possible is considered key. + +The python wrapper botan2.py targets CPython 2.7, 3.x, and latest PyPy. Note that +a single file is used to avoid dealing with any of Python's various crazy module +distribution issues. + +For random scripts not typically run by an end-user (codegen, visualization, and +so on) there isn't any need to worry about 2.6 and even just running under +Python2 xor Python3 is acceptable if needed. Here it's fine to depend on any +useful modules such as graphviz or matplotlib, regardless if it is available +from a stock CPython install. Since Python2 is now end of life, prefer Python3 +for new scripts of this sort. + +Build Tools and Hints +---------------------------------------- + +If you don't already use it for all your C/C++ development, install ``ccache`` +(or on Windows, ``sccache``) right now, and configure a large cache on a fast +disk. It allows for very quick rebuilds by caching the compiler output. + +Use ``--enable-sanitizers=`` flag to enable various sanitizer checks. Supported +values including "address" and "undefined" for GCC and Clang. GCC also supports +"iterator" (checked iterators), and Clang supports "memory" (MSan) and +"coverage" (for fuzzing). + +On Linux if you have the ``lcov`` and ``gcov`` tools installed, then running +``./src/scripts/ci_build.py coverage`` will produce a coverage enabled build, +run the tests, test the fuzzers against a corpus, and produce an HTML report +of total coverage. This coverage build requires the development headers for +zlib, bzip2, liblzma, OpenSSL, TrouSerS (libtspi), and Sqlite3. + +Copyright Notice +---------------------------------------- + +At the top of any new file add a comment with a copyright and a reference to the +license, for example:: + + /* + * (C) 20xx Copyright Holder + * Botan is released under the Simplified BSD License (see license.txt) + */ + +If you are making a substantial or non-trivial change to an existing file, add +or update your own copyright statement at the top of each file. + +Style Conventions +---------------------------------------- + +When writing your code remember the need for it to be easily understood by +reviewers and auditors, both at the time of the patch submission and in the +future. + +Avoid complicated template metaprogramming where possible. It has its places but +should be used judiciously. + +When designing a new API (for use either by library users or just internally) +try writing out the calling code first. That is, write out some code calling +your idealized API, then just implement that API. This can often help avoid +cut-and-paste by creating the correct abstractions needed to solve the problem +at hand. + +The C++11 ``auto`` keyword is very convenient but only use it when the type +truly is obvious (considering also the potential for unexpected integer +conversions and the like, such as an apparent uint8_t being promoted to an int). + +If a variable is defined and not modified, declare it ``const``. Some exception +for very short-lived variables, but generally speaking being able to read the +declaration and know it will not be modified is useful. + +Use ``override`` annotations whenever overriding a virtual function. If +introducing a new type that is not intended for derivation, mark it ``final``. + +Avoid explicit ``delete`` - use RAII. + +Use ``m_`` prefix on all member variables. + +For formatting, there are configs for emacs and astyle in ``src/configs``. +No tabs, and remove trailing whitespace. + +Prefer using braces on both sides of if/else blocks, even if only using a single +statement. The current code doesn't always do this. + +Avoid ``using namespace`` declarations, even inside of single functions. One +allowed exception is ``using namespace std::placeholders`` in functions which +use ``std::bind``. (But, don't use ``std::bind`` - use a lambda instead). + +Use ``::`` to explicitly refer to the global namespace (eg, when calling an OS +or external library function like ``::select`` or ``::sqlite3_open``). + +Use of External Dependencies +---------------------------------------- + +Compiler Dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + +The library should always be as functional as possible when compiled with just +C++11. However, feel free to use the full C++11 language. No accomodations are +made for compilers that are incomplete or buggy. + +Use of compiler extensions is fine whenever appropriate; this is typically +restricted to a single file or an internal header. Compiler extensions used +currently include native uint128_t, SIMD intrinsics, inline asm syntax and so +on, so there are some existing examples of appropriate use. + +Generally intrinsics or inline asm is preferred over bare assembly to avoid +calling convention issues among different platforms; the improvement in +maintainability is seen as worth any potential performance tradeoff. One risk +with intrinsics is that the compiler might rewrite your clever const-time SIMD +into something with a conditional jump, but code intended to be const-time +should in any case be annotated so it can be checked at runtime with tools. + +Operating System Dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're adding a small OS dependency in some larger piece of code, try to +contain the actual non-portable operations to utils/os_utils.* and then call +them from there. + +As a policy, operating systems which are not supported by their original vendor +are not supported by Botan either. Patches that complicate the code in order to +support obsolete operating systems will likely be rejected. In writing OS +specific code, feel free to assume roughly POSIX 2008, or for Windows, Windows 8 +/Server 2012 (which are as of this writing the oldest versions still supported +by Microsoft). + +Some operating systems, such as OpenBSD, only support the latest release. For +such cases, it's acceptable to add code that requires APIs added in the most +recent release of that OS as soon as the release is available. + +Library Dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Any external library dependency - even optional ones - is met with as one PR +submitter put it "great skepticism". + +At every API boundary there is potential for confusion that does not exist when +the call stack is all contained within the boundary. So the additional API +really needs to pull its weight. For example a simple text parser or such which +can be trivially implemented is not really for consideration. As a rough idea of +the bar, equate the viewed cost of an external dependency as at least 1000 +additional lines of code in the library. That is, if the library really does +need this functionality, and it can be done in the library for less than that, +then it makes sense to just write the code. Yup. + +Currently the (optional) external dependencies of the library are OpenSSL (for +access to fast and side channel hardened RSA, ECDSA, AES), some compression +libraries (zlib, bzip2, lzma), sqlite3 database, Trousers (TPM integration), +plus various operating system utilities like basic filesystem operations. These +provide major pieces of functionality which seem worth the trouble of +maintaining an integration with. + +At this point the most plausible examples of an appropriate new external +dependency are all deeper integrations with system level cryptographic systems +(CommonCrypto, CryptoAPI, /dev/crypto, iOS keychain, TPM 2.0, etc) diff --git a/comm/third_party/botan/doc/dev_ref/fuzzing.rst b/comm/third_party/botan/doc/dev_ref/fuzzing.rst new file mode 100644 index 0000000000..46e60fb533 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/fuzzing.rst @@ -0,0 +1,91 @@ +Fuzzing The Library +============================ + +Botan comes with a set of fuzzing endpoints which can be used to test +the library. + +.. highlight:: shell + +Fuzzing with libFuzzer +------------------------ + +To fuzz with libFuzzer (https://llvm.org/docs/LibFuzzer.html), you'll first +need to compile libFuzzer:: + + $ svn co https://llvm.org/svn/llvm-project/compiler-rt/trunk/lib/fuzzer libFuzzer + $ cd libFuzzer && clang -c -g -O2 -std=c++11 *.cpp + $ ar cr libFuzzer.a libFuzzer/*.o + +Then build the fuzzers:: + + $ ./configure.py --cc=clang --build-fuzzer=libfuzzer --unsafe-fuzzer-mode \ + --enable-sanitizers=coverage,address,undefined + $ make fuzzers + +Enabling 'coverage' sanitizer flags is required for libFuzzer to work. +Address sanitizer and undefined sanitizer are optional. + +The fuzzer binaries will be in `build/fuzzer`. Simply pick one and run it, optionally +also passing a directory containing corpus inputs. + +Using `libfuzzer` build mode implicitly assumes the fuzzers need to +link with `libFuzzer`; if another library is needed (for example in +OSS-Fuzz, which uses `libFuzzingEngine`), use the flag +`--with-fuzzer-lib` to specify the desired name. + +Fuzzing with AFL +-------------------- + +To fuzz with AFL (http://lcamtuf.coredump.cx/afl/):: + + $ ./configure.py --with-sanitizers --build-fuzzer=afl --unsafe-fuzzer-mode --cc-bin=afl-g++ + $ make fuzzers + +For AFL sanitizers are optional. You can also use `afl-clang-fast++` +or `afl-clang++`, be sure to set `--cc=clang` also. + +The fuzzer binaries will be in `build/fuzzer`. To run them you need to +run under `afl-fuzz`:: + + $ afl-fuzz -i corpus_path -o output_path ./build/fuzzer/binary + +Fuzzing with TLS-Attacker +-------------------------- + +TLS-Attacker (https://github.com/RUB-NDS/TLS-Attacker) includes a mode for fuzzing +TLS servers. A prebuilt copy of TLS-Attacker is available in a git repository:: + + $ git clone --depth 1 https://github.com/randombit/botan-ci-tools.git + +To run it against Botan's server:: + + $ ./configure.py --with-sanitizers + $ make botan + $ ./src/scripts/run_tls_attacker.py ./botan ./botan-ci-tools + +Output and logs from the fuzzer are placed into `/tmp`. See the +TLS-Attacker documentation for more information about how to use this +tool. + +Input Corpus +----------------------- + +AFL requires an input corpus, and libFuzzer can certainly make good +use of it. + +Some crypto corpus repositories include + +* https://github.com/randombit/crypto-corpus +* https://github.com/mozilla/nss-fuzzing-corpus +* https://github.com/google/boringssl/tree/master/fuzz +* https://github.com/openssl/openssl/tree/master/fuzz/corpora + +Adding new fuzzers +--------------------- + +New fuzzers are created by adding a source file to `src/fuzzers` which +have the signature: + +``void fuzz(const uint8_t in[], size_t len)`` + +After adding your fuzzer, rerun ``./configure.py`` and build. diff --git a/comm/third_party/botan/doc/dev_ref/mistakes.rst b/comm/third_party/botan/doc/dev_ref/mistakes.rst new file mode 100644 index 0000000000..6ea9ea927e --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/mistakes.rst @@ -0,0 +1,77 @@ + +Mistakes Were Made +=================== + +These are mistakes made early on in the project's history which are difficult to +fix now, but mentioned in the hope they may serve as an example for others. + +C++ API +--------- + +As an implementation language, I still think C++ is the best choice (or at least +the best choice available in early '00s) at offering good performance, +reasonable abstractions, and low overhead. But the user API should have been +pure C with opaque structs (rather like the FFI layer, which was added much +later). Then an expressive C++ API could be built on top of the C API. This +would have given us a stable ABI, allowed C applications to use the library, and +(these days) make it easier to progressively rewrite the library in Rust. + +Public Algorithm Specific Classes +------------------------------------ + +Classes like AES_128 and SHA_256 should never have been exposed to applications. +Intead such operations should have been accessible only via the higher level +interfaces (here BlockCipher and HashFunction). This would substantially reduce +the overall API and ABI surface. + +These interfaces are now deprecated, and perhaps will be able to be +removed eventually. + +Header Directories +------------------- + +It would have been better to install all headers as ``X/header.h`` +where ``X`` is the base dir in the source, eg ``block/aes128.h``, +``hash/md5.h``, ... + +Exceptions +----------- + +Constant ABI headaches from this, and it impacts performance and makes APIs +harder to understand. Should have been handled with a result<> type instead. + +Virtual inheritance +--------------------- + +This was used in the public key interfaces and the hierarchy is a tangle. +Public and private keys should be distinct classes, with a function on private +keys that creates a new object corresponding to the public key. + +Cipher Interface +------------------ + +The cipher interface taking a secure_vector that it reads from and writes to was +an artifact of an earlier design which supported both compression and encryption +in a single API. But it leads to inefficient copies. + +(I am hoping this issue can be somewhat fixed by introducing a new cipher API +and implementing the old API in terms of the new one.) + +Pipe Interface +---------------- + +On the surface this API seems very convenient and easy to use. And it is. But +the downside is it makes the application code totally opaque; some bytes go into +a Pipe object and then come out the end transformed in some way. What happens in +between? Unless the Pipe was built in the same function and you can see the +parameters to the constructor, there is no way to find out. + +The problems with the Pipe API are documented, and it is no longer used within +the library itself. But since many people seem to like it and many applications +use it, we are stuck at least with maintaining it as it currently exists. + +License +--------- + +MIT is more widely used and doesn't have the ambiguity surrounding the +various flavors of BSD. diff --git a/comm/third_party/botan/doc/dev_ref/oids.rst b/comm/third_party/botan/doc/dev_ref/oids.rst new file mode 100644 index 0000000000..6aac1a5a72 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/oids.rst @@ -0,0 +1,43 @@ +Private OID Assignments +========================== + +The library uses some OIDs under a private arc assigned by IANA, +1.3.6.1.4.1.25258 + +Values currently assigned are:: + + randombit OBJECT IDENTIFIER ::= { 1 3 6 1 4 1 25258 } + + publicKey OBJECT IDENTIFIER ::= { randombit 1 } + + mceliece OBJECT IDENTIFIER ::= { publicKey 3 } + -- { publicKey 4 } previously used as private X25519 + -- { publicKey 5 } previously used for XMSS draft 6 + gost-3410-with-sha256 OBJECT IDENTIFIER ::= { publicKey 6 1 } + kyber OBJECT IDENTIFIER ::= { publicKey 7 } + xmss OBJECT IDENTIFIER ::= { publicKey 8 } + + symmetricKey OBJECT IDENTIFIER ::= { randombit 3 } + + ocbModes OBJECT IDENTIFIER ::= { symmetricKey 2 } + + aes-128-ocb OBJECT IDENTIFIER ::= { ocbModes 1 } + aes-192-ocb OBJECT IDENTIFIER ::= { ocbModes 2 } + aes-256-ocb OBJECT IDENTIFIER ::= { ocbModes 3 } + serpent-256-ocb OBJECT IDENTIFIER ::= { ocbModes 4 } + twofish-256-ocb OBJECT IDENTIFIER ::= { ocbModes 5 } + camellia-128-ocb OBJECT IDENTIFIER ::= { ocbModes 6 } + camellia-192-ocb OBJECT IDENTIFIER ::= { ocbModes 7 } + camellia-256-ocb OBJECT IDENTIFIER ::= { ocbModes 8 } + + sivModes OBJECT IDENTIFIER ::= { symmetricKey 4 } + + aes-128-siv OBJECT IDENTIFIER ::= { sivModes 1 } + aes-192-siv OBJECT IDENTIFIER ::= { sivModes 2 } + aes-256-siv OBJECT IDENTIFIER ::= { sivModes 3 } + serpent-256-siv OBJECT IDENTIFIER ::= { sivModes 4 } + twofish-256-siv OBJECT IDENTIFIER ::= { sivModes 5 } + camellia-128-siv OBJECT IDENTIFIER ::= { sivModes 6 } + camellia-192-siv OBJECT IDENTIFIER ::= { sivModes 7 } + camellia-256-siv OBJECT IDENTIFIER ::= { sivModes 8 } + sm4-128-siv OBJECT IDENTIFIER ::= { sivModes 9 } diff --git a/comm/third_party/botan/doc/dev_ref/os.rst b/comm/third_party/botan/doc/dev_ref/os.rst new file mode 100644 index 0000000000..eee3e09b76 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/os.rst @@ -0,0 +1,61 @@ +OS Features +======================================== + +A summary of OS features as defined in ``src/build-data/os``. + +:: + + a: aix + a: android + c: cygwin + d: dragonfly + e: emscripten + f: freebsd + h: haiku + h: hpux + h: hurd + i: includeos + i: ios + l: linux + l: llvm + m: macos + m: mingw + n: nacl + n: netbsd + o: openbsd + q: qnx + s: solaris + u: uwp + w: windows + +.. csv-table:: + :header: "Feature", "a", "a", "c", "d", "e", "f", "h", "h", "h", "i", "i", "l", "l", "m", "m", "n", "n", "o", "q", "s", "u", "w" + + "apple_keychain", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " " + "arc4random", " ", "X", " ", "X", " ", "X", " ", " ", " ", " ", "X", " ", " ", "X", " ", " ", "X", "X", " ", " ", " ", " " + "cap_enter", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " " + "certificate_store", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X" + "clock_gettime", "X", "X", " ", "X", " ", "X", "X", "X", "X", " ", " ", "X", " ", "X", " ", " ", "X", "X", "X", "X", " ", " " + "commoncrypto", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " " + "crypto_ng", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " " + "dev_random", "X", "X", "X", "X", "X", "X", "X", "X", "X", "X", " ", "X", " ", "X", " ", " ", "X", "X", "X", "X", " ", " " + "elf_aux_info", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " " + "explicit_bzero", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " " + "explicit_memset", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " " + "filesystem", "X", "X", "X", "X", "X", "X", "X", "X", "X", " ", "X", "X", "X", "X", "X", " ", "X", "X", "X", "X", "X", "X" + "getauxval", " ", "X", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", " ", " ", " " + "getentropy", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", "X", " ", " ", " ", " " + "pledge", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " " + "posix1", "X", "X", "X", "X", "X", "X", "X", "X", "X", "X", "X", "X", " ", "X", " ", " ", "X", "X", "X", "X", " ", " " + "posix_mlock", "X", "X", " ", "X", " ", "X", " ", "X", "X", " ", "X", "X", " ", "X", " ", " ", "X", "X", "X", "X", " ", " " + "proc_fs", "X", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", " ", "X", " ", " " + "rtlgenrandom", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", "X" + "rtlsecurezeromemory", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", "X" + "sockets", "X", "X", "X", "X", " ", "X", "X", "X", "X", " ", "X", "X", " ", "X", " ", " ", "X", "X", "X", "X", " ", " " + "threads", "X", "X", "X", "X", " ", "X", "X", "X", "X", " ", "X", "X", " ", "X", "X", "X", "X", "X", "X", "X", "X", "X" + "virtual_lock", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", " ", "X" + "win32", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", " ", " ", " ", " ", " ", "X", "X" + "winsock2", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "X", "X" + +.. note:: + This file is auto generated by ``src/scripts/gen_os_features.py``. Dont modify it manually. diff --git a/comm/third_party/botan/doc/dev_ref/reading_list.rst b/comm/third_party/botan/doc/dev_ref/reading_list.rst new file mode 100644 index 0000000000..1b27d05d69 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/reading_list.rst @@ -0,0 +1,93 @@ +Reading List +================ + +These are papers, articles and books that are interesting or useful from the +perspective of crypto implementation. + +Papers +-------- + +Implementation Techniques +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* "Randomizing the Montgomery Powering Ladder" + Le, Tan, Tunstall https://eprint.iacr.org/2015/657 + A variant of Algorithm 7 is used for GF(p) point multplications when + BOTAN_POINTGFP_BLINDED_MULTIPLY_USE_MONTGOMERY_LADDER is set + +* "Accelerating AES with vector permute instructions" + Mike Hamburg https://shiftleft.org/papers/vector_aes/ + His public doman assembly code was rewritten into SSS3 intrinsics + for aes_ssse3. + +* "Elliptic curves and their implementation" Langley + http://www.imperialviolet.org/2010/12/04/ecc.html + Describes sparse representations for ECC math + +Random Number Generation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* "On Extract-then-Expand Key Derivation Functions and an HMAC-based KDF" + Hugo Krawczyk http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.131.8254 + RNG design underlying HMAC_RNG + +AES Side Channels +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* "Software mitigations to hedge AES against cache-based software side + channel vulnerabilities" https://eprint.iacr.org/2006/052.pdf + +* "Cache Games - Bringing Access-Based Cache Attacks on AES to Practice" + http://www.ieee-security.org/TC/SP2011/PAPERS/2011/paper031.pdf + +* "Cache-Collision Timing Attacks Against AES" Bonneau, Mironov + http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.88.4753 + +Public Key Side Channels +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* "Fast Elliptic Curve Multiplications Resistant against Side Channel Attacks" + http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.1028&rep=rep1&type=pdf + +* "Resistance against Differential Power Analysis for Elliptic Curve Cryptosystems" + Coron http://www.jscoron.fr/publications/dpaecc.pdf + +* "Further Results and Considerations on Side Channel Attacks on RSA" + Klima, Rosa https://eprint.iacr.org/2002/071 + Side channel attacks on RSA-KEM and MGF1-SHA1 + +* "Side-Channel Attacks on the McEliece and Niederreiter Public-Key Cryptosystems" + Avanzi, Hoerder, Page, and Tunstall https://eprint.iacr.org/2010/479 + +* "Minimum Requirements for Evaluating Side-Channel Attack Resistance + of Elliptic Curve Implementations" BSI + https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Zertifizierung/Interpretationen/AIS_46_ECCGuide_e_pdf.pdf + +Books +------ + +* "Handbook of Elliptic and Hyperelliptic Curve Cryptography" + Cohen and Frey https://www.hyperelliptic.org/HEHCC/ + An excellent reference for ECC math, algorithms, and side channels + +* "Post-Quantum Cryptography" Bernstein, Buchmann, Dahmen + Covers code, lattice, and hash based cryptography + +Standards +----------- + +* IEEE 1363 http://grouper.ieee.org/groups/1363/ + Very influential early in the library lifetime, so a lot of terminology used + in the public key (such as "EME" for message encoding) code comes from here. + +* ISO/IEC 18033-2 http://www.shoup.net/iso/std4.pdf + RSA-KEM, PSEC-KEM + +* NIST SP 800-108 + http://csrc.nist.gov/publications/nistpubs/800-108/sp800-108.pdf + KDF schemes + +* NIST SP 800-90A + http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf + HMAC_DRBG, Hash_DRBG, CTR_DRBG, maybe one other thing? + diff --git a/comm/third_party/botan/doc/dev_ref/release_process.rst b/comm/third_party/botan/doc/dev_ref/release_process.rst new file mode 100644 index 0000000000..1dd0842643 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/release_process.rst @@ -0,0 +1,129 @@ +Release Process and Checklist +======================================== + +Releases are done quarterly, normally on the first non-holiday Monday +of January, April, July and October. A feature freeze goes into effect +starting 9 days before the release. + +.. highlight:: shell + +.. note:: + + This information is only useful if you are a developer of botan who + is creating a new release of the library. + +Pre Release Testing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Kick off a Coverity scan a day or so before the planned release. + +Do maintainer-mode builds with Clang and GCC to catch any warnings +that should be corrected. Also check Visual C++ build logs for any +warnings that should be addressed. + +And remember that CI doesn't test everything. In particular, not all +tests run under valgrind or on the qemu cross builds due to time +constraints. So before release: + + - Run under valgrind, building with ``--with-valgrind`` flag + - Using Clang sanitizers (ASan + UbSan) + - Native compile on FreeBSD x86-64 + - Native compile on at least one unusual platform (AIX, NetBSD, ...) + - Build the website content to detect any Doxygen problems + - Test many build configurations (using `src/scripts/test_all_configs.py`) + - Build/test SoftHSM + +Confirm that the release notes in ``news.rst`` are accurate and +complete and that the version number in ``version.txt`` is correct. + +Tag the Release +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Update the release date in the release notes and change the entry for +the appropriate branch in ``readme.rst`` to point to the new release. + +Now check in, and backport changes to the release branch:: + + $ git commit readme.rst news.rst -m "Update for 2.6.13 release" + $ git checkout release-2 + $ git merge master + $ git tag 2.6.13 + +Build The Release Tarballs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The release script is ``src/scripts/dist.py`` and must be run from a +git workspace. + + $ src/scripts/dist.py 2.6.13 + +One useful option is ``--output-dir``, which specifies where the +output will be placed. + +Now do a final build/test of the released tarball. + +The ``--pgp-key-id`` option is used to specify a PGP keyid. If set, +the script assumes that it can execute GnuPG and will attempt to +create signatures for the tarballs. The default value is ``EFBADFBC``, +which is the official signing key. You can use ``--pgp-key-id=none`` +to avoid creating any signature, though official distributed releases +*should not* be released without signatures. + +The releases served on the official site are taken from the contents +in a git repository:: + + $ git checkout git@botan.randombit.net:/srv/git/botan-releases.git + $ src/scripts/dist.py 2.6.13 --output-dir=botan-releases + $ cd botan-releases + $ sha256sum Botan-2.6.13.tgz >> sha256sums.txt + $ git add . + $ git commit -m "Release version 2.6.13" + $ git push origin master + +A cron job updates the live site every 10 minutes. + +Push to GitHub +^^^^^^^^^^^^^^^^^^ + +Don't forget to also push tags:: + + $ git push origin --tags release-2 master + +Build The Windows Installer +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + We haven't distributed Windows binaries for some time. + +On Windows, run ``configure.py`` to setup a build:: + + $ python ./configure.py --cc=msvc --cpu=$ARCH --distribution-info=unmodified + +After completing the build (and running the tests), use `InnoSetup +<http://www.jrsoftware.org/isinfo.php>`_ to create the installer. A +InnoSetup script is created from ``src/build-data/innosetup.in`` and +placed in ``build/botan.iss`` by ``configure.py``. Create the +installer either via the InnoSetup GUI by opening the ``iss`` file and +selecting the 'Compile' option, or using the ``iscc`` command line +tool. If all goes well it will produce an executable with a name like +``botan-2.6.13-x86_64.exe``. Sign the installers with GPG. + +Update The Website +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The website content is created by ``src/scripts/website.py``. + +The website is mirrored automatically from a git repository which must be updated:: + + $ git checkout git@botan.randombit.net:/srv/git/botan-website.git + $ ./src/scripts/website.py --output botan-website + $ cd botan-website + $ git add . + $ git commit -m "Update for 2.6.13" + $ git push origin master + +Announce The Release +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Send an email to the botan-announce and botan-devel mailing lists +noting that a new release is available. diff --git a/comm/third_party/botan/doc/dev_ref/test_framework.rst b/comm/third_party/botan/doc/dev_ref/test_framework.rst new file mode 100644 index 0000000000..241c51bd1a --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/test_framework.rst @@ -0,0 +1,314 @@ +Test Framework +================ + +Botan uses a custom-built test framework. Some portions of it are +quite similar to assertion-based test frameworks such as Catch or +Gtest, but it also includes many features which are well suited for +testing cryptographic algorithms. + +The intent is that the test framework and the test suite evolve +symbiotically; as a general rule of thumb if a new function would make +the implementation of just two distinct tests simpler, it is worth +adding to the framework on the assumption it will prove useful again. +Feel free to propose changes to the test system. + +When writing a new test, there are three key classes that are used, +namely ``Test``, ``Test::Result``, and ``Text_Based_Test``. A ``Test`` +(or ``Test_Based_Test``) runs and returns one or more ``Test::Result``. + +Namespaces in Test +------------------- + +The test code lives in a distinct namespace (``Botan_Tests``) and all +code in the tests which calls into the library should use the +namespace prefix ``Botan::`` rather than a ``using namespace`` +declaration. This makes it easier to see where the test is actually +invoking the library, and makes it easier to reuse test code for +applications. + +Test Data +----------- + +The test framework is heavily data driven. As of this writing, there +is about 1 Mib of test code and 17 MiB of test data. For most (though +certainly not all) tests, it is better to add a data file representing +the input and outputs, and run the tests over it. Data driven tests +make adding or editing tests easier, for example by writing scripts +which produce new test data and output it in the expected format. + +Test +-------- + +.. cpp:class:: Test + + .. cpp:function:: virtual std::vector<Test::Result> run() = 0 + + This is the key function of a ``Test``: it executes and returns a + list of results. Almost all other functions on ``Test`` are + static functions which just serve as helper functions for ``run``. + + .. cpp:function:: static std::string read_data_file(const std::string& path) + + Return the contents of a data file and return it as a string. + + .. cpp:function:: static std::vector<uint8_t> read_binary_data_file(const std::string& path) + + Return the contents of a data file and return it as a vector of + bytes. + + .. cpp:function:: static std::string data_file(const std::string& what) + + An alternative to ``read_data_file`` and ``read_binary_file``, + use only as a last result, typically for library APIs which + themselves accept a filename rather than a data blob. + + .. cpp:function:: static bool run_long_tests() const + + Returns true if the user gave option ``--run-long-tests``. Use + this to gate particularly time-intensive tests. + + .. cpp:function:: static Botan::RandomNumberGenerator& rng() + + Returns a reference to a fast, not cryptographically secure + random number generator. It is deterministicly seeded with the + seed logged by the test runner, so it is possible to reproduce + results in "random" tests. + +Tests are registered using the macro ``BOTAN_REGISTER_TEST`` which +takes 2 arguments: the name of the test and the name of the test class. +For example given a ``Test`` instance named ``MyTest``, use:: + + BOTAN_REGISTER_TEST("mytest", MyTest); + +All test names should contain only lowercase letters, numbers, and +underscore. + +Test::Result +------------- + +.. cpp:class:: Test::Result + + A ``Test::Result`` records one or more tests on a particular topic + (say "AES-128/CBC" or "ASN.1 date parsing"). Most of the test functions + return true or false if the test was successful or not; this allows + performing conditional blocks as a result of earlier tests:: + + if(result.test_eq("first value", produced, expected)) + { + // further tests that rely on the initial test being correct + } + + Only the most commonly used functions on ``Test::Result`` are documented here, + see the header ``tests.h`` for more. + + .. cpp:function:: Test::Result(const std::string& who) + + Create a test report on a particular topic. This will be displayed in the + test results. + + .. cpp:function:: bool test_success() + + Report a test that was successful. + + .. cpp:function:: bool test_success(const std::string& note) + + Report a test that was successful, including some comment. + + .. cpp:function:: bool test_failure(const std::string& err) + + Report a test failure of some kind. The error string will be logged. + + .. cpp:function:: bool test_failure(const std::string& what, const std::string& error) + + Report a test failure of some kind, with a description of what failed and + what the error was. + + .. cpp:function:: void test_failure(const std::string& what, const uint8_t buf[], size_t buf_len) + + Report a test failure due to some particular input, which is provided as + arguments. Normally this is only used if the test was using some + randomized input which unexpectedly failed, since if the input is + hardcoded or from a file it is easier to just reference the test number. + + .. cpp:function:: bool test_eq(const std::string& what, const std::string& produced, const std::string& expected) + + Compare to strings for equality. + + .. cpp:function:: bool test_ne(const std::string& what, const std::string& produced, const std::string& expected) + + Compare to strings for non-equality. + + .. cpp:function:: bool test_eq(const char* producer, const std::string& what, \ + const uint8_t produced[], size_t produced_len, \ + const uint8_t expected[], size_t expected_len) + + Compare two arrays for equality. + + .. cpp:function:: bool test_ne(const char* producer, const std::string& what, \ + const uint8_t produced[], size_t produced_len, \ + const uint8_t expected[], size_t expected_len) + + Compare two arrays for non-equality. + + .. cpp:function:: bool test_eq(const std::string& producer, const std::string& what, \ + const std::vector<uint8_t>& produced, \ + const std::vector<uint8_t>& expected) + + Compare two vectors for equality. + + .. cpp:function:: bool test_ne(const std::string& producer, const std::string& what, \ + const std::vector<uint8_t>& produced, \ + const std::vector<uint8_t>& expected) + + Compare two vectors for non-equality. + + .. cpp:function:: bool confirm(const std::string& what, bool expr) + + Test that some expression evaluates to ``true``. + + .. cpp:function:: template<typename T> bool test_not_null(const std::string& what, T* ptr) + + Verify that the pointer is not null. + + .. cpp:function:: bool test_lt(const std::string& what, size_t produced, size_t expected) + + Test that ``produced`` < ``expected``. + + .. cpp:function:: bool test_lte(const std::string& what, size_t produced, size_t expected) + + Test that ``produced`` <= ``expected``. + + .. cpp:function:: bool test_gt(const std::string& what, size_t produced, size_t expected) + + Test that ``produced`` > ``expected``. + + .. cpp:function:: bool test_gte(const std::string& what, size_t produced, size_t expected) + + Test that ``produced`` >= ``expected``. + + .. cpp:function:: bool test_throws(const std::string& what, std::function<void ()> fn) + + Call a function and verify it throws an exception of some kind. + + .. cpp:function:: bool test_throws(const std::string& what, const std::string& expected, std::function<void ()> fn) + + Call a function and verify it throws an exception of some kind + and that the exception message exactly equals ``expected``. + +Text_Based_Test +----------------- + +A ``Text_Based_Text`` runs tests that are produced from a text file +with a particular format which looks somewhat like an INI-file:: + + # Comments begin with # and continue to end of line + [Header] + # Test 1 + Key1 = Value1 + Key2 = Value2 + + # Test 2 + Key1 = Value1 + Key2 = Value2 + +.. cpp:class:: VarMap + + An object of this type is passed to each invocation of the text-based test. + It is used to access the test variables. All access takes a key, which is + one of the strings which was passed to the constructor of ``Text_Based_Text``. + Accesses are either required (``get_req_foo``), in which case an exception is + throwing if the key is not set, or optional (``get_opt_foo``) in which case + the test provides a default value which is returned if the key was not set + for this particular instance of the test. + + .. cpp:function:: std::vector<uint8_t> get_req_bin(const std::string& key) const + + Return a required binary string. The input is assumed to be hex encoded. + + .. cpp:function:: std::vector<uint8_t> get_opt_bin(const std::string& key) const + + Return an optional binary string. The input is assumed to be hex encoded. + + .. cpp:function:: std::vector<std::vector<uint8_t>> get_req_bin_list(const std::string& key) const + + .. cpp:function:: Botan::BigInt get_req_bn(const std::string& key) const + + Return a required BigInt. The input can be decimal or (with "0x" prefix) hex encoded. + + .. cpp:function:: Botan::BigInt get_opt_bn(const std::string& key, const Botan::BigInt& def_value) const + + Return an optional BigInt. The input can be decimal or (with "0x" prefix) hex encoded. + + .. cpp:function:: std::string get_req_str(const std::string& key) const + + Return a required text string. + + .. cpp:function:: std::string get_opt_str(const std::string& key, const std::string& def_value) const + + Return an optional text string. + + .. cpp:function:: size_t get_req_sz(const std::string& key) const + + Return a required integer. The input should be decimal. + + .. cpp:function:: size_t get_opt_sz(const std::string& key, const size_t def_value) const + + Return an optional integer. The input should be decimal. + +.. cpp:class:: Text_Based_Test : public Test + + .. cpp:function:: Text_Based_Test(const std::string& input_file, \ + const std::string& required_keys, \ + const std::string& optional_keys = "") + + This constructor is + + .. note:: + The final element of required_keys is the "output key", that is + the key which signifies the boundary between one test and the next. + When this key is seen, ``run_one_test`` will be invoked. In the + test input file, this key must always appear least for any particular + test. All the other keys may appear in any order. + + .. cpp:function:: Test::Result run_one_test(const std::string& header, \ + const VarMap& vars) + + Runs a single test and returns the result of it. The ``header`` + parameter gives the value (if any) set in a ``[Header]`` block. + This can be useful to distinguish several types of tests within a + single file, for example "[Valid]" and "[Invalid]". + + .. cpp:function:: bool clear_between_callbacks() const + + By default this function returns ``false``. If it returns + ``true``, then when processing the data in the file, variables + are not cleared between tests. This can be useful when several + tests all use some common parameters. + +Test Runner +------------- + +If you are simply writing a new test there should be no need to modify +the runner, however it can be useful to be aware of its abilities. + +The runner can run tests concurrently across many cores. By default single +threaded execution is used, but you can use ``--test-threads`` option to +specify the number of threads to use. If you use ``--test-threads=0`` then +the runner will probe the number of active CPUs and use that (but limited +to at most 16). If you want to run across many cores on a large machine, +explicitly specify a thread count. The speedup is close to linear. + +The RNG used in the tests is deterministic, and the seed is logged for each +execution. You can cause the random sequence to repeat using ``--drbg-seed`` +option. + +.. note:: + Currently the RNG is seeded just once at the start of execution. So you + must run the exact same sequence of tests as the original test run in + order to get reproducible results. + +If you are trying to track down a bug that happens only occasionally, two very +useful options are ``--test-runs`` and ``--abort-on-first-fail``. The first +takes an integer and runs the specified test cases that many times. The second +causes abort to be called on the very first failed test. This is sometimes +useful when tracing a memory corruption bug. diff --git a/comm/third_party/botan/doc/dev_ref/todo.rst b/comm/third_party/botan/doc/dev_ref/todo.rst new file mode 100644 index 0000000000..891475caa9 --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/todo.rst @@ -0,0 +1,199 @@ +Todo List +======================================== + +Feel free to take one of these on if it interests you. Before starting +out on something, send an email to the dev list or open a discussion +ticket on GitHub to make sure you're on the right track. + +Request a new feature by opening a pull request to update this file. + +Ciphers, Hashes, PBKDF +---------------------------------------- + +* Stiched AES/GCM mode for CPUs supporting both AES and CLMUL +* Combine AES-NI, ARMv8 and POWER AES implementations (as already done for CLMUL) +* Vector permute AES only supports little-endian systems; fix for big-endian +* SM4 using AES-NI (https://github.com/mjosaarinen/sm4ni) or vector permute +* Poly1305 using AVX2 +* ChaCha using SSSE3 +* Skein-MAC +* PMAC +* SIV-PMAC +* GCM-SIV (RFC 8452) +* EME* tweakable block cipher (https://eprint.iacr.org/2004/125) +* FFX format preserving encryption (NIST 800-38G) +* SHA-512 using BMI2+AVX2 +* Constant time DES using bitslicing and/or BMI2 +* Threefish-1024 +* SIMD evaluation of SHA-2 and SHA-3 compression functions +* Adiantum (https://eprint.iacr.org/2018/720) +* CRC using clmul/pmull + +Public Key Crypto, Math +---------------------------------------- + +* Short vector optimization for BigInt +* Abstract representation of ECC point elements to allow specific + implementations of the field arithmetic depending upon the curve. +* Use NAF (joint sparse form) for ECC multi-exponentiation +* Curves for pairings (BN-256, BLS12-381) +* Identity based encryption +* Paillier homomorphic cryptosystem +* Socialist Millionaires Protocol (needed for OTRv3) +* Hashing onto an elliptic curve (draft-irtf-cfrg-hash-to-curve) +* New PAKEs (pending CFRG bakeoff results) +* New post quantum schemes (pending NIST contest results) +* SPHINX password store (https://eprint.iacr.org/2018/695) +* X448 and Ed448 +* Use GLV decomposition to speed up secp256k1 operations + +Utility Functions +------------------ + +* Add a memory span type +* Make Memory_Pool more concurrent (currently uses a global lock) +* Guarded integer type to prevent overflow bugs +* Add logging callbacks +* Add latency tracing framework + +Multiparty Protocols +---------------------- + +* Distributed key generation for DL, RSA +* Threshold signing, decryption + +External Providers, Hardware Support +---------------------------------------- + +* Add support ARMv8.4-A SHA-512, SHA-3, SM3 and RNG +* Aarch64 inline asm for BigInt +* Extend OpenSSL provider (DH, HMAC, CMAC, GCM) +* Support using BoringSSL instead of OpenSSL or LibreSSL +* /dev/crypto provider (ciphers, hashes) +* Windows CryptoNG provider (ciphers, hashes) +* Extend Apple CommonCrypto provider (HMAC, CMAC, RSA, ECDSA, ECDH) +* Add support for iOS keychain access +* POWER8 SHA-2 extensions (GH #1486 + #1487) +* Add support VPSUM on big-endian PPC64 (GH #2252) +* Better TPM support: NVRAM, PCR measurements, sealing +* Add support for TPM 2.0 hardware +* Support Intel QuickAssist accelerator cards + +TLS +---------------------------------------- + +* Make DTLS support optional at build time +* Improve/optimize DTLS defragmentation and retransmission +* Implement logging callbacks for TLS +* Make RSA optional at build time +* Make finite field DH optional at build time +* Authentication using TOFU (sqlite3 storage) +* Certificate pinning (using TACK?) +* Certificate Transparency extensions +* TLS supplemental authorization data (RFC 4680, RFC 5878) +* DTLS-SCTP (RFC 6083) + +PKIX +---------------------------------------- + +* Further tests of validation API (see GH #785) +* Test suite for validation of 'real world' cert chains (GH #611) +* Improve output of X509_Certificate::to_string + This is a free-form string for human consumption so the only constraints + are being informative and concise. (GH #656) +* X.509 policy constraints +* OCSP responder logic + +New Protocols / Formats +---------------------------------------- + +* ACME protocol +* PKCS7 / Cryptographic Message Syntax +* PKCS12 / PFX +* Off-The-Record v3 https://otr.cypherpunks.ca/ +* Certificate Management Protocol (RFC 5273); requires CMS +* Fernet symmetric encryption (https://cryptography.io/en/latest/fernet/) +* RNCryptor format (https://github.com/RNCryptor/RNCryptor) +* Useful OpenPGP subset 1: symmetrically encrypted files. + Not aiming to process arbitrary OpenPGP, but rather produce + something that happens to be readable by `gpg` and is relatively + simple to process for decryption. Require AEAD mode (EAX/OCB). +* Useful OpenPGP subset 2: Process OpenPGP public keys +* Useful OpenPGP subset 3: Verification of OpenPGP signatures + +Cleanups +----------- + +* Split test_ffi.cpp into multiple files +* Unicode path support on Windows (GH #1615) +* The X.509 path validation tests have much duplicated logic + +Compat Headers +---------------- + +* OpenSSL compatible API headers: EVP, TLS, certificates, etc + +New C APIs +---------------------------------------- + +* PKCS10 requests +* Certificate signing +* Expose TLS +* Expose NIST key wrap with padding +* Expose secret sharing +* Expose deterministic PRNG +* base32 +* base58 +* DL_Group +* EC_Group + +Python +---------------- + +* Anywhere Pylint warnings too-many-locals, too-many-branches, or + too-many-statements are skipped, fix the code so Pylint no longer warns. + +* Write a CLI or HTTPS client in Python + +Build/Test +---------------------------------------- + +* Start using GitHub Actions for CI, especially Windows builds +* Create Docker image for Travis that runs 18.04 and has all + the tools we need pre-installed. +* Code signing for Windows installers +* Test runner python script that captures backtraces and other + debug info during CI +* Support hardcoding all test vectors into the botan-test binary + so it can run as a standalone item (copied to a device, etc) +* Run iOS binary under simulator in CI +* Run Android binary under simulator in CI +* Run the TPM tests against an emulator + (https://github.com/PeterHuewe/tpm-emulator) +* Add clang-tidy, clang-analyzer, cppcheck to CI +* Add support for vxWorks +* Add support for Fuschia OS +* Add support for CloudABI +* Add support for SGX + +CLI +---------------------------------------- + +* Add a ``--completion`` option to dump autocomplete info, write + support for autocompletion in bash/zsh. +* Refactor ``speed`` +* Change `tls_server` to be a tty<->socket app, like `tls_client` is, + instead of a bogus echo server. +* `encrypt` / `decrypt` tools providing password based file encryption +* Add ECM factoring +* Clone of `minisign` signature utility +* Implementation of `tlsdate` +* Password store utility +* TOTP calculator + +Documentation +---------------------------------------- + +* X.509 certs, path validation +* Specific docs covering one major topic (RSA, ECDSA, AES/GCM, ...) +* Some howto style docs (setting up CA, ...) |