diff options
Diffstat (limited to 'comm/third_party/botan/doc/dev_ref/configure.rst')
| -rw-r--r-- | comm/third_party/botan/doc/dev_ref/configure.rst | 407 |
1 files changed, 407 insertions, 0 deletions
diff --git a/comm/third_party/botan/doc/dev_ref/configure.rst b/comm/third_party/botan/doc/dev_ref/configure.rst new file mode 100644 index 0000000000..7eb8358edf --- /dev/null +++ b/comm/third_party/botan/doc/dev_ref/configure.rst @@ -0,0 +1,407 @@ +Understanding configure.py +============================ + +.. highlight:: none + +Botan's build is handled with a custom Python script, ``configure.py``. +This document tries to explain how configure works. + +.. note:: + You only need to read this if you are modifying the library, + or debugging some problem with your build. For how to use it, + see :ref:`building`. + +Build Structure +-------------------- + +Modules are a group of related source and header files, which can be +individually enabled or disabled at build time. Modules can depend on +other modules; if a dependency is not available then the module itself +is also removed from the list. Examples of modules in the existing +codebase are ``asn1`` and ``x509``, Since ``x509`` depends on (among +other things) ``asn1``, disabling ``asn1`` will also disable ``x509``. + +Most modules define one or more macros, which application code can use +to detect the modules presence or absence. The value of each macro is +a datestamp, in the form YYYYMMDD which indicates the last time this +module changed in a way that would be visible to an application. For +example if a class gains a new function, the datestamp should be +incremented. That allows applications to detect if the new feature is +available. + +What ``configure.py`` does +----------------------------- + +First, all command line options are parsed. + +Then all of the files giving information about target CPUs, compilers, +etc are parsed and sanity checked. + +In ``calculate_cc_min_version`` the compiler version is detected using +the preprocessor. + +Then in ``check_compiler_arch`` the target architecture are detected, again +using the preprocessor. + +Now that the target is identified and options have been parsed, the modules to +include into the artifact are picked, in ``ModulesChooser``. + +In ``create_template_vars``, a dictionary of variables is created which describe +different aspects of the build. These are serialized to +``build/build_config.json``. + +Up until this point no changes have been made on disk. This occurs in +``do_io_for_build``. Build output directories are created, and header files are +linked into ``build/include/botan``. Templates are processed to create the +Makefile, ``build.h`` and other artifacts. + +When Modifying ``configure.py`` +-------------------------------- + +For now, any changes to ``configure.py`` must work under both CPython 2.7 and +CPython 3.x. In a future major release, support for CPython2 will be dropped, +but until then if making modifications verify the code works as expected on +both versions. + +Run ``./src/scripts/ci_build.py lint`` to run Pylint checks after any change. + +Template Language +-------------------- + +Various output files are generated by processing input files using a simple +template language. All input files are stored in ``src/build-data`` and use the +suffix ``.in``. Anything not recognized as a template command is passed through +to the output unmodified. The template elements are: + + * Variable substitution, ``%{variable_name}``. The configure script creates + many variables for various purposes, this allows getting their value within + the output. If a variable is not defined, an error occurs. + + If a variable reference ends with ``|upper``, the value is uppercased before + being inserted into the template output. + + * Iteration, ``%{for variable} block %{endfor}``. This iterates over a list and + repeats the block as many times as it is included. Variables within the block + are expanded. The two template elements ``%{for ...}`` and ``%{endfor}`` must + appear on lines with no text before or after. + + * Conditional inclusion, ``%{if variable} block %{endif}``. If the variable + named is defined and true (in the Python sense of the word; if the variable + is empty or zero it is considered false), then the block will be included and + any variables expanded. As with the for loop syntax, both the start and end + of the conditional must be on their own lines with no additional text. + +Adding a new module +-------------------- + +Create a directory in the appropriate place and create a info.txt file. + +Syntax of ``info.txt`` +------------------------ + +.. warning:: + + The syntax described here is documented to make it easier to use + and understand, but it is not considered part of the public API + contract. That is, the developers are allowed to change the syntax + at any time on the assumption that all users are contained within + the library itself. If that happens this document will be updated. + +Modules and files describing information about the system use the same +parser and have common syntactical elements. + +Comments begin with '#' and continue to end of line. + +There are three main types: maps, lists, and variables. + +A map has a syntax like:: + + <MAP_NAME> + NAME1 -> VALUE1 + NAME2 -> VALUE2 + ... + </MAP_NAME> + +The interpretation of the names and values will depend on the map's name +and what type of file is being parsed. + +A list has similar syntax, it just doesn't have values:: + + <LIST_NAME> + ELEM1 + ELEM2 + ... + </LIST_NAME> + +Lastly there are single value variables like:: + + VAR1 SomeValue + VAR2 "Quotes Can Be Used (And will be stripped out)" + VAR3 42 + +Variables can have string, integer or boolean values. Boolean values +are specified with 'yes' or 'no'. + +Module Syntax +--------------------- + +The ``info.txt`` files have the following elements. Not all are required; a minimal +file for a module with no dependencies might just contain a macro define. + +Lists: + * ``comment`` and ``warning`` provides block-comments which + are displayed to the user at build time. + * ``requires`` is a list of module dependencies. An ``os_features`` can be + specified as a condition for needing the dependency by writing it before + the module name and separated by a ``?``, e.g. ``rtlgenrandom?dyn_load``. + * ``header:internal`` is the list of headers (from the current module) + which are internal-only. + * ``header:public`` is a the list of headers (from the + current module) which should be exported for public use. If neither + ``header:internal`` nor ``header:public`` are used then all headers + in the current directory are assumed public. + + .. note:: If you omit a header from both internal and public lists, it will + be ignored. + + * ``header:external`` is used when naming headers which are included + in the source tree but might be replaced by an external version. This is used + for the PKCS11 headers. + * ``arch`` is a list of architectures this module may be used on. + * ``isa`` lists ISA features which must be enabled to use this module. + Can be proceeded by an ``arch`` name followed by a ``:`` if it is only needed + on a specific architecture, e.g. ``x86_64:ssse3``. + * ``cc`` is a list of compilers which can be used with this module. If the + compiler name is suffixed with a version (like "gcc:5.0") then only compilers + with that minimum version can use the module. + * ``os_features`` is a list of OS features which are required in order to use this + module. Each line can specify one or more features combined with ','. Alternatives + can be specified on additional lines. + +Maps: + * ``defines`` is a map from macros to datestamps. These macros will be defined in + the generated ``build.h``. + * ``libs`` specifies additional libraries which should be linked if this module is + included. It maps from the OS name to a list of libraries (comma seperated). + * ``frameworks`` is a macOS/iOS specific feature which maps from an OS name to + a framework. + +Variables: + * ``load_on`` Can take on values ``never``, ``always``, ``auto``, ``dep`` or ``vendor``. + TODO describe the behavior of these + * ``endian`` Required endian for the module (``any`` (default), ``little``, ``big``) + +An example:: + + # Disable this by default + load_on never + + <isa> + sse2 + </isa> + + <defines> + DEFINE1 -> 20180104 + DEFINE2 -> 20190301 + </defines> + + <comment> + I have eaten + the plums + that were in + the icebox + </comment> + + <warning> + There are no more plums + </warning> + + <header:public> + header1.h + </header:public> + + <header:internal> + header_helper.h + whatever.h + </header:internal> + + <arch> + x86_64 + </arch> + + <cc> + gcc:4.9 # gcc 4.8 doesn't work for <reasons> + clang + </cc> + + # Can work with POSIX+getentropy or Win32 + <os_features> + posix1,getentropy + win32 + </os_features> + + <frameworks> + macos -> FramyMcFramerson + </frameworks> + + <libs> + qnx -> foo,bar,baz + solaris -> socket + </libs> + +Supporting a new CPU type +--------------------------- + +CPU information is stored in ``src/build-data/arch``. + +There is also a file ``src/build-data/detect_arch.cpp`` which is used +for build-time architecture detection using the compiler preprocessor. +Supporting this is optional but recommended. + +Lists: + * ``aliases`` is a list of alternative names for the CPU architecture. + * ``isa_extensions`` is a list of possible ISA extensions that can be used on + this architecture. For example x86-64 has extensions "sse2", "ssse3", + "avx2", "aesni", ... + +Variables: + * ``endian`` if defined should be "little" or "big". This can also be + controlled or overridden at build time. + * ``family`` can specify a family group for several related architecture. + For example both x86_32 and x86_64 use ``family`` of "x86". + * ``wordsize`` is the default wordsize, which controls the size of limbs + in the multi precision integers. If not set, defaults to 32. + +Supporting a new compiler +--------------------------- + +Compiler information is stored in ``src/build-data/cc``. Looking over +those files will probably help understanding, especially the ones for +GCC and Clang which are most complete. + +In addition to the info file, for compilers there is a file +``src/build-data/detect_version.cpp``. The ``configure.py`` script runs the +preprocessor over this file to attempt to detect the compiler +version. Supporting this is not strictly necessary. + +Maps: + * ``binary_link_commands`` gives the command to use to run the linker, + it maps from operating system name to the command to use. It uses + the entry "default" for any OS not otherwise listed. + * ``cpu_flags_no_debug`` unused, will be removed + * ``cpu_flags`` used to emit CPU specific flags, for example LLVM + bitcode target uses ``-emit-llvm`` flag. Rarely needed. + * ``isa_flags`` maps from CPU extensions (like NEON or AES-NI) to + compiler flags which enable that extension. These have the same name + as the ISA flags listed in the architecture files. + * ``lib_flags`` has a single possible entry "debug" which if set maps + to additional flags to pass when building a debug library. + Rarely needed. + * ``mach_abi_linking`` specifies flags to enable when building and + linking on a particular CPU. This is usually flags that modify + ABI. There is a special syntax supported here + "all!os1,arch1,os2,arch2" which allows setting ABI flags which are + used for all but the named operating systems and/or architectures. + * ``sanitizers`` is a map of sanitizers the compiler supports. It must + include "default" which is a list of sanitizers to include by default + when sanitizers are requested. The other keys should map to compiler + flags. + * ``so_link_commands`` maps from operating system to the command to + use to build a shared object. + +Variables: + * ``binary_name`` the default name of the compiler binary. + * ``linker_name`` the name of the linker to use with this compiler. + * ``macro_name`` a macro of the for ``BOTAN_BUILD_COMPILER_IS_XXX`` + will be defined. + * ``output_to_object`` (default "-o") gives the compiler option used to + name the output object. + * ``output_to_exe`` (default "-o") gives the compiler option used to + name the output object. + * ``add_include_dir_option`` (default "-I") gives the compiler option used + to specify an additional include dir. + * ``add_lib_dir_option`` (default "-L") gives the compiler option used + to specify an additional library dir. + * ``add_sysroot_option`` gives the compiler option used to specify the sysroot. + * ``add_lib_option`` (default "-l%s") gives the compiler option to + link in a library. ``%s`` will be replaced with the library name. + * ``add_framework_option`` (default "-framework") gives the compiler option + to add a macOS framework. + * ``preproc_flags`` (default "-E") gives the compiler option used to run + the preprocessor. + * ``compile_flags`` (default "-c") gives the compiler option used to compile a file. + * ``debug_info_flags`` (default "-g") gives the compiler option used to enable debug info. + * ``optimization_flags`` gives the compiler optimization flags to use. + * ``size_optimization_flags`` gives compiler optimization flags to use when + compiling for size. If not set then ``--optimize-for-size`` will use + the default optimization flags. + * ``sanitizer_optimization_flags`` gives compiler optimization flags to use + when building with sanitizers. + * ``coverage_flags`` gives the compiler flags to use when generating coverage + information. + * ``stack_protector_flags`` gives compiler flags to enable stack overflow checking. + * ``shared_flags`` gives compiler flags to use when generation shared libraries. + * ``lang_flags`` gives compiler flags used to enable the required version of C++. + * ``warning_flags`` gives warning flags to enable. + * ``maintainer_warning_flags`` gives extra warning flags to enable during maintainer + mode builds. + * ``visibility_build_flags`` gives compiler flags to control symbol visibility + when generation shared libraries. + * ``visibility_attribute`` gives the attribute to use in the ``BOTAN_DLL`` macro + to specify visibility when generation shared libraries. + * ``ar_command`` gives the command to build static libraries + * ``ar_options`` gives the options to pass to ``ar_command``, if not set here + takes this from the OS specific information. + * ``ar_output_to`` gives the flag to pass to ``ar_command`` to specify where to + output the static library. + * ``werror_flags`` gives the complier flags to treat warnings as errors. + +Supporting a new OS +--------------------------- + +Operating system information is stored in ``src/build-data/os``. + +Lists: + * ``aliases`` is a list of alternative names which will be accepted + * ``target_features`` is a list of target specific OS features. Some of these + are supported by many OSes (for example "posix1") others are specific to + just one or two OSes (such as "getauxval"). Adding a value here causes a new + macro ``BOTAN_TARGET_OS_HAS_XXX`` to be defined at build time. Use + ``configure.py --list-os-features`` to list the currently defined OS + features. + * ``feature_macros`` is a list of macros to define. + +Variables: + * ``ar_command`` gives the command to build static libraries + * ``ar_options`` gives the options to pass to ``ar_command`` + * ``ar_output_to`` gives the flag to pass to ``ar_command`` to specify where to + output the static library. + * ``bin_dir`` (default "bin") specifies where binaries should be installed, + relative to install_root. + * ``cli_exe_name`` (default "botan") specifies the name of the command line utility. + * ``default_compiler`` specifies the default compiler to use for this OS. + * ``doc_dir`` (default "doc") specifies where documentation should be installed, + relative to install_root + * ``header_dir`` (default "include") specifies where include files + should be installed, relative to install_root + * ``install_root`` (default "/usr/local") specifies where to install + by default. + * ``lib_dir`` (default "lib") specifies where library should be installed, + relative to install_root. + * ``lib_prefix`` (default "lib") prefix to add to the library name + * ``library_name`` + * ``man_dir`` specifies where man files should be installed, relative to install_root + * ``obj_suffix`` (default "o") specifies the suffix used for object files + * ``program_suffix`` (default "") specifies the suffix used for executables + * ``shared_lib_symlinks`` (default "yes) specifies if symbolic names should be + created from the base and patch soname to the library name. + * ``soname_pattern_abi`` + * ``soname_pattern_base`` + * ``soname_pattern_patch`` + * ``soname_suffix`` file extension to use for shared library if ``soname_pattern_base`` + is not specified. + * ``static_suffix`` (default "a") file extension to use for static library. + * ``use_stack_protector`` (default "true") specify if by default stack smashing + protections should be enabled. + * ``uses_pkg_config`` (default "yes") specify if by default a pkg-config file + should be created. |