diff options
Diffstat (limited to 'src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc')
38 files changed, 3126 insertions, 0 deletions
diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wamr.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wamr.md new file mode 100644 index 000000000..a66f27be0 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wamr.md @@ -0,0 +1,213 @@ + +# Build WAMR vmcore + +WAMR vmcore is a set of runtime libraries for loading and running Wasm modules. This document introduces how to build the WAMR vmcore. + +References: +- [how to build iwasm](../product-mini/README.md): building different target platforms such as Linux, Windows, Mac etc +- [Blog: Introduction to WAMR running modes](https://bytecodealliance.github.io/wamr.dev/blog/introduction-to-wamr-running-modes/) + + +## WAMR vmcore cmake building configurations + +By including the script `runtime_lib.cmake` under folder [build-scripts](../build-scripts) in CMakeList.txt, it is easy to use vmcore to build host software with cmake. + +```cmake +# add this into your CMakeList.txt +include (${WAMR_ROOT_DIR}/build-scripts/runtime_lib.cmake) +add_library(vmlib ${WAMR_RUNTIME_LIB_SOURCE}) +``` + +The script `runtime_lib.cmake` defines a number of variables for configuring the WAMR runtime features. You can set these variables in your CMakeList.txt or pass the configurations from cmake command line. + +#### **Configure platform and architecture** + +- **WAMR_BUILD_PLATFORM**: set the target platform. It can be set to any platform name (folder name) under folder [core/shared/platform](../core/shared/platform). + +- **WAMR_BUILD_TARGET**: set the target CPU architecture. Current supported targets are: X86_64, X86_32, AARCH64, ARM, THUMB, XTENSA, ARC, RISCV32, RISCV64 and MIPS. + - For ARM and THUMB, the format is \<arch>\[\<sub-arch>]\[_VFP], where \<sub-arch> is the ARM sub-architecture and the "_VFP" suffix means using VFP coprocessor registers s0-s15 (d0-d7) for passing arguments or returning results in standard procedure-call. Both \<sub-arch> and "_VFP" are optional, e.g. ARMV7, ARMV7_VFP, THUMBV7, THUMBV7_VFP and so on. + - For AARCH64, the format is\<arch>[\<sub-arch>], VFP is enabled by default. \<sub-arch> is optional, e.g. AARCH64, AARCH64V8, AARCH64V8.1 and so on. + - For RISCV64, the format is \<arch\>[_abi], where "_abi" is optional, currently the supported formats are RISCV64, RISCV64_LP64D and RISCV64_LP64: RISCV64 and RISCV64_LP64D are identical, using [LP64D](https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#-named-abis) as abi (LP64 with hardware floating-point calling convention for FLEN=64). And RISCV64_LP64 uses [LP64](https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#-named-abis) as abi (Integer calling-convention only, and hardware floating-point calling convention is not used). + - For RISCV32, the format is \<arch\>[_abi], where "_abi" is optional, currently the supported formats are RISCV32, RISCV32_ILP32D and RISCV32_ILP32: RISCV32 and RISCV32_ILP32D are identical, using [ILP32D](https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#-named-abis) as abi (ILP32 with hardware floating-point calling convention for FLEN=64). And RISCV32_ILP32 uses [ILP32](https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#-named-abis) as abi (Integer calling-convention only, and hardware floating-point calling convention is not used). + +```bash +cmake -DWAMR_BUILD_PLATFORM=linux -DWAMR_BUILD_TARGET=ARM +``` + +#### **Configure interpreters** + +- **WAMR_BUILD_INTERP**=1/0: enable or disable WASM interpreter + +- **WAMR_BUILD_FAST_INTERP**=1/0: build fast (default) or classic WASM interpreter. + + NOTE: the fast interpreter runs ~2X faster than classic interpreter, but consumes about 2X memory to hold the pre-compiled code. + +#### **Configure AOT and JITs** + +- **WAMR_BUILD_AOT**=1/0, enable AOT or not, default to enable if not set +- **WAMR_BUILD_JIT**=1/0, enable LLVM JIT or not, default to disable if not set +- **WAMR_BUILD_FAST_JIT**=1/0, enable Fast JIT or not, default to disable if not set +- **WAMR_BUILD_FAST_JIT**=1 and **WAMR_BUILD_JIT**=1, enable Multi-tier JIT, default to disable if not set + +#### **Configure LIBC** + +- **WAMR_BUILD_LIBC_BUILTIN**=1/0, build the built-in libc subset for WASM app, default to enable if not set + +- **WAMR_BUILD_LIBC_WASI**=1/0, build the [WASI](https://github.com/WebAssembly/WASI) libc subset for WASM app, default to enable if not set + +- **WAMR_BUILD_LIBC_UVWASI**=1/0 (Experiment), build the [WASI](https://github.com/WebAssembly/WASI) libc subset for WASM app based on [uvwasi](https://github.com/nodejs/uvwasi) implementation, default to disable if not set + +> Note: for platform which doesn't support **WAMR_BUILD_LIBC_WASI**, e.g. Windows, developer can try using **WAMR_BUILD_LIBC_UVWASI**. + +#### **Enable Multi-Module feature** + +- **WAMR_BUILD_MULTI_MODULE**=1/0, default to disable if not set + +#### **Enable WASM mini loader** + +- **WAMR_BUILD_MINI_LOADER**=1/0, default to disable if not set + +> Note: the mini loader doesn't check the integrity of the WASM binary file, developer must ensure that the WASM file is well-formed. + +#### **Enable shared memory feature** +- **WAMR_BUILD_SHARED_MEMORY**=1/0, default to disable if not set + +#### **Enable bulk memory feature** +- **WAMR_BUILD_BULK_MEMORY**=1/0, default to disable if not set + +#### **Enable thread manager** +- **WAMR_BUILD_THREAD_MGR**=1/0, default to disable if not set + +#### **Enable lib-pthread** +- **WAMR_BUILD_LIB_PTHREAD**=1/0, default to disable if not set +> Note: The dependent feature of lib pthread such as the `shared memory` and `thread manager` will be enabled automatically. + +#### **Enable lib-pthread-semaphore** +- **WAMR_BUILD_LIB_PTHREAD_SEMAPHORE**=1/0, default to disable if not set +> Note: This feature depends on `lib-pthread`, it will be enabled automatically if this feature is enabled. + +#### **Enable lib wasi-threads** +- **WAMR_BUILD_LIB_WASI_THREADS**=1/0, default to disable if not set +> Note: The dependent feature of lib wasi-threads such as the `shared memory` and `thread manager` will be enabled automatically. + +#### **Enable lib wasi-nn** +- **WAMR_BUILD_WASI_NN**=1/0, default to disable if not set + +#### **Enable lib wasi-nn GPU mode** +- **WASI_NN_ENABLE_GPU**=1/0, default to disable if not set + +#### **Disable boundary check with hardware trap** +- **WAMR_DISABLE_HW_BOUND_CHECK**=1/0, default to enable if not set and supported by platform +> Note: by default only platform linux/darwin/android/windows/vxworks 64-bit will enable the boundary check with hardware trap feature, and the wamrc tool will generate AOT code without boundary check instructions in all 64-bit targets except SGX to improve performance. The boundary check includes linear memory access boundary and native stack access boundary, if `WAMR_DISABLE_STACK_HW_BOUND_CHECK` below isn't set. + +#### **Disable native stack boundary check with hardware trap** +- **WAMR_DISABLE_STACK_HW_BOUND_CHECK**=1/0, default to enable if not set and supported by platform, same as `WAMR_DISABLE_HW_BOUND_CHECK`. +> Note: When boundary check with hardware trap is disabled, or `WAMR_DISABLE_HW_BOUND_CHECK` is set to 1, the native stack boundary check with hardware trap will be disabled too, no matter what value is set to `WAMR_DISABLE_STACK_HW_BOUND_CHECK`. And when boundary check with hardware trap is enabled, the status of this feature is set according to the value of `WAMR_DISABLE_STACK_HW_BOUND_CHECK`. + +#### **Enable tail call feature** +- **WAMR_BUILD_TAIL_CALL**=1/0, default to disable if not set + +#### **Enable 128-bit SIMD feature** +- **WAMR_BUILD_SIMD**=1/0, default to enable if not set +> Note: only supported in AOT mode x86-64 target. + +#### **Configure Debug** + +- **WAMR_BUILD_CUSTOM_NAME_SECTION**=1/0, load the function name from custom name section, default to disable if not set + +#### **Enable dump call stack feature** +- **WAMR_BUILD_DUMP_CALL_STACK**=1/0, default to disable if not set + +> Note: if it is enabled, the call stack will be dumped when exception occurs. + +> - For interpreter mode, the function names are firstly extracted from *custom name section*, if this section doesn't exist or the feature is not enabled, then the name will be extracted from the import/export sections +> - For AOT/JIT mode, the function names are extracted from import/export section, please export as many functions as possible (for `wasi-sdk` you can use `-Wl,--export-all`) when compiling wasm module, and add `--enable-dump-call-stack` option to wamrc during compiling AOT module. + +#### **Enable memory profiling (Experiment)** +- **WAMR_BUILD_MEMORY_PROFILING**=1/0, default to disable if not set +> Note: if it is enabled, developer can use API `void wasm_runtime_dump_mem_consumption(wasm_exec_env_t exec_env)` to dump the memory consumption info. +Currently we only profile the memory consumption of module, module_instance and exec_env, the memory consumed by other components such as `wasi-ctx`, `multi-module` and `thread-manager` are not included. + +#### **Enable performance profiling (Experiment)** +- **WAMR_BUILD_PERF_PROFILING**=1/0, default to disable if not set +> Note: if it is enabled, developer can use API `void wasm_runtime_dump_perf_profiling(wasm_module_inst_t module_inst)` to dump the performance consumption info. Currently we only profile the performance consumption of each WASM function. + +> The function name searching sequence is the same with dump call stack feature. + +#### **Enable the global heap** +- **WAMR_BUILD_GLOBAL_HEAP_POOL**=1/0, default to disable if not set for all *iwasm* applications, except for the platforms Alios and Zephyr. + +> **WAMR_BUILD_GLOBAL_HEAP_POOL** is used in the *iwasm* applications provided in the directory `product-mini`. When writing your own host application using WAMR, if you want to use a global heap and allocate memory from it, you must set the initialization argument `mem_alloc_type` to `Alloc_With_Pool`. +> The global heap is defined in the documentation [Memory model and memory usage tunning](memory_tune.md). + +#### **Set the global heap size** +- **WAMR_BUILD_GLOBAL_HEAP_SIZE**=n, default to 10 MB (10485760) if not set for all *iwasm* applications, except for the platforms Alios (256 kB), Riot (256 kB) and Zephyr (128 kB). + +> **WAMR_BUILD_GLOBAL_HEAP_SIZE** is used in the *iwasm* applications provided in the directory `product-mini`. When writing your own host application using WAMR, if you want to set the amount of memory dedicated to the global heap pool, you must set the initialization argument `mem_alloc_option.pool` with the appropriate values. +> The global heap is defined in the documentation [Memory model and memory usage tunning](memory_tune.md). +> Note: if `WAMR_BUILD_GLOBAL_HEAP_SIZE` is not set and the flag `WAMR_BUILD_SPEC_TEST` is set, the global heap size is equal to 300 MB (314572800), or 100 MB (104857600) when compiled for Intel SGX (Linux). + +#### **Set maximum app thread stack size** +- **WAMR_APP_THREAD_STACK_SIZE_MAX**=n, default to 8 MB (8388608) if not set +> Note: the AOT boundary check with hardware trap mechanism might consume large stack since the OS may lazily grow the stack mapping as a guard page is hit, we may use this configuration to reduce the total stack usage, e.g. -DWAMR_APP_THREAD_STACK_SIZE_MAX=131072 (128 KB). + +#### **WAMR_BH_VPRINTF**=<vprintf_callback>, default to disable if not set +> Note: if the vprintf_callback function is provided by developer, the os_printf() and os_vprintf() in Linux, Darwin, Windows and VxWorks platforms, besides WASI Libc output will call the callback function instead of libc vprintf() function to redirect the stdout output. For example, developer can define the callback function like below outside runtime lib: +> +> ```C +> int my_vprintf(const char *format, va_list ap) +> { +> /* output to pre-opened file stream */ +> FILE *my_file = ...; +> return vfprintf(my_file, format, ap); +> /* or output to pre-opened file descriptor */ +> int my_fd = ...; +> return vdprintf(my_fd, format, ap); +> /* or output to string buffer and print the string */ +> char buf[128]; +> vsnprintf(buf, sizeof(buf), format, ap); +> return my_printf("%s", buf); +> } +> ``` +> +> and then use `cmake -DWAMR_BH_VPRINTF=my_vprintf ..` to pass the callback function, or add `BH_VPRINTF=my_vprintf` macro for the compiler, e.g. add line `add_defintions(-DBH_VPRINTF=my_vprintf)` in CMakeListst.txt. + +#### **Enable reference types feature** +- **WAMR_BUILD_REF_TYPES**=1/0, default to disable if not set + +#### **Exclude WAMR application entry functions** +- **WAMR_DISABLE_APP_ENTRY**=1/0, default to disable if not set + +> Note: The WAMR application entry (`core/iwasm/common/wasm_application.c`) encapsulate some common process to instantiate, execute the wasm functions and print the results. Some platform related APIs are used in these functions, so you can enable this flag to exclude this file if your platform doesn't support those APIs. +> *Don't enable this flag if you are building `product-mini`* + +#### **Enable source debugging features** +- **WAMR_BUILD_DEBUG_INTERP**=1/0, default to 0 if not set +> Note: There are some other setup required by source debugging, please refer to [source_debugging.md](./source_debugging.md) for more details. + +#### **Enable load wasm custom sections** +- **WAMR_BUILD_LOAD_CUSTOM_SECTION**=1/0, default to disable if not set + +> Note: By default, the custom sections are ignored. If the embedder wants to get custom sections from `wasm_module_t`, then `WAMR_BUILD_LOAD_CUSTOM_SECTION` should be enabled, and then `wasm_runtime_get_custom_section` can be used to get a custom section by name. + +> Note: If `WAMR_BUILD_CUSTOM_NAME_SECTION` is enabled, then the `custom name section` will be treated as a special section and consumed by the runtime, not available to the embedder. + +> For AoT file, must use `--emit-custom-sections` to specify which sections need to be emitted into AoT file, otherwise all custom sections (except custom name section) will be ignored. + +### **Stack guard size** +- **WAMR_BUILD_STACK_GUARD_SIZE**=n, default to N/A if not set. +> Note: By default, the stack guard size is 1K (1024) or 24K (if uvwasi enabled). + +**Combination of configurations:** + +We can combine the configurations. For example, if we want to disable interpreter, enable AOT and WASI, we can run command: + +``` Bash +cmake .. -DWAMR_BUILD_INTERP=0 -DWAMR_BUILD_AOT=1 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_PLATFORM=linux +``` + +Or if we want to enable interpreter, disable AOT and WASI, and build as X86_32, we can run command: + +``` Bash +cmake .. -DWAMR_BUILD_INTERP=1 -DWAMR_BUILD_AOT=0 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_TARGET=X86_32 +``` diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wasm_app.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wasm_app.md new file mode 100644 index 000000000..40f1b89dd --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/build_wasm_app.md @@ -0,0 +1,423 @@ +# Build WASM applications + +Prepare WASM building environments +================================== + +For C and C++, WASI-SDK version 19.0+ is the major tool supported by WAMR to build WASM applications. Also, we can use [Emscripten SDK (EMSDK)](https://github.com/emscripten-core/emsdk), but it is not recommended. And there are some other compilers such as the standard clang compiler, which might also work [here](./other_wasm_compilers.md). + +To install WASI SDK, please download the [wasi-sdk release](https://github.com/CraneStation/wasi-sdk/releases) and extract the archive to default path `/opt/wasi-sdk`. + +The official *wasi-sdk release* doesn't fully support *latest 128-bit SIMD spec* yet. WAMR provides a script in [build-wasi-sdk](../test-tools/build-wasi-sdk/) to generate +another wasi-sdk with *llvm-15* from source code and installs it at *../test-tools/wasi-sdk*. If you plan to build WASM applications with *latest 128-bit SIMD*, please use it instead of the official release. + +And [sample workloads](../samples/workload) are using the self-compiled wasi-sdk. + +For [AssemblyScript](https://github.com/AssemblyScript/assemblyscript), please refer to [AssemblyScript quick start](https://www.assemblyscript.org/quick-start.html) and [AssemblyScript compiler](https://www.assemblyscript.org/compiler.html#command-line-options) for how to install `asc` compiler and build WASM applications. + +For Rust, please refer to [Install Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) to install *cargo*, *rustc* and *rustup*. By default they are under ~/.cargo/bin. + +And then run such a command to install `wasm32-wasi` target. + +``` bash +$ rustup target add wasm32-wasi +``` + +To build WASM applications, run + +``` bash +$ cargo build --target wasm32-wasi +``` + +The output files are under `target/wasm32-wasi`. + +To build a release version + +``` bash +$ cargo build --release --target wasm32-wasi +``` + + +Build WASM applications with wasi-sdk +===================================== + +You can write a simple ```test.c``` as the first sample. + +``` C +#include <stdio.h> +#include <stdlib.h> + +int main(int argc, char **argv) +{ + char *buf; + + printf("Hello world!\n"); + + buf = malloc(1024); + if (!buf) { + printf("malloc buf failed\n"); + return -1; + } + + printf("buf ptr: %p\n", buf); + + sprintf(buf, "%s", "1234\n"); + printf("buf: %s", buf); + + free(buf); + return 0; +} +``` + +To build the source file to WASM bytecode, we can input the following command: + +``` Bash +/opt/wasi-sdk/bin/clang -O3 -o test.wasm test.c +``` + +## 1. wasi-sdk options + +There are some useful options that are used to compile C/C++ to Wasm (for a full introduction, please refer to [clang command line argument reference](https://clang.llvm.org/docs/ClangCommandLineReference.html) and [wasm-ld command line argument manual](https://lld.llvm.org/WebAssembly.html)): + +- **-nostdlib** Do not use the standard system startup files or libraries when linking. In this mode, the **libc-builtin** library of WAMR must be built to run the wasm app, otherwise, the **libc-wasi** library must be built. You can specify **-DWAMR_BUILD_LIBC_BUILTIN=1** or **-DWAMR_BUILD_LIBC_WASI=1** for CMake to build WAMR with libc-builtin support or libc-wasi support. + +- **-Wl,--no-entry** Do not output any entry point + +- **-Wl,--export=\<value\>** Force a symbol to be exported, e.g. **-Wl,--export=foo** to export foo function + +- **-Wl,--export-all** Export all symbols (normally combined with --no-gc-sections) + +- **-Wl,--initial-memory=\<value\>** Initial size of the linear memory, which must be a multiple of 65536 + +- **-Wl,--max-memory=\<value\>** Maximum size of the linear memory, which must be a multiple of 65536 + +- **-z stack-size=\<vlaue\>** The auxiliary stack size, which is an area of linear memory, must be smaller than the initial memory size. + +- **-Wl,--strip-all** Strip all symbols + +- **-Wl,--shared-memory** Use shared linear memory + +- **-Wl,--allow-undefined** Allow undefined symbols in linked binary + +- **-Wl,--allow-undefined-file=\<value\>** Allow symbols listed in \<file\> to be undefined in linked binary + +- **-pthread** Support POSIX threads in generated code + +For example, we can build the wasm app with the command: + +``` Bash +/opt/wasi-sdk/bin/clang -O3 -nostdlib \ + -z stack-size=8192 -Wl,--initial-memory=65536 \ + -o test.wasm test.c \ + -Wl,--export=main -Wl,--export=__main_argc_argv \ + -Wl,--export=__heap_base -Wl,--export=__data_end \ + -Wl,--no-entry -Wl,--strip-all -Wl,--allow-undefined +``` +to generate a wasm binary with nostdlib mode, the auxiliary stack size is 8192 bytes, initial memory size is 64 KB, main function, heap base global and data end global are exported, no entry function is generated (no _start function is exported), and all symbols are stripped. Note that it is nostdlib mode, so libc-builtin should be enabled by runtime embedder or iwasm (with `cmake -DWAMR_BUILD_LIBC_BUILT=1`, enabled by iwasm in Linux by default). + +If we want to build the wasm app with wasi mode, we may build the wasm app with the command: + +```bash +/opt/wasi-sdk/bin/clang -O3 \ + -z stack-size=8192 -Wl,--initial-memory=65536 \ + -o test.wasm test.c \ + -Wl,--export=__heap_base -Wl,--export=__data_end \ + -Wl,--strip-all +``` + +to generate a wasm binary with wasi mode, the auxiliary stack size is 8192 bytes, initial memory size is 64 KB, heap base global and data end global are exported, wasi entry function exported (_start function), and all symbols are stripped. Note that it is wasi mode, so libc-wasi should be enabled by runtime embedder or iwasm (with `cmake -DWAMR_BUILD_LIBC_WASI=1`, enabled by iwasm in Linux by default), and normally no need to export main function, by default _start function is executed by iwasm. + +> Note: for the Rust project, we can set these flags by setting the `rustflags` in the Cargo configuration file, e.g. `<project_dir>/.cargo/config.toml` or `$CARGO_HOME/config.toml`, for example: +> ```toml +> [build] +> rustflags = [ +> "-C", "link-arg=--initial-memory=65536", +> "-C", "link-arg=-zstack-size=8192", +> "-C", "link-arg=--export=__heap_base", +> "-C", "link-arg=--export=__data_end", +> "-C", "link-arg=--strip-all", +> ] +> ``` + +## 2. How to reduce the footprint? + +Firstly if libc-builtin (-nostdlib) mode meets the requirements, e.g. there are no file io operations in wasm app, we should build the wasm app with -nostdlib option as possible as we can, since the compiler doesn't build the libc source code into wasm bytecodes, which greatly reduces the binary size. + +### (1) Methods to reduce the libc-builtin (-nostdlib) mode footprint + +- export \_\_heap_base global and \_\_data_end global + ```bash + -Wl,--export=__heap_base -Wl,--export=__data_end + ``` + If the two globals are exported, and there are no memory.grow and memory.size opcodes (normally nostdlib mode doesn't introduce these opcodes since the libc malloc function isn't built into wasm bytecode), WAMR runtime will truncate the linear memory at the place of \__heap_base and append app heap to the end, so we don't need to allocate the memory specified by `-Wl,--initial-memory=n` which must be at least 64 KB. This is helpful for some embedded devices whose memory resource might be limited. + +> For the Rust project, please set the flags in the Cargo configuration file, for example: +> ```toml +> [build] +> rustflags = [ +> "-C", "link-arg=--export=__heap_base", +> "-C", "link-arg=--export=__data_end", +> "-C", "link-arg=--initial-memory=65536", +> ] +> ``` + +- reduce auxiliary stack size + + The auxiliary stack is an area of linear memory, normally the size is 64 KB by default which might be a little large for embedded devices and partly used, we can use `-z stack-size=n` to set its size. + +> For the Rust project, please set the flag in the Cargo configuration file, for example: +> ```toml +> [build] +> rustflags = [ +> "-C", "link-arg=-zstack-size=8192" +> ] +> ``` + +- use -O3 and -Wl,--strip-all + +> For the Rust project, please set the flag in the Cargo configuration file, for example: +> ```toml +> [build] +> rustflags = [ +> "-C", "link-arg=--strip-all" +> ] +> ``` + +- reduce app heap size when running iwasm + + We can pass `--heap-size=n` option to set the maximum app heap size for iwasm, by default it is 16 KB. For the runtime embedder, we can set the `uint32_t heap_size` argument when calling API ` wasm_runtime_instantiate`. + +- reduce wasm operand stack size when running iwasm + + WebAssembly is a binary instruction format for a stack-based virtual machine, which requires a stack to execute the bytecodes. We can pass `--stack-size=n` option to set the maximum stack size for iwasm, by default it is 16 KB. For the runtime embedder, we can set the `uint32_t stack_size` argument when calling API ` wasm_runtime_instantiate` and `wasm_runtime_create_exec_env`. + +- decrease block_addr_cache size for classic interpreter + + The block_addr_cache is a hash cache to store the else/end addresses for WebAssembly blocks (BLOCK/IF/LOOP) to speed up address lookup. This is only available in the classic interpreter. We can set it by defineing macro `-DBLOCK_ADDR_CACHE_SIZE=n`, e.g. add `add_defintion (-DBLOCK_ADDR_CACHE_SIZE=n)` in CMakeLists.txt, by default it is 64, and the total block_addr_cache size is 3072 bytes in 64-bit platform and 1536 bytes in 32-bit platform. + +### (2) Methods to reduce the libc-wasi (without -nostdlib) mode footprint + +Most of the above methods are also available for libc-wasi mode, besides them, we can export malloc and free functions with `-Wl,--export=malloc -Wl,--export=free` option, so WAMR runtime will disable its app heap and call the malloc/free function exported to allocate/free the memory from/to the heap space managed by libc. + +Note: wasm-ld from LLVM 13 and later automatically inserts ctor/dtor calls +for all exported functions for a command. (vs reactor) +It breaks the malloc/free exports mentioned above. + +## 3. Build wasm app with pthread support + +Please ref to [pthread library](./pthread_library.md) for more details. + +## 4. Build wasm app with SIMD support + +The official *wasi-sdk release* doesn't fully support *latest 128-bit SIMD spec* yet. WARM provides a script in [build-wasi-sdk](../test-tools/build-wasi-sdk/) to generate +another wasi-sdk with *llvm-13* from source code and installs it at *../test-tools/wasi-sdk*. If you plan to build WASM applications with *latest 128-bit SIMD*, please use it instead of the official release. + +And also you can install emsdk and use its SSE header files, please ref to workload samples, e.g. [bwa CMakeLists.txt](../samples/workload/bwa/CMakeLists.txt) and [wasm-av1 CMakeLists.txt](../samples/workload/wasm-av1/CMakeLists.txt) for more details. + +For both wasi-sdk and emsdk, please add the option `-msimd128` for clang or emcc to generate WASM application with SIMD bytecodes. + +# Build WASM applications with emsdk + +## 1. Install emsdk + +Assuming you are using Linux, you may install emcc and em++ from Emscripten EMSDK following the steps below: + +``` +git clone https://github.com/emscripten-core/emsdk.git +cd emsdk +./emsdk install latest +./emsdk activate latest +# And then source the emsdk_env.sh script before build wasm app +source emsdk_env.sh (or add it to ~/.bashrc if you don't want to run it each time) +``` + +The Emscripten website provides other installation methods beyond Linux. + +## 2. emsdk options + +To build the wasm C source code into wasm binary, we can use the following command: + +```bash +EMCC_ONLY_FORCED_STDLIBS=1 emcc -O3 -s STANDALONE_WASM=1 \ + -o test.wasm test.c \ + -s TOTAL_STACK=4096 -s TOTAL_MEMORY=65536 \ + -s "EXPORTED_FUNCTIONS=['_main']" \ + -s ERROR_ON_UNDEFINED_SYMBOLS=0 +``` + +There are some useful options: + +- **EMCC_ONLY_FORCED_STDLIBS=1** whether to link libc library into the output binary or not, similar to `-nostdlib` option of wasi-sdk clang. If specified, then no libc library is linked and the **libc-builtin** library of WAMR must be built to run the wasm app, otherwise, the **libc-wasi** library must be built. You can specify **-DWAMR_BUILD_LIBC_BUILTIN=1** or **-DWAMR_BUILD_LIBC_WASI=1** for CMake to build WAMR with libc-builtin support or libc-wasi support. + + The emsdk's wasi implementation is incomplete, e.g. open a file might just return fail, so it is strongly not recommended to use this mode, especially when there are file io operations in wasm app, please use wasi-sdk instead. + +- **-s STANDALONE_WASM=1** build wasm app in standalone mode (non-web mode), if the output file has suffix ".wasm", then only wasm file is generated (without html file and JavaScript file). + +- **-s TOTAL_STACK=\<value\>** the auxiliary stack size, same as `-z stack-size=\<value\>` of wasi-sdk + +- **-s TOTAL_MEMORY=\<value\>** or **-s INITIAL_MEORY=\<value\>** the initial linear memory size + +- **-s MAXIMUM_MEMORY=\<value\>** the maximum linear memory size, only take effect if **-s ALLOW_MEMORY_GROWTH=1** is set + +- **-s ALLOW_MEMORY_GROWTH=1/0** whether the linear memory is allowed to grow or not + +- **-s "EXPORTED_FUNCTIONS=['func name1', 'func name2']"** to export functions + +- **-s ERROR_ON_UNDEFINED_SYMBOLS=0** disable the errors when there are undefined symbols + +For more options, please ref to <EMSDK_DIR>/upstream/emscripten/src/settings.js, or [Emscripten document](https://emscripten.org/docs/compiling/Building-Projects.html). + +# Build a project with cmake + +If you have a complex WASM application project which contains dozens of source files, you can consider using cmake for project building. + +You can cross compile your project by using the toolchain provided by WAMR. + +Assume the original `CMakeLists.txt` for `test.c` likes below: + +``` cmake +cmake_minimum_required (VERSION 3.5) +project(hello_world) +add_executable(hello_world test.c) +``` + +It is easy to use *wasi-sdk* in CMake by setting *CMAKE_TOOLCHAIN_FILE* without any modification on the original *CMakeLists.txt*. + +``` +$ cmake -DWASI_SDK_PREFIX=${WASI_SDK_INSTALLTION_DIR} + -DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_INSTALLTION_DIR}/share/cmake/wasi-sdk.cmake + -DCMAKE_SYSROOT=<a sysroot directory> + .. +``` + +`WASI_SDK_INSTALLTION_DIR` is the directory in where you install the *wasi-sdk*. like */opt/wasi-sdk* + +If you prefer WASI, set *CMAKE_SYSROOT* to *wasi-sysroot* + +``` +$ cmake <same as above> + -DCMAKE_SYSROOT=${WASI_SDK_INSTALLTION_DIR}/share/wasi-sysroot + .. +``` + +If you prefer *WAMR builtin libc*, set *CMAKE_SYSROOT* to *libc-builtin-sysroot* + +> Note: If you have already built a SDK profile + +``` +$ cmake <same as above> + -DCMAKE_SYSROOT=${WAMR_SOURCE_ROOT}/wamr-sdk/app/libc-builtin-sysroot + .. +``` + +You will get ```hello_world``` which is the WASM app binary. + + +# Compile WASM to AOT module + +Please ensure the wamrc was already generated and available in your shell PATH. Then we can use wamrc to compile WASM app binary to WAMR AOT binary. + +``` Bash +wamrc -o test.aot test.wasm +``` + +wamrc supports a number of compilation options through the command line arguments: + +``` Bash +wamrc --help +Usage: wamrc [options] -o output_file wasm_file + --target=<arch-name> Set the target arch, which has the general format: <arch><sub> + <arch> = x86_64, i386, aarch64, arm, thumb, xtensa, mips, + riscv64, riscv32. + Default is host arch, e.g. x86_64 + <sub> = for ex. on arm or thumb: v5, v6m, v7a, v7m, etc. + Use --target=help to list supported targets + --target-abi=<abi> Set the target ABI, e.g. gnu, eabi, gnueabihf, msvc, etc. + Default is gnu if target isn't riscv64 or riscv32 + For target riscv64 and riscv32, default is lp64d and ilp32d + Use --target-abi=help to list all the ABI supported + --cpu=<cpu> Set the target CPU (default: host CPU, e.g. skylake) + Use --cpu=help to list all the CPU supported + --cpu-features=<features> Enable or disable the CPU features + Use +feature to enable a feature, or -feature to disable it + For example, --cpu-features=+feature1,-feature2 + Use --cpu-features=+help to list all the features supported + --opt-level=n Set the optimization level (0 to 3, default is 3) + --size-level=n Set the code size level (0 to 3, default is 3) + -sgx Generate code for SGX platform (Intel Software Guard Extention) + --bounds-checks=1/0 Enable or disable the bounds checks for memory access: + by default it is disabled in all 64-bit platforms except SGX and + in these platforms runtime does bounds checks with hardware trap, + and by default it is enabled in all 32-bit platforms + --format=<format> Specifies the format of the output file + The format supported: + aot (default) AoT file + object Native object file + llvmir-unopt Unoptimized LLVM IR + llvmir-opt Optimized LLVM IR + --disable-bulk-memory Disable the MVP bulk memory feature + --enable-multi-thread Enable multi-thread feature, the dependent features bulk-memory and + thread-mgr will be enabled automatically + --enable-tail-call Enable the post-MVP tail call feature + --disable-simd Disable the post-MVP 128-bit SIMD feature: + currently 128-bit SIMD is only supported for x86-64 target, + and by default it is enabled in x86-64 target and disabled + in other targets + --disable-ref-types Disable the MVP reference types feature + --disable-aux-stack-check Disable auxiliary stack overflow/underflow check + --enable-dump-call-stack Enable stack trace feature + --enable-perf-profiling Enable function performance profiling + -v=n Set log verbose level (0 to 5, default is 2), larger with more log +Examples: wamrc -o test.aot test.wasm + wamrc --target=i386 -o test.aot test.wasm + wamrc --target=i386 --format=object -o test.o test.wasm +``` + +## AoT compilation with 3rd-party toolchains + +`wamrc` uses LLVM to compile wasm bytecode to AoT file, this works for most of the architectures, but there may be circumstances where you want to use 3rd-party toolchains to take over some steps of the compilation pipeline, e.g. + +1. The upstream LLVM doesn't support generating object file for your CPU architecture (such as ARC), then we may need some other assembler to do such things. +2. You may get some other LLVM-based toolchains which may have better optimizations for the specific target, then you may want your toolchain to take over all optimization steps. + +`wamrc` provides two environment variables to achieve these: +- `WAMRC_LLC_COMPILER` + + When specified, `wamrc` will emit the optimized LLVM-IR (.bc) to a file, and invoke `$WAMRC_LLC_COMPILER` with ` -c -O3 ` to generate the object file. + + Optionally, you can use environment variable `WAMRC_LLC_FLAGS` to overwrite the default flags. + +- `WAMRC_ASM_COMPILER` + + When specified, `wamrc` will emit the text based assembly file (.s), and invoke `$WAMRC_ASM_COMPILER` with ` -c -O3 ` to generate the object file. + + Optionally, you can use environment variable `WAMRC_ASM_FLAGS` to overwrite the default flags. + +### Usage example +``` bash +WAMRC_LLC_COMPILER=<path/to/your/compiler/driver> ./wamrc -o test.aot test.wasm +``` + +> Note: `wamrc` will verify whether the specified file exists and executable. If verification failed, `wamrc` will report a warning and fallback to normal pipeline. Since the verification is based on file, you **must specify the absolute path to the binary** even if it's in `$PATH` + +> Note: `WAMRC_LLC_COMPILER` has higher priority than `WAMRC_ASM_COMPILER`, if `WAMRC_LLC_COMPILER` is set and verified, then `WAMRC_ASM_COMPILER` will be ignored. + +> Note: the `LLC` and `ASM` in the env name just means this compiler will be used to compile the `LLVM IR file`/`assembly file` to object file, usually passing the compiler driver is the simplest way. (e.g. for LLVM toolchain, you don't need to pass `/usr/bin/llc`, using `/usr/bin/clang` is OK) + +Run WASM app in WAMR mini product build +======================================= + +Run the test.wasm or test.aot with WAMR mini product build: +``` Bash +./iwasm test.wasm or +./iwasm test.aot +``` +You will get the following output: +``` +Hello world! +buf ptr: 0xffffc2c8 +buf: 1234 +``` +If you would like to run the test app on Zephyr, we have embedded a test sample into its OS image. You will need to execute: +``` +ninja run +``` diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/devcontainer.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/devcontainer.md new file mode 100644 index 000000000..b75f13ec9 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/devcontainer.md @@ -0,0 +1,25 @@ +# Visual Studio Code development container + +We all know Docker containers and may use them a lot in school or work. It resolves dependency management for our projects/applications, prevents package version confusion and conflict, and contamination of the local environment. + +Now WAMR has a Dockerfile under path `.devontainer` to create a container image, dev container images that you could easily use in VS Code. In case you prefer other IDE like Clion, you can also build it and use for the IDE you like. + +## How to use it + +It's straightforward to use Docker in VS Code! First, you have VS Code and Docker installed(if not yet, check [next section](#learn-more-about-docker-and-vs-code) for howto). Then you need to download Docker in VS Code extensions marketplace. + +And that's it, and you are good to go! When you open the root folder of WAMR, in the bottom right corner, the Docker extension will pop a notification and ask if you like to reopen the folder in a container. + +If you encounter any problems or get stuck somewhere, may this video [demo](https://youtu.be/Uvf2FVS1F8k) for docker usage in VS Code will help. + +## Learn more about Docker and VS Code + +[Install Docker](https://docs.docker.com/get-docker/) + +[Install VS Code](https://code.visualstudio.com/) + +[Docker extension for VS Code](https://code.visualstudio.com/docs/containers/overview) + +[Remote development with Docker in VS Code](https://code.visualstudio.com/docs/remote/containers#_getting-started) + +[What is dev container image in VS Code](https://code.visualstudio.com/docs/remote/containers#_prebuilding-dev-container-images)
\ No newline at end of file diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/embed_wamr.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/embed_wamr.md new file mode 100644 index 000000000..050384027 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/embed_wamr.md @@ -0,0 +1,336 @@ +Embedding WAMR guideline +===================================== + +**Note**: This document is about how to embed WAMR into C/C++ host applications, for other languages, please refer to: [Embed WAMR into Python](../language-bindings/python), [Embed WAMR into Go](../language-bindings/go). + +All the embedding APIs supported by the runtime are defined under folder [core/iwasm/include](../core/iwasm/include). The API details are available in the header files. + +## Embed WAMR into developer's project + +WAMR is designed to be easy embeddable in any project, a typical way of building WAMR is to use cmake, developer can configure features by setting cmake variables and then include the script `runtime_lib.cmake` under folder [build-scripts](../build-scripts) in his CMakeList.txt, for example: +``` cmake +set (WAMR_BUILD_PLATFORM "linux") +set (WAMR_BUILD_TARGET "X86_64") +set (WAMR_BUILD_INTERP 1) +set (WAMR_BUILD_FAST_INTERP 1) +set (WAMR_BUILD_AOT 1) +set (WAMR_BUILD_LIBC_BUILTIN 1) +set (WAMR_BUILD_LIBC_WASI 1) +set (WAMR_BUILD_SIMD 1) +set (WAMR_ROOT_DIR path/to/wamr/root) + +include (${WAMR_ROOT_DIR}/build-scripts/runtime_lib.cmake) +add_library(vmlib ${WAMR_RUNTIME_LIB_SOURCE}) + +target_link_libraries (your_project vmlib) +``` +Examples can be found in [CMakeLists.txt of linux platform](../product-mini/platforms/linux/CMakeLists.txt) and [other platforms](../product-mini/platforms). The available features to configure can be found in [Build WAMR vmcore](./build_wamr.md#wamr-vmcore-cmake-building-configurations). + +Developer can also use Makefile to embed WAMR, by defining macros and including directories, and adding the source files, examples can be found in [makefile of alios-things platform](../product-mini/platforms/alios-things/aos.mk) and [makefile of nuttx platform](../product-mini/platforms/nuttx/wamr.mk). + +## The runtime initialization + +``` C + char *buffer, error_buf[128]; + wasm_module_t module; + wasm_module_inst_t module_inst; + wasm_function_inst_t func; + wasm_exec_env_t exec_env; + uint32 size, stack_size = 8092, heap_size = 8092; + + /* initialize the wasm runtime by default configurations */ + wasm_runtime_init(); + + /* read WASM file into a memory buffer */ + buffer = read_wasm_binary_to_buffer(…, &size); + + /* add line below if we want to export native functions to WASM app */ + wasm_runtime_register_natives(...); + + /* parse the WASM file from buffer and create a WASM module */ + module = wasm_runtime_load(buffer, size, error_buf, sizeof(error_buf)); + + /* create an instance of the WASM module (WASM linear memory is ready) */ + module_inst = wasm_runtime_instantiate(module, stack_size, heap_size, + error_buf, sizeof(error_buf)); +``` + +The `wasm_runtime_init()` uses the default memory allocator os_malloc/os_free function from the [`core/shared/platform`](../core/shared/platform) for the runtime memory management. + +WAMR supports to restrict its all memory allocations in a raw buffer. It ensures the dynamic memories used by the WASM applications won't harm the system availability, which is extremely important for embedded systems. This can be done by using `wasm_runtime_full_init()`. This function also allows you to configure the native API's for exporting to WASM app, and set the maximum thread number when multi-thread feature is enabled. + +Refer to the following sample: + +```c +/* the native functions that will be exported to WASM app */ +static NativeSymbol native_symbols[] = { + EXPORT_WASM_API_WITH_SIG(display_input_read, "(*)i"), + EXPORT_WASM_API_WITH_SIG(display_flush, "(iiii*)") +}; + +/* all the runtime memory allocations are retricted in the global_heap_buf array */ +static char global_heap_buf[512 * 1024]; +RuntimeInitArgs init_args; +memset(&init_args, 0, sizeof(RuntimeInitArgs)); + +/* configure the memory allocator for the runtime */ +init_args.mem_alloc_type = Alloc_With_Pool; +init_args.mem_alloc_option.pool.heap_buf = global_heap_buf; +init_args.mem_alloc_option.pool.heap_size = sizeof(global_heap_buf); + +/* configure the native functions being exported to WASM app */ +init_args.native_module_name = "env"; +init_args.n_native_symbols = sizeof(native_symbols) / sizeof(NativeSymbol); +init_args.native_symbols = native_symbols; + +/* set maximum thread number if needed when multi-thread is enabled, + the default value is 4 */ +init_args.max_thread_num = max_thread_num; + +/* initialize runtime environment with user configurations*/ +if (!wasm_runtime_full_init(&init_args)) { + return -1; +} +``` + +## Native calls WASM functions and passes parameters + +After a module is instantiated, the runtime embedder can lookup the target WASM function by name, and create execution environment to call the function. + +```c + /* lookup a WASM function by its name + The function signature can NULL here */ + func = wasm_runtime_lookup_function(module_inst, "fib", NULL); + + /* creat an execution environment to execute the WASM functions */ + exec_env = wasm_runtime_create_exec_env(module_inst, stack_size); +``` + +There are several ways to call WASM function: + +1. Function call with parameters in an array of 32 bits elements and size: + +```c + uint32 argv[2]; + + /* arguments are always transferred in 32-bit element */ + argv[0] = 8; + + /* call the WASM function */ + if (wasm_runtime_call_wasm(exec_env, func, 1, argv) ) { + /* the return value is stored in argv[0] */ + printf("fib function return: %d\n", argv[0]); + } + else { + /* exception is thrown if call fails */ + printf("%s\n", wasm_runtime_get_exception(module_inst)); + } +``` + +The parameters are transferred in an array of 32 bits elements. For parameters that occupy 4 or fewer bytes, each parameter can be a single array element. For parameters in types like double or int64, each parameter will take two array elements. The function return value will be sent back in the first one or two elements of the array according to the value type. See the sample code below: + +```c + uint32 argv[6]; + char arg1 = 'a'; + int arg2 = 10; + double arg3 = 1.0; + int64 arg4 = 100; + double ret; + + argv[0] = arg1; + argv[1] = arg2; + + /** + * use memory copy for 8-byte parameters rather than + * *(double*)(&argv[2]) = arg3 here because some archs + * like ARM, MIPS require the address must be 8-byte aligned. + * Or use the aligned malloc or compiler align attribute + * to ensure the array address is 8-byte aligned + */ + memcpy(&argv[2], &arg3, sizeof(arg3)); + memcpy(&argv[4], &arg4, sizeof(arg4)); + + /* attention: the arg number is 6 here since both + arg3 and arg4 each takes 2 elements */ + wasm_runtime_call_wasm(exec_env, func, 6, argv); + + /* if the return value is type of 8 bytes, it takes + the first two array elements */ + memcpy(&ret, &argv[0], sizeof(ret)); +``` + +2. Function call with results and arguments both in `wasm_val_t` struct and size: + +```c + uint32 num_args = 1, num_results = 1; + wasm_val_t args[1], results[1]; + + /* set the argument type and value */ + args[0].kind = WASM_I32; + args[0].of.i32 = 8; + + /* call the WASM function */ + if (wasm_runtime_call_wasm_a(exec_env, func, num_results, results, num_args, args)) { + /* the return value is stored in results */ + printf("fib function return: %d\n", results[0].of.i32); + } + else { + /* exception is thrown if call fails */ + printf("%s\n", wasm_runtime_get_exception(module_inst)); + } +``` + +3. Function call with variant argument support: + +```c + uint32 num_args = 1, num_results = 1; + wasm_val_t results[1]; + + /* call the WASM function */ + if (wasm_runtime_call_wasm_v(exec_env, func, 1, results, 1, 8)) { + /* the return value is stored in results */ + printf("fib function return: %d\n", results[0].of.i32); + } + else { + /* exception is thrown if call fails */ + printf("%s\n", wasm_runtime_get_exception(module_inst)); + } +``` + +## Pass buffer to WASM function + +If we need to transfer a buffer to WASM function, we can pass the buffer address through a parameter. **Attention**: The sandbox will forbid the WASM code to access outside memory, we must **allocate the buffer from WASM instance's own memory space and pass the buffer address in instance's space (not the runtime native address)**. + +There are two runtime APIs available for this purpose. + +```c +/** + * malloc a buffer from instance's private memory space. + * + * return: the buffer address in instance's memory space (pass to the WASM funciton) + * p_native_addr: return the native address of allocated memory + * size: the buffer size to allocate + */ +uint32_t +wasm_runtime_module_malloc(wasm_module_inst_t module_inst, + uint32_t size, void **p_native_addr); + +/** + * malloc a buffer from instance's private memory space, + * and copy the data from another native buffer to it. + * + * return: the buffer address in instance's memory space (pass to the WASM funciton) + * src: the native buffer address + * size: the size of buffer to be allocated and copy data + */ +uint32_t +wasm_runtime_module_dup_data(wasm_module_inst_t module_inst, + const char *src, uint32_t size); + +/* free the memory allocated from module memory space */ +void +wasm_runtime_module_free(wasm_module_inst_t module_inst, uint32_t ptr); +``` + +Usage sample: + +```c +char * buffer = NULL; +uint32_t buffer_for_wasm; + +buffer_for_wasm = wasm_runtime_module_malloc(module_inst, 100, &buffer); +if (buffer_for_wasm != 0) { + uint32 argv[2]; + strncpy(buffer, "hello", 100); /* use native address for accessing in runtime */ + argv[0] = buffer_for_wasm; /* pass the buffer address for WASM space */ + argv[1] = 100; /* the size of buffer */ + wasm_runtime_call_wasm(exec_env, func, 2, argv); + + /* it is runtime embedder's responsibility to release the memory, + unless the WASM app will free the passed pointer in its code */ + wasm_runtime_module_free(module_inst, buffer_for_wasm); +} +``` + +## Pass structured data to WASM function + +We can't pass structure data or class objects through the pointer since the memory layout can different in two worlds. The way to do it is serialization. Refer to [export_native_api.md](./export_native_api.md) for the details. + +## Execute wasm functions in multiple threads + +The `exec_env` is not thread safety, it will cause unexpected behavior if the same `exec_env` is used in multiple threads. However, we've provided two ways to execute wasm functions concurrently: + +- You can use `pthread` APIs in your wasm application, see [pthread library](./pthread_library.md) for more details. + +- The `spawn exec_env` and `spawn thread` APIs are available, you can use these APIs to manage the threads in native: + + *spawn exec_env:* + + `spawn exec_env` API spawns a `new_exec_env` base on the original `exec_env`, use can use it in other threads: + + ```C + new_exec_env = wasm_runtime_spawn_exec_env(exec_env); + + /* Then you can use new_exec_env in your new thread */ + module_inst = wasm_runtime_get_module_inst(new_exec_env); + func_inst = wasm_runtime_lookup_function(module_inst, ...); + wasm_runtime_call_wasm(new_exec_env, func_inst, ...); + + /* you need to use this API to manually destroy the spawned exec_env */ + wasm_runtime_destroy_spawned_exec_env(new_exec_env); + ``` + + *spawn thread:* + + You can also use `spawn thread` API to avoid manually manage the spawned exec_env: + + ```C + wasm_thread_t wasm_tid; + void *wamr_thread_cb(wasm_exec_env_t exec_env, void *arg) + { + module_inst = wasm_runtime_get_module_inst(exec_env); + func_inst = wasm_runtime_lookup_function(module_inst, ...); + wasm_runtime_call_wasm(exec_env, func_inst, ...); + } + wasm_runtime_spawn_thread(exec_env, &wasm_tid, wamr_thread_cb, NULL); + /* Use wasm_runtime_join_thread to join the spawned thread */ + wasm_runtime_join_thread(wasm_tid, NULL); + ``` + +**Note1: You can manage the maximum number of threads can be created:** + +```C +init_args.max_thread_num = THREAD_NUM; +/* If this init argument is not set, the default maximum thread number is 4 */ +``` + +**Note2: The wasm application should be built with `--shared-memory` and `-pthread` enabled:** + +```bash + /opt/wasi-sdk/bin/clang -o test.wasm test.c -nostdlib -pthread \ + -Wl,--shared-memory,--max-memory=131072 \ + -Wl,--no-entry,--export=__heap_base,--export=__data_end \ + -Wl,--export=__wasm_call_ctors,--export=${your_func_name} +``` + + **Note3: The pthread library feature should be enabled while building the runtime:** + + ```bash + cmake .. -DWAMR_BUILD_LIB_PTHREAD=1 + ``` + +[Here](../samples/spawn-thread) is a sample to show how to use these APIs. + +## The deinitialization procedure + +``` + wasm_runtime_destroy_exec_env(exec_env); + wasm_runtime_deinstantiate(module_inst); + wasm_runtime_unload(module); + wasm_runtime_destroy(); + +``` + +## Native calling WASM function working flow + + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/export_native_api.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/export_native_api.md new file mode 100644 index 000000000..b87d92552 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/export_native_api.md @@ -0,0 +1,264 @@ + +Export native API to WASM application +======================================================= + + + +Exporting native API steps +-------------------------- + +#### Step 1: Declare the function interface in WASM app + +Create a header file in a WASM app and declare the functions that are exported from native. In this example, we declare foo and foo2 as below in the header file `example.h` + +```c +/*** file name: example.h ***/ + +int foo(int a, int b); +void foo2(char * msg, char * buffer, int buf_len); +``` + + + +#### Step 2: Define the native API + +Then we should define the native functions in runtime source tree for handling the calls from the WASM app. The native function can be any name, for example **foo_native** and **foo2** here: + +``` C +int foo_native(wasm_exec_env_t exec_env , int a, int b) +{ + return a+b; +} + +void foo2(wasm_exec_env_t exec_env, char * msg, uint8 * buffer, int buf_len) +{ + strncpy(buffer, msg, buf_len); +} +``` + +The first parameter exec_env must be defined using type **wasm_exec_env_t** which is the calling convention by WAMR. + +The rest parameters should be in the same types as the parameters of WASM function foo(), but there are a few special cases that are explained in section "Buffer address conversion and boundary check". Regarding the parameter names, they don't have to be the same, but we would suggest using the same names for easy maintenance. + + + +#### Step 3: Register the native APIs + +Register the native APIs in the runtime, then everything is fine. It is ready to build the runtime software. + +``` C +// Define an array of NativeSymbol for the APIs to be exported. +// Note: the array must be static defined since runtime +// will keep it after registration +static NativeSymbol native_symbols[] = +{ + { + "foo", // the name of WASM function name + foo_native, // the native function pointer + "(ii)i" // the function prototype signature + }, + { + "foo2", // the name of WASM function name + foo2, // the native function pointer + "($*~)" // the function prototype signature + } +}; + +// initialize the runtime before registering the native functions +wasm_runtime_init(); + +int n_native_symbols = sizeof(native_symbols) / sizeof(NativeSymbol); +if (!wasm_runtime_register_natives("env", + native_symbols, + n_native_symbols)) { + goto fail1; +} + +// natives registeration must be done before loading WASM modules +module = wasm_runtime_load(buffer, size, error_buf, sizeof(error_buf)); + +``` + +**Function signature**: + +The function signature field in **NativeSymbol** structure is a string for describing the function prototype. It is critical to ensure the function signature is correctly mapping the native function interface. + +Each letter in the "()" represents a parameter type, and the one following after ")" represents the return value type. The meaning of each letter: + +- '**i**': i32 +- '**I**': i64 +- '**f**': f32 +- '**F**': f64 +- '**r**': externref (has to be the value of a `uintptr_t` variable) +- '**\***': the parameter is a buffer address in WASM application +- '**~**': the parameter is the byte length of WASM buffer as referred by preceding argument "\*". It must follow after '*', otherwise, registration will fail +- '**$**': the parameter is a string in WASM application + +The signature can defined as NULL, then all function parameters are assumed as i32 data type. + +**Use EXPORT_WASM_API_WITH_SIG** + +The `NativeSymbol` element for `foo2 ` above can be also defined with macro EXPORT_WASM_API_WITH_SIG. This macro can be used when the native function name is the same as the WASM symbol name. + +```c +static NativeSymbol native_symbols[] = +{ + EXPORT_WASM_API_WITH_SIG(foo2, "($*~)") // wasm symbol name will be "foo2" +}; +``` + +​ + +## Call exported API in WASM application + +Now we can call the exported native API in wasm application like this: +``` C +#include <stdio.h> +#include "example.h" // where the APIs are declared + +int main(int argc, char **argv) +{ + int a = 0, b = 1; + char * msg = "hello"; + char buffer[100]; + + int c = foo(a, b); // call into native foo_native() + foo2(msg, buffer, sizeof(buffer)); // call into native foo2() + + return 0; +} +``` + +## Build native lib into shared library and register it with `iwasm` application + +Developer can also build the native library into a shared library and register it with iwasm application: +```bash +iwasm --native-lib=<lib file> <wasm file> +``` + +Refer to [native lib sample](../samples/native-lib) for more details. + + +## Buffer address conversion and boundary check + +A WebAssembly sandbox ensures applications only access to its own memory with a private address space. When passing a pointer address from WASM to native, the address value must be converted to native address before the native function can access it. It is also the native world's responsibility to check the buffer length is not over its sandbox boundary. + + + +The signature letter '$', '\*' and '\~' help the runtime do automatic address conversion and buffer boundary check, so the native function directly uses the string and buffer address. **Notes**: if '\*' is not followed by '\~', the native function should not assume the length of the buffer is more than 1 byte. + + + +As function parameters are always passed in 32 bits numbers, you can also use 'i' for the pointer type argument, then you must do all the address conversion and boundary checking in your native function. For example, if you change the foo2 signature to "(iii)", then you will implement the native part as the following sample: + +```c +// +// If the function signature used i32 data type ("i") +// for buffer address or string parameters, here +// is how to do address conversation and boundary check manually +// +void foo2(wasm_exec_env_t exec_env, + uint32 msg_offset, + uint32 buffer_offset, + int32 buf_len) +{ + wasm_module_inst_t module_inst = get_module_inst(exec_env); + char *buffer; + char * msg ; + + // do boundary check + if (!wasm_runtime_validate_app_str_add(msg_offset)) + return 0; + + if (!wasm_runtime_validate_app_addr(buffer_offset, buf_len)) + return; + + // do address conversion + buffer = wasm_runtime_addr_app_to_native(buffer_offset); + msg = wasm_runtime_addr_app_to_native(msg_offset); + + strncpy(buffer, msg, buf_len); +} +``` + + + + + +## Sandbox security attention + +The runtime builder should ensure not broking the memory sandbox when exporting the native function to WASM. + +A ground rule: + +- Do the pointer address conversion in the native API if "$\*" is not used for the pointer in the function signature + +A few recommendations: + +- Never pass any structure/class object pointer to native (do data serialization instead) +- Never pass a function pointer to the native + +Note: while not recommended here, nothing prevents you from passing +structure/function pointers as far as the native API is aware of +and careful about the ABI used in the wasm module. For example, +C function pointers are usually represented as table indexes which +the native API can call with wasm_runtime_call_indirect() or similar. +However, in this document, we don't recommend to implement your native +API that way unless necessary because it needs extra carefulness. + + +## Pass structured data or class object + +We must do data serialization for passing structured data or class objects between the two worlds of WASM and native. There are two serialization methods available in WASM as below, and yet you can introduce more like json, cbor etc. + +- [attributes container](../core/app-framework/app-native-shared/attr_container.c) +- [restful request/response](../core/app-framework/app-native-shared/restful_utils.c) + +Note the serialization library is separately compiled into WASM and runtime. And the source files are located in the folder "[core/app-framework/app-native-shared](../core/app-framework/app-native-shared)“ where all source files will be compiled into both worlds. + + + +The following sample code demonstrates WASM app packs a response structure to buffer, then pass the buffer pointer to the native: + +```c +/*** file name: core/app-framework/base/app/request.c ***/ + +void api_response_send(response_t *response) +{ + int size; + char * buffer = pack_response(response, &size); + if (buffer == NULL) + return; + + wasm_response_send(buffer, size); // calling exported native API + free_req_resp_packet(buffer); +} +``` + + + +The following code demonstrates the native API unpack the WASM buffer to local native data structure: + +```c +/*** file name: core/app-framework/base/native/request_response.c ***/ + +bool +wasm_response_send(wasm_exec_env_t exec_env, char *buffer, int size) +{ + if (buffer != NULL) { + response_t response[1]; + + if (NULL == unpack_response(buffer, size, response)) + return false; + + am_send_response(response); + + return true; + } + + return false; +} +``` + + + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/linux_sgx.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/linux_sgx.md new file mode 100644 index 000000000..e7a32753a --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/linux_sgx.md @@ -0,0 +1,271 @@ + +Build and Port WAMR vmcore (iwasm) for Linux SGX +================================================ + +Build WAMR vmcore (iwasm) for Linux SGX +--------------------------------------- + +First of all please install the [Intel SGX SDK](https://software.intel.com/en-us/sgx/sdk), v2.8 or later is required, and it is recommended to install the SDK to /opt/intel/sgxsdk. + +After installing the dependencies, build the source code: +``` Bash +source <SGX_SDK dir>/environment +cd product-mini/platforms/linux-sgx/ +mkdir build && cd build +cmake .. +make +``` + +By default the `fast interpreter` and `AOT` is enabled. If to enable `Fast JIT`, run: +```Bash +mkdir build && cd build +cmake .. -DWAMR_BUILD_FAST_JIT=1 +make +``` + +This builds two libraries required by SGX application: + - libvmlib.a for Enclave part + - libvmlib_untrusted.a for App part + +**Note:** WAMR provides some features which can be easily configured by passing options to cmake, please see [WAMR vmcore cmake building configurations](./build_wamr.md#wamr-vmcore-cmake-building-configurations) for the details. Currently in Linux SGX, fast interpreter, AOT, libc-builtin, libc-WASI and lib-pthread are enabled by default. + +Then build the enclave sample: +``` Bash +source <SGX_SDK dir>/environment +cd enclave-sample +make +``` + +**Note:** By default, the generated SGX application assumes it is signed with production key and running on simulation mode. The user can explicitly specify the relative variables in commandline to overwrite the default settings. For example, to build a debug enclave, please build the enclave with `make SGX_DEBUG=1`. To build the enclave running on a hardware-based SGX platform, execute `make SGX_MODE=HW`. + +The binary file iwasm will be generated. To run the sample: + +``` Bash +source <SGX_SDK dir>/environment +iwasm [-options] wasm_file [args...] +or: +iwasm [-options] aot_file [args...] +``` + +### Minimal build +The libc-WASI and lib-pthread features require a lot of ocalls, if you don't need so much ocalls in your application, you can use the `minimal` version + +``` Bash +# replace the build files with minimal version +cd product-mini/platforms/linux-sgx/ +cp CMakeLists_minimal.txt CMakeLists.txt +cp enclave-sample/Makefile_minimal enclave-sample/Makefile +cp enclave-sample/Enclave/Enclave_minimal.edl enclave-sample/Enclave/Enclave.edl +# follow the building process above +``` + +Port WAMR vmcore for Linux SGX +------------------------------ + +The enclave-sample creates a sample to embed wamr vmlib of Enclave part and App part to an SGX application. To port WAMR vmcore lib to SGX application, there are some steps to do: + +**Step 1: Add "sgx_wamr.edl" and "sgx_pthread.edl" into EDL file, e.g. Enclave.edl:** +> This step is not required in minimal version + +```bash +from "sgx_pthread.edl" import *; +from "sgx_wamr.edl" import *; +``` + +The sgx_wamr.edl is under ${WAMR_ROOT}/core/shared/platform/linux-sgx, so please **add it to the search path list** when generating Enclave_u.c and Enclave_t.c from Enclave.edl: + +```bash +@cd App && $(SGX_EDGER8R) --untrusted ../Enclave/Enclave.edl \ + --search-path ../Enclave \ + --search-path $(SGX_SDK)/include \ + --search-path $(WAMR_ROOT)/core/shared/platform/linux-sgx +``` + +```bash +@cd Enclave && $(SGX_EDGER8R) --trusted ../Enclave/Enclave.edl \ + --search-path ../Enclave \ + --search-path $(SGX_SDK)/include \ + --search-path $(WAMR_ROOT)/core/shared/platform/linux-sgx +``` + +**Step 2: Link libvmlib.a to Enclave part and link libvmlib_untrusted.a to App part:** +> libvmlib_untrusted.a is not required in minimal version + +```makefile +Enclave_Link_Flags := ... libvmlib.a ... +``` + +```makefile +App_Link_Flags := ... libvmlib_untrusted.a ... +``` + +**And link SGX pthread lib to Enclave part:** +> SGX pthread lib is not required in minimal version + +```makefile +Enclave_Link_Flags := ... -lsgx_pthread ... +``` + +**Step 3: Add WAMR folders and SGX SDK folders to Enclave include path:** + +```makefile +Enclave_Include_Paths := ... -I$(WAMR_ROOT)/core/iwasm/include \ + -I$(WAMR_ROOT)/core/shared/utils \ + -I$(WAMR_ROOT)/core/shared/platform/linux-sgx \ + -I$(SGX_SDK)/include \ + -I$(SGX_SDK)/include/tlibc \ + -I$(SGX_SDK)/include/stlport +``` + +**Step 4: Configure reserved memory and thread info in file Enclave config file (e.g. Enclave.config.xml) to support WAMR AOT and multi-thread, e.g:** + +```xml +<ReservedMemMaxSize>0x400000</ReservedMemMaxSize> +<ReservedMemExecutable>1</ReservedMemExecutable> +<TCSNum>10</TCSNum> +``` + +**Step 5: To support log output and os_printf() function in Enclave, please implement an ocall_print function, e.g. in Enclave.edl, add:** + +```cpp +untrusted { + void ocall_print([in, string]const char* str); +}; +``` + +In App part, add: + +```cpp +void +ocall_print(const char* str) +{ + printf("%s", str); +} +``` + +And in Enclave part, set the print function: + +```cpp +#include "wasm_export.h" +#include "bh_platform.h" + +extern "C" { + typedef void (*os_print_function_t)(const char* message); + extern void os_set_print_function(os_print_function_t pf); + + void + enclave_print(const char *message) + { + ocall_print(message); + } +} + +// In the beginning of Enclave initialization, add: +os_set_print_function(enclave_print); +``` + +Embed WAMR vmcore in Linux SGX +------------------------------ + +Normally we can embed WAMR vmcore in Linux SGX by calling the vmcore exported API's, see [Embed WAMR guide](./embed_wamr.md) for the details. And the the ecall_iwasm_main() function in file Enclave.cpp of enclave-sample also provides sample to invoke wasm app main function with wasm file buffer: + +```cpp +void +ecall_iwasm_main(uint8_t *wasm_file_buf, uint32_t wasm_file_size); +``` + +The enclave-sample also wraps an ecall function to receive commands from App to Enclave, and handle the commands in Enclave by calling the related WAMR vmcore API. The commands and related API's are: + +```cpp +typedef enum EcallCmd { + CMD_INIT_RUNTIME = 0, /* wasm_runtime_init/full_init() */ + CMD_LOAD_MODULE, /* wasm_runtime_load() */ + CMD_INSTANTIATE_MODULE, /* wasm_runtime_instantiate() */ + CMD_LOOKUP_FUNCTION, /* wasm_runtime_lookup_function() */ + CMD_CREATE_EXEC_ENV, /* wasm_runtime_create_exec_env() */ + CMD_CALL_WASM, /* wasm_runtime_call_wasm */ + CMD_EXEC_APP_FUNC, /* wasm_application_execute_func() */ + CMD_EXEC_APP_MAIN, /* wasm_application_execute_main() */ + CMD_GET_EXCEPTION, /* wasm_runtime_get_exception() */ + CMD_DEINSTANTIATE_MODULE, /* wasm_runtime_deinstantiate() */ + CMD_UNLOAD_MODULE, /* wasm_runtime_unload() */ + CMD_DESTROY_RUNTIME, /* wasm_runtime_destroy() */ + CMD_SET_WASI_ARGS, /* wasm_runtime_set_wasi_args() */ + CMD_SET_LOG_LEVEL, /* bh_log_set_verbose_level() */ +}; +``` + +SGX Intel Protected File System +------------------------------- +Intel SGX introduced a feature called [Intel Protection File System Library (IPFS)](https://www.intel.com/content/www/us/en/developer/articles/technical/overview-of-intel-protected-file-system-library-using-software-guard-extensions.html) to create, operate and delete files inside the enclave. +WAMR supports the mapping of IPFS on WASI functions related to file interactions, providing seamless persistence with confidentiality and integrity to the hosted WebAssembly applications in the enclave. + +The usage of SGX IPFS is an optional feature. +To opt-in, the support of IPFS requires the following changes: + - set the flag `WAMR_BUILD_SGX_IPFS=1` when running `cmake`, + - the enclave must be linked with the trusted IPFS library (`-lsgx_tprotected_fs`), + - the application outside of the enclave must be linked with the untrusted IPFS library (`-lsgx_uprotected_fs`), + - the EDL file must include the following import statement: + +```edl +from "sgx_tprotected_fs.edl" import *; +``` + +When using the [enclave-sample](../product-mini/platforms/linux-sgx/enclave-sample/) project, setting the flag `WAMR_BUILD_SGX_IPFS=1` when running `cmake` enables these changes automatically. + + +### Verification of SGX IPFS +One can observe the usage of IPFS by running the [file sample](../samples/file/) WebAssembly application. +Enabling the SGX IPFS on this sample project leads to the generation of an encrypted text file. + + +### Mapping of WASI/POSIX to IPFS +This table summarizes how WASI is mapped to POSIX and IPFS. +Since IPFS is a subset of the WASI/POSIX, emulation is performed to fill the missing implementation. + +| WASI | POSIX | IPFS | +|------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------| +| `fd_read` | `readv` | `sgx_fread` | +| `fd_write` | `writev` | `sgx_fwrite` | +| `fd_close` | `close` | `sgx_fclose` | +| `path_open` | `openat` | `sgx_fopen` | +| `fd_datasync` | `fsync` | `sgx_fflush` | +| `fd_tell` | `lseek` | `sgx_ftell` | +| `fd_filestat_set_size` | `ftruncate` | Shrinking files is not supported, nor emulated. Extending files is emulated using `sgx_fseek`/`sgx_ftell`/`sgx_fwrite`. | +| `fd_seek` | `lseek` | The POSIX and IPFS behaviors differ. Emulated using `sgx_fseek`/`sgx_ftell`/`sgx_fwrite`. | +| `fd_pwrite` | `pwrite` | Not supported. Emulated using `sgx_fseek`/`sgx_ftell`/`sgx_fwrite`. | +| `fd_pread` | `pread` | Not supported. Emulated using `sgx_fseek`/`sgx_ftell`/`sgx_fread`. | +| `fd_allocate` | `posix_fallocate` | Not supported. Emulated using `sgx_fseek`/`sgx_ftell`/`sgx_fwrite`/`sgx_fflush`. | + + +### Performance overheads +Many benchmarks have assessed the overheads caused by IPFS through WASI functions using Twine, an early and academic adaptation of WAMR in Intel SGX with WASI support. +The results can be found in [this paper](https://arxiv.org/abs/2103.15860). + +### Limitations +The threat model and the limitations of SGX IPFS can be found in [the official documentation](https://www.intel.com/content/dam/develop/external/us/en/documents/overviewofintelprotectedfilesystemlibrary.pdf). + + +Others +------ + +- Please add "-sgx" option when generating AoT file for SGX platform, e.g.: + + ```bash + wamrc -sgx -o test.aot test.wasm + ``` + +- The default max heap size of Enclave is 16 MB, it might be not enough when executing some workloads, please modify it in Enclave/Enclave.config.xml with a larger size when exception was thrown: + + ```bash + Exception: fail to enlarge memory. + or + Exception: allocate memory failed. + ``` + + Enclave/Enclave.config.xml, default max heap size is 16 MB: + + ```xml + <HeapMaxSize>0x1000000</HeapMaxSize> + ``` + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_tune.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_tune.md new file mode 100644 index 000000000..e14a1a164 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_tune.md @@ -0,0 +1,32 @@ +# Memory model and memory usage tunning + +References: +- [Blog: Understand WAMR heap](https://bytecodealliance.github.io/wamr.dev/blog/understand-the-wamr-heaps/) +- [Blog: Understand WAMR stacks](https://bytecodealliance.github.io/wamr.dev/blog/understand-the-wamr-stacks/) + +## The memory model + +The memory model of WAMR can be basically described as below: + +<center><img src="./pics/wamr_memory_model.png" width="75%" height="75%"></img></center> + +Note: +- **global heap**: the heap to allocate memory for runtime data structures, including wasm module, wasm module instance, exec env, wasm operand stack and so on. It is initialized by `wasm_runtime_init` or `wasm_runtime_full_init`. And for `wasm_runtime_full_init`, developer can specify the memory allocation mode with `RuntimeInitArgs *init_args`: allocate memory from a user defined byte buffer, from user defined allocation function, or from the platform's os_malloc function. Refer to [wasm_export.h](../core/iwasm/include/wasm_export.h#L98-L141) and [Embedding WAMR guideline](./embed_wamr.md#the-runtime-initialization) for more details. And developer can use `wasm_runtime_malloc/wasm_runtime_free` to allocate/free memory from/to the global heap. +- **wasm operand stack**: the stack to store the operands required by wasm bytecodes as WebAssembly is based on a stack machine. If the exec_env is created by developer with `wasm_runtime_create_exec_env`, then its size is specified by `wasm_runtime_create_exec_env`, otherwise if the exec_env is created by runtime internally, e.g. by `wasm_application_execute_main` or `wasm_application_execute_func`, then the size is specified by `wasm_runtime_instantiate`. +- **linear memory**: a contiguous, mutable array of raw bytes. It is created with an initial size but might be grown dynamically. For most compilers, e.g. wasi-sdk, emsdk, rustc or asc, normally it includes three parts, data area, auxiliary stack area and heap area. For wasi-sdk, the initial/max size can be specified with `-Wl,--initial-memory=n1,--max-memory=n2`, for emsdk, the initial/max size can be specified with `-s INITIAL_MEMORY=n1 -s MAXIMUM_MEMORY=n2 -s ALLOW_MEMORY_GROWTH=1` or `-s TOTAL_MEMORY=n`, and for asc, they can be specified with `--initialMemory` and `--maximumMemory` flags. + - If the memory access boundary check with hardware trap feature is enabled, e.g. in Linux/MacOS/Windows x86-64 by default, the linear memory is allocated by `os_mmap` from virtual address space instead of global heap. +- **aux stack**: the auxiliary stack resides in linear memory to store some temporary data when calling wasm functions, for example, calling a wasm function with complex struct arguments. For wasi-sdk, the size can be specified with `-z stack-size=n`, for emsdk, the size can be specified with `-s TOTAL_STACK=n`. +- **app heap and libc heap**: the heap to allocate memory for wasm app, note that app heap is created only when the malloc/free functions (or __new/__release functions for AssemblyScript) are not exported and runtime can not detect the libc heap. To export the malloc/free functions, for wasi-sdk and emsdk, developer can use `-Wl,--export=malloc -Wl,--export=free` options, for asc, developer can use `--exportRuntime` option. For app heap, the size is specified by `wasm_runtime_instantiate`. It is recommended to export the malloc/free functions and disable app heap in single thread mode, and for multi-threading, as the libc heap isn't thread-safe, it is recommended to remove the dlmalloc.o from libc.a for wasi-sdk and use `-s MALLOC="none"` for emsdk, refer to [WAMR pthread library](./pthread_library.md) for more details. And developer can use `wasm_runtime_module_malloc/wasm_runtime_module_free` to allocate/free memory from/to app heap (or libc heap if malloc/free functions are exported). +- **__data_end global and __heap_base global**: two globals exported by wasm application to indicate the end of data area and the base address of libc heap. For WAMR, it is recommended to export them as when there are no possible memory grow operations, runtime will truncate the linear memory into the size indicated by `__heap_base`, so as to reduce the footprint, or at least one page (64KB) is required by linear memory. + +## Tune the memory usage + +Normally there are some methods to tune the memory usage: +- set the global heap size with `wasm_runtime_full_init` +- set the wasm operand stack size with `wasm_runtime_create_exec_env` or `wasm_runtime_instantiate` +- set the linear memory size +- set the auxiliary stack size +- export `malloc/free` functions to use libc heap and disable app heap +- set the app heap size with `wasm_runtime_instantiate` +- use `nostdlib` mode, add `-Wl,--strip-all`: refer to [How to reduce the footprint](./build_wasm_app.md#2-how-to-reduce-the-footprint) of building wasm app for more details +- use XIP mode, refer to [WAMR XIP (Execution In Place) feature introduction](./xip.md) for more details diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_usage.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_usage.md new file mode 100644 index 000000000..ec0624c6f --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/memory_usage.md @@ -0,0 +1,134 @@ +Memory usage estimation for a module +==================================== + +This document aims to provide information useful to make a rough estimation +of necessary memory to execute a WASM module. + +Instead of trying to cover every possible configurations, +the following configuration is assumed in this document: + +* Module is built with `wasi-sdk` +* Module is loaded with `wasm_runtime_load` +* AOT is used +* WASI is used +* libc heap is used +* app heap is not used +* The pthread implementation in `wasi-libc`, which is based on `wasi-threads` + (`WASM_ENABLE_LIB_WASI_THREADS`) might be used +* The another pthread implementation (`WASM_ENABLE_LIB_PTHREAD`) is not used + +Module +------ + +The memory to store the module binary is allocated by the embedder and +passed to `wasm_runtime_load`. +While WAMR owns the buffer, WAMR might make in-place modifications to +its contents. + +Loaded module and its instances +------------------------------- + +Many of data structures for module and instances are allocated from +the global heap. (aka. `wasm_runtime_malloc`) + +AOT code section +---------------- + +Memory to load AOT machine code section. + +Because this memory needs to be executable, depending on platforms, +it's allocated from a separate allocator. +For example, `mmap` and `mprotect` are used on POSIX-like platforms. + +Linear memory +------------- + +A WASM linear memory is either shared or non-shared. + +A WASM linear memory has `min` and `max` sizes. +(They correspond to `wasm-ld`'s `--init-memory` and `--max-memory` options.) +They are in the number of WASM pages, each of which is of 65536 bytes. +The `max` is optional for non-shared memory. When omitted, it effectivily +means unlimited. + +If `OS_ENABLE_HW_BOUND_CHECK` is enabled, the memory is allocated via +`os_mmap` and `os_mem_commit`/`os_mprotect`. +Otherwise, it's allocated from the global heap. + +If the memory is shared and `OS_ENABLE_HW_BOUND_CHECK` is not enabled, +the `max` size of memory is allocated on instantiation. + +Otherwise, the `min` size of memory is allocated on instantiation. +It can later grow up to the `max` size via the `memory.grow` instruction. + +Libc heap +--------- + +The libc heap is the last (highest address) part of linear memory, +which might be dynamically grown with `memory.grow` instruction, when +necessary to serve memory allocations within the module. + +App heap +-------- + +Not used for the above mentioned configuration. + +You can safely disable the app heap creation by specifying `0` for +the `heap_size` argument of `wasm_runtime_instantiate`. +(It's automatically disabled if malloc/free are exported from the module.) + +WASM stack +---------- + +Operand stack is not used for AOT. + +However, a small amount of WASM stack is used for call frames when +certain features are enabled. +(`WASM_ENABLE_DUMP_CALL_STACK` or `WASM_ENABLE_PERF_PROFILING`) + +It's allocated from the global heap. + +You can specify its size with the `stack_size` argument of +`wasm_runtime_instantiate` and `wasm_runtime_create_exec_env`. +(1 is the minimum because 0 means the default.) + +AUX stack (aka. C shadow stack) +------------------------------- + +For the main thread, it's a part of the linear memory, +between `__data_end` and `__heap_base` symbols. +You can control the size of this stack with `wasm-ld`'s +`-z stack-size` option. + +For threads created by `pthread_create`, libc allocates the stack for +them dynamically from the libc heap. +The size of this stack is inherited from the main thread's one +unless overwritten with `pthread_attr_setstacksize` etc. + +WAMR tries to detect overflow/underflow when updating the stack pointer +global. For threads created by `pthread_create`, the detection mechanism +is disabled as of writing this. + +Native stack +------------ + +The stack of the host environment thread which runs WAMR. + +For threads created by `pthread_create`, WAMR automatically creates +host threads to run those WASM threads. The stack size of these host +threads are controlled by a build-time configuration. +(`APP_THREAD_STACK_SIZE_DEFAULT`) + +In some configurations, runtime overflow can be detected using hardware traps. +(`OS_ENABLE_HW_BOUND_CHECK`) + +In some configurations, explicit overflow detection logic can be emitted +into AOT modules themselves. (cf. `os_thread_get_stack_boundary`, +`check_stack_boundary`, `wamrc --stack-bounds-checks=1/0`) + +Memory profiling +================ + +You can collect and dump detailed information about memory usage +by actually running a module with the `WASM_ENABLE_MEMORY_PROFILING` +build-time option. diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/multi_module.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/multi_module.md new file mode 100644 index 000000000..1c22863f9 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/multi_module.md @@ -0,0 +1,161 @@ +# Multiple Modules as Dependencies + +A WASM module can _import_ _functions_, _globals_, _memories_ and _tables_ from other modules as dependencies. A module can also _export_ those entities for other modules like a library. + +WAMR loads all dependencies recursively according to the _import section_ of a module. + +> WAMR only implements the load-time dynamic linking. Please refer to [dynamic linking](https://webassembly.org/docs/dynamic-linking/) for more details. + +WAMR follows [WASI Command/Reactor Model](https://github.com/WebAssembly/WASI/blob/main/design/application-abi.md#current-unstable-abi). The WASI model separates modules into commands and reactors. A Command is the main module that requires exports of reactors(submodules). + +if `WASM_ENABLE_LIBC_WASI` is enabled, any module imports a WASI APIs, like `(import "wasi_snapshot_preview1" "XXX")`, should follow restrictions of the _WASI application ABI_: + +- a main module(a command) should include `_start()` +- a submodule(a reactor) should include `_initialize()` +- both a command and a reactor should include an exported `memory` + +## Multi-Module Related APIs + +### Register a module + +```c +bool +wasm_runtime_register_module(const char *module_name, + wasm_module_t module, + char *error_buf, + uint32_t error_buf_size); +``` + +It is used to register a _module_ with a _module_name_ to WASM runtime, especially for the main module, which is loaded by `wasm_runtime_load()` and doesn't have a chance to tell runtime its _module name_. + +WAMR will get submodules' names(according to the _import section_ of the main module) and load .wasm files from the filesystem or stream and then register them internally. + +### Find a registered module + +```c +wasm_module_t +wasm_runtime_find_module_registered( + const char *module_name); +``` + +It is used to check whether a module with a given _module_name_ has been registered before or not. Return the module if yes. + +### Module reader and destroyer + +```c +typedef bool (*module_reader)(const char *module_name, + uint8_t **p_buffer, + uint32_t *p_size); + +typedef void (*module_destroyer)(uint8_t *buffer, + uint32_t size); + +void +wasm_runtime_set_module_reader(const module_reader reader, + const module_destroyer destroyer); +``` + +WAMR hopes that the native host or embedding environment loads/unloads the module WASM files by themselves and only passes runtime the binary content without worrying about filesystem or storage issues. `module_reader` and `module_destroyer` are two callbacks called when dynamic-loading/unloading submodules. Developers must implement the two callbacks by themselves. + +### Call function of a submodule + +```c +wasm_function_inst_t +wasm_runtime_lookup_function(wasm_module_inst_t const module_inst, + const char *name, + const char *signature); +``` + +Multi-module allows one to look up an exported function of a submodule. There are two ways to indicate the function _name_: + +- parent function name only by default, used to look up the function of the parent module +- submodule name, function name and two $ symbols, e.g. `$submodule_name$function_name`, used to lookup function of submodule +- `signature` can be NULL + +## Example + +### Attributes in C/C++ + +Suppose there are three C files, _mA.c_, _mB.c_ and _mC.c_. Each of them exports functions and imports from others except mA. + +import/export with two kinds of `__attribute__`: + +- `__attribute__((import_module("MODULE_NAME"))) __attribute__((import_name("FUNCTION_NAME")))`. to indicate dependencies of the current module. + +- `__attribute__((export_name("FUNCTION_NAME")))`. to expose functions. + +```C +// mA.c +__attribute__((export_name("A1"))) int +A1() +{ + return 11; +} +``` + +```C +// mB.c +__attribute__((import_module("mA"))) +__attribute__((import_name("A1"))) extern int +A1(); + +__attribute__((export_name("B1"))) int +B1() +{ + return 21; +} +``` + +### Compile Options + +to generate a wasm module as a command + +``` +$ /path/to/wasi-sdk/bin/clang -o command.wasm main_module.c +``` + +to generate a wasm module as a reactor + +``` +$ /path/to/wasi-sdk/bin/clang -mexec-model=reactor -o reactor.wasm submodule.c +``` + +In the above case, _mA_ and _mB_ are reactors(submodules), _mC_ is the command(main module). Their _import relationships_ will be like: + + + +### libvmlib + +We need to enable _WAMR_BUILD_MULTI_MODULE_ option when building WAMR vmlib. Please ref to [Build WAMR core](./build_wamr.md) for a thoughtful guide. + +### code + +After all the preparation, we can call some functions from native code with APIs + +First, create two callbacks to load WASM module files into memory and unload them later + +```c +static bool +module_reader_cb(const char *module_name, uint8 **p_buffer, uint32 *p_size) +{ + // ... + *p_buffer = (uint8_t *)bh_read_file_to_buffer(wasm_file_path, p_size); + // ... +} + +static void +module_destroyer_cb(uint8 *buffer, uint32 size) +{ + BH_FREE(buffer); +} +``` + +Second, create a large buffer and tell WAMR malloc any resource only from this buffer later. + +More details + +```c +static char sandbox_memory_space[10 * 1024 * 1024] = { 0 }; +``` + +Third, put all together. Please refer to [main.c](../samples/multi-module/src/main.c) diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/other_wasm_compilers.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/other_wasm_compilers.md new file mode 100644 index 000000000..5aa505ab3 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/other_wasm_compilers.md @@ -0,0 +1,78 @@ +
+## Use clang compiler
+
+The recommended method to build a WASM binary is to use clang compiler ```clang-8```. You can refer to [apt.llvm.org](https://apt.llvm.org) for the detailed instructions. Here are referenced steps to install clang-8 in Ubuntu 16.04 and Ubuntu 18.04.
+
+(1) Add source to your system source list from llvm website
+
+For Ubuntu 16.04, add the following lines to /etc/apt/sources.list:
+
+``` Bash
+deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial main
+deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial main
+# 8
+deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main
+deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main
+# 9
+deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-9 main
+deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-9 main
+```
+
+For Ubuntu 18.04, add the following lines to /etc/apt/sources.list:
+
+``` Bash
+# i386 not available
+deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic main
+deb-src http://apt.llvm.org/bionic/ llvm-toolchain-bionic main
+# 8
+deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-8 main
+deb-src http://apt.llvm.org/bionic/ llvm-toolchain-bionic-8 main
+# 9
+deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main
+deb-src http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main
+```
+
+(2) Download and install clang-8 tool-chain using following commands:
+
+``` Bash
+sudo wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
+# Fingerprint: 6084 F3CF 814B 57C1 CF12 EFD5 15CF 4D18 AF4F 7421
+sudo apt-get update
+sudo apt-get install llvm-8 lld-8 clang-8
+```
+
+(3) Create a soft link under /usr/bin:
+
+``` Bash
+cd /usr/bin
+sudo ln -s wasm-ld-8 wasm-ld
+```
+
+(4) Use the clang-8 command below to build the WASM C source code into the WASM binary.
+
+``` Bash
+clang-8 --target=wasm32 -O3 \
+ -z stack-size=4096 -Wl,--initial-memory=65536 \
+ -Wl,--allow-undefined,--export=main \
+ -Wl,--strip-all,--no-entry -nostdlib \
+ -o test.wasm test.c
+```
+
+You will get ```test.wasm``` which is the WASM app binary.
+
+## Using Docker
+
+Another method availble is using [Docker](https://www.docker.com/). We assume you've already configured Docker (see Platform section above) and have a running interactive shell. Currently the Dockerfile only supports compiling apps with clang, with Emscripten planned for the future.
+
+Use the clang-8 command below to build the WASM C source code into the WASM binary.
+
+``` Bash
+clang-8 --target=wasm32 -O3 \
+ -z stack-size=4096 -Wl,--initial-memory=65536 \
+ -Wl,--allow-undefined,--export=main \
+ -Wl,--strip-all,--no-entry -nostdlib \
+ -o test.wasm test.c
+```
+
+You will get ```test.wasm``` which is the WASM app binary.
+
diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/app_framework.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/app_framework.PNG Binary files differnew file mode 100644 index 000000000..802c45ba2 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/app_framework.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/embed.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/embed.PNG Binary files differnew file mode 100644 index 000000000..68340ee93 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/embed.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/extend_library.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/extend_library.PNG Binary files differnew file mode 100644 index 000000000..abf10cab3 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/extend_library.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/multi_module_pic1.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/multi_module_pic1.png Binary files differnew file mode 100644 index 000000000..f0c8e33a3 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/multi_module_pic1.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/native_call_wasm.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/native_call_wasm.PNG Binary files differnew file mode 100644 index 000000000..934af72f5 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/native_call_wasm.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/request.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/request.PNG Binary files differnew file mode 100644 index 000000000..281115be2 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/request.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/safe.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/safe.PNG Binary files differnew file mode 100644 index 000000000..ee3ed21e6 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/safe.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sensor_callflow.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sensor_callflow.PNG Binary files differnew file mode 100644 index 000000000..12839aa5a --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sensor_callflow.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sub.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sub.PNG Binary files differnew file mode 100644 index 000000000..25607a538 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/sub.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo.png Binary files differnew file mode 100644 index 000000000..3cb7eb464 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo2.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo2.png Binary files differnew file mode 100644 index 000000000..8b07e6275 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo2.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo_linux.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo_linux.png Binary files differnew file mode 100644 index 000000000..848537382 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_demo_linux.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_linux.PNG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_linux.PNG Binary files differnew file mode 100644 index 000000000..65ce06307 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/vgl_linux.PNG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr-arch.JPG b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr-arch.JPG Binary files differnew file mode 100644 index 000000000..759ed3888 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr-arch.JPG diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_memory_model.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_memory_model.png Binary files differnew file mode 100644 index 000000000..d01d37820 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_memory_model.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_menu_config.png b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_menu_config.png Binary files differnew file mode 100644 index 000000000..c85409584 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pics/wamr_menu_config.png diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/port_wamr.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/port_wamr.md new file mode 100644 index 000000000..7899ff318 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/port_wamr.md @@ -0,0 +1,75 @@ + +WAMR porting guide +========================= + + +This document describes how to port WAMR to a new platform "**new-os**" + + + +# Step 1: Implement platform API layer + +------------------------- +Firstly create the folder for platform API layer implementations: +* for common platforms, create a folder in **`core/shared/platform/new-os`** in WAMR repository folder, so the implementation can be upstreamed +* for platforms that are internal and its implementation shouldn't be published, it's recommended to create a folder outside of the WAMR repository folder (e.g. have separate repository for platform API layer implementation) + +In the folder you just created, you must provide the following files: + +- `platform_internal.h`: It can be used for any platform-specific definitions such as macros, data types and internal APIs. + +- `shared_platform.cmake`: the cmake file will be included by the building script. It is recommended to add a definition for your platform: + + - ```cmake + add_definitions(-DBH_PLATFORM_YOUR_NAME) + ``` + +Then go to implement the APIs defined in the following header files for the platform abstraction layer: + +- [`platform_api_vmcore.h`](../core/shared/platform/include/platform_api_vmcore.h): mandatory for building mini-product (vmcore only). Part of the APIs is needed only for Ahead of Time compilation support. +- [`platform_api_extension.h`](../core/shared/platform/include/platform_api_extension.h): mandatory for app-mgr and app-framework. Given that the app-mgr and app-framework are not required for your target platform, you won't have to implement the API defined in the `platform_api_extension.h`. + + + +**common/posix:** + +There is posix based implementation of the platform API located in the `platform/common/posix` folder. You can include it if your platform supports posix API. refer to platform linux implementation. + + + +**common/math:** + +Some platforms such as ZephyrOS don't provide math functions e.g. sqrt, fabs and isnan, then you should include source files under the folder `platform/common/math`. + + + +# Step 2: Create the mini product for the platform + +------------------------- +You can build a mini WAMR product which is only the vmcore for your platform. Normally you need to implement the main function which loads a WASM file and run it with the WASM runtime. You don't have to do this step if there is no mini-product need for your platform porting. + + + +Firstly create folder **product-mini/platforms/new-os** for the platform mini product build, then refer to the linux platform mini-product for creating the CMakeList.txt and the C implementations. + + + +You should set cmake variable `WAMR_BUILD_PLATFORM` to your platform name while building the mini product. It can be done in the mini product CMakeList.txt file, or pass arguments to cmake command line like: + +``` +mkdir build +cd build +cmake .. -DWAMR_BUILD_PLATFORM=new-os +``` + +For platform implementations that are outside of the WAMR repository (e.g. internal platforms), you also need to provide `SHARED_PLATFORM_CONFIG` path: + +``` +cmake .. -DWAMR_BUILD_PLATFORM=new-os -DSHARED_PLATFORM_CONFIG=/path/to/new-os/shared_platform.cmake +``` + + +Refer to [build_wamr.md](./build_wamr.md) for the building configurations and parameters. + + + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_impls.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_impls.md new file mode 100644 index 000000000..92b51cb4a --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_impls.md @@ -0,0 +1,59 @@ +# Pthread implementations + +WAMR has two pthread implementations available as of writing this. + +These implementations are not ABI-compatible. You at least need to rebuild +your wasm modules when migrating from one pthread implementation to another. + +For new users, we recommend to use (or at least experiment) +the new wasi-threads based implementation. +In future, we might remove the old implementation. + +## WAMR lib-pthread (old) + + * The pthread API is directly implemented as host functions in WAMR. + (`WAMR_BUILD_LIB_PTHREAD`) + + * Only minimum API is implemented as of writing this. + (eg. no pthread barriers) + + * WAMR-specific ABI + + * [Known limitations](pthread_library.md#known-limits) + +## wasi-threads (new) + + * The pthread API is implemented in wasi-libc, based on + [wasi-threads](https://github.com/WebAssembly/wasi-threads) + and [WASM threads](https://github.com/WebAssembly/threads) proposals. + + * It requires a recent-enough version of wasi-libc. The experimental support + is included in + [wasi-sdk 20.0](https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-20) + or later. + To build your application, cmake users can use the + [cmake toolchain file](https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk-pthread.cmake) + provided by wasi-sdk. + + * wasi-threads is implemented as a host function in WAMR. + (`WAMR_BUILD_LIB_WASI_THREADS`) + + * The ABI is specified in wasi-threads proposal. + You can run the same wasm modules on other runtimes which implement + the proposal. (wasmtime, toywasm, ...) + + * Basically more feature-rich and complete than WAMR lib-pthread. + + **EXCEPTION**: `pthread_exit` is not available as of writing this. + If `pthread_exit` is important for your use cases, please speak up in + the [GitHub issue](https://github.com/WebAssembly/wasi-threads/issues/7). + + **EXCEPTION**: For threads created by `pthread_create`, the AUX stack + (aka C shadow stack) overflow detection mechanism is disabled as of + writing this. + If it's important for your use cases, please speak up in the + [GitHub issue](https://github.com/WebAssembly/wasi-threads/issues/12). + +# References + +* https://github.com/bytecodealliance/wasm-micro-runtime/issues/1790 diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_library.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_library.md new file mode 100644 index 000000000..ba43ee1c0 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/pthread_library.md @@ -0,0 +1,198 @@ +# WAMR pthread library + +**Note**: This document describes the old pthread implementation. +See [Pthread implementations](pthread_impls.md). + +WAMR provides a built-in library to support pthread APIs. You can call pthread APIs in your application source code. + +## Build and run +Suppose you have written a C program calling pthread_create() to create a thread, and the file name is main.c +``` C +#include <stdio.h> +#include <pthread.h> + +void *thread_routine(void *arg) +{ + printf("Enter thread\n"); + pthread_exit(NULL); + return NULL; +} + +int main(int argc, char** argv) +{ + pthread_t tid; + + if (0 != pthread_create(&tid, NULL, thread_routine, NULL)) { + printf("Failed to create thread\n"); + } + + if (0 != pthread_join(tid, NULL)) { + printf("Failed to join thread %d.\n", tid); + } + + printf("Exit\n"); + + return 0; +} +``` +**Build with libc-builtin** + +To build this C program into WebAssembly app with libc-builtin, you can use this command: +``` bash +/opt/wasi-sdk/bin/clang --target=wasm32 \ + --sysroot=${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot \ + -O3 -pthread -nostdlib -z stack-size=32768 \ + -Wl,--shared-memory \ + -Wl,--initial-memory=131072,--max-memory=131072 \ + -Wl,--allow-undefined-file=${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot/share/defined-symbols.txt \ + -Wl,--no-entry -Wl,--export=main \ + -Wl,--export=__heap_base,--export=__data_end \ + -Wl,--export=__wasm_call_ctors \ + main.c -o test.wasm +# -pthread: it will enable some dependent WebAssembly features for thread +# -nostdlib: disable the WASI standard library as we are using WAMR builtin-libc +# -z stack-size=: specify the total aux stack size +# -Wl,--export=__heap_base,--export=__data_end: export these globals so the runtime can resolve the total aux stack size and the start offset of the stack top +# -Wl,--export=__wasm_call_ctors: export the init function to initialize the passive data segments +``` + +**Build with libc-WASI** + +You can also build this program with WASI, but we need to make some changes to wasi-sysroot: + +1. disable malloc/free of wasi, as they are not atomic operations: + ``` bash + /opt/wasi-sdk/bin/llvm-ar -d /opt/wasi-sdk/share/wasi-sysroot/lib/wasm32-wasi/libc.a dlmalloc.o + ``` +2. copy the pthread.h to wasi-sysroot so the compiler can find it: + ``` bash + cp ${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot/include/pthread.h /opt/wasi-sdk/share/wasi-sysroot/include + ``` +> Note: </br> +>1. Remember to back up the original sysroot files + +Then build the program with this command: +``` bash +/opt/wasi-sdk/bin/clang -pthread -O3 \ + -Wl,--shared-memory,--max-memory=196608 \ + -Wl,--allow-undefined,--no-check-features \ + -Wl,--export=__heap_base,--export=__data_end \ + main.c -o test.wasm +# -Wl,--no-check-features: the errno.o in wasi-sysroot is not compatible with pthread feature, pass this option to avoid errors +``` + +**Build with EMCC** + +> Note: This document is based on `emcc 2.0.26`, other version may not work with these commands + +EMCC's `-pthread` option is not compatible with standalone mode, we need to pass `-mbulk-memory -matomics` to the compiler and `--shared-memory,--no-check-features` to linker manually + +EMCC provides some empty implementation for pthread related APIs, we need to remove them from emcc's libc. +``` bash +cd ${emsdk_dir}/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten +emar d libc.a library_pthread_stub.o +emranlib libc.a +``` + +``` bash +emcc -O3 -mbulk-memory -matomics -s MALLOC="none" \ + -Wl,--export=__data_end,--export=__heap_base \ + -Wl,--shared-memory,--no-check-features \ + -s ERROR_ON_UNDEFINED_SYMBOLS=0 \ + main.c -o test.wasm +``` + +**Build AOT module** + +You can build the wasm module into AOT module with pthread support, please pass option `--enable-multi-thread` to wamrc: +``` bash +wamrc --enable-multi-thread -o test.aot test.wasm +``` + +Currently WAMR disables pthread library by default. To run the module with pthread support, please build the runtime with `-DWAMR_BUILD_LIB_PTHREAD=1` +``` bash +cd ${WAMR_ROOT}/product-mini/platforms/linux +mkdir build && cd build +cmake .. -DWAMR_BUILD_LIB_PTHREAD=1 +make +# Then you can run the wasm module above: +./iwasm test.wasm +# Or the AOT module: +# ./iwasm test.aot +``` + +[Here](../samples/multi-thread) is also a sample to show how wasm-apps use pthread APIs to create threads, and how to build it with cmake. You can build this sample and have a try: +``` bash +cd ${WAMR_ROOT}/samples/multi-thread +mkdir build && cd build +cmake .. +make +# Run wasm application +./iwasm wasm-apps/test.wasm +``` + + +## Aux stack seperation +The compiler may use some spaces in the linear memory as an auxiliary stack. When pthread is enabled, every thread should have its own aux stack space, so the total aux stack space reserved by the compiler will be divided into N + 1 parts, where N is the maximum number of threads that can be created by the user code. + +The default value of N is 4, which means you can create 4 threads at most. This value can be changed by an option if you are using product-mini: +``` bash +./iwasm --max-threads=n test.wasm +``` +If you are going to develop your own runtime product, you can use the API `wasm_runtime_set_max_thread_num` or init arg `init_args.max_thread_num` to set the value, or you can change the macro `CLUSTER_MAX_THREAD_NUM` in [config.h](../core/config.h). + +> Note: the total size of aux stack reserved by compiler can be set with `-z stack-size` option during compilation. If you need to create more threads, please set a larger value, otherwise it is easy to cause aux stack overflow. + +## Supported APIs +``` C +/* Thread APIs */ +int pthread_create(pthread_t *thread, const void *attr, + void *(*start_routine) (void *), void *arg); + +int pthread_join(pthread_t thread, void **retval); + +int pthread_detach(pthread_t thread); + +int pthread_cancel(pthread_t thread); + +pthread_t pthread_self(void); + +void pthread_exit(void *retval); + +/* Mutex APIs */ +int pthread_mutex_init(pthread_mutex_t *mutex, const void *attr); + +int pthread_mutex_lock(pthread_mutex_t *mutex); + +int pthread_mutex_unlock(pthread_mutex_t *mutex); + +int pthread_mutex_destroy(pthread_mutex_t *mutex); + +/* Cond APIs */ +int pthread_cond_init(pthread_cond_t *cond, const void *attr); + +int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex); + +int pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, + unsigned int useconds); + +int pthread_cond_signal(pthread_cond_t *cond); + +int pthread_cond_broadcast(pthread_cond_t *cond); + +int pthread_cond_destroy(pthread_cond_t *cond); + +/* Pthread key APIs */ +int pthread_key_create(pthread_key_t *key, void (*destructor)(void *)); + +int pthread_setspecific(pthread_key_t key, const void *value); + +void *pthread_getspecific(pthread_key_t key); + +int pthread_key_delete(pthread_key_t key); +``` + +## Known limits +- `pthread_attr_t`, `pthread_mutexattr_t` and `pthread_condattr_t` are not supported yet, so please pass `NULL` as the second argument of `pthread_create`, `pthread_mutex_init` and `pthread_cond_init`. +- The `errno.o` in wasi-sysroot is not compatible with this feature, so using errno in multi-thread may cause unexpected behavior. +- Currently `struct timespec` is not supported, so the prototype of `pthread_cond_timedwait` is different from the native one, it takes an unsigned int argument `useconds` to indicate the waiting time. diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/ref_types.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/ref_types.md new file mode 100644 index 000000000..7fefa999d --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/ref_types.md @@ -0,0 +1,9 @@ +# WAMR reference-types introduction + +WebAssembly [reference-types](https://github.com/WebAssembly/reference-types) proposal introduces two new types `funcref` and `externref`. With `externref`, It is easier and more efficient to interoperate with host environment. Host references are able to be represented directly by type `externref`. + +WAMR has implemented the reference-types proposal. WAMR allows a native method to pass a host object to a WASM application as an `externref` parameter or receives a host object from a WASM application as an `externref` result. Internally, WAMR won't try to parse or dereference `externref`. It is an opaque type. + +The restriction of using `externref` in a native method is the host object has to be the value of a `unintptr_t` variable. In other words, it takes **8 bytes** on 64-bit machine and **4 bytes** on 32-bit machines. Please keep that in mind especially when calling `wasm_runtime_call_wasm`. + +Please ref to the [sample](../samples/ref-types) for more details. diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/release_ack.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/release_ack.md new file mode 100644 index 000000000..a0adfff41 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/release_ack.md @@ -0,0 +1,61 @@ +Major feature releases and contributors +========================================= + + +**May 07, 2019: WAMR first GitHub release** + +- Contributors: Wenyong Huang, Weining Lu, Lei Shi, Li Tian, Jizhao Zhang, Yi Zhang, Daoming Qiu, Xin Wang (Intel) + +**May 17, 2019: Application manager, WASM APP API, samples and test tools** + +- Contributors: Wenyong Huang, Weining Lu, Lei Shi, Li Tian, Jizhao Zhang, Yi Zhang, Daoming Qiu, Xin Wang (Intel) + + +**May 23, 2019: Support AliOS Things** + +- Contributor: JinZhou Zhu (Alibaba) + +**May 24, 2019: Support memory usage profiler** + +- Contributors Wenyong Huang (Intel) + +**Jun 11, 2019: Add WASM APP API connection** + + +- Contributor: Weining Lu (Intel) + +**Jun 10, 2019: Support VxWorks** + +- Contributor: Yiting Wang (WindRiver) + +**Aug 1, 2019: Add WGL graphic user interface API** + +- Contributor: Weining Lu + +**Aug 14, 2019: Add Docker support** + + +- Contributor: beriberikix + + +**Aug 14, 2019: WASM IoT app store demo** + + +- Contributor: Luhanzhi Li, Jun Xu (Intel) + + +**Aug 28, 2019: SGX support** + + +- Contributor: Mic Bowman (Intel) + + +**Sep 6, 2019: Mac platform support** + + +- Contributor: Jonathan Dong (Alibaba) + +**Nov 2019: WASI support** (Intel) + +**Jan 2020: Ahead of time and Just-in-Time compilation support** (Intel) + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/roadmap.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/roadmap.md new file mode 100644 index 000000000..c5c7b84a9 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/roadmap.md @@ -0,0 +1,23 @@ + +# WebAssembly Micro Runtime Roadmap + + + +## Data serialization +Evaluating using cbor as the default data serialization + +No plan yet. + + + +## Threading +Plan: 2020 Q1 + + + +## AssemblyScript Support and API + +Currently under evaluation + + + diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/semantic_version.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/semantic_version.md new file mode 100644 index 000000000..d0d46a5e2 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/semantic_version.md @@ -0,0 +1,21 @@ +# WAMR uses semantic versioning + +WAMR uses the _semantic versioning_ to replace the current _date versioning_ system. + +There are three parts in the new version string: + +- _major_. Any incompatible modification, on both ABI and APIs, will lead an increment + in the value of _major_. APIs includes: `wasm_export.h`, `wasm_c_api.h`, + _sections in AOT files_, and so on. +- _minor_. It represents new features. It includes not just MVP or POST-MVP features + but also WASI features and WAMR private ones. +- _patch_. It represents patches. + +## Legacy versions + +All legacy versions(tags) will keep their current status. No existed releasings names +and links will be changed. + +## Reference + +- [Semantic Versioning 2.0.0](https://semver.org/) diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/socket_api.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/socket_api.md new file mode 100644 index 000000000..9e65d33cd --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/socket_api.md @@ -0,0 +1,89 @@ +# How to use Berkeley/Posix Socket APIs in WebAssembly + +**_Berkeley sockets_** usually means an API for Internet sockets and Unix domain +sockets. A socket is an abstract representation of the local endpoint of a +network communication path. + +Currently, WAMR supports some Socket API features: +- Support TCP and UDP +- Support IPv4 and IPv6 +- Support get/set socket options +- Support access control + +This document introduces a way to support the _Berkeley/POSIX Socket API_ in +WebAssembly code. + +## Patch the native code + +The first step is to include a header file of the WAMR socket extension in the +native source code. + +```c +#ifdef __wasi__ +#include <wasi_socket_ext.h> +#endif +``` + +`__wasi__` is a macro defined by WASI. The host compiler will not enable it. + +## CMake files + +It is recommended that the project should use CMake as its build system. Use +[_wasi-sdk_](https://github.com/WebAssembly/wasi-sdk) +as a toolchain to compile C/C++ to WebAssembly + +```bash +$ cmake -DWASI_SDK_PREFIX=${WASI_SDK_DIR} + -DCMAKE_TOOLCHAIN_FILE=${WASI_TOOLCHAIN_FILE} + -DCMAKE_SYSROOT=${WASI_SYS_ROOT} + .. +``` + +In the *CMakeLists.txt*, include an extension of socket support and link with it. + +```cmake +include(${CMAKE_CURRENT_SOURCE_DIR}/../../../core/iwasm/libraries/lib-socket/lib_socket_wasi.cmake) +add_executable(socket_example tcp_server.c) +target_link_libraries(socket_example socket_wasi_ext) +``` + +Now, the native code with socket APIs is ready for compilation. + +## Run with iwasm + +If having the _.wasm_, the last step is to run it with _iwasm_. + +The _iwasm_ should be compiled with `WAMR_BUILD_LIBC_WASI=1`. By default, it is +enabled. + +_iwasm_ accepts address ranges via an option, `--addr-pool`, to implement +the capability control. All IP address the WebAssembly application may need to `bind()` or `connect()` +should be announced first. Every IP address should be in CIRD notation. + +```bash +$ iwasm --addr-pool=1.2.3.4/15,2.3.4.6/16 socket_example.wasm +``` + +_iwasm_ also accepts list of domain names and domain name patterns for the address resolution via an option, `--allow-resolve`, to implement the capability control. Every domain that will be resolved using `sock_addr_resolve` needs to be added to the allowlist first. + +```bash +$ iwasm --allow-resolve=*.example.com --allow-resolve=domain.com +``` + +The example above shows how to allow for resolving all `example.com`'s subdomains (e.g. `x.example.com`, `a.b.c.example.com`) and `domain.com` domain. + +Refer to [socket api sample](../samples/socket-api) for more details. + +## Intel SGX support + +WAMR also supports the socket API within Intel SGX enclaves. + +The _iwasm_ should be compiled with `WAMR_BUILD_LIBC_WASI=1` and `WAMR_BUILD_LIB_PTHREAD=1`, which are enabled by default. + +Similarly to running _iwasm_ outside of an enclave, the allowed address ranges are given via the option `--addr-pool`. + +```bash +$ iwasm --addr-pool=1.2.3.4/15,2.3.4.6/16 socket_example.wasm +``` + +Refer to [socket api sample](../samples/socket-api) for the compilation of the Wasm applications and [_iwasm_ for Intel SGX](../product-mini/platforms/linux-sgx) for the Wasm runtime. diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/source_debugging.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/source_debugging.md new file mode 100644 index 000000000..a9fa09307 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/source_debugging.md @@ -0,0 +1,224 @@ +# WAMR source debugging + +References: +- [Blog: WAMR source debugging basic](https://bytecodealliance.github.io/wamr.dev/blog/wamr-source-debugging-basic/) +- [Blog: Debugging wasm with VSCode](https://bytecodealliance.github.io/wamr.dev/blog/debugging-wasm-with-vscode/) + +WAMR supports source level debugging based on DWARF (normally used in C/C++/Rust), source map (normally used in AssemblyScript) is not supported. + +**The lldb's ability to debug wasm application is based on the patch [Add class WasmProcess for WebAssembly debugging](https://reviews.llvm.org/D78801). Thanks very much to the author @paolosev for such a great work!** + +## Build wasm application with debug information +To debug your application, you need to compile them with debug information. You can use `-g` option when compiling the source code if you are using wasi-sdk (also work for emcc and rustc): +``` bash +/opt/wasi-sdk/bin/clang -g test.c -o test.wasm +``` + +Then you will get `test.wasm` which is a WebAssembly module with embedded DWARF sections. Further, you can use `llvm-dwarfdump` to check if the generated wasm file contains DWARF information: +``` bash +llvm-dwarfdump-12 test.wasm +``` + +## Debugging with interpreter +1. Install dependent libraries +``` bash +apt update && apt install cmake make g++ libxml2-dev -y +``` + +2. Build iwasm with source debugging feature +``` bash +cd ${WAMR_ROOT}/product-mini/platforms/linux +mkdir build && cd build +cmake .. -DWAMR_BUILD_DEBUG_INTERP=1 +make +``` +> Note: On MacOS M1 environment, pass the additional `-DWAMR_DISABLE_HW_BOUND_CHECK=1` cmake configuration. + +3. Execute iwasm with debug engine enabled +``` bash +iwasm -g=127.0.0.1:1234 test.wasm +# Use port = 0 to allow a random assigned debug port +``` + +4. Build customized lldb +``` bash +git clone --branch release/13.x --depth=1 https://github.com/llvm/llvm-project +cd llvm-project +git apply ${WAMR_ROOT}/build-scripts/lldb-wasm.patch +mkdir build-lldb +cmake -S ./llvm -B build-lldb \ + -G Ninja \ + -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;lldb" \ + -DLLVM_TARGETS_TO_BUILD:STRING="X86;WebAssembly" \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DLLVM_BUILD_BENCHMARKS:BOOL=OFF \ + -DLLVM_BUILD_DOCS:BOOL=OFF -DLLVM_BUILD_EXAMPLES:BOOL=OFF \ + -DLLVM_BUILD_LLVM_DYLIB:BOOL=OFF -DLLVM_BUILD_TESTS:BOOL=OFF \ + -DLLVM_ENABLE_BINDINGS:BOOL=OFF -DLLVM_INCLUDE_BENCHMARKS:BOOL=OFF \ + -DLLVM_INCLUDE_DOCS:BOOL=OFF -DLLVM_INCLUDE_EXAMPLES:BOOL=OFF \ + -DLLVM_INCLUDE_TESTS:BOOL=OFF -DLLVM_ENABLE_LIBXML2:BOOL=ON +cmake --build build-lldb --target lldb --parallel $(nproc) +# The lldb is generated under build-lldb/bin/lldb +``` +> Note: If using `CommandLineTools` on MacOS, make sure only one SDK is present in `/Library/Developer/CommandLineTools/SDKs`. + +> You can download pre-built `wamr-lldb` binaries from [here](https://github.com/bytecodealliance/wasm-micro-runtime/releases). + +5. Launch customized lldb and connect to iwasm +``` bash +lldb +(lldb) process connect -p wasm connect://127.0.0.1:1234 +``` +Then you can use lldb commands to debug your applications. Please refer to [lldb document](https://lldb.llvm.org/use/tutorial.html) for command usage. + +## Debugging with AOT + +> Note: AOT debugging is experimental and only a few debugging capabilities are supported. + +1. Build lldb (assume you have already built llvm) +``` bash +cd ${WAMR_ROOT}/core/deps/llvm/build +cmake ../llvm -DLLVM_ENABLE_PROJECTS="clang;lldb" -DLLDB_INCLUDE_TESTS=OFF +make -j $(nproc) +``` + +2. Build wamrc with debugging feature +``` bash +cd ${WAMR_ROOT}/wamr-compiler +mkdir build && cd build +cmake .. -DWAMR_BUILD_DEBUG_AOT=1 +make -j $(nproc) +``` + +3. Build iwasm with debugging feature +``` bash +cd ${WAMR_ROOT}/product-mini/platforms/linux +mkdir build && cd build +cmake .. -DWAMR_BUILD_DEBUG_AOT=1 +make +``` + +4. Compile wasm module to AOT module +``` bash +wamrc -o test.aot test.wasm +``` + +5. Execute iwasm using lldb + + Then you can use lldb commands to debug both wamr runtime and your wasm application in ***current terminal***. + + ``` bash + % lldb iwasm -- test.aot + (lldb) target create "iwasm" + Current executable set to 'iwasm' (x86_64). + (lldb) settings set -- target.run-args "test.aot" + (lldb) settings set plugin.jit-loader.gdb.enable on + (lldb) b main + Breakpoint 1: where = iwasm`main + 48 at main.c:294:11, address = 0x0000000100001020 + (lldb) run + Process 27954 launched: '/tmp/bin/iwasm' (x86_64) + Process 27954 stopped + * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 + frame #0: 0x0000000100001020 iwasm`main(argc=2, argv=0x00007ff7bfeff678) at main.c:294:11 + 291 int + 292 main(int argc, char *argv[]) + 293 { + -> 294 int32 ret = -1; + 295 char *wasm_file = NULL; + 296 const char *func_name = NULL; + 297 uint8 *wasm_file_buf = NULL; + Target 0: (iwasm) stopped. + (lldb) c + Process 27954 resuming + 1 location added to breakpoint 1 + error: need to add support for DW_TAG_base_type 'void' encoded with DW_ATE = 0x0, bit_size = 0 + Process 27954 stopped + * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2 + frame #0: 0x00000001002980a0 JIT(0x100298004)`main(exenv=0x0000000301808200) at hello.c:6:9 + 3 int + 4 main(void) + 5 { + -> 6 printf("hello\n"); + 7 + 8 return 0; + 9 } + Target 0: (iwasm) stopped. + (lldb) br l + Current breakpoints: + 1: name = 'main', locations = 2, resolved = 2, hit count = 2 + 1.1: where = iwasm`main + 48 at main.c:294:11, address = 0x0000000100001020, resolved, hit count = 1 + 1.2: where = JIT(0x100298004)`main + 12 at hello.c:6:9, address = 0x00000001002980a0, resolved, hit count = 1 + + (lldb) + ``` + + * In the above example, + + * The first `main` function, which is in `main.c`, is the main + function of the iwasm command. + + * The second `main` function, which is in `hello.c`, is the main + function of the AOT-compiled wasm module. + + * WAMR AOT debugging uses the GDB JIT loader mechanism to load + the debug info of the debugee module. + On some platforms including macOS, you need to enable it explicitly. + (`settings set plugin.jit-loader.gdb.enable on`) + + References: + + * https://github.com/llvm/llvm-project/blob/main/llvm/docs/DebuggingJITedCode.rst + * https://sourceware.org/gdb/current/onlinedocs/gdb/JIT-Interface.html + +## Enable debugging in embedders (for interpreter) + +There are three steps to enable debugging in embedders + +1. Set the debug parameters when initializing the runtime environment: + ``` c + RuntimeInitArgs init_args; + memset(&init_args, 0, sizeof(RuntimeInitArgs)); + + /* ... */ + strcpy(init_args.ip_addr, "127.0.0.1"); + init_args.instance_port = 1234; + /* + * Or set port to 0 to use a port assigned by os + * init_args.instance_port = 0; + */ + + if (!wasm_runtime_full_init(&init_args)) { + return false; + } + ``` + +2. Use `wasm_runtime_start_debug_instance` to create the debug instance: + ``` c + /* + initialization, loading and instantiating + ... + */ + exec_env = wasm_runtime_create_exec_env(module_inst, stack_size); + uint32_t debug_port = wasm_runtime_start_debug_instance(exec_env); + ``` + +3. Enable source debugging features during building + + You can use `-DWAMR_BUILD_DEBUG_INTERP=1` during cmake configuration + + Or you can set it directly in `cmake` files: + ``` cmake + set (WAMR_BUILD_DEBUG_INTERP 1) + ``` + +### Attentions +- Debugging `multi-thread wasm module` is not supported, if your wasm module use pthread APIs (see [pthread_library.md](./pthread_library.md)), or the embedder use `wasm_runtime_spawn_thread` to create new wasm threads, then there may be **unexpected behaviour** during debugging. + + > Note: This attention is about "wasm thread" rather than native threads. Executing wasm functions in several different native threads will **not** affect the normal behaviour of debugging feature. + +- When using source debugging features, **don't** create multiple `wasm_instance` from the same `wasm_module`, because the debugger may change the bytecode (set/unset breakpoints) of the `wasm_module`. If you do need several instance from the same bytecode, you need to copy the bytecode to a new butter, then load a new `wasm_module`, and then instantiate the new wasm module to get the new instance. + +- If you are running `lldb` on non-linux platforms, please use `platform select remote-linux` command in lldb before connecting to the runtime: + ``` + (lldb) platform select remote-linux + (lldb) process connect -p wasm connect://127.0.0.1:1234 + ``` diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wamr_api.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wamr_api.md new file mode 100644 index 000000000..3eff86927 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wamr_api.md @@ -0,0 +1,351 @@ + +WAMR application framework +======================== + +## Application system callbacks +The `on_init` and `on_destroy` functions are wamr application system callbacks which must be implemented in the wasm application if you want to use the APP framework. +``` C +void on_init() +{ + /* + Your init functions here, for example: + * platform initialization + * timer registration + * service / event registration + * ...... + */ +} + +void on_destroy() +{ + /* + your destroy functions here + */ +} +``` + +## Base App library + +The base library of application framework supports the essential API for WASM applications, such as inter-app communication, timers, etc. Other application framework components rely on the base library. + +When building the WAMR SDK, once application framework is enabled, the base library will automatically enabled. + +### Timer +The *timer* API's can be used to create some `soft timers` with single-shot mode or periodic mode. Here is a reference of how to use timer API's to execute a function every one second. +``` C +/* User global variable */ +static int num = 0; + +/* Timer callback */ +void timer1_update(user_timer_t timer) +{ + printf("Timer update %d\n", num++); +} + +void on_init() +{ + user_timer_t timer; + + /* set up a timer */ + timer = api_timer_create(1000, true, false, timer1_update); + api_timer_restart(timer, 1000); +} + +void on_destroy() +{ + +} +``` + +### Micro-service model (request/response) +The microservice model is also known as request and response model. One WASM application acts as the server which provides a specific service. Other WASM applications or host/cloud applications request that service and get the response. + +<center><img src="./pics/request.PNG" width="60%" height="60%"></center> + +Below is the reference implementation of the server application. It provides room temperature measurement service. + +``` C +void on_init() +{ + api_register_resource_handler("/room_temp", room_temp_handler); +} + +void on_destroy() +{ +} + +void room_temp_handler(request_t *request) +{ + response_t response[1]; + attr_container_t *payload; + payload = attr_container_create("room_temp payload"); + if (payload == NULL) + return; + + attr_container_set_string(&payload, "temp unit", "centigrade"); + attr_container_set_int(&payload, "value", 26); + + make_response_for_request(request, response); + set_response(response, + CONTENT_2_05, + FMT_ATTR_CONTAINER, + payload, + attr_container_get_serialize_length(payload)); + + api_response_send(response); + attr_container_destroy(payload); +} +``` + + +### Pub/sub model +One WASM application acts as the event publisher. It publishes events to notify WASM applications or host/cloud applications which subscribe to the events. + +<center><img src="./pics/sub.PNG" width="60%" height="60%"></center> + +Below is the reference implementation of the pub application. It utilizes a timer to repeatedly publish an overheat alert event to the subscriber applications. Then the subscriber applications receive the events immediately. + +``` C +/* Timer callback */ +void timer_update(user_timer_t timer) +{ + attr_container_t *event; + + event = attr_container_create("event"); + attr_container_set_string(&event, + "warning", + "temperature is over high"); + + api_publish_event("alert/overheat", + FMT_ATTR_CONTAINER, + event, + attr_container_get_serialize_length(event)); + + attr_container_destroy(event); +} + +void on_init() +{ + user_timer_t timer; + timer = api_timer_create(1000, true, true, timer_update); +} + +void on_destroy() +{ +} +``` + +Below is the reference implementation of the sub application. +``` C +void overheat_handler(request_t *event) +{ + printf("Event: %s\n", event->url); + + if (event->payload != NULL && event->fmt == FMT_ATTR_CONTAINER) + attr_container_dump((attr_container_t *) event->payload); +} + +void on_init( +{ + api_subscribe_event ("alert/overheat", overheat_handler); +} + +void on_destroy() +{ +} +``` +**Note:** You can also subscribe this event from host side by using host tool. Please refer `samples/simple` project for detail usage. + + +## Sensor API + +The API set is defined in the header file ```core/app-framework/sensor/app/wa-inc/sensor.h```. + +Here is a reference of how to use sensor API's: + +``` C +static sensor_t sensor = NULL; + +/* Sensor event callback*/ +void sensor_event_handler(sensor_t sensor, attr_container_t *event, + void *user_data) +{ + printf("### app get sensor event\n"); + attr_container_dump(event); +} + +void on_init() +{ + char *user_data; + attr_container_t *config; + + printf("### app on_init 1\n"); + /* open a sensor */ + user_data = malloc(100); + printf("### app on_init 2\n"); + sensor = sensor_open("sensor_test", 0, sensor_event_handler, user_data); + printf("### app on_init 3\n"); + + /* config the sensor */ + sensor_config(sensor, 1000, 0, 0); + printf("### app on_init 4\n"); +} + +void on_destroy() +{ + if (NULL != sensor) { + sensor_config(sensor, 0, 0, 0); + } +} +``` + +## Connection API: + +The API set is defined in the header file `core/app-framework/connection/app/wa-inc/connection.h` + +Here is a reference of how to use connection API's: +``` C +/* User global variable */ +static int num = 0; +static user_timer_t g_timer; +static connection_t *g_conn = NULL; + +void on_data1(connection_t *conn, + conn_event_type_t type, + const char *data, + uint32 len, + void *user_data) +{ + if (type == CONN_EVENT_TYPE_DATA) { + char message[64] = {0}; + memcpy(message, data, len); + printf("Client got a message from server -> %s\n", message); + } else if (type == CONN_EVENT_TYPE_DISCONNECT) { + printf("connection is close by server!\n"); + } else { + printf("error: got unknown event type!!!\n"); + } +} + +/* Timer callback */ +void timer1_update(user_timer_t timer) +{ + char message[64] = {0}; + /* Reply to server */ + snprintf(message, sizeof(message), "Hello %d", num++); + api_send_on_connection(g_conn, message, strlen(message)); +} + +void my_close_handler(request_t * request) +{ + response_t response[1]; + + if (g_conn != NULL) { + api_timer_cancel(g_timer); + api_close_connection(g_conn); + } + + make_response_for_request(request, response); + set_response(response, DELETED_2_02, 0, NULL, 0); + api_response_send(response); +} + +void on_init() +{ + user_timer_t timer; + attr_container_t *args; + char *str = "this is client!"; + + api_register_resource_handler("/close", my_close_handler); + + args = attr_container_create(""); + attr_container_set_string(&args, "address", "127.0.0.1"); + attr_container_set_uint16(&args, "port", 7777); + + g_conn = api_open_connection("TCP", args, on_data1, NULL); + if (g_conn == NULL) { + printf("connect to server fail!\n"); + return; + } + + printf("connect to server success! handle: %p\n", g_conn); + + /* set up a timer */ + timer = api_timer_create(1000, true, false, timer1_update); + api_timer_restart(timer, 1000); +} + +void on_destroy() +{ + +} +``` + +## GUI API + +The API's is listed in header file ```core/app-framework/wgl/app/wa-inc/wgl.h``` which is implemented based on open source 2D graphic library [LVGL](https://docs.lvgl.io/master/index.html). + +``` C +static void btn_event_cb(wgl_obj_t btn, wgl_event_t event); + +uint32_t count = 0; +char count_str[11] = { 0 }; +wgl_obj_t hello_world_label; +wgl_obj_t count_label; +wgl_obj_t btn1; +wgl_obj_t label_count1; +int label_count1_value = 0; +char label_count1_str[11] = { 0 }; + +void timer1_update(user_timer_t timer1) +{ + if ((count % 100) == 0) { + snprintf(count_str, sizeof(count_str), "%d", count / 100); + wgl_label_set_text(count_label, count_str); + } + ++count; +} + +void on_init() +{ + hello_world_label = wgl_label_create((wgl_obj_t)NULL, (wgl_obj_t)NULL); + wgl_label_set_text(hello_world_label, "Hello world!"); + wgl_obj_align(hello_world_label, (wgl_obj_t)NULL, WGL_ALIGN_IN_TOP_LEFT, 0, 0); + + count_label = wgl_label_create((wgl_obj_t)NULL, (wgl_obj_t)NULL); + wgl_obj_align(count_label, (wgl_obj_t)NULL, WGL_ALIGN_IN_TOP_MID, 0, 0); + + btn1 = wgl_btn_create((wgl_obj_t)NULL, (wgl_obj_t)NULL); /*Create a button on the currently loaded screen*/ + wgl_obj_set_event_cb(btn1, btn_event_cb); /*Set function to be called when the button is released*/ + wgl_obj_align(btn1, (wgl_obj_t)NULL, WGL_ALIGN_CENTER, 0, 0); /*Align below the label*/ + + /*Create a label on the button*/ + wgl_obj_t btn_label = wgl_label_create(btn1, (wgl_obj_t)NULL); + wgl_label_set_text(btn_label, "Click ++"); + + label_count1 = wgl_label_create((wgl_obj_t)NULL, (wgl_obj_t)NULL); + wgl_label_set_text(label_count1, "0"); + wgl_obj_align(label_count1, (wgl_obj_t)NULL, WGL_ALIGN_IN_BOTTOM_MID, 0, 0); + + /* set up a timer */ + user_timer_t timer; + timer = api_timer_create(10, true, false, timer1_update); + if (timer) + api_timer_restart(timer, 10); + else + printf("Fail to create timer.\n"); +} + +static void btn_event_cb(wgl_obj_t btn, wgl_event_t event) +{ + if(event == WGL_EVENT_RELEASED) { + label_count1_value++; + snprintf(label_count1_str, sizeof(label_count1_str), + "%d", label_count1_value); + wgl_label_set_text(label_count1, label_count1_str); + } +} + +``` + +Currently supported widgets include button, label, list and check box and more widgets would be provided in future. diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wasm_c_api.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wasm_c_api.md new file mode 100644 index 000000000..131adadd0 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/wasm_c_api.md @@ -0,0 +1,65 @@ +# wasm-c-api introduction + +wasm-c-api is an engine-agnostic API to embed a WASM engine. +In wasm-micro-runtime, it's provided by the header file `wasm_c_api.h`. +Its functionalities are overlapping with `wasm_export.h`, which is +a native API of wasm-micro-runtime. An embedder is supposed to pick +one of these APIs, rather than mixing both of them. + +All samples come from the commit 340fd9528cc3b26d22fe30ee1628c8c3f2b8c53b +of [wasm-c-api](https://github.com/WebAssembly/wasm-c-api). + +Developer can learn these _APIs_ from +[wasm.h](https://github.com/WebAssembly/wasm-c-api/blob/master/include/wasm.h). + +And here are [examples](https://github.com/WebAssembly/wasm-c-api/tree/master/example) which +are helpful. + +## FYI + +- The thread model of _wasm_c_api_ is + + - An `wasm_engine_t` instance may only be created once per process + - Every `wasm_store_t` and its objects may only be accessed in a single thread + +- `wasm_engine_new`, `wasm_engine_new_with_config`, `wasm_engine_new_with_args`, + `wasm_engine_delete`should be called in a thread-safe environment. Such + behaviors are not recommended, and please make sure an appropriate calling + sequence if it has to be. + + - call `wasm_engine_new` and `wasm_engine_delete` in different threads + - call `wasm_engine_new` or `wasm_engine_delete` multiple times in + different threads + +## unspported list + +Currently WAMR supports most of the APIs, the unsupported APIs are listed as below: + +- References + +```c +WASM_API_EXTERN own wasm_shared_##name##_t* wasm_##name##_share(const wasm_##name##_t*); +WASM_API_EXTERN own wasm_##name##_t* wasm_##name##_obtain(wasm_store_t*, const wasm_shared_##name##_t*); +``` + +- Several Module APIs + +```c +WASM_API_EXTERN void wasm_module_serialize(const wasm_module_t*, own wasm_byte_vec_t* out); +WASM_API_EXTERN own wasm_module_t* wasm_module_deserialize(wasm_store_t*, const wasm_byte_vec_t*); +``` + +Currently growing a table or memory by wasm opcode is supported and it is not supported to grow them +by host-side function callings. + +- Table Grow APIs + +```c +WASM_API_EXTERN bool wasm_table_grow(wasm_table_t*, wasm_table_size_t delta, wasm_ref_t* init); +``` + +- Memory Grow APIs + +```c +WASM_API_EXTERN bool wasm_memory_grow(wasm_memory_t*, wasm_memory_pages_t delta); +``` diff --git a/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/xip.md b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/xip.md new file mode 100644 index 000000000..d6c5a3701 --- /dev/null +++ b/src/fluent-bit/lib/wasm-micro-runtime-WAMR-1.2.2/doc/xip.md @@ -0,0 +1,14 @@ +# WAMR XIP (Execution In Place) feature introduction + +Some IoT devices may require to run the AOT file from flash or ROM which is read-only, so as to reduce the memory consumption, or resolve the issue that there is no executable memory available to run AOT code. In such case, the AOT code inside the AOT file shouldn't be duplicated into memory and shouldn't be modified (or patched) by the AOT relocations. To address this, WAMR implements the XIP (Execution In Place) feature, which generates the AOT relocations as few as possible: +- In the AOT code, an AOT function calls other functions with indirect mode: it doesn't call other functions directly, but looks up their pointers from the function pointer table passed by its first argument exec_env, and then calls the function pointer found. By this way the relocations to other functions are eliminated. +- Eliminate the calls to the LLVM intrinsic functions, or, replace calling them with calling runtime self implemented functions instead, e.g. the calling to `llvm.experimental.constrained.fadd.f32` is replaced by the calling to `aot_intrinsic_fadd_f32`. + +The XIP file is an AOT file without (or with few) relocations to patch the AOT code (or text section). Developer can use the option `--enable-indirect-mode --disable-llvm-intrinsics` for wamrc to generate the AOT file, e.g.: +```bash +wamrc --enable-indirect-mode --disable-llvm-intrinsics -o <aot_file> <wasm_file> +``` + +## Known issues + +There may be some relocations to the ".rodata" like sections which require to patch the AOT code. More work will be done to resolve it in the future. |
