diff options
Diffstat (limited to 'src/collectors/windows-events.plugin')
18 files changed, 5502 insertions, 0 deletions
diff --git a/src/collectors/windows-events.plugin/README.md b/src/collectors/windows-events.plugin/README.md new file mode 100644 index 000000000..ecaa4349a --- /dev/null +++ b/src/collectors/windows-events.plugin/README.md @@ -0,0 +1,289 @@ +# Windows Events plugin
+
+[KEY FEATURES](#key-features) | [EVENTS SOURCES](#events-sources) | [EVENT FIELDS](#event-fields) |
+[PLAY MODE](#play-mode) | [FULL TEXT SEARCH](#full-text-search) | [PERFORMANCE](#query-performance) |
+[CONFIGURATION](#configuration-and-maintenance) | [FAQ](#faq)
+
+The Windows Events plugin by Netdata makes viewing, exploring and analyzing Windows Events simple and
+efficient.
+
+
+
+## Key features
+
+- Supports **Windows Event Logs (WEL)**.
+- Supports **Event Tracing for Windows (ETW)** and **TraceLogging (TL)**, when events are routed to Event Log.
+- Allows filtering on all System Events fields.
+- Allows **full text search** (`grep`) on all System and User fields.
+- Provides a **histogram** for log entries over time, with a break down per field-value, for any System Event field and any
+ time-frame.
+- Supports coloring log entries based on severity.
+- In PLAY mode it "tails" all the Events, showing new log entries immediately after they are received.
+
+### Prerequisites
+
+`windows-events.plugin` is a Netdata Function Plugin.
+
+To protect your privacy, as with all Netdata Functions, a free Netdata Cloud user account is required to access it.
+For more information check [this discussion](https://github.com/netdata/netdata/discussions/16136).
+
+## Events Sources
+
+The plugin automatically detects all the available channels and offers a list of "Event Channels".
+
+By default, it aggregates events from all event channels, providing a unified view of all events.
+
+> To improve query performance, we recommend selecting the relevant event channels, before doing more
+> analysis on the events.
+
+In the list of events channels, several shortcuts are added, aggregating events according to various attributes:
+
+- `All`, aggregates events from all available channels. This provides a holistic view of all events in the system.
+- `All-Admin`, `All-Operational`, `All-Analytic` and `All-Debug` aggregates events from channels marked `Admin`, `Operational`, `Analytic` and `Debug`, respectively.
+- `All-Windows`, aggregates events from `Application`, `Security`, `System` and `Setup`.
+- `All-Enabled` and `All-Disabled` aggregates events from channels depending on their status.
+- `All-Forwarded` aggregates events from channels owned by `Microsoft-Windows-EventCollector`.
+- `All-Classic` aggregates events from channels using the Classic Event Log API.
+- `All-Of-X`, where `X` is a provider name, is offered for all providers having more than a channel.
+- `All-In-X`, where `X` is `Backup-Mode`, `Overwrite-Mode`, `StopWhenFull-Mode` and `RetainAndBackup-Mode`, aggregate events based on their channel retention policy.
+
+Channels that are configured but are not queryable, and channels that do not have any events in them, are automatically excluded from the channels list.
+
+## Event Fields
+
+Windows Events are structured with both system-defined fields and user-defined fields.
+The Windows Events plugin primarily works with the system-defined fields, which are consistently available
+across all event types.
+
+### System-defined fields
+
+The system-defined fields are:
+
+1. **EventRecordID**
+ A unique, sequential identifier for the event within the channel. This ID increases as new events are logged.
+
+2. **Version**
+ The version of the event, indicating possible structural changes or updates to the event definition.
+
+ Netdata adds this field automatically when it is not zero.
+
+3. **Level**
+ The severity or importance of the event. Levels can include:
+ - 0: LogAlways (reserved)
+ - 1: Critical
+ - 2: Error
+ - 3: Warning
+ - 4: Informational
+ - 5: Verbose
+
+ Additionally, applications may define their own levels.
+
+ Netdata provides 2 fields: `Level` and `LevelID` for the text and numeric representation of it.
+
+4. **Opcode**
+ The action or state within a provider when the event was logged.
+
+ Netdata provides 2 fields: `Opcode` and `OpcodeID` for the text and numeric representation of it.
+
+5. **EventID**
+ This identifies the event template, linking it to a specific message or event type. Event IDs are provider-specific.
+
+6. **Task**
+ Defines a higher-level categorization or logical grouping for the event, often related to a specific function within the application or provider.
+
+ Netdata provides 2 fields: `Task` and `TaskID` for the text and numeric representation of it.
+
+7. **Qualifiers**
+ Provides additional detail for interpreting the event and is often specific to the event source.
+
+ Netdata adds this field automatically when it is not zero.
+
+8. **ProcessID**
+ The ID of the process that generated the event, useful for pinpointing the source of the event within the system.
+
+9. **ThreadID**
+ The ID of the thread within the process that generated the event, which helps in more detailed debugging scenarios.
+
+10. **Keywords**
+ A categorization field that can be used for event filtering. Keywords are bit flags that represent categories or purposes of the event, providing additional context.
+
+ Netdata provides 2 fields: `Keywords` and `keywordsID` for the text and numeric representation of it.
+
+11. **Provider**
+ The unique identifier (GUID) of the event provider. This is essential for knowing which application or system component generated the event.
+
+ Netdata provides 2 fields: `Provider` and `ProviderGUID` for its name and GUID of it.
+
+12. **ActivityID**
+ A GUID that correlates events generated as part of the same operation or transaction, helping to track activities across different components or stages.
+
+ Netdata adds this field automatically when it is not zero.
+
+13. **RelatedActivityID**
+ A GUID that links related operations or transactions, allowing for tracing complex workflows where one event triggers or relates to another.
+
+ Netdata adds this field automatically when it is not zero.
+
+14. **Timestamp**
+ The timestamp when the event was created. This provides precise timing information about when the event occurred.
+
+15. **User**
+ The system user who logged this event.
+
+ Netdata provides 3 fields: `UserAccount`, `UserDomain` and `UserSID`.
+
+### User-defined fields
+Each event log entry can include up to 100 user-defined fields (per event-id).
+
+Unfortunately, accessing these fields is significantly slower, to a level that is not practical to do so
+when there are more than few thousand log entries to explore. So, Netdata presents them
+with lazy loading.
+
+This prevents Netdata for offering filtering for user-defined fields, although Netdata does support
+full text search on user-defined field values.
+
+### Event fields as columns in the table
+
+The system fields mentioned above are offered as columns on the UI. Use the gear button above the table to
+select visible columns.
+
+### Event fields as filters
+
+The plugin presents the system fields as filters for the query, with counters for each of the possible values
+for the field. This list can be used to quickly check which fields and values are available for the entire
+time-frame of the query, across multiple providers and channels.
+
+### Event fields as histogram sources
+
+The histogram can be based on any of the system fields that are available as filters. This allows you to
+visualize the distribution of events over time based on different criteria such as Level, Provider, or EventID.
+
+## PLAY mode
+
+The PLAY mode in this plugin allows real-time monitoring of new events as they are added to the Windows Event
+Log. This feature works by continuously querying for new events and updating the display.
+
+## Full-text search
+
+The plugin supports searching for text within all system and user fields of the events. This means that while
+user-defined fields are not directly filterable, they are searchable through the full-text search feature.
+
+Keep in mind that query performance is slower while doing full text search, mainly because the plugin
+needs to ask from the system to provide all the user fields values.
+
+## Query performance
+
+The plugin is optimized to work efficiently with Event Logs. It uses several layers of caching and
+similar techniques to offload as much work as possible from the system, offering quick responses even when
+hundreds of thousands of events are within the visible timeframe.
+
+To achieve this level of efficiency, the plugin:
+
+- pre-loads ETW providers' manifests for resolving numeric Levels, Opcodes, Tasks and Keywords to text.
+- caches number to text maps for Levels, Opcodes, Tasks and Keywords per provider for WEL providers.
+- caches user SID to account and domain maps.
+- lazy loads the "expensive" event Message and XML, so that the system is queried only for the visible events.
+
+For Full Text Search:
+
+- requests only the Message and the values of the user-fields from the system, avoiding the "expensive" XML call (which is still lazy-loaded).
+
+The result is a system that is highly efficient for working with moderate volumes (hundreds of thousands) of events.
+
+## Configuration and maintenance
+
+This Netdata plugin does not require any specific configuration. It automatically detects available event logs
+on the system.
+
+## FAQ
+
+### Can I use this plugin on event centralization servers?
+
+Yes. You can centralize your Windows Events using Windows Event Forwarding (WEF) or other event collection
+mechanisms, and then install Netdata on this events centralization server to explore the events of all your
+infrastructure.
+
+This plugin will automatically provide multi-node views of your events and also give you the ability to
+combine the events of multiple servers, as you see fit.
+
+### Can I use this plugin from a parent Netdata?
+
+Yes. When your nodes are connected to a Netdata parent, all their functions are available via the parent's UI.
+So, from the parent UI, you can access the functions of all your nodes.
+
+Keep in mind that to protect your privacy, in order to access Netdata functions, you need a free Netdata Cloud
+account.
+
+### Is any of my data exposed to Netdata Cloud from this plugin?
+
+No. When you access the agent directly, none of your data passes through Netdata Cloud. You need a free Netdata
+Cloud account only to verify your identity and enable the use of Netdata Functions. Once this is done, all the
+data flow directly from your Netdata agent to your web browser.
+
+When you access Netdata via https://app.netdata.cloud, your data travel via Netdata Cloud, but they are not stored
+in Netdata Cloud. This is to allow you access your Netdata agents from anywhere. All communication from/to
+Netdata Cloud is encrypted.
+
+### What are the different types of event logs supported by this plugin?
+
+The plugin supports all the kinds of event logs currently supported by the Windows Event Viewer:
+
+- Windows Event Logs (WEL): The traditional event logging system in Windows.
+- Event Tracing for Windows (ETW): A more detailed and efficient event tracing system.
+- TraceLogging (TL): An extension of ETW that simplifies the process of adding events to your code.
+
+The plugin can access all of these when they are routed to the Windows Event Log.
+
+### How does this plugin handle user-defined fields in Windows Events?
+
+User-defined fields are not directly exposed as table columns or filters in the plugin interface. However,
+they are included in the XML representation of each event, which can be viewed in the info sidebar when
+clicking on an event entry. Additionally, the full-text search feature does search through these
+user-defined fields, allowing you to find specific information even if it's not in the main system fields.
+
+### Can I use this plugin to monitor real-time events?
+
+Yes, the plugin supports a PLAY mode that allows you to monitor events in real-time. When activated, it
+continuously updates to show new events as they are logged, similar to the "tail" functionality in
+Unix-like systems.
+
+### How does the plugin handle large volumes of events?
+
+The plugin is designed to handle moderate volumes of events (hundreds of thousands of events) efficiently.
+
+It is in our roadmap to port the `systemd-journal` sampling techniques to it, for working with very large
+datasets to provide quick responses while still giving accurate representations of the data. However, for
+the best performance, we recommend querying smaller time frames or using more specific filters when dealing
+with extremely large event volumes.
+
+### Can I use this plugin to analyze events from multiple servers?
+
+Yes, if you have set up Windows Event Forwarding (WEF) or another method of centralizing your Windows Events,
+you can use this plugin on the central server to analyze events from multiple sources. The plugin will
+automatically detect the available event sources.
+
+### How does the histogram feature work in this plugin?
+
+The histogram feature provides a visual representation of event frequency over time. You can base the
+histogram on any of the system fields available as filters (such as Level, Provider, or EventID). This
+allows you to quickly identify patterns or anomalies in your event logs.
+
+### Is it possible to export or share the results from this plugin?
+
+While the plugin doesn't have a direct export feature, you can use browser-based methods to save or share
+the results. This could include taking screenshots, using browser print/save as PDF functionality, or
+copying data from the table view. For more advanced data export needs, you might need to use the Windows
+Event Log API directly or other Windows administrative tools.
+
+### How often does the plugin update its data?
+
+The plugin updates its data in real-time when in PLAY mode. In normal mode, it refreshes data based on the
+query you've submitted. The plugin is designed to provide the most up-to-date information available in the
+Windows Event Logs at the time of the query.
+
+## TODO
+
+1. Support Sampling, so that the plugin can respond faster even on very busy systems (millions of events visible).
+2. Support exploring events from live Tracing sessions.
+3. Support exploring events in saved Event Trace Log files (`.etl` files).
+4. Support exploring events in saved Event Logs files (`.evtx` files).
diff --git a/src/collectors/windows-events.plugin/windows-events-fields-cache.c b/src/collectors/windows-events.plugin/windows-events-fields-cache.c new file mode 100644 index 000000000..4b4b72fa4 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-fields-cache.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-3.0-or-later
+
+#include "windows-events-fields-cache.h"
+
+typedef struct field_key {
+ uint64_t value;
+ ND_UUID provider;
+} WEVT_FIELD_KEY;
+
+typedef struct field_value {
+ WEVT_FIELD_KEY key;
+ uint32_t name_size;
+ char name[];
+} WEVT_FIELD_VALUE;
+
+#define SIMPLE_HASHTABLE_NAME _FIELDS_CACHE
+#define SIMPLE_HASHTABLE_VALUE_TYPE WEVT_FIELD_VALUE
+#define SIMPLE_HASHTABLE_KEY_TYPE WEVT_FIELD_KEY
+#define SIMPLE_HASHTABLE_VALUE2KEY_FUNCTION field_cache_value_to_key
+#define SIMPLE_HASHTABLE_COMPARE_KEYS_FUNCTION field_cache_cache_compar
+#define SIMPLE_HASHTABLE_SAMPLE_IMPLEMENTATION 1
+#include "libnetdata/simple_hashtable/simple_hashtable.h"
+
+static inline WEVT_FIELD_KEY *field_cache_value_to_key(WEVT_FIELD_VALUE *p) {
+ return &p->key;
+}
+
+static inline bool field_cache_cache_compar(WEVT_FIELD_KEY *a, WEVT_FIELD_KEY *b) {
+ return memcmp(a, b, sizeof(WEVT_FIELD_KEY)) == 0;
+}
+
+struct ht {
+ SPINLOCK spinlock;
+ size_t allocations;
+ size_t bytes;
+ struct simple_hashtable_FIELDS_CACHE ht;
+};
+
+static struct {
+ bool initialized;
+ struct ht ht[WEVT_FIELD_TYPE_MAX];
+} fdc = {
+ .initialized = false,
+};
+
+void field_cache_init(void) {
+ for(size_t type = 0; type < WEVT_FIELD_TYPE_MAX ; type++) {
+ spinlock_init(&fdc.ht[type].spinlock);
+ simple_hashtable_init_FIELDS_CACHE(&fdc.ht[type].ht, 10000);
+ }
+}
+
+static inline bool should_zero_provider(WEVT_FIELD_TYPE type, uint64_t value) {
+ switch(type) {
+ case WEVT_FIELD_TYPE_LEVEL:
+ return !is_valid_provider_level(value, true);
+
+ case WEVT_FIELD_TYPE_KEYWORD:
+ return !is_valid_provider_keyword(value, true);
+
+ case WEVT_FIELD_TYPE_OPCODE:
+ return !is_valid_provider_opcode(value, true);
+
+ case WEVT_FIELD_TYPE_TASK:
+ return !is_valid_provider_task(value, true);
+
+ default:
+ return false;
+ }
+}
+
+bool field_cache_get(WEVT_FIELD_TYPE type, const ND_UUID *uuid, uint64_t value, TXT_UTF8 *dst) {
+ fatal_assert(type < WEVT_FIELD_TYPE_MAX);
+
+ struct ht *ht = &fdc.ht[type];
+
+ WEVT_FIELD_KEY t = {
+ .value = value,
+ .provider = should_zero_provider(type, value) ? UUID_ZERO : *uuid,
+ };
+ XXH64_hash_t hash = XXH3_64bits(&t, sizeof(t));
+
+ spinlock_lock(&ht->spinlock);
+ SIMPLE_HASHTABLE_SLOT_FIELDS_CACHE *slot = simple_hashtable_get_slot_FIELDS_CACHE(&ht->ht, hash, &t, true);
+ WEVT_FIELD_VALUE *v = SIMPLE_HASHTABLE_SLOT_DATA(slot);
+ spinlock_unlock(&ht->spinlock);
+
+ if(v) {
+ txt_utf8_resize(dst, v->name_size, false);
+ memcpy(dst->data, v->name, v->name_size);
+ dst->used = v->name_size;
+ dst->src = TXT_SOURCE_FIELD_CACHE;
+ return true;
+ }
+
+ return false;
+}
+
+static WEVT_FIELD_VALUE *wevt_create_cache_entry(WEVT_FIELD_KEY *t, TXT_UTF8 *name, size_t *bytes) {
+ *bytes = sizeof(WEVT_FIELD_VALUE) + name->used;
+ WEVT_FIELD_VALUE *v = callocz(1, *bytes);
+ v->key = *t;
+ memcpy(v->name, name->data, name->used);
+ v->name_size = name->used;
+ return v;
+}
+
+//static bool is_numeric(const char *s) {
+// while(*s) {
+// if(!isdigit((uint8_t)*s++))
+// return false;
+// }
+//
+// return true;
+//}
+
+void field_cache_set(WEVT_FIELD_TYPE type, const ND_UUID *uuid, uint64_t value, TXT_UTF8 *name) {
+ fatal_assert(type < WEVT_FIELD_TYPE_MAX);
+
+ struct ht *ht = &fdc.ht[type];
+
+ WEVT_FIELD_KEY t = {
+ .value = value,
+ .provider = should_zero_provider(type, value) ? UUID_ZERO : *uuid,
+ };
+ XXH64_hash_t hash = XXH3_64bits(&t, sizeof(t));
+
+ spinlock_lock(&ht->spinlock);
+ SIMPLE_HASHTABLE_SLOT_FIELDS_CACHE *slot = simple_hashtable_get_slot_FIELDS_CACHE(&ht->ht, hash, &t, true);
+ WEVT_FIELD_VALUE *v = SIMPLE_HASHTABLE_SLOT_DATA(slot);
+ if(!v) {
+ size_t bytes;
+ v = wevt_create_cache_entry(&t, name, &bytes);
+ simple_hashtable_set_slot_FIELDS_CACHE(&ht->ht, slot, hash, v);
+
+ ht->allocations++;
+ ht->bytes += bytes;
+ }
+// else {
+// if((v->name_size == 1 && name->used > 0) || is_numeric(v->name)) {
+// size_t bytes;
+// WEVT_FIELD_VALUE *nv = wevt_create_cache_entry(&t, name, &bytes);
+// simple_hashtable_set_slot_FIELDS_CACHE(&ht->ht, slot, hash, nv);
+// ht->bytes += name->used;
+// ht->bytes -= v->name_size;
+// freez(v);
+// }
+// else if(name->used > 2 && !is_numeric(name->data) && (v->name_size != name->used || strcasecmp(v->name, name->data) != 0)) {
+// const char *a = v->name;
+// const char *b = name->data;
+// int x = 0;
+// x++;
+// }
+// }
+
+ spinlock_unlock(&ht->spinlock);
+}
+
diff --git a/src/collectors/windows-events.plugin/windows-events-fields-cache.h b/src/collectors/windows-events.plugin/windows-events-fields-cache.h new file mode 100644 index 000000000..a76170d68 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-fields-cache.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-3.0-or-later
+
+#ifndef NETDATA_WINDOWS_EVENTS_FIELDS_CACHE_H
+#define NETDATA_WINDOWS_EVENTS_FIELDS_CACHE_H
+
+#include "windows-events.h"
+
+typedef enum __attribute__((packed)) {
+ WEVT_FIELD_TYPE_LEVEL = 0,
+ WEVT_FIELD_TYPE_OPCODE,
+ WEVT_FIELD_TYPE_KEYWORD,
+ WEVT_FIELD_TYPE_TASK,
+
+ // terminator
+ WEVT_FIELD_TYPE_MAX,
+} WEVT_FIELD_TYPE;
+
+void field_cache_init(void);
+bool field_cache_get(WEVT_FIELD_TYPE type, const ND_UUID *uuid, uint64_t value, TXT_UTF8 *dst);
+void field_cache_set(WEVT_FIELD_TYPE type, const ND_UUID *uuid, uint64_t value, TXT_UTF8 *name);
+
+#endif //NETDATA_WINDOWS_EVENTS_FIELDS_CACHE_H
diff --git a/src/collectors/windows-events.plugin/windows-events-providers.c b/src/collectors/windows-events.plugin/windows-events-providers.c new file mode 100644 index 000000000..d4c4d35ea --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-providers.c @@ -0,0 +1,678 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events-providers.h" + +#define MAX_OPEN_HANDLES_PER_PROVIDER 5 + +struct provider; + +// typedef as PROVIDER_META_HANDLE in include file +struct provider_meta_handle { + pid_t owner; // the owner of the handle, or zero + uint32_t locks; // the number of locks the owner has on this handle + EVT_HANDLE hMetadata; // the handle + struct provider *provider; // a pointer back to the provider + + usec_t created_monotonic_ut; // the monotonic timestamp this handle was created + + // double linked list + PROVIDER_META_HANDLE *prev; + PROVIDER_META_HANDLE *next; +}; + +struct provider_data { + uint64_t value; // the mask of the keyword + XXH64_hash_t hash; // the hash of the name + uint32_t len; // the length of the name + char *name; // the name of the keyword in UTF-8 +}; + +struct provider_list { + uint64_t min, max, mask; + bool exceeds_data_type; // true when the manifest values exceed the capacity of the EvtXXX() API + uint32_t total; // the number of entries in the array + struct provider_data *array; // the array of entries, sorted (for binary search) +}; + +typedef struct provider_key { + ND_UUID uuid; // the Provider GUID + DWORD len; // the length of the Provider Name + const wchar_t *wname; // the Provider wide-string Name (UTF-16) +} PROVIDER_KEY; + +typedef struct provider { + PROVIDER_KEY key; + const char *name; // the Provider Name (UTF-8) + uint32_t total_handles; // the number of handles allocated + uint32_t available_handles; // the number of available handles + uint32_t deleted_handles; // the number of deleted handles + PROVIDER_META_HANDLE *handles; // a double linked list of all the handles + + WEVT_PROVIDER_PLATFORM platform; + + struct provider_list keyword; + struct provider_list tasks; + struct provider_list opcodes; + struct provider_list levels; +} PROVIDER; + +// A hashtable implementation for Providers +// using the Provider GUID as key and PROVIDER as value +#define SIMPLE_HASHTABLE_NAME _PROVIDER +#define SIMPLE_HASHTABLE_VALUE_TYPE PROVIDER +#define SIMPLE_HASHTABLE_KEY_TYPE PROVIDER_KEY +#define SIMPLE_HASHTABLE_VALUE2KEY_FUNCTION provider_value_to_key +#define SIMPLE_HASHTABLE_COMPARE_KEYS_FUNCTION provider_cache_compar +#define SIMPLE_HASHTABLE_SAMPLE_IMPLEMENTATION 1 +#include "libnetdata/simple_hashtable/simple_hashtable.h" + +static struct { + SPINLOCK spinlock; + uint32_t total_providers; + uint32_t total_handles; + uint32_t deleted_handles; + struct simple_hashtable_PROVIDER hashtable; + ARAL *aral_providers; + ARAL *aral_handles; +} pbc = { + .spinlock = NETDATA_SPINLOCK_INITIALIZER, +}; + +static void provider_load_list(PROVIDER_META_HANDLE *h, WEVT_VARIANT *content, WEVT_VARIANT *property, + TXT_UTF16 *dst, struct provider_list *l, EVT_PUBLISHER_METADATA_PROPERTY_ID property_id); + +const char *provider_get_name(PROVIDER_META_HANDLE *p) { + return (p && p->provider && p->provider->name) ? p->provider->name : "__UNKNOWN PROVIDER__"; +} + +ND_UUID provider_get_uuid(PROVIDER_META_HANDLE *p) { + return (p && p->provider) ? p->provider->key.uuid : UUID_ZERO; +} + +static inline PROVIDER_KEY *provider_value_to_key(PROVIDER *p) { + return &p->key; +} + +static inline bool provider_cache_compar(PROVIDER_KEY *a, PROVIDER_KEY *b) { + return a->len == b->len && UUIDeq(a->uuid, b->uuid) && memcmp(a->wname, b->wname, a->len) == 0; +} + +void provider_cache_init(void) { + simple_hashtable_init_PROVIDER(&pbc.hashtable, 100000); + pbc.aral_providers = aral_create("wevt_providers", sizeof(PROVIDER), 0, 4096, NULL, NULL, NULL, false, true); + pbc.aral_handles = aral_create("wevt_handles", sizeof(PROVIDER_META_HANDLE), 0, 4096, NULL, NULL, NULL, false, true); +} + +static bool provider_property_get(PROVIDER_META_HANDLE *h, WEVT_VARIANT *content, EVT_PUBLISHER_METADATA_PROPERTY_ID property_id) { + DWORD bufferUsed = 0; + + if(!EvtGetPublisherMetadataProperty(h->hMetadata, property_id, 0, 0, NULL, &bufferUsed)) { + DWORD status = GetLastError(); + if (status != ERROR_INSUFFICIENT_BUFFER) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetPublisherMetadataProperty() failed"); + goto cleanup; + } + } + + wevt_variant_resize(content, bufferUsed); + if (!EvtGetPublisherMetadataProperty(h->hMetadata, property_id, 0, content->size, content->data, &bufferUsed)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetPublisherMetadataProperty() failed after resize"); + goto cleanup; + } + + return true; + +cleanup: + return false; +} + +static bool provider_string_property_exists(PROVIDER_META_HANDLE *h, WEVT_VARIANT *content, EVT_PUBLISHER_METADATA_PROPERTY_ID property_id) { + if(!provider_property_get(h, content, property_id)) + return false; + + if(content->data->Type != EvtVarTypeString) + return false; + + if(!content->data->StringVal[0]) + return false; + + return true; +} + +static void provider_detect_platform(PROVIDER_META_HANDLE *h, WEVT_VARIANT *content) { + if(UUIDiszero(h->provider->key.uuid)) + h->provider->platform = WEVT_PLATFORM_WEL; + else if(h->hMetadata) { + if (provider_string_property_exists(h, content, EvtPublisherMetadataMessageFilePath) || + provider_string_property_exists(h, content, EvtPublisherMetadataResourceFilePath) || + provider_string_property_exists(h, content, EvtPublisherMetadataParameterFilePath)) + h->provider->platform = WEVT_PLATFORM_ETW; + else + // The provider cannot be opened, does not have any resource files (message, resource, parameter) + h->provider->platform = WEVT_PLATFORM_TL; + } + else h->provider->platform = WEVT_PLATFORM_ETW; +} + +WEVT_PROVIDER_PLATFORM provider_get_platform(PROVIDER_META_HANDLE *p) { + return p->provider->platform; +} + +PROVIDER_META_HANDLE *provider_get(ND_UUID uuid, LPCWSTR providerName) { + if(!providerName || !providerName[0]) + return NULL; + + PROVIDER_KEY key = { + .uuid = uuid, + .len = wcslen(providerName), + .wname = providerName, + }; + XXH64_hash_t hash = XXH3_64bits(providerName, wcslen(key.wname) * sizeof(*key.wname)); + + spinlock_lock(&pbc.spinlock); + + SIMPLE_HASHTABLE_SLOT_PROVIDER *slot = + simple_hashtable_get_slot_PROVIDER(&pbc.hashtable, hash, &key, true); + + bool load_it = false; + PROVIDER *p = SIMPLE_HASHTABLE_SLOT_DATA(slot); + if(!p) { + p = aral_callocz(pbc.aral_providers); + p->key.uuid = key.uuid; + p->key.len = key.len; + p->key.wname = wcsdup(key.wname); + p->name = strdupz(provider2utf8(key.wname)); + simple_hashtable_set_slot_PROVIDER(&pbc.hashtable, slot, hash, p); + load_it = true; + pbc.total_providers++; + } + + pid_t me = gettid_cached(); + PROVIDER_META_HANDLE *h; + for(h = p->handles; h ;h = h->next) { + // find the first that is mine, + // or the first not owned by anyone + if(!h->owner || h->owner == me) + break; + } + + if(!h) { + h = aral_callocz(pbc.aral_handles); + h->provider = p; + h->created_monotonic_ut = now_monotonic_usec(); + h->hMetadata = EvtOpenPublisherMetadata( + NULL, // Local machine + providerName, // Provider name + NULL, // Log file path (NULL for default) + 0, // Locale (0 for default locale) + 0 // Flags + ); + + // we put it at the beginning of the list + // to find it first if the same owner needs more locks on it + DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(p->handles, h, prev, next); + pbc.total_handles++; + p->total_handles++; + p->available_handles++; + } + + if(!h->owner) { + fatal_assert(p->available_handles > 0); + p->available_handles--; + h->owner = me; + } + + h->locks++; + + if(load_it) { + WEVT_VARIANT content = { 0 }; + WEVT_VARIANT property = { 0 }; + TXT_UTF16 unicode = { 0 }; + + provider_detect_platform(h, &content); + provider_load_list(h, &content, &property, &unicode, &p->keyword, EvtPublisherMetadataKeywords); + provider_load_list(h, &content, &property, &unicode, &p->levels, EvtPublisherMetadataLevels); + provider_load_list(h, &content, &property, &unicode, &p->opcodes, EvtPublisherMetadataOpcodes); + provider_load_list(h, &content, &property, &unicode, &p->tasks, EvtPublisherMetadataTasks); + + txt_utf16_cleanup(&unicode); + wevt_variant_cleanup(&content); + wevt_variant_cleanup(&property); + } + + spinlock_unlock(&pbc.spinlock); + + return h; +} + +EVT_HANDLE provider_handle(PROVIDER_META_HANDLE *h) { + return h ? h->hMetadata : NULL; +} + +PROVIDER_META_HANDLE *provider_dup(PROVIDER_META_HANDLE *h) { + if(h) h->locks++; + return h; +} + +static void provider_meta_handle_delete(PROVIDER_META_HANDLE *h) { + PROVIDER *p = h->provider; + + DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(p->handles, h, prev, next); + + if(h->hMetadata) + EvtClose(h->hMetadata); + + aral_freez(pbc.aral_handles, h); + + fatal_assert(pbc.total_handles && p->total_handles && p->available_handles); + + pbc.total_handles--; + p->total_handles--; + + pbc.deleted_handles++; + p->deleted_handles++; + + p->available_handles--; +} + +void providers_release_unused_handles(void) { + usec_t now_ut = now_monotonic_usec(); + + spinlock_lock(&pbc.spinlock); + for(size_t i = 0; i < pbc.hashtable.size ; i++) { + SIMPLE_HASHTABLE_SLOT_PROVIDER *slot = &pbc.hashtable.hashtable[i]; + PROVIDER *p = SIMPLE_HASHTABLE_SLOT_DATA(slot); + if(!p) continue; + + PROVIDER_META_HANDLE *h = p->handles; + while(h) { + PROVIDER_META_HANDLE *next = h->next; + + if(!h->locks && (now_ut - h->created_monotonic_ut) >= WINDOWS_EVENTS_RELEASE_IDLE_PROVIDER_HANDLES_TIME_UT) + provider_meta_handle_delete(h); + + h = next; + } + } + spinlock_unlock(&pbc.spinlock); +} + +void provider_release(PROVIDER_META_HANDLE *h) { + if(!h) return; + pid_t me = gettid_cached(); + fatal_assert(h->owner == me); + fatal_assert(h->locks > 0); + if(--h->locks == 0) { + PROVIDER *p = h->provider; + + spinlock_lock(&pbc.spinlock); + h->owner = 0; + + if(++p->available_handles > MAX_OPEN_HANDLES_PER_PROVIDER) { + // there are too many idle handles on this provider + provider_meta_handle_delete(h); + } + else if(h->next) { + // it is not the last, put it at the end + DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(p->handles, h, prev, next); + DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(p->handles, h, prev, next); + } + + spinlock_unlock(&pbc.spinlock); + } +} + +// -------------------------------------------------------------------------------------------------------------------- +// load provider lists + +static bool wevt_get_property_from_array(WEVT_VARIANT *property, EVT_HANDLE handle, DWORD dwIndex, EVT_PUBLISHER_METADATA_PROPERTY_ID PropertyId) { + DWORD used = 0; + + if (!EvtGetObjectArrayProperty(handle, PropertyId, dwIndex, 0, property->size, property->data, &used)) { + DWORD status = GetLastError(); + if (status != ERROR_INSUFFICIENT_BUFFER) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetObjectArrayProperty() failed"); + return false; + } + + wevt_variant_resize(property, used); + if (!EvtGetObjectArrayProperty(handle, PropertyId, dwIndex, 0, property->size, property->data, &used)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetObjectArrayProperty() failed"); + return false; + } + } + + property->used = used; + return true; +} + +// Comparison function for ascending order (for Levels, Opcodes, Tasks) +static int compare_ascending(const void *a, const void *b) { + struct provider_data *d1 = (struct provider_data *)a; + struct provider_data *d2 = (struct provider_data *)b; + + if (d1->value < d2->value) return -1; + if (d1->value > d2->value) return 1; + return 0; +} + +//// Comparison function for descending order (for Keywords) +//static int compare_descending(const void *a, const void *b) { +// struct provider_data *d1 = (struct provider_data *)a; +// struct provider_data *d2 = (struct provider_data *)b; +// +// if (d1->value > d2->value) return -1; +// if (d1->value < d2->value) return 1; +// return 0; +//} + +static void provider_load_list(PROVIDER_META_HANDLE *h, WEVT_VARIANT *content, WEVT_VARIANT *property, + TXT_UTF16 *dst, struct provider_list *l, EVT_PUBLISHER_METADATA_PROPERTY_ID property_id) { + if(!h || !h->hMetadata) return; + + EVT_PUBLISHER_METADATA_PROPERTY_ID name_id, message_id, value_id; + uint8_t value_bits = 32; + int (*compare_func)(const void *, const void *); + bool (*is_valid)(uint64_t, bool); + + switch(property_id) { + case EvtPublisherMetadataLevels: + name_id = EvtPublisherMetadataLevelName; + message_id = EvtPublisherMetadataLevelMessageID; + value_id = EvtPublisherMetadataLevelValue; + value_bits = 32; + compare_func = compare_ascending; + is_valid = is_valid_provider_level; + break; + + case EvtPublisherMetadataOpcodes: + name_id = EvtPublisherMetadataOpcodeName; + message_id = EvtPublisherMetadataOpcodeMessageID; + value_id = EvtPublisherMetadataOpcodeValue; + value_bits = 32; + is_valid = is_valid_provider_opcode; + compare_func = compare_ascending; + break; + + case EvtPublisherMetadataTasks: + name_id = EvtPublisherMetadataTaskName; + message_id = EvtPublisherMetadataTaskMessageID; + value_id = EvtPublisherMetadataTaskValue; + value_bits = 32; + is_valid = is_valid_provider_task; + compare_func = compare_ascending; + break; + + case EvtPublisherMetadataKeywords: + name_id = EvtPublisherMetadataKeywordName; + message_id = EvtPublisherMetadataKeywordMessageID; + value_id = EvtPublisherMetadataKeywordValue; + value_bits = 64; + is_valid = is_valid_provider_keyword; + compare_func = NULL; + break; + + default: + nd_log(NDLS_COLLECTORS, NDLP_ERR, "Internal Error: Can't handle property id %u", property_id); + return; + } + + EVT_HANDLE hMetadata = h->hMetadata; + EVT_HANDLE hArray = NULL; + DWORD itemCount = 0; + + // Get the metadata array for the list (e.g., opcodes, tasks, or levels) + if(!provider_property_get(h, content, property_id)) + goto cleanup; + + // Get the number of items (e.g., levels, tasks, or opcodes) + hArray = content->data->EvtHandleVal; + if (!EvtGetObjectArraySize(hArray, &itemCount)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetObjectArraySize() failed"); + goto cleanup; + } + + if (itemCount == 0) { + l->total = 0; + l->array = NULL; + goto cleanup; + } + + // Allocate memory for the list items + l->array = callocz(itemCount, sizeof(struct provider_data)); + l->total = itemCount; + + uint64_t min = UINT64_MAX, max = 0, mask = 0; + + // Iterate over the list and populate the entries + for (DWORD i = 0; i < itemCount; ++i) { + struct provider_data *d = &l->array[i]; + + // Get the value (e.g., opcode, task, or level) + if (wevt_get_property_from_array(property, hArray, i, value_id)) { + switch(value_bits) { + case 64: + d->value = wevt_field_get_uint64(property->data); + break; + + case 32: + d->value = wevt_field_get_uint32(property->data); + break; + } + + if(d->value < min) + min = d->value; + + if(d->value > max) + max = d->value; + + mask |= d->value; + + if(!is_valid(d->value, false)) + l->exceeds_data_type = true; + } + + // Get the message ID + if (wevt_get_property_from_array(property, hArray, i, message_id)) { + uint32_t messageID = wevt_field_get_uint32(property->data); + + if (messageID != (uint32_t)-1) { + if (EvtFormatMessage_utf16(dst, hMetadata, NULL, messageID, EvtFormatMessageId)) { + size_t len; + d->name = utf16_to_utf8_strdupz(dst->data, &len); + d->len = len; + } + } + } + + // Get the name if the message is missing + if (!d->name && wevt_get_property_from_array(property, hArray, i, name_id)) { + fatal_assert(property->data->Type == EvtVarTypeString); + size_t len; + d->name = utf16_to_utf8_strdupz(property->data->StringVal, &len); + d->len = len; + } + + // Calculate the hash for the name + if (d->name) + d->hash = XXH3_64bits(d->name, d->len); + } + + l->min = min; + l->max = max; + l->mask = mask; + + if(itemCount > 1 && compare_func != NULL) { + // Sort the array based on the value (ascending for all except keywords, descending for keywords) + qsort(l->array, itemCount, sizeof(struct provider_data), compare_func); + } + +cleanup: + if (hArray) + EvtClose(hArray); +} + +// -------------------------------------------------------------------------------------------------------------------- +// lookup functions + +// lookup bitmap metdata (returns a comma separated list of strings) +static bool provider_bitmap_metadata(TXT_UTF8 *dst, struct provider_list *l, uint64_t value) { + if(!(value & l->mask) || !l->total || !l->array || l->exceeds_data_type) + return false; + + // do not empty the buffer, there may be reserved keywords in it + // dst->used = 0; + + if(dst->used) + dst->used--; + + size_t added = 0; + for(size_t k = 0; value && k < l->total; k++) { + struct provider_data *d = &l->array[k]; + + if(d->value && (value & d->value) == d->value && d->name && d->len) { + const char *s = d->name; + size_t slen = d->len; + + // remove the mask from the value + value &= ~(d->value); + + txt_utf8_resize(dst, dst->used + slen + 2 + 1, true); + + if(dst->used) { + // add a comma and a space + dst->data[dst->used++] = ','; + dst->data[dst->used++] = ' '; + } + + memcpy(&dst->data[dst->used], s, slen); + dst->used += slen; + dst->src = TXT_SOURCE_PROVIDER; + added++; + } + } + + if(dst->used > 1) { + txt_utf8_resize(dst, dst->used + 1, true); + dst->data[dst->used++] = 0; + } + + fatal_assert(dst->used <= dst->size); + return added; +} + +//// lookup a single value (returns its string) +//static bool provider_value_metadata_linear(TXT_UTF8 *dst, struct provider_list *l, uint64_t value) { +// if(value < l->min || value > l->max || !l->total || !l->array || l->exceeds_data_type) +// return false; +// +// dst->used = 0; +// +// for(size_t k = 0; k < l->total; k++) { +// struct provider_data *d = &l->array[k]; +// +// if(d->value == value && d->name && d->len) { +// const char *s = d->name; +// size_t slen = d->len; +// +// txt_utf8_resize(dst, slen + 1, false); +// +// memcpy(dst->data, s, slen); +// dst->used = slen; +// dst->src = TXT_SOURCE_PROVIDER; +// +// break; +// } +// } +// +// if(dst->used) { +// txt_utf8_resize(dst, dst->used + 1, true); +// dst->data[dst->used++] = 0; +// } +// +// fatal_assert(dst->used <= dst->size); +// +// return (dst->used > 0); +//} + +static bool provider_value_metadata(TXT_UTF8 *dst, struct provider_list *l, uint64_t value) { + if(value < l->min || value > l->max || !l->total || !l->array || l->exceeds_data_type) + return false; + + // if(l->total < 3) return provider_value_metadata_linear(dst, l, value); + + dst->used = 0; + + size_t left = 0; + size_t right = l->total - 1; + + // Binary search within bounds + while (left <= right) { + size_t mid = left + (right - left) / 2; + struct provider_data *d = &l->array[mid]; + + if (d->value == value) { + // Value found, now check if it has a valid name and length + if (d->name && d->len) { + const char *s = d->name; + size_t slen = d->len; + + txt_utf8_resize(dst, slen + 1, false); + memcpy(dst->data, s, slen); + dst->used = slen; + dst->data[dst->used++] = 0; + dst->src = TXT_SOURCE_PROVIDER; + } + break; + } + + if (d->value < value) + left = mid + 1; + else { + if (mid == 0) break; + right = mid - 1; + } + } + + fatal_assert(dst->used <= dst->size); + return (dst->used > 0); +} + +// -------------------------------------------------------------------------------------------------------------------- +// public API to lookup metadata + +bool provider_keyword_cacheable(PROVIDER_META_HANDLE *h) { + return h && !h->provider->keyword.exceeds_data_type; +} + +bool provider_tasks_cacheable(PROVIDER_META_HANDLE *h) { + return h && !h->provider->tasks.exceeds_data_type; +} + +bool is_useful_provider_for_levels(PROVIDER_META_HANDLE *h) { + return h && !h->provider->levels.exceeds_data_type; +} + +bool provider_opcodes_cacheable(PROVIDER_META_HANDLE *h) { + return h && !h->provider->opcodes.exceeds_data_type; +} + +bool provider_get_keywords(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value) { + if(!h) return false; + return provider_bitmap_metadata(dst, &h->provider->keyword, value); +} + +bool provider_get_level(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value) { + if(!h) return false; + return provider_value_metadata(dst, &h->provider->levels, value); +} + +bool provider_get_task(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value) { + if(!h) return false; + return provider_value_metadata(dst, &h->provider->tasks, value); +} + +bool provider_get_opcode(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value) { + if(!h) return false; + return provider_value_metadata(dst, &h->provider->opcodes, value); +} diff --git a/src/collectors/windows-events.plugin/windows-events-providers.h b/src/collectors/windows-events.plugin/windows-events-providers.h new file mode 100644 index 000000000..b6d476c5c --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-providers.h @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_PROVIDERS_H +#define NETDATA_WINDOWS_EVENTS_PROVIDERS_H + +typedef enum __attribute__((packed)) { + WEVT_PLATFORM_UNKNOWN = 0, + WEVT_PLATFORM_WEL, + WEVT_PLATFORM_ETW, + WEVT_PLATFORM_TL, +} WEVT_PROVIDER_PLATFORM; + +#include "windows-events.h" + +struct provider_meta_handle; +typedef struct provider_meta_handle PROVIDER_META_HANDLE; + +PROVIDER_META_HANDLE *provider_get(ND_UUID uuid, LPCWSTR providerName); +void provider_release(PROVIDER_META_HANDLE *h); +EVT_HANDLE provider_handle(PROVIDER_META_HANDLE *h); +PROVIDER_META_HANDLE *provider_dup(PROVIDER_META_HANDLE *h); + +void providers_release_unused_handles(void); + +const char *provider_get_name(PROVIDER_META_HANDLE *p); +ND_UUID provider_get_uuid(PROVIDER_META_HANDLE *p); + +void provider_cache_init(void); + +bool provider_keyword_cacheable(PROVIDER_META_HANDLE *h); +bool provider_tasks_cacheable(PROVIDER_META_HANDLE *h); +bool is_useful_provider_for_levels(PROVIDER_META_HANDLE *h); +bool provider_opcodes_cacheable(PROVIDER_META_HANDLE *h); + +bool provider_get_keywords(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value); +bool provider_get_level(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value); +bool provider_get_task(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value); +bool provider_get_opcode(TXT_UTF8 *dst, PROVIDER_META_HANDLE *h, uint64_t value); +WEVT_PROVIDER_PLATFORM provider_get_platform(PROVIDER_META_HANDLE *p); + +#endif //NETDATA_WINDOWS_EVENTS_PROVIDERS_H diff --git a/src/collectors/windows-events.plugin/windows-events-query-builder.c b/src/collectors/windows-events.plugin/windows-events-query-builder.c new file mode 100644 index 000000000..75c6fbdca --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-query-builder.c @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events-query-builder.h" + +// -------------------------------------------------------------------------------------------------------------------- +// query without XPath + +typedef struct static_utf8_8k { + char buffer[8192]; + size_t size; + size_t len; +} STATIC_BUF_8K; + +typedef struct static_unicode_16k { + wchar_t buffer[16384]; + size_t size; + size_t len; +} STATIC_UNI_16K; + +static bool wevt_foreach_selected_value_cb(FACETS *facets __maybe_unused, size_t id, const char *key, const char *value, void *data) { + STATIC_BUF_8K *b = data; + + b->len += snprintfz(&b->buffer[b->len], b->size - b->len, + "%s%s=%s", + id ? " or " : "", key, value); + + return b->len < b->size; +} + +wchar_t *wevt_generate_query_no_xpath(LOGS_QUERY_STATUS *lqs, BUFFER *wb) { + static __thread STATIC_UNI_16K q = { + .size = sizeof(q.buffer) / sizeof(wchar_t), + .len = 0, + }; + static __thread STATIC_BUF_8K b = { + .size = sizeof(b.buffer) / sizeof(char), + .len = 0, + }; + + lqs_query_timeframe(lqs, ANCHOR_DELTA_UT); + + usec_t seek_to = lqs->query.start_ut; + if(lqs->rq.direction == FACETS_ANCHOR_DIRECTION_BACKWARD) + // windows events queries are limited to millisecond resolution + // so, in order not to lose data, we have to add + // a millisecond when the direction is backward + seek_to += USEC_PER_MS; + + // Convert the microseconds since Unix epoch to FILETIME (used in Windows APIs) + FILETIME fileTime = os_unix_epoch_ut_to_filetime(seek_to); + + // Convert FILETIME to SYSTEMTIME for use in XPath + SYSTEMTIME systemTime; + if (!FileTimeToSystemTime(&fileTime, &systemTime)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "FileTimeToSystemTime() failed"); + return NULL; + } + + // Format SYSTEMTIME into ISO 8601 format (YYYY-MM-DDTHH:MM:SS.sssZ) + q.len = swprintf(q.buffer, q.size, + L"Event/System[TimeCreated[@SystemTime%ls\"%04d-%02d-%02dT%02d:%02d:%02d.%03dZ\"]", + lqs->rq.direction == FACETS_ANCHOR_DIRECTION_BACKWARD ? L"<=" : L">=", + systemTime.wYear, systemTime.wMonth, systemTime.wDay, + systemTime.wHour, systemTime.wMinute, systemTime.wSecond, systemTime.wMilliseconds); + + if(lqs->rq.slice) { + b.len = snprintf(b.buffer, b.size, " and ("); + if (facets_foreach_selected_value_in_key( + lqs->facets, + WEVT_FIELD_LEVEL, + sizeof(WEVT_FIELD_LEVEL) - 1, + used_hashes_registry, + wevt_foreach_selected_value_cb, + &b)) { + b.len += snprintf(&b.buffer[b.len], b.size - b.len, ")"); + if (b.len < b.size) { + utf82unicode(&q.buffer[q.len], q.size - q.len, b.buffer); + q.len = wcslen(q.buffer); + } + } + + b.len = snprintf(b.buffer, b.size, " and ("); + if (facets_foreach_selected_value_in_key( + lqs->facets, + WEVT_FIELD_EVENTID, + sizeof(WEVT_FIELD_EVENTID) - 1, + used_hashes_registry, + wevt_foreach_selected_value_cb, + &b)) { + b.len += snprintf(&b.buffer[b.len], b.size - b.len, ")"); + if (b.len < b.size) { + utf82unicode(&q.buffer[q.len], q.size - q.len, b.buffer); + q.len = wcslen(q.buffer); + } + } + } + + q.len += swprintf(&q.buffer[q.len], q.size - q.len, L"]"); + + buffer_json_member_add_string(wb, "_query", channel2utf8(q.buffer)); + + return q.buffer; +} + +// -------------------------------------------------------------------------------------------------------------------- +// query with XPath + diff --git a/src/collectors/windows-events.plugin/windows-events-query-builder.h b/src/collectors/windows-events.plugin/windows-events-query-builder.h new file mode 100644 index 000000000..80136e0aa --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-query-builder.h @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_QUERY_BUILDER_H +#define NETDATA_WINDOWS_EVENTS_QUERY_BUILDER_H + +#include "windows-events.h" + +wchar_t *wevt_generate_query_no_xpath(LOGS_QUERY_STATUS *lqs, BUFFER *wb); + +#endif //NETDATA_WINDOWS_EVENTS_QUERY_BUILDER_H diff --git a/src/collectors/windows-events.plugin/windows-events-query-evt-variant.c b/src/collectors/windows-events.plugin/windows-events-query-evt-variant.c new file mode 100644 index 000000000..ee3aa382b --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-query-evt-variant.c @@ -0,0 +1,354 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events.h" +#include <sddl.h> // For SID string conversion + +// Function to append the separator if the buffer is not empty +static inline void append_separator_if_needed(BUFFER *b, const char *separator) { + if (buffer_strlen(b) > 0 && separator != NULL) + buffer_strcat(b, separator); +} + +// Helper function to convert UTF16 strings to UTF8 and append to the buffer +static inline void append_utf16(BUFFER *b, LPCWSTR utf16Str, const char *separator) { + if (!utf16Str || !*utf16Str) return; + + append_separator_if_needed(b, separator); + + size_t remaining = b->size - b->len; + if(remaining < 128) { + buffer_need_bytes(b, 128); + remaining = b->size - b->len; + } + + bool truncated = false; + size_t used = utf16_to_utf8(&b->buffer[b->len], remaining, utf16Str, -1, &truncated); + if(truncated) { + // we need to resize + size_t needed = utf16_to_utf8(NULL, 0, utf16Str, -1, NULL); // find the size needed + buffer_need_bytes(b, needed); + remaining = b->size - b->len; + used = utf16_to_utf8(&b->buffer[b->len], remaining, utf16Str, -1, NULL); + } + + if(used) { + b->len += used - 1; + + internal_fatal(buffer_strlen(b) != strlen(buffer_tostring(b)), + "Buffer length mismatch."); + } +} + +// Function to append binary data to the buffer +static inline void append_binary(BUFFER *b, PBYTE data, DWORD size, const char *separator) { + if (data == NULL || size == 0) return; + + append_separator_if_needed(b, separator); + + buffer_need_bytes(b, size * 4); + for (DWORD i = 0; i < size; i++) { + uint8_t value = data[i]; + b->buffer[b->len++] = hex_digits[(value & 0xf0) >> 4]; + b->buffer[b->len++] = hex_digits[(value & 0x0f)]; + } +} + +// Function to append size_t to the buffer +static inline void append_size_t(BUFFER *b, size_t size, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64(b, size); +} + +// Function to append HexInt32 in hexadecimal format +static inline void append_uint32_hex(BUFFER *b, UINT32 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64_hex(b, n); +} + +// Function to append HexInt64 in hexadecimal format +static inline void append_uint64_hex(BUFFER *b, UINT64 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64_hex(b, n); +} + +// Function to append various data types to the buffer +static inline void append_uint64(BUFFER *b, UINT64 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64(b, n); +} + +static inline void append_int64(BUFFER *b, INT64 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_int64(b, n); +} + +static inline void append_double(BUFFER *b, double n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_netdata_double(b, n); +} + +static inline void append_guid(BUFFER *b, GUID *guid, const char *separator) { + fatal_assert(sizeof(GUID) == sizeof(nd_uuid_t)); + + append_separator_if_needed(b, separator); + + ND_UUID *uuid = (ND_UUID *)guid; + buffer_need_bytes(b, UUID_STR_LEN); + uuid_unparse_lower(uuid->uuid, &b->buffer[b->len]); + b->len += UUID_STR_LEN - 1; + + internal_fatal(buffer_strlen(b) != strlen(buffer_tostring(b)), + "Buffer length mismatch."); +} + +static inline void append_systime(BUFFER *b, SYSTEMTIME *st, const char *separator) { + append_separator_if_needed(b, separator); + buffer_sprintf(b, "%04d-%02d-%02d %02d:%02d:%02d", + st->wYear, st->wMonth, st->wDay, st->wHour, st->wMinute, st->wSecond); +} + +static inline void append_filetime(BUFFER *b, FILETIME *ft, const char *separator) { + SYSTEMTIME st; + if (FileTimeToSystemTime(ft, &st)) + append_systime(b, &st, separator); +} + +static inline void append_sid(BUFFER *b, PSID sid, const char *separator) { + cached_sid_to_buffer_append(sid, b, separator); +} + +static inline void append_sbyte(BUFFER *b, INT8 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_int64(b, n); +} + +static inline void append_byte(BUFFER *b, UINT8 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64(b, n); +} + +static inline void append_int16(BUFFER *b, INT16 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_int64(b, n); +} + +static inline void append_uint16(BUFFER *b, UINT16 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64(b, n); +} + +static inline void append_int32(BUFFER *b, INT32 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_int64(b, n); +} + +static inline void append_uint32(BUFFER *b, UINT32 n, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64(b, n); +} + +// Function to append EVT_HANDLE to the buffer +static inline void append_evt_handle(BUFFER *b, EVT_HANDLE h, const char *separator) { + append_separator_if_needed(b, separator); + buffer_print_uint64_hex(b, (uintptr_t)h); +} + +// Function to append XML data (UTF-16) to the buffer +static inline void append_evt_xml(BUFFER *b, LPCWSTR xmlData, const char *separator) { + append_utf16(b, xmlData, separator); // XML data is essentially UTF-16 string +} + +void evt_variant_to_buffer(BUFFER *b, EVT_VARIANT *ev, const char *separator) { + if(ev->Type == EvtVarTypeNull) return; + + if (ev->Type & EVT_VARIANT_TYPE_ARRAY) { + for (DWORD i = 0; i < ev->Count; i++) { + switch (ev->Type & EVT_VARIANT_TYPE_MASK) { + case EvtVarTypeString: + append_utf16(b, ev->StringArr[i], separator); + break; + + case EvtVarTypeAnsiString: + if (ev->AnsiStringArr[i] != NULL) { + append_utf16(b, (LPCWSTR)ev->AnsiStringArr[i], separator); + } + break; + + case EvtVarTypeSByte: + append_sbyte(b, ev->SByteArr[i], separator); + break; + + case EvtVarTypeByte: + append_byte(b, ev->ByteArr[i], separator); + break; + + case EvtVarTypeInt16: + append_int16(b, ev->Int16Arr[i], separator); + break; + + case EvtVarTypeUInt16: + append_uint16(b, ev->UInt16Arr[i], separator); + break; + + case EvtVarTypeInt32: + append_int32(b, ev->Int32Arr[i], separator); + break; + + case EvtVarTypeUInt32: + append_uint32(b, ev->UInt32Arr[i], separator); + break; + + case EvtVarTypeInt64: + append_int64(b, ev->Int64Arr[i], separator); + break; + + case EvtVarTypeUInt64: + append_uint64(b, ev->UInt64Arr[i], separator); + break; + + case EvtVarTypeSingle: + append_double(b, ev->SingleArr[i], separator); + break; + + case EvtVarTypeDouble: + append_double(b, ev->DoubleArr[i], separator); + break; + + case EvtVarTypeGuid: + append_guid(b, &ev->GuidArr[i], separator); + break; + + case EvtVarTypeFileTime: + append_filetime(b, &ev->FileTimeArr[i], separator); + break; + + case EvtVarTypeSysTime: + append_systime(b, &ev->SysTimeArr[i], separator); + break; + + case EvtVarTypeSid: + append_sid(b, ev->SidArr[i], separator); + break; + + case EvtVarTypeBinary: + append_binary(b, ev->BinaryVal, ev->Count, separator); + break; + + case EvtVarTypeSizeT: + append_size_t(b, ev->SizeTArr[i], separator); + break; + + case EvtVarTypeHexInt32: + append_uint32_hex(b, ev->UInt32Arr[i], separator); + break; + + case EvtVarTypeHexInt64: + append_uint64_hex(b, ev->UInt64Arr[i], separator); + break; + + case EvtVarTypeEvtHandle: + append_evt_handle(b, ev->EvtHandleVal, separator); + break; + + case EvtVarTypeEvtXml: + append_evt_xml(b, ev->XmlValArr[i], separator); + break; + + default: + // Skip unknown array types + break; + } + } + } else { + switch (ev->Type & EVT_VARIANT_TYPE_MASK) { + case EvtVarTypeNull: + // Do nothing for null types + break; + + case EvtVarTypeString: + append_utf16(b, ev->StringVal, separator); + break; + + case EvtVarTypeAnsiString: + append_utf16(b, (LPCWSTR)ev->AnsiStringVal, separator); + break; + + case EvtVarTypeSByte: + append_sbyte(b, ev->SByteVal, separator); + break; + + case EvtVarTypeByte: + append_byte(b, ev->ByteVal, separator); + break; + + case EvtVarTypeInt16: + append_int16(b, ev->Int16Val, separator); + break; + + case EvtVarTypeUInt16: + append_uint16(b, ev->UInt16Val, separator); + break; + + case EvtVarTypeInt32: + append_int32(b, ev->Int32Val, separator); + break; + + case EvtVarTypeUInt32: + append_uint32(b, ev->UInt32Val, separator); + break; + + case EvtVarTypeInt64: + append_int64(b, ev->Int64Val, separator); + break; + + case EvtVarTypeUInt64: + append_uint64(b, ev->UInt64Val, separator); + break; + + case EvtVarTypeSingle: + append_double(b, ev->SingleVal, separator); + break; + + case EvtVarTypeDouble: + append_double(b, ev->DoubleVal, separator); + break; + + case EvtVarTypeBoolean: + append_separator_if_needed(b, separator); + buffer_strcat(b, ev->BooleanVal ? "true" : "false"); + break; + + case EvtVarTypeGuid: + append_guid(b, ev->GuidVal, separator); + break; + + case EvtVarTypeBinary: + append_binary(b, ev->BinaryVal, ev->Count, separator); + break; + + case EvtVarTypeSizeT: + append_size_t(b, ev->SizeTVal, separator); + break; + + case EvtVarTypeHexInt32: + append_uint32_hex(b, ev->UInt32Val, separator); + break; + + case EvtVarTypeHexInt64: + append_uint64_hex(b, ev->UInt64Val, separator); + break; + + case EvtVarTypeEvtHandle: + append_evt_handle(b, ev->EvtHandleVal, separator); + break; + + case EvtVarTypeEvtXml: + append_evt_xml(b, ev->XmlVal, separator); + break; + + default: + // Skip unknown types + break; + } + } +} diff --git a/src/collectors/windows-events.plugin/windows-events-query.c b/src/collectors/windows-events.plugin/windows-events-query.c new file mode 100644 index 000000000..fefa72829 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-query.c @@ -0,0 +1,717 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events.h" + +static void wevt_event_done(WEVT_LOG *log); + +static uint64_t wevt_log_file_size(const wchar_t *channel); + +// -------------------------------------------------------------------------------------------------------------------- + +static const char *EvtGetExtendedStatus_utf8(void) { + static __thread wchar_t wbuf[4096]; + static __thread char buf[4096]; + DWORD wbuf_used = 0; + + if(EvtGetExtendedStatus(sizeof(wbuf) / sizeof(wchar_t), wbuf, &wbuf_used) == ERROR_SUCCESS) { + wbuf[sizeof(wbuf) / sizeof(wchar_t) - 1] = 0; + unicode2utf8(buf, sizeof(buf), wbuf); + } + else + buf[0] = '\0'; + + // the EvtGetExtendedStatus() may be successful with an empty message + if(!buf[0]) + strncpyz(buf, "no additional information", sizeof(buf) - 1); + + return buf; +} + +// -------------------------------------------------------------------------------------------------------------------- + +bool EvtFormatMessage_utf16( + TXT_UTF16 *dst, EVT_HANDLE hMetadata, EVT_HANDLE hEvent, DWORD dwMessageId, EVT_FORMAT_MESSAGE_FLAGS flags) { + dst->used = 0; + + DWORD size = 0; + if(!dst->data) { + EvtFormatMessage(hMetadata, hEvent, dwMessageId, 0, NULL, flags, 0, NULL, &size); + if(!size) { + // nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtFormatMessage() to get message size failed."); + goto cleanup; + } + txt_utf16_resize(dst, size, false); + } + + // First, try to get the message using the existing buffer + if (!EvtFormatMessage(hMetadata, hEvent, dwMessageId, 0, NULL, flags, dst->size, dst->data, &size) || !dst->data) { + if (dst->data && GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + // nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtFormatMessage() failed."); + goto cleanup; + } + + // Try again with the resized buffer + txt_utf16_resize(dst, size, false); + if (!EvtFormatMessage(hMetadata, hEvent, dwMessageId, 0, NULL, flags, dst->size, dst->data, &size)) { + // nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtFormatMessage() failed after resizing buffer."); + goto cleanup; + } + } + + // make sure it is null terminated + if(size <= dst->size) + dst->data[size - 1] = 0; + else + dst->data[dst->size - 1] = 0; + + // unfortunately we have to calculate the length every time + // the size returned may not be the length of the dst string + dst->used = wcslen(dst->data) + 1; + + return true; + +cleanup: + dst->used = 0; + return false; +} + +static bool EvtFormatMessage_utf8( + TXT_UTF16 *tmp, PROVIDER_META_HANDLE *p, EVT_HANDLE hEvent, + TXT_UTF8 *dst, EVT_FORMAT_MESSAGE_FLAGS flags) { + + dst->src = TXT_SOURCE_EVENT_LOG; + + if(EvtFormatMessage_utf16(tmp, provider_handle(p), hEvent, 0, flags)) + return txt_utf16_to_utf8(dst, tmp); + + txt_utf8_empty(dst); + return false; +} + +bool EvtFormatMessage_Event_utf8(TXT_UTF16 *tmp, PROVIDER_META_HANDLE *p, EVT_HANDLE hEvent, TXT_UTF8 *dst) { + return EvtFormatMessage_utf8(tmp, p, hEvent, dst, EvtFormatMessageEvent); +} + +bool EvtFormatMessage_Xml_utf8(TXT_UTF16 *tmp, PROVIDER_META_HANDLE *p, EVT_HANDLE hEvent, TXT_UTF8 *dst) { + return EvtFormatMessage_utf8(tmp, p, hEvent, dst, EvtFormatMessageXml); +} + +// -------------------------------------------------------------------------------------------------------------------- + +static void wevt_get_field_from_cache( + WEVT_LOG *log, uint64_t value, PROVIDER_META_HANDLE *h, + TXT_UTF8 *dst, const ND_UUID *provider, + WEVT_FIELD_TYPE cache_type, EVT_FORMAT_MESSAGE_FLAGS flags) { + + if (field_cache_get(cache_type, provider, value, dst)) + return; + + EvtFormatMessage_utf8(&log->ops.unicode, h, log->hEvent, dst, flags); + field_cache_set(cache_type, provider, value, dst); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Level + +#define SET_LEN_AND_RETURN(constant) *len = sizeof(constant) - 1; return constant + +static inline const char *wevt_level_hardcoded(uint64_t level, size_t *len) { + switch(level) { + case WEVT_LEVEL_NONE: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_NONE); + case WEVT_LEVEL_CRITICAL: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_CRITICAL); + case WEVT_LEVEL_ERROR: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_ERROR); + case WEVT_LEVEL_WARNING: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_WARNING); + case WEVT_LEVEL_INFORMATION: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_INFORMATION); + case WEVT_LEVEL_VERBOSE: SET_LEN_AND_RETURN(WEVT_LEVEL_NAME_VERBOSE); + default: *len = 0; return NULL; + } +} + +static void wevt_get_level(WEVT_LOG *log, WEVT_EVENT *ev, PROVIDER_META_HANDLE *h) { + TXT_UTF8 *dst = &log->ops.level; + uint64_t value = ev->level; + + txt_utf8_empty(dst); + + EVT_FORMAT_MESSAGE_FLAGS flags = EvtFormatMessageLevel; + WEVT_FIELD_TYPE cache_type = WEVT_FIELD_TYPE_LEVEL; + bool is_provider = is_valid_provider_level(value, true); + + if(!is_provider) { + size_t len; + const char *hardcoded = wevt_level_hardcoded(value, &len); + if(hardcoded) { + txt_utf8_set(dst, hardcoded, len); + dst->src = TXT_SOURCE_HARDCODED; + } + else { + // since this is not a provider value + // we expect to get the system description of it + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + } + else if (!provider_get_level(dst, h, value)) { + // not found in the manifest, get it from the cache + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + + txt_utf8_set_numeric_if_empty( + dst, WEVT_PREFIX_LEVEL, sizeof(WEVT_PREFIX_LEVEL) - 1, ev->level); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Opcode + +static inline const char *wevt_opcode_hardcoded(uint64_t opcode, size_t *len) { + switch(opcode) { + case WEVT_OPCODE_INFO: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_INFO); + case WEVT_OPCODE_START: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_START); + case WEVT_OPCODE_STOP: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_STOP); + case WEVT_OPCODE_DC_START: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_DC_START); + case WEVT_OPCODE_DC_STOP: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_DC_STOP); + case WEVT_OPCODE_EXTENSION: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_EXTENSION); + case WEVT_OPCODE_REPLY: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_REPLY); + case WEVT_OPCODE_RESUME: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_RESUME); + case WEVT_OPCODE_SUSPEND: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_SUSPEND); + case WEVT_OPCODE_SEND: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_SEND); + case WEVT_OPCODE_RECEIVE: SET_LEN_AND_RETURN(WEVT_OPCODE_NAME_RECEIVE); + default: *len = 0; return NULL; + } +} + +static void wevt_get_opcode(WEVT_LOG *log, WEVT_EVENT *ev, PROVIDER_META_HANDLE *h) { + TXT_UTF8 *dst = &log->ops.opcode; + uint64_t value = ev->opcode; + + txt_utf8_empty(dst); + + EVT_FORMAT_MESSAGE_FLAGS flags = EvtFormatMessageOpcode; + WEVT_FIELD_TYPE cache_type = WEVT_FIELD_TYPE_OPCODE; + bool is_provider = is_valid_provider_opcode(value, true); + + if(!is_provider) { + size_t len; + const char *hardcoded = wevt_opcode_hardcoded(value, &len); + if(hardcoded) { + txt_utf8_set(dst, hardcoded, len); + dst->src = TXT_SOURCE_HARDCODED; + } + else { + // since this is not a provider value + // we expect to get the system description of it + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + } + else if (!provider_get_opcode(dst, h, value)) { + // not found in the manifest, get it from the cache + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + + txt_utf8_set_numeric_if_empty( + dst, WEVT_PREFIX_OPCODE, sizeof(WEVT_PREFIX_OPCODE) - 1, ev->opcode); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Task + +static const char *wevt_task_hardcoded(uint64_t task, size_t *len) { + switch(task) { + case WEVT_TASK_NONE: SET_LEN_AND_RETURN(WEVT_TASK_NAME_NONE); + default: *len = 0; return NULL; + } +} + +static void wevt_get_task(WEVT_LOG *log, WEVT_EVENT *ev, PROVIDER_META_HANDLE *h) { + TXT_UTF8 *dst = &log->ops.task; + uint64_t value = ev->task; + + txt_utf8_empty(dst); + + EVT_FORMAT_MESSAGE_FLAGS flags = EvtFormatMessageTask; + WEVT_FIELD_TYPE cache_type = WEVT_FIELD_TYPE_TASK; + bool is_provider = is_valid_provider_task(value, true); + + if(!is_provider) { + size_t len; + const char *hardcoded = wevt_task_hardcoded(value, &len); + if(hardcoded) { + txt_utf8_set(dst, hardcoded, len); + dst->src = TXT_SOURCE_HARDCODED; + } + else { + // since this is not a provider value + // we expect to get the system description of it + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + } + else if (!provider_get_task(dst, h, value)) { + // not found in the manifest, get it from the cache + wevt_get_field_from_cache(log, value, h, dst, &ev->provider, cache_type, flags); + } + + txt_utf8_set_numeric_if_empty( + dst, WEVT_PREFIX_TASK, sizeof(WEVT_PREFIX_TASK) - 1, ev->task); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Keyword + +#define SET_BITS(msk, txt) { .mask = msk, .name = txt, .len = sizeof(txt) - 1, } + +static uint64_t wevt_keyword_handle_reserved(uint64_t value, TXT_UTF8 *dst) { + struct { + uint64_t mask; + const char *name; + size_t len; + } bits[] = { + SET_BITS(WEVT_KEYWORD_EVENTLOG_CLASSIC, WEVT_KEYWORD_NAME_EVENTLOG_CLASSIC), + SET_BITS(WEVT_KEYWORD_CORRELATION_HINT, WEVT_KEYWORD_NAME_CORRELATION_HINT), + SET_BITS(WEVT_KEYWORD_AUDIT_SUCCESS, WEVT_KEYWORD_NAME_AUDIT_SUCCESS), + SET_BITS(WEVT_KEYWORD_AUDIT_FAILURE, WEVT_KEYWORD_NAME_AUDIT_FAILURE), + SET_BITS(WEVT_KEYWORD_SQM, WEVT_KEYWORD_NAME_SQM), + SET_BITS(WEVT_KEYWORD_WDI_DIAG, WEVT_KEYWORD_NAME_WDI_DIAG), + SET_BITS(WEVT_KEYWORD_WDI_CONTEXT, WEVT_KEYWORD_NAME_WDI_CONTEXT), + SET_BITS(WEVT_KEYWORD_RESPONSE_TIME, WEVT_KEYWORD_NAME_RESPONSE_TIME), + }; + + txt_utf8_empty(dst); + + for(size_t i = 0; i < sizeof(bits) / sizeof(bits[0]) ;i++) { + if((value & bits[i].mask) == bits[i].mask) { + txt_utf8_add_keywords_separator_if_needed(dst); + txt_utf8_append(dst, bits[i].name, bits[i].len); + value &= ~(bits[i].mask); + dst->src = TXT_SOURCE_HARDCODED; + } + } + + // return it without any remaining reserved bits + return value & 0x0000FFFFFFFFFFFF; +} + +static void wevt_get_keyword(WEVT_LOG *log, WEVT_EVENT *ev, PROVIDER_META_HANDLE *h) { + TXT_UTF8 *dst = &log->ops.keywords; + + if(ev->keywords == WEVT_KEYWORD_NONE) { + txt_utf8_set(dst, WEVT_KEYWORD_NAME_NONE, sizeof(WEVT_KEYWORD_NAME_NONE) - 1); + dst->src = TXT_SOURCE_HARDCODED; + } + + uint64_t value = wevt_keyword_handle_reserved(ev->keywords, dst); + + EVT_FORMAT_MESSAGE_FLAGS flags = EvtFormatMessageKeyword; + WEVT_FIELD_TYPE cache_type = WEVT_FIELD_TYPE_KEYWORD; + + if(!value && dst->used <= 1) { + // no hardcoded info in the buffer, make it None + txt_utf8_set(dst, WEVT_KEYWORD_NAME_NONE, sizeof(WEVT_KEYWORD_NAME_NONE) - 1); + dst->src = TXT_SOURCE_HARDCODED; + } + else if (value && !provider_get_keywords(dst, h, value) && dst->used <= 1) { + // the provider did not provide any info and the description is still empty. + // the system returns 1 keyword, the highest bit, not a list + // so, when we call the system, we pass the original value (ev->keywords) + wevt_get_field_from_cache(log, ev->keywords, h, dst, &ev->provider, cache_type, flags); + } + + txt_utf8_set_hex_if_empty( + dst, WEVT_PREFIX_KEYWORDS, sizeof(WEVT_PREFIX_KEYWORDS) - 1, ev->keywords); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Fetching Events + +static inline bool wEvtRender(WEVT_LOG *log, EVT_HANDLE context, WEVT_VARIANT *raw) { + DWORD bytes_used = 0, property_count = 0; + if (!EvtRender(context, log->hEvent, EvtRenderEventValues, raw->size, raw->data, &bytes_used, &property_count)) { + // information exceeds the allocated space + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "EvtRender() failed, hRenderSystemContext: 0x%lx, hEvent: 0x%lx, content: 0x%lx, size: %u, extended info: %s", + (uintptr_t)context, (uintptr_t)log->hEvent, (uintptr_t)raw->data, raw->size, + EvtGetExtendedStatus_utf8()); + return false; + } + + wevt_variant_resize(raw, bytes_used); + if (!EvtRender(context, log->hEvent, EvtRenderEventValues, raw->size, raw->data, &bytes_used, &property_count)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "EvtRender() failed, after bytes_used increase, extended info: %s", + EvtGetExtendedStatus_utf8()); + return false; + } + } + raw->used = bytes_used; + raw->count = property_count; + + return true; +} + +static bool wevt_get_next_event_one(WEVT_LOG *log, WEVT_EVENT *ev) { + bool ret = false; + + if(!wEvtRender(log, log->hRenderSystemContext, &log->ops.raw.system)) + goto cleanup; + + EVT_VARIANT *content = log->ops.raw.system.data; + + ev->id = wevt_field_get_uint64(&content[EvtSystemEventRecordId]); + ev->event_id = wevt_field_get_uint16(&content[EvtSystemEventID]); + ev->level = wevt_field_get_uint8(&content[EvtSystemLevel]); + ev->opcode = wevt_field_get_uint8(&content[EvtSystemOpcode]); + ev->keywords = wevt_field_get_uint64_hex(&content[EvtSystemKeywords]); + ev->version = wevt_field_get_uint8(&content[EvtSystemVersion]); + ev->task = wevt_field_get_uint16(&content[EvtSystemTask]); + ev->qualifiers = wevt_field_get_uint16(&content[EvtSystemQualifiers]); + ev->process_id = wevt_field_get_uint32(&content[EvtSystemProcessID]); + ev->thread_id = wevt_field_get_uint32(&content[EvtSystemThreadID]); + ev->created_ns = wevt_field_get_filetime_to_ns(&content[EvtSystemTimeCreated]); + + if(log->type & WEVT_QUERY_EXTENDED) { + wevt_field_get_string_utf8(&content[EvtSystemChannel], &log->ops.channel); + wevt_field_get_string_utf8(&content[EvtSystemComputer], &log->ops.computer); + wevt_field_get_string_utf8(&content[EvtSystemProviderName], &log->ops.provider); + wevt_get_uuid_by_type(&content[EvtSystemProviderGuid], &ev->provider); + wevt_get_uuid_by_type(&content[EvtSystemActivityID], &ev->activity_id); + wevt_get_uuid_by_type(&content[EvtSystemRelatedActivityID], &ev->related_activity_id); + wevt_field_get_sid(&content[EvtSystemUserID], &log->ops.account, &log->ops.domain, &log->ops.sid); + + PROVIDER_META_HANDLE *p = log->provider = + provider_get(ev->provider, content[EvtSystemProviderName].StringVal); + + ev->platform = provider_get_platform(p); + + wevt_get_level(log, ev, p); + wevt_get_task(log, ev, p); + wevt_get_opcode(log, ev, p); + wevt_get_keyword(log, ev, p); + + if(log->type & WEVT_QUERY_EVENT_DATA && wEvtRender(log, log->hRenderUserContext, &log->ops.raw.user)) { +#if (ON_FTS_PRELOAD_MESSAGE == 1) + EvtFormatMessage_Event_utf8(&log->ops.unicode, log->provider, log->hEvent, &log->ops.event); +#endif +#if (ON_FTS_PRELOAD_XML == 1) + EvtFormatMessage_Xml_utf8(&log->ops.unicode, log->provider, log->hEvent, &log->ops.xml); +#endif +#if (ON_FTS_PRELOAD_EVENT_DATA == 1) + for(size_t i = 0; i < log->ops.raw.user.count ;i++) + evt_variant_to_buffer(log->ops.event_data, &log->ops.raw.user.data[i], " ||| "); +#endif + } + } + + ret = true; + +cleanup: + return ret; +} + +bool wevt_get_next_event(WEVT_LOG *log, WEVT_EVENT *ev) { + DWORD size = (log->type & WEVT_QUERY_EXTENDED) ? BATCH_NEXT_EVENT : 1; + DWORD max_failures = 10; + + fatal_assert(log && log->hQuery && log->hRenderSystemContext); + + while(max_failures > 0) { + if (log->batch.used >= log->batch.size) { + log->batch.size = 0; + log->batch.used = 0; + DWORD err; + if(!EvtNext(log->hQuery, size, log->batch.hEvents, INFINITE, 0, &log->batch.size)) { + err = GetLastError(); + if(err == ERROR_NO_MORE_ITEMS) + return false; // no data available, return failure + } + + if(!log->batch.size) { + if(size == 1) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "EvtNext() failed, hQuery: 0x%lx, size: %zu, extended info: %s", + (uintptr_t)log->hQuery, (size_t)size, EvtGetExtendedStatus_utf8()); + return false; + } + + // EvtNext() returns true when it can full the array + // so, let's retry with a smaller array. + size /= 2; + if(size < 1) size = 1; + continue; + } + } + + log->query_stats.event_count++; + log->log_stats.event_count++; + + // cleanup any previous event data + wevt_event_done(log); + + log->hEvent = log->batch.hEvents[log->batch.used]; + log->batch.hEvents[log->batch.used] = NULL; + log->batch.used++; + + if(wevt_get_next_event_one(log, ev)) + return true; + else { + log->query_stats.failed_count++; + log->log_stats.failed_count++; + max_failures--; + } + } + + return false; +} + +static void wevt_event_done(WEVT_LOG *log) { + if (log->provider) { + provider_release(log->provider); + log->provider = NULL; + } + + if (log->hEvent) { + EvtClose(log->hEvent); + log->hEvent = NULL; + } + + log->ops.channel.src = TXT_SOURCE_UNKNOWN; + log->ops.provider.src = TXT_SOURCE_UNKNOWN; + log->ops.computer.src = TXT_SOURCE_UNKNOWN; + log->ops.account.src = TXT_SOURCE_UNKNOWN; + log->ops.domain.src = TXT_SOURCE_UNKNOWN; + log->ops.sid.src = TXT_SOURCE_UNKNOWN; + + log->ops.event.src = TXT_SOURCE_UNKNOWN; + log->ops.level.src = TXT_SOURCE_UNKNOWN; + log->ops.keywords.src = TXT_SOURCE_UNKNOWN; + log->ops.opcode.src = TXT_SOURCE_UNKNOWN; + log->ops.task.src = TXT_SOURCE_UNKNOWN; + log->ops.xml.src = TXT_SOURCE_UNKNOWN; + + log->ops.channel.used = 0; + log->ops.provider.used = 0; + log->ops.computer.used = 0; + log->ops.account.used = 0; + log->ops.domain.used = 0; + log->ops.sid.used = 0; + + log->ops.event.used = 0; + log->ops.level.used = 0; + log->ops.keywords.used = 0; + log->ops.opcode.used = 0; + log->ops.task.used = 0; + log->ops.xml.used = 0; + + if(log->ops.event_data) + log->ops.event_data->len = 0; +} + +// -------------------------------------------------------------------------------------------------------------------- +// Query management + +bool wevt_query(WEVT_LOG *log, LPCWSTR channel, LPCWSTR query, EVT_QUERY_FLAGS direction) { + wevt_query_done(log); + log->log_stats.queries_count++; + + EVT_HANDLE hQuery = EvtQuery(NULL, channel, query, EvtQueryChannelPath | (direction & (EvtQueryReverseDirection | EvtQueryForwardDirection)) | EvtQueryTolerateQueryErrors); + if (!hQuery) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtQuery() failed, query: %s | extended info: %s", + query2utf8(query), EvtGetExtendedStatus_utf8()); + + log->log_stats.queries_failed++; + return false; + } + + log->hQuery = hQuery; + return true; +} + +void wevt_query_done(WEVT_LOG *log) { + // close the last working hEvent + wevt_event_done(log); + + // close all batched hEvents + for(DWORD i = log->batch.used; i < log->batch.size ;i++) { + if(log->batch.hEvents[i]) + EvtClose(log->batch.hEvents[i]); + + log->batch.hEvents[i] = NULL; + } + log->batch.used = 0; + log->batch.size = 0; + + if (log->hQuery) { + EvtClose(log->hQuery); + log->hQuery = NULL; + } + + log->query_stats.event_count = 0; + log->query_stats.failed_count = 0; +} + +// -------------------------------------------------------------------------------------------------------------------- +// Log management + +WEVT_LOG *wevt_openlog6(WEVT_QUERY_TYPE type) { + WEVT_LOG *log = callocz(1, sizeof(*log)); + log->type = type; + + // create the system render + log->hRenderSystemContext = EvtCreateRenderContext(0, NULL, EvtRenderContextSystem); + if (!log->hRenderSystemContext) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "EvtCreateRenderContext() on system context failed, extended info: %s", + EvtGetExtendedStatus_utf8()); + goto cleanup; + } + + if(type & WEVT_QUERY_EVENT_DATA) { + log->hRenderUserContext = EvtCreateRenderContext(0, NULL, EvtRenderContextUser); + if (!log->hRenderUserContext) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "EvtCreateRenderContext failed, on user context failed, extended info: %s", + EvtGetExtendedStatus_utf8()); + goto cleanup; + } + + log->ops.event_data = buffer_create(4096, NULL); + } + + return log; + +cleanup: + wevt_closelog6(log); + return NULL; +} + +void wevt_closelog6(WEVT_LOG *log) { + wevt_query_done(log); + + if (log->hRenderSystemContext) + EvtClose(log->hRenderSystemContext); + + if (log->hRenderUserContext) + EvtClose(log->hRenderUserContext); + + wevt_variant_cleanup(&log->ops.raw.system); + wevt_variant_cleanup(&log->ops.raw.user); + txt_utf16_cleanup(&log->ops.unicode); + txt_utf8_cleanup(&log->ops.channel); + txt_utf8_cleanup(&log->ops.provider); + txt_utf8_cleanup(&log->ops.computer); + txt_utf8_cleanup(&log->ops.account); + txt_utf8_cleanup(&log->ops.domain); + txt_utf8_cleanup(&log->ops.sid); + + txt_utf8_cleanup(&log->ops.event); + txt_utf8_cleanup(&log->ops.level); + txt_utf8_cleanup(&log->ops.keywords); + txt_utf8_cleanup(&log->ops.opcode); + txt_utf8_cleanup(&log->ops.task); + txt_utf8_cleanup(&log->ops.xml); + + buffer_free(log->ops.event_data); + + freez(log); +} + +// -------------------------------------------------------------------------------------------------------------------- +// Retention + +bool wevt_channel_retention(WEVT_LOG *log, const wchar_t *channel, const wchar_t *query, EVT_RETENTION *retention) { + bool ret = false; + + // get the number of the oldest record in the log + // "EvtGetLogInfo()" does not work properly with "EvtLogOldestRecordNumber" + // we have to get it from the first EventRecordID + + // query the eventlog + log->hQuery = EvtQuery(NULL, channel, query, EvtQueryChannelPath | EvtQueryForwardDirection | EvtQueryTolerateQueryErrors); + if (!log->hQuery) { + if (GetLastError() == ERROR_EVT_CHANNEL_NOT_FOUND) + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtQuery() for retention failed, channel '%s' not found, cannot get retention, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + else + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtQuery() for retention on channel '%s' failed, cannot get retention, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + + goto cleanup; + } + + if (!wevt_get_next_event(log, &retention->first_event)) + goto cleanup; + + if (!retention->first_event.id) { + // no data in the event log + retention->first_event = retention->last_event = WEVT_EVENT_EMPTY; + ret = true; + goto cleanup; + } + EvtClose(log->hQuery); + + log->hQuery = EvtQuery(NULL, channel, query, EvtQueryChannelPath | EvtQueryReverseDirection | EvtQueryTolerateQueryErrors); + if (!log->hQuery) { + if (GetLastError() == ERROR_EVT_CHANNEL_NOT_FOUND) + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtQuery() for retention failed, channel '%s' not found, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + else + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtQuery() for retention on channel '%s' failed, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + + goto cleanup; + } + + if (!wevt_get_next_event(log, &retention->last_event) || retention->last_event.id == 0) { + // no data in eventlog + retention->last_event = retention->first_event; + } + retention->last_event.id += 1; // we should read the last record + ret = true; + +cleanup: + wevt_query_done(log); + + if(ret) { + retention->entries = (channel && !query) ? retention->last_event.id - retention->first_event.id : 0; + + if(retention->last_event.created_ns >= retention->first_event.created_ns) + retention->duration_ns = retention->last_event.created_ns - retention->first_event.created_ns; + else + retention->duration_ns = retention->first_event.created_ns - retention->last_event.created_ns; + + retention->size_bytes = wevt_log_file_size(channel); + } + else + memset(retention, 0, sizeof(*retention)); + + return ret; +} + +static uint64_t wevt_log_file_size(const wchar_t *channel) { + EVT_HANDLE hLog = NULL; + EVT_VARIANT evtVariant; + DWORD bufferUsed = 0; + uint64_t file_size = 0; + + // Open the event log channel + hLog = EvtOpenLog(NULL, channel, EvtOpenChannelPath); + if (!hLog) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtOpenLog() on channel '%s' failed, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + goto cleanup; + } + + // Get the file size of the log + if (!EvtGetLogInfo(hLog, EvtLogFileSize, sizeof(evtVariant), &evtVariant, &bufferUsed)) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "EvtGetLogInfo() on channel '%s' failed, extended info: %s", + channel2utf8(channel), EvtGetExtendedStatus_utf8()); + goto cleanup; + } + + // Extract the file size from the EVT_VARIANT structure + file_size = evtVariant.UInt64Val; + +cleanup: + if (hLog) + EvtClose(hLog); + + return file_size; +} diff --git a/src/collectors/windows-events.plugin/windows-events-query.h b/src/collectors/windows-events.plugin/windows-events-query.h new file mode 100644 index 000000000..3136b23df --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-query.h @@ -0,0 +1,296 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_QUERY_H +#define NETDATA_WINDOWS_EVENTS_QUERY_H + +#include "libnetdata/libnetdata.h" +#include "windows-events.h" + +#define BATCH_NEXT_EVENT 500 + +typedef struct wevt_event { + uint64_t id; // EventRecordId (unique and sequential per channel) + uint8_t version; + uint8_t level; // The severity of event + uint8_t opcode; // we receive this as 8bit, but providers use 32bit + uint16_t event_id; // This is the template that defines the message to be shown + uint16_t task; + uint16_t qualifiers; + uint32_t process_id; + uint32_t thread_id; + uint64_t keywords; // Categorization of the event + ND_UUID provider; + ND_UUID activity_id; + ND_UUID related_activity_id; + nsec_t created_ns; + WEVT_PROVIDER_PLATFORM platform; +} WEVT_EVENT; + +#define WEVT_EVENT_EMPTY (WEVT_EVENT){ .id = 0, .created_ns = 0, } + +typedef struct { + EVT_VARIANT *data; + DWORD size; + DWORD used; + DWORD count; +} WEVT_VARIANT; + +typedef struct { + WEVT_EVENT first_event; + WEVT_EVENT last_event; + + uint64_t entries; + nsec_t duration_ns; + uint64_t size_bytes; +} EVT_RETENTION; + +struct provider_meta_handle; + +typedef enum __attribute__((packed)) { + WEVT_QUERY_BASIC = (1 << 0), + WEVT_QUERY_EXTENDED = (1 << 1), + WEVT_QUERY_EVENT_DATA = (1 << 2), +} WEVT_QUERY_TYPE; + +#define WEVT_QUERY_RETENTION WEVT_QUERY_BASIC +#define WEVT_QUERY_NORMAL (WEVT_QUERY_BASIC | WEVT_QUERY_EXTENDED) +#define WEVT_QUERY_FTS (WEVT_QUERY_BASIC | WEVT_QUERY_EXTENDED | WEVT_QUERY_EVENT_DATA) + +typedef struct wevt_log { + struct { + DWORD size; + DWORD used; + EVT_HANDLE hEvents[BATCH_NEXT_EVENT]; + } batch; + + EVT_HANDLE hEvent; + EVT_HANDLE hQuery; + EVT_HANDLE hRenderSystemContext; + EVT_HANDLE hRenderUserContext; + struct provider_meta_handle *provider; + + WEVT_QUERY_TYPE type; + + struct { + struct { + // temp buffer used for rendering event log messages + // never use directly + WEVT_VARIANT system; + WEVT_VARIANT user; + } raw; + + // temp buffer used for fetching and converting UNICODE and UTF-8 + // every string operation overwrites it, multiple times per event log entry + // it can be used within any function, for its own purposes, + // but never share between functions + TXT_UTF16 unicode; + + // string attributes of the current event log entry + // valid until another event if fetched + + // IMPORTANT: + // EVERY FIELD NEEDS ITS OWN BUFFER! + // the way facets work, all the field value pointers need to be valid + // until the entire row closes, so reusing a buffer for the same field + // actually copies the same value to all fields using the same buffer. + + TXT_UTF8 channel; + TXT_UTF8 provider; + TXT_UTF8 computer; + TXT_UTF8 account; + TXT_UTF8 domain; + TXT_UTF8 sid; + + TXT_UTF8 event; // the message to be shown to the user + TXT_UTF8 level; + TXT_UTF8 keywords; + TXT_UTF8 opcode; + TXT_UTF8 task; + TXT_UTF8 xml; + + BUFFER *event_data; + } ops; + + struct { + size_t event_count; + size_t failed_count; + } query_stats; + + struct { + size_t queries_count; + size_t queries_failed; + + size_t event_count; + size_t failed_count; + } log_stats; + +} WEVT_LOG; + +WEVT_LOG *wevt_openlog6(WEVT_QUERY_TYPE type); +void wevt_closelog6(WEVT_LOG *log); + +bool wevt_channel_retention(WEVT_LOG *log, const wchar_t *channel, const wchar_t *query, EVT_RETENTION *retention); + +bool wevt_query(WEVT_LOG *log, LPCWSTR channel, LPCWSTR query, EVT_QUERY_FLAGS direction); +void wevt_query_done(WEVT_LOG *log); + +bool wevt_get_next_event(WEVT_LOG *log, WEVT_EVENT *ev); + +bool EvtFormatMessage_utf16( + TXT_UTF16 *dst, EVT_HANDLE hMetadata, EVT_HANDLE hEvent, DWORD dwMessageId, EVT_FORMAT_MESSAGE_FLAGS flags); + +bool EvtFormatMessage_Event_utf8(TXT_UTF16 *tmp, struct provider_meta_handle *p, EVT_HANDLE hEvent, TXT_UTF8 *dst); +bool EvtFormatMessage_Xml_utf8(TXT_UTF16 *tmp, struct provider_meta_handle *p, EVT_HANDLE hEvent, TXT_UTF8 *dst); + +void evt_variant_to_buffer(BUFFER *b, EVT_VARIANT *ev, const char *separator); + +static inline void wevt_variant_cleanup(WEVT_VARIANT *v) { + freez(v->data); +} + +static inline void wevt_variant_resize(WEVT_VARIANT *v, size_t required_size) { + if(required_size < v->size) + return; + + wevt_variant_cleanup(v); + v->size = txt_compute_new_size(v->size, required_size); + v->data = mallocz(v->size); +} + +static inline void wevt_variant_count_from_used(WEVT_VARIANT *v) { + v->count = v->used / sizeof(*v->data); +} + +static inline uint8_t wevt_field_get_uint8(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeByte); + return ev->ByteVal; +} + +static inline uint16_t wevt_field_get_uint16(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeUInt16); + return ev->UInt16Val; +} + +static inline uint32_t wevt_field_get_uint32(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeUInt32); + return ev->UInt32Val; +} + +static inline uint64_t wevt_field_get_uint64(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeUInt64); + return ev->UInt64Val; +} + +static inline uint64_t wevt_field_get_uint64_hex(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeHexInt64); + return ev->UInt64Val; +} + +static inline bool wevt_field_get_string_utf8(EVT_VARIANT *ev, TXT_UTF8 *dst) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) { + txt_utf8_empty(dst); + return false; + } + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeString); + return wchar_to_txt_utf8(dst, ev->StringVal, -1); +} + +bool cached_sid_to_account_domain_sidstr(PSID sid, TXT_UTF8 *dst_account, TXT_UTF8 *dst_domain, TXT_UTF8 *dst_sid_str); +static inline bool wevt_field_get_sid(EVT_VARIANT *ev, TXT_UTF8 *dst_account, TXT_UTF8 *dst_domain, TXT_UTF8 *dst_sid_str) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) { + txt_utf8_empty(dst_account); + txt_utf8_empty(dst_domain); + txt_utf8_empty(dst_sid_str); + return false; + } + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeSid); + return cached_sid_to_account_domain_sidstr(ev->SidVal, dst_account, dst_domain, dst_sid_str); +} + +static inline uint64_t wevt_field_get_filetime_to_ns(EVT_VARIANT *ev) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) + return 0; + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeFileTime); + return os_windows_ulonglong_to_unix_epoch_ns(ev->FileTimeVal); +} + +static inline bool wevt_GUID_to_ND_UUID(ND_UUID *nd_uuid, const GUID *guid) { + if(guid && sizeof(GUID) == sizeof(ND_UUID)) { + memcpy(nd_uuid->uuid, guid, sizeof(ND_UUID)); + return true; + } + else { + *nd_uuid = UUID_ZERO; + return false; + } +} + +static inline bool wevt_get_uuid_by_type(EVT_VARIANT *ev, ND_UUID *dst) { + if((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeNull) { + wevt_GUID_to_ND_UUID(dst, NULL); + return false; + } + + fatal_assert((ev->Type & EVT_VARIANT_TYPE_MASK) == EvtVarTypeGuid); + return wevt_GUID_to_ND_UUID(dst, ev->GuidVal); +} + +// https://learn.microsoft.com/en-us/windows/win32/wes/defining-severity-levels +static inline bool is_valid_provider_level(uint64_t level, bool strict) { + if(strict) + // when checking if the name is provider independent + return level >= 16 && level <= 255; + else + // when checking acceptable values in provider manifests + return level <= 255; +} + +// https://learn.microsoft.com/en-us/windows/win32/wes/defining-tasks-and-opcodes +static inline bool is_valid_provider_opcode(uint64_t opcode, bool strict) { + if(strict) + // when checking if the name is provider independent + return opcode >= 10 && opcode <= 239; + else + // when checking acceptable values in provider manifests + return opcode <= 255; +} + +// https://learn.microsoft.com/en-us/windows/win32/wes/defining-tasks-and-opcodes +static inline bool is_valid_provider_task(uint64_t task, bool strict) { + if(strict) + // when checking if the name is provider independent + return task > 0 && task <= 0xFFFF; + else + // when checking acceptable values in provider manifests + return task <= 0xFFFF; +} + +// https://learn.microsoft.com/en-us/windows/win32/wes/defining-keywords-used-to-classify-types-of-events +static inline bool is_valid_provider_keyword(uint64_t keyword, bool strict) { + if(strict) + // when checking if the name is provider independent + return keyword > 0 && keyword <= 0x0000FFFFFFFFFFFF; + else + // when checking acceptable values in provider manifests + return true; +} + +#endif //NETDATA_WINDOWS_EVENTS_QUERY_H diff --git a/src/collectors/windows-events.plugin/windows-events-sources.c b/src/collectors/windows-events.plugin/windows-events-sources.c new file mode 100644 index 000000000..b931ed059 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-sources.c @@ -0,0 +1,644 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events.h" + +//struct { +// const char *name; +// const wchar_t *query; +//} custom_queries[] = { +// { +// .name = "All-Administrative-Events", +// .query = L"<QueryList>\n" +// " <Query Id=\"0\" Path=\"Application\">\n" +// " <Select Path=\"Application\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Security\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"System\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"HardwareEvents\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Internet Explorer\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Key Management Service\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-AppV-Client/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-AppV-Client/Virtual Applications\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-All-User-Install-Agent/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-AppHost/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Application Server-Applications/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-AppModel-Runtime/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-AppReadiness/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-AssignedAccess/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-AssignedAccessBroker/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Storage-ATAPort/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-BitLocker-DrivePreparationTool/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Client-Licensing-Platform/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DataIntegrityScan/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DataIntegrityScan/CrashRecovery\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DSC/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Autopilot\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DeviceSetupManager/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Dhcp-Client/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Dhcpv6-Client/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Diagnosis-Scripted/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Storage-Disk/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-DxgKrnl-Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-EDP-Application-Learning/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-EDP-Audit-Regular/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-EDP-Audit-TCB/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Client-License-Flexible-Platform/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-GenericRoaming/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Hyper-V-Guest-Drivers/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Hyper-V-Hypervisor-Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Hyper-V-VID-Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Kernel-EventTracing/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-KeyboardFilter/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-ModernDeployment-Diagnostics-Provider/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-ModernDeployment-Diagnostics-Provider/Autopilot\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-ModernDeployment-Diagnostics-Provider/Diagnostics\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-ModernDeployment-Diagnostics-Provider/ManagementService\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-MUI/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-PowerShell/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-PrintBRM/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-PrintService/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Provisioning-Diagnostics-Provider/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Provisioning-Diagnostics-Provider/AutoPilot\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Provisioning-Diagnostics-Provider/ManagementService\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-PushNotification-Platform/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteApp and Desktop Connections/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteAssistance/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-RetailDemo/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-SecurityMitigationsBroker/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-SmartCard-TPM-VCard-Module/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-SMBDirect/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-SMBWitnessClient/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Storage-Tiering/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Storage-ClassPnP/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Storage-Storport/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ClientUSBDevices/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-LocalSessionManager/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-PnPDevices/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-Printers/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ServerUSBDevices/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Troubleshooting-Recommended/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-User Device Registration/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-VerifyHardwareSecurity/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-WindowsBackup/ActionCenter\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Microsoft-Windows-Workplace Join/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"OAlerts\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"OneApp_IGCC\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"OpenSSH/Admin\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"USER_ESRV_SVC_QUEENCREEK\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Visual Studio\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " <Select Path=\"Windows PowerShell\">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\n" +// " </Query>\n" +// "</QueryList>", +// }, +// { +// .name = "All-Remote-Desktop-Services", +// .query = L"<QueryList>\n" +// " <Query Id=\"0\" Path=\"Microsoft-Rdms-UI/Admin\">\n" +// " <Select Path=\"Microsoft-Rdms-UI/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Rdms-UI/Operational\">*</Select>\n" +// " <Select Path=\"Remote-Desktop-Management-Service/Admin\">*</Select>\n" +// " <Select Path=\"Remote-Desktop-Management-Service/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-SessionBroker-Client/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-SessionBroker-Client/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-PnPDevices/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-PnPDevices/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteApp and Desktop Connections/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteApp and Desktop Connection Management/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-RemoteApp and Desktop Connection Management/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-SessionBroker/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-SessionBroker/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-TSV-VmHostAgent/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-TSV-VmHostAgent/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ServerUSBDevices/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ServerUSBDevices/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-LocalSessionManager/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-LocalSessionManager/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ClientUSBDevices/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-ClientUSBDevices/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-RDPClient/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-Licensing/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-Licensing/Operational\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-Gateway/Admin\">*</Select>\n" +// " <Select Path=\"Microsoft-Windows-TerminalServices-Gateway/Operational\">*</Select>\n" +// " </Query>\n" +// "</QueryList>", +// }, +// { +// .name = "All-Security-SPP", +// .query = L"<QueryList>\n" +// " <Query Id=\"0\" Path=\"Microsoft-Windows-HelloForBusiness/Operational\">\n" +// " <Select Path=\"Microsoft-Windows-HelloForBusiness/Operational\">*[System[(Level>5 )]]</Select>\n" +// " </Query>\n" +// "</QueryList>", +// } +//}; + +ENUM_STR_MAP_DEFINE(WEVT_SOURCE_TYPE) = { + { .id = WEVTS_ALL, .name = WEVT_SOURCE_ALL_NAME }, + { .id = WEVTS_ADMIN, .name = WEVT_SOURCE_ALL_ADMIN_NAME }, + { .id = WEVTS_OPERATIONAL, .name = WEVT_SOURCE_ALL_OPERATIONAL_NAME }, + { .id = WEVTS_ANALYTIC, .name = WEVT_SOURCE_ALL_ANALYTIC_NAME }, + { .id = WEVTS_DEBUG, .name = WEVT_SOURCE_ALL_DEBUG_NAME }, + { .id = WEVTS_WINDOWS, .name = WEVT_SOURCE_ALL_WINDOWS_NAME }, + { .id = WEVTS_ENABLED, .name = WEVT_SOURCE_ALL_ENABLED_NAME }, + { .id = WEVTS_DISABLED, .name = WEVT_SOURCE_ALL_DISABLED_NAME }, + { .id = WEVTS_FORWARDED, .name = WEVT_SOURCE_ALL_FORWARDED_NAME }, + { .id = WEVTS_CLASSIC, .name = WEVT_SOURCE_ALL_CLASSIC_NAME }, + { .id = WEVTS_BACKUP_MODE, .name = WEVT_SOURCE_ALL_BACKUP_MODE_NAME }, + { .id = WEVTS_OVERWRITE_MODE, .name = WEVT_SOURCE_ALL_OVERWRITE_MODE_NAME }, + { .id = WEVTS_STOP_WHEN_FULL_MODE, .name = WEVT_SOURCE_ALL_STOP_WHEN_FULL_MODE_NAME }, + { .id = WEVTS_RETAIN_AND_BACKUP_MODE, .name = WEVT_SOURCE_ALL_RETAIN_AND_BACKUP_MODE_NAME }, + + // terminator + { . id = 0, .name = NULL } +}; + +BITMAP_STR_DEFINE_FUNCTIONS(WEVT_SOURCE_TYPE, WEVTS_NONE, ""); + +DICTIONARY *wevt_sources = NULL; +DICTIONARY *used_hashes_registry = NULL; +static usec_t wevt_session = 0; + +void wevt_sources_del_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) { + LOGS_QUERY_SOURCE *src = value; + freez((void *)src->fullname); + string_freez(src->source); + + src->fullname = NULL; + src->source = NULL; +} + +static bool wevt_sources_conflict_cb(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) { + LOGS_QUERY_SOURCE *src_old = old_value; + LOGS_QUERY_SOURCE *src_new = new_value; + + bool ret = false; + if(src_new->last_scan_monotonic_ut > src_old->last_scan_monotonic_ut) { + src_old->last_scan_monotonic_ut = src_new->last_scan_monotonic_ut; + + if (src_old->source != src_new->source) { + string_freez(src_old->source); + src_old->source = src_new->source; + src_new->source = NULL; + } + src_old->source_type = src_new->source_type; + + src_old->msg_first_ut = src_new->msg_first_ut; + src_old->msg_last_ut = src_new->msg_last_ut; + src_old->msg_first_id = src_new->msg_first_id; + src_old->msg_last_id = src_new->msg_last_id; + src_old->entries = src_new->entries; + src_old->size = src_new->size; + + ret = true; + } + + freez((void *)src_new->fullname); + string_freez(src_new->source); + src_new->fullname = NULL; + src_new->source = NULL; + + return ret; +} + +void wevt_sources_init(void) { + wevt_session = now_realtime_usec(); + + used_hashes_registry = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE); + + wevt_sources = dictionary_create_advanced(DICT_OPTION_FIXED_SIZE | DICT_OPTION_DONT_OVERWRITE_VALUE, + NULL, sizeof(LOGS_QUERY_SOURCE)); + + dictionary_register_delete_callback(wevt_sources, wevt_sources_del_cb, NULL); + dictionary_register_conflict_callback(wevt_sources, wevt_sources_conflict_cb, NULL); +} + +void buffer_json_wevt_versions(BUFFER *wb __maybe_unused) { + buffer_json_member_add_object(wb, "versions"); + { + buffer_json_member_add_uint64(wb, "sources", + wevt_session + dictionary_version(wevt_sources)); + } + buffer_json_object_close(wb); +} + +// -------------------------------------------------------------------------------------------------------------------- + +int wevt_sources_dict_items_backward_compar(const void *a, const void *b) { + const DICTIONARY_ITEM **da = (const DICTIONARY_ITEM **)a, **db = (const DICTIONARY_ITEM **)b; + LOGS_QUERY_SOURCE *sa = dictionary_acquired_item_value(*da); + LOGS_QUERY_SOURCE *sb = dictionary_acquired_item_value(*db); + + // compare the last message timestamps + if(sa->msg_last_ut < sb->msg_last_ut) + return 1; + + if(sa->msg_last_ut > sb->msg_last_ut) + return -1; + + // compare the first message timestamps + if(sa->msg_first_ut < sb->msg_first_ut) + return 1; + + if(sa->msg_first_ut > sb->msg_first_ut) + return -1; + + return 0; +} + +int wevt_sources_dict_items_forward_compar(const void *a, const void *b) { + return -wevt_sources_dict_items_backward_compar(a, b); +} + +// -------------------------------------------------------------------------------------------------------------------- + +typedef enum { + wevt_source_type_internal, + wevt_source_type_provider, + wevt_source_type_channel, +} wevt_source_type; + +struct wevt_source { + wevt_source_type type; + usec_t first_ut; + usec_t last_ut; + size_t count; + size_t entries; + uint64_t size; +}; + +static int wevt_source_to_json_array_cb(const DICTIONARY_ITEM *item, void *entry, void *data) { + const struct wevt_source *s = entry; + BUFFER *wb = data; + + const char *name = dictionary_acquired_item_name(item); + + if(s->count == 1 && strncmp(name, WEVT_SOURCE_ALL_OF_PROVIDER_PREFIX, sizeof(WEVT_SOURCE_ALL_OF_PROVIDER_PREFIX) - 1) == 0) + // do not include "All-Of-X" when there is only 1 channel + return 0; + + bool default_selected = (s->type == wevt_source_type_channel); + if(default_selected && (strcmp(name, "NetdataWEL") == 0 || strcmp(name, "Netdata/Access") == 0)) + // do not select Netdata Access logs by default + default_selected = false; + + buffer_json_add_array_item_object(wb); + { + char size_for_humans[128]; + size_snprintf(size_for_humans, sizeof(size_for_humans), s->size, "B", false); + + char duration_for_humans[128]; + duration_snprintf(duration_for_humans, sizeof(duration_for_humans), + (time_t)((s->last_ut - s->first_ut) / USEC_PER_SEC), "s", true); + + char entries_for_humans[128]; + entries_snprintf(entries_for_humans, sizeof(entries_for_humans), s->entries, "", false); + + char info[1024]; + snprintfz(info, sizeof(info), "%zu channel%s, with a total size of %s, covering %s%s%s%s", + s->count, s->count > 1 ? "s":"", size_for_humans, duration_for_humans, + s->entries ? ", having " : "", s->entries ? entries_for_humans : "", s->entries ? " entries" : ""); + + buffer_json_member_add_string(wb, "id", name); + buffer_json_member_add_string(wb, "name", name); + buffer_json_member_add_string(wb, "pill", size_for_humans); + buffer_json_member_add_string(wb, "info", info); + buffer_json_member_add_boolean(wb, "default_selected", default_selected); + } + buffer_json_object_close(wb); // options object + + return 1; +} + +static bool wevt_source_merge_sizes(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value , void *data __maybe_unused) { + struct wevt_source *old_v = old_value; + const struct wevt_source *new_v = new_value; + + old_v->count += new_v->count; + old_v->size += new_v->size; + old_v->entries += new_v->entries; + + if(new_v->first_ut && new_v->first_ut < old_v->first_ut) + old_v->first_ut = new_v->first_ut; + + if(new_v->last_ut && new_v->last_ut > old_v->last_ut) + old_v->last_ut = new_v->last_ut; + + return false; +} + +void wevt_sources_to_json_array(BUFFER *wb) { + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED|DICT_OPTION_NAME_LINK_DONT_CLONE|DICT_OPTION_DONT_OVERWRITE_VALUE); + dictionary_register_conflict_callback(dict, wevt_source_merge_sizes, NULL); + + struct wevt_source t = { 0 }; + + LOGS_QUERY_SOURCE *src; + dfe_start_read(wevt_sources, src) { + t.first_ut = src->msg_first_ut; + t.last_ut = src->msg_last_ut; + t.count = 1; + t.size = src->size; + t.entries = src->entries; + + src->source_type |= WEVTS_ALL; + t.type = wevt_source_type_internal; + for(size_t i = 0; WEVT_SOURCE_TYPE_names[i].name ;i++) { + if(src->source_type & WEVT_SOURCE_TYPE_names[i].id) + dictionary_set(dict, WEVT_SOURCE_TYPE_names[i].name, &t, sizeof(t)); + } + + if(src->provider) { + t.type = wevt_source_type_provider; + dictionary_set(dict, string2str(src->provider), &t, sizeof(t)); + } + + if(src->source) { + t.type = wevt_source_type_channel; + dictionary_set(dict, string2str(src->source), &t, sizeof(t)); + } + } + dfe_done(jf); + + dictionary_sorted_walkthrough_read(dict, wevt_source_to_json_array_cb, wb); +} + +static bool ndEvtGetChannelConfigProperty(EVT_HANDLE hChannelConfig, WEVT_VARIANT *pr, EVT_CHANNEL_CONFIG_PROPERTY_ID id) { + if (!EvtGetChannelConfigProperty(hChannelConfig, id, 0, pr->size, pr->data, &pr->used)) { + DWORD status = GetLastError(); + if (ERROR_INSUFFICIENT_BUFFER == status) { + wevt_variant_resize(pr, pr->used); + if(!EvtGetChannelConfigProperty(hChannelConfig, id, 0, pr->size, pr->data, &pr->used)) { + pr->used = 0; + pr->count = 0; + return false; + } + } + } + + wevt_variant_count_from_used(pr); + return true; +} + +WEVT_SOURCE_TYPE categorize_channel(const wchar_t *channel_path, const char **provider, WEVT_VARIANT *property) { + EVT_HANDLE hChannelConfig = NULL; + WEVT_SOURCE_TYPE result = WEVTS_ALL; + + // Open the channel configuration + hChannelConfig = EvtOpenChannelConfig(NULL, channel_path, 0); + if (!hChannelConfig) + goto cleanup; + + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelConfigType) & + property->count && + property->data[0].Type == EvtVarTypeUInt32) { + switch (property->data[0].UInt32Val) { + case EvtChannelTypeAdmin: + result |= WEVTS_ADMIN; + break; + + case EvtChannelTypeOperational: + result |= WEVTS_OPERATIONAL; + break; + + case EvtChannelTypeAnalytic: + result |= WEVTS_ANALYTIC; + break; + + case EvtChannelTypeDebug: + result |= WEVTS_DEBUG; + break; + + default: + break; + } + } + + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelConfigClassicEventlog) && + property->count && + property->data[0].Type == EvtVarTypeBoolean && + property->data[0].BooleanVal) + result |= WEVTS_CLASSIC; + + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelConfigOwningPublisher) && + property->count && + property->data[0].Type == EvtVarTypeString) { + *provider = provider2utf8(property->data[0].StringVal); + if(wcscasecmp(property->data[0].StringVal, L"Microsoft-Windows-EventCollector") == 0) + result |= WEVTS_FORWARDED; + } + else + *provider = NULL; + + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelConfigEnabled) && + property->count && + property->data[0].Type == EvtVarTypeBoolean) { + if(property->data[0].BooleanVal) + result |= WEVTS_ENABLED; + else + result |= WEVTS_DISABLED; + } + + bool got_retention = false; + bool retained = false; + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelLoggingConfigRetention) && + property->count && + property->data[0].Type == EvtVarTypeBoolean) { + got_retention = true; + retained = property->data[0].BooleanVal; + } + + bool got_auto_backup = false; + bool auto_backup = false; + if(ndEvtGetChannelConfigProperty(hChannelConfig, property, EvtChannelLoggingConfigAutoBackup) && + property->count && + property->data[0].Type == EvtVarTypeBoolean) { + got_auto_backup = true; + auto_backup = property->data[0].BooleanVal; + } + + if(got_retention && got_auto_backup) { + if(!retained) { + if(auto_backup) + result |= WEVTS_BACKUP_MODE; + else + result |= WEVTS_OVERWRITE_MODE; + } + else { + if(auto_backup) + result |= WEVTS_STOP_WHEN_FULL_MODE; + else + result |= WEVTS_RETAIN_AND_BACKUP_MODE; + } + } + +cleanup: + if (hChannelConfig) + EvtClose(hChannelConfig); + + return result; +} + +void wevt_sources_scan(void) { + static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER; + LPWSTR channel = NULL; + EVT_HANDLE hChannelEnum = NULL; + + if(spinlock_trylock(&spinlock)) { + const usec_t started_ut = now_monotonic_usec(); + + WEVT_VARIANT property = { 0 }; + DWORD dwChannelBufferSize = 0; + DWORD dwChannelBufferUsed = 0; + DWORD status = ERROR_SUCCESS; + + // Open a handle to enumerate the event channels + hChannelEnum = EvtOpenChannelEnum(NULL, 0); + if (!hChannelEnum) { + nd_log(NDLS_COLLECTORS, NDLP_ERR, "WINDOWS EVENTS: EvtOpenChannelEnum() failed with %" PRIu64 "\n", + (uint64_t)GetLastError()); + goto cleanup; + } + + WEVT_LOG *log = wevt_openlog6(WEVT_QUERY_RETENTION); + if(!log) goto cleanup; + + while (true) { + if (!EvtNextChannelPath(hChannelEnum, dwChannelBufferSize, channel, &dwChannelBufferUsed)) { + status = GetLastError(); + if (status == ERROR_NO_MORE_ITEMS) + break; // No more channels + else if (status == ERROR_INSUFFICIENT_BUFFER) { + dwChannelBufferSize = dwChannelBufferUsed; + freez(channel); + channel = mallocz(dwChannelBufferSize * sizeof(WCHAR)); + continue; + } else { + nd_log(NDLS_COLLECTORS, NDLP_ERR, + "WINDOWS EVENTS: EvtNextChannelPath() failed\n"); + break; + } + } + + EVT_RETENTION retention; + if(!wevt_channel_retention(log, channel, NULL, &retention)) + continue; + + LOGS_QUERY_SOURCE *found = dictionary_get(wevt_sources, channel2utf8(channel)); + if(found) { + // we just need to update its retention + + found->last_scan_monotonic_ut = now_monotonic_usec(); + found->msg_first_id = retention.first_event.id; + found->msg_last_id = retention.last_event.id; + found->msg_first_ut = retention.first_event.created_ns / NSEC_PER_USEC; + found->msg_last_ut = retention.last_event.created_ns / NSEC_PER_USEC; + found->size = retention.size_bytes; + continue; + } + + const char *name = channel2utf8(channel); + const char *fullname = strdupz(name); + const char *provider; + + WEVT_SOURCE_TYPE sources = categorize_channel(channel, &provider, &property); + char *slash = strchr(name, '/'); + if(slash) *slash = '\0'; + + if(strcasecmp(name, "Application") == 0) + sources |= WEVTS_WINDOWS; + if(strcasecmp(name, "Security") == 0) + sources |= WEVTS_WINDOWS; + if(strcasecmp(name, "Setup") == 0) + sources |= WEVTS_WINDOWS; + if(strcasecmp(name, "System") == 0) + sources |= WEVTS_WINDOWS; + + LOGS_QUERY_SOURCE src = { + .entries = retention.entries, + .fullname = fullname, + .fullname_len = strlen(fullname), + .last_scan_monotonic_ut = now_monotonic_usec(), + .msg_first_id = retention.first_event.id, + .msg_last_id = retention.last_event.id, + .msg_first_ut = retention.first_event.created_ns / NSEC_PER_USEC, + .msg_last_ut = retention.last_event.created_ns / NSEC_PER_USEC, + .size = retention.size_bytes, + .source_type = sources, + .source = string_strdupz(fullname), + }; + + if(strncmp(fullname, "Netdata", 7) == 0) + // WEL based providers of Netdata are named NetdataX + provider = "Netdata"; + + if(provider && *provider) { + char buf[sizeof(WEVT_SOURCE_ALL_OF_PROVIDER_PREFIX) + strlen(provider)]; // sizeof() includes terminator + snprintf(buf, sizeof(buf), WEVT_SOURCE_ALL_OF_PROVIDER_PREFIX "%s", provider); + + if(trim_all(buf) != NULL) { + for (size_t i = 0; i < sizeof(buf) - 1; i++) { + // remove character that may interfere with our parsing + if (isspace((uint8_t) buf[i]) || buf[i] == '%' || buf[i] == '+' || buf[i] == '|' || buf[i] == ':') + buf[i] = '_'; + } + src.provider = string_strdupz(buf); + } + } + + dictionary_set(wevt_sources, src.fullname, &src, sizeof(src)); + } + +// // add custom queries +// for(size_t i = 0; i < sizeof(custom_queries) / sizeof(custom_queries[0]) ;i++) { +// EVT_RETENTION retention; +// if(!wevt_channel_retention(log, NULL, custom_queries[i].query, &retention)) +// continue; +// +// LOGS_QUERY_SOURCE src = { +// .entries = 0, +// .fullname = strdupz(custom_queries[i].name), +// .fullname_len = strlen(custom_queries[i].name), +// .last_scan_monotonic_ut = now_monotonic_usec(), +// .msg_first_id = retention.first_event.id, +// .msg_last_id = retention.last_event.id, +// .msg_first_ut = retention.first_event.created_ns / NSEC_PER_USEC, +// .msg_last_ut = retention.last_event.created_ns / NSEC_PER_USEC, +// .size = retention.size_bytes, +// .source_type = WEVTS_ALL, +// .source = string_strdupz(custom_queries[i].name), +// }; +// +// dictionary_set(wevt_sources, src.fullname, &src, sizeof(src)); +// } +// + wevt_closelog6(log); + + LOGS_QUERY_SOURCE *src; + dfe_start_write(wevt_sources, src) + { + if(src->last_scan_monotonic_ut < started_ut) { + src->msg_first_id = 0; + src->msg_last_id = 0; + src->msg_first_ut = 0; + src->msg_last_ut = 0; + src->size = 0; + dictionary_del(wevt_sources, src->fullname); + } + } + dfe_done(src); + dictionary_garbage_collect(wevt_sources); + + spinlock_unlock(&spinlock); + + wevt_variant_cleanup(&property); + } + +cleanup: + freez(channel); + EvtClose(hChannelEnum); +} diff --git a/src/collectors/windows-events.plugin/windows-events-sources.h b/src/collectors/windows-events.plugin/windows-events-sources.h new file mode 100644 index 000000000..4ad4880d7 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-sources.h @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_SOURCES_H +#define NETDATA_WINDOWS_EVENTS_SOURCES_H + +#include "libnetdata/libnetdata.h" + +typedef enum { + WEVTS_NONE = 0, + WEVTS_ALL = (1 << 0), + WEVTS_ADMIN = (1 << 1), + WEVTS_OPERATIONAL = (1 << 2), + WEVTS_ANALYTIC = (1 << 3), + WEVTS_DEBUG = (1 << 4), + WEVTS_WINDOWS = (1 << 5), + WEVTS_ENABLED = (1 << 6), + WEVTS_DISABLED = (1 << 7), + WEVTS_FORWARDED = (1 << 8), + WEVTS_CLASSIC = (1 << 9), + WEVTS_BACKUP_MODE = (1 << 10), + WEVTS_OVERWRITE_MODE = (1 << 11), + WEVTS_STOP_WHEN_FULL_MODE = (1 << 12), + WEVTS_RETAIN_AND_BACKUP_MODE = (1 << 13), +} WEVT_SOURCE_TYPE; + +BITMAP_STR_DEFINE_FUNCTIONS_EXTERN(WEVT_SOURCE_TYPE) + +#define WEVT_SOURCE_ALL_NAME "All" +#define WEVT_SOURCE_ALL_ADMIN_NAME "All-Admin" +#define WEVT_SOURCE_ALL_OPERATIONAL_NAME "All-Operational" +#define WEVT_SOURCE_ALL_ANALYTIC_NAME "All-Analytic" +#define WEVT_SOURCE_ALL_DEBUG_NAME "All-Debug" +#define WEVT_SOURCE_ALL_WINDOWS_NAME "All-Windows" +#define WEVT_SOURCE_ALL_ENABLED_NAME "All-Enabled" +#define WEVT_SOURCE_ALL_DISABLED_NAME "All-Disabled" +#define WEVT_SOURCE_ALL_FORWARDED_NAME "All-Forwarded" +#define WEVT_SOURCE_ALL_CLASSIC_NAME "All-Classic" +#define WEVT_SOURCE_ALL_BACKUP_MODE_NAME "All-In-Backup-Mode" +#define WEVT_SOURCE_ALL_OVERWRITE_MODE_NAME "All-In-Overwrite-Mode" +#define WEVT_SOURCE_ALL_STOP_WHEN_FULL_MODE_NAME "All-In-StopWhenFull-Mode" +#define WEVT_SOURCE_ALL_RETAIN_AND_BACKUP_MODE_NAME "All-In-RetainAndBackup-Mode" + +#define WEVT_SOURCE_ALL_OF_PROVIDER_PREFIX "All-Of-" + +typedef struct { + const char *fullname; + size_t fullname_len; + + const wchar_t *custom_query; + + STRING *source; + STRING *provider; + WEVT_SOURCE_TYPE source_type; + usec_t msg_first_ut; + usec_t msg_last_ut; + size_t size; + + usec_t last_scan_monotonic_ut; + + uint64_t msg_first_id; + uint64_t msg_last_id; + uint64_t entries; +} LOGS_QUERY_SOURCE; + +extern DICTIONARY *wevt_sources; +extern DICTIONARY *used_hashes_registry; + +void wevt_sources_init(void); +void wevt_sources_scan(void); +void buffer_json_wevt_versions(BUFFER *wb); + +void wevt_sources_to_json_array(BUFFER *wb); +WEVT_SOURCE_TYPE wevt_internal_source_type(const char *value); + +int wevt_sources_dict_items_backward_compar(const void *a, const void *b); +int wevt_sources_dict_items_forward_compar(const void *a, const void *b); + +#endif //NETDATA_WINDOWS_EVENTS_SOURCES_H diff --git a/src/collectors/windows-events.plugin/windows-events-unicode.c b/src/collectors/windows-events.plugin/windows-events-unicode.c new file mode 100644 index 000000000..81da31107 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-unicode.c @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events-unicode.h" + +inline void utf82unicode(wchar_t *dst, size_t dst_size, const char *src) { + if (src) { + // Convert from UTF-8 to wide char (UTF-16) + if (utf8_to_utf16(dst, dst_size, src, -1) == 0) + wcsncpy(dst, L"[failed conv.]", dst_size - 1); + } + else + wcsncpy(dst, L"[null]", dst_size - 1); +} + +inline void unicode2utf8(char *dst, size_t dst_size, const wchar_t *src) { + if (src) { + if(WideCharToMultiByte(CP_UTF8, 0, src, -1, dst, (int)dst_size, NULL, NULL) == 0) + strncpyz(dst, "[failed conv.]", dst_size - 1); + } + else + strncpyz(dst, "[null]", dst_size - 1); +} + +wchar_t *channel2unicode(const char *utf8str) { + static __thread wchar_t buffer[1024]; + utf82unicode(buffer, _countof(buffer), utf8str); + return buffer; +} + +char *channel2utf8(const wchar_t *channel) { + static __thread char buffer[1024]; + unicode2utf8(buffer, sizeof(buffer), channel); + return buffer; +} + +char *query2utf8(const wchar_t *query) { + static __thread char buffer[16384]; + unicode2utf8(buffer, sizeof(buffer), query); + return buffer; +} + +char *provider2utf8(const wchar_t *provider) { + static __thread char buffer[256]; + unicode2utf8(buffer, sizeof(buffer), provider); + return buffer; +} diff --git a/src/collectors/windows-events.plugin/windows-events-unicode.h b/src/collectors/windows-events.plugin/windows-events-unicode.h new file mode 100644 index 000000000..e932bb5df --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-unicode.h @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_UNICODE_H +#define NETDATA_WINDOWS_EVENTS_UNICODE_H + +#include "libnetdata/libnetdata.h" + +#define WINEVENT_NAME_KEYWORDS_SEPARATOR ", " +static inline void txt_utf8_add_keywords_separator_if_needed(TXT_UTF8 *dst) { + if(dst->used > 1) + txt_utf8_append(dst, WINEVENT_NAME_KEYWORDS_SEPARATOR, sizeof(WINEVENT_NAME_KEYWORDS_SEPARATOR) - 1); +} + +static inline void txt_utf8_set_numeric_if_empty(TXT_UTF8 *dst, const char *prefix, size_t len, uint64_t value) { + if(dst->used <= 1) { + txt_utf8_resize(dst, len + UINT64_MAX_LENGTH + 1, false); + memcpy(dst->data, prefix, len); + dst->used = len + print_uint64(&dst->data[len], value) + 1; + } +} + +static inline void txt_utf8_set_hex_if_empty(TXT_UTF8 *dst, const char *prefix, size_t len, uint64_t value) { + if(dst->used <= 1) { + txt_utf8_resize(dst, len + UINT64_HEX_MAX_LENGTH + 1, false); + memcpy(dst->data, prefix, len); + dst->used = len + print_uint64_hex_full(&dst->data[len], value) + 1; + } +} + +// -------------------------------------------------------------------------------------------------------------------- +// conversions + +void unicode2utf8(char *dst, size_t dst_size, const wchar_t *src); +void utf82unicode(wchar_t *dst, size_t dst_size, const char *src); + +char *channel2utf8(const wchar_t *channel); +wchar_t *channel2unicode(const char *utf8str); + +char *query2utf8(const wchar_t *query); +char *provider2utf8(const wchar_t *provider); + +#endif //NETDATA_WINDOWS_EVENTS_UNICODE_H diff --git a/src/collectors/windows-events.plugin/windows-events-xml.c b/src/collectors/windows-events.plugin/windows-events-xml.c new file mode 100644 index 000000000..931ea6c54 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-xml.c @@ -0,0 +1,344 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "windows-events-xml.h" + +#include <string.h> +#include <stdio.h> + +#define INDENT_STEP 2 +#define A_LOT_OF_SPACES " " + +// Helper: Add indentation +static inline void buffer_add_xml_indent(BUFFER *buffer, const int level) { + size_t total_spaces = (size_t)level * INDENT_STEP; + const size_t step = sizeof(A_LOT_OF_SPACES) - 1; + while (total_spaces > 0) { + const size_t spaces_to_add = (total_spaces > step) ? step : total_spaces; + buffer_fast_strcat(buffer, A_LOT_OF_SPACES, spaces_to_add); + total_spaces -= spaces_to_add; + } +} + +const char *append_the_rest(BUFFER *buffer, const char *xml, const char *end) { + if(xml >= end) return end; + buffer_fast_strcat(buffer, xml, end - xml); + return end; +} + +static const char *parse_node(BUFFER *buffer, const char *xml, const char *end, int level); + +// Helper: Parse the value (between > and <) and return the next position to parse +const char *parse_value_and_closing_tag(BUFFER *buffer, const char *xml, const char *end, int level) { + const char *start = xml; + bool has_subnodes = false; + + // const char *tag_start = NULL, *tag_end = NULL; + while (xml < end) { + if(*xml == '<') { + if(xml + 1 < end && *(xml + 1) == '/') { + // a closing tag + xml += 2; + +// tag_start = xml; + + while(xml < end && *xml != '>') + xml++; + +// tag_end = xml; + + if(xml < end && *xml == '>') + xml++; + + if(has_subnodes) { + buffer_putc(buffer, '\n'); + buffer_add_xml_indent(buffer, level); + } + + buffer_fast_strcat(buffer, start, xml - start); + return xml; + } + else { + // an opening tag + buffer_fast_strcat(buffer, start, xml - start); + xml = start = parse_node(buffer, xml, end, level + 1); + while(xml < end && isspace((uint8_t)*xml)) + xml++; + has_subnodes = true; + } + } + else + xml++; + } + + return append_the_rest(buffer, start, end); +} + +// Parse a field value and return the next position to parse +const char *parse_field_value(BUFFER *buffer, const char *xml, const char *end) { + const char quote = *xml; + + if(quote != '"' && quote != '\'') + return append_the_rest(buffer, xml, end); + + const char *start = xml++; + + while (xml < end && *xml != quote) { + if (*xml == '\\') { + xml++; // Skip escape character + + if(xml < end) + xml++; + + continue; + } + + xml++; + } + + if(xml < end && *xml == quote) { + xml++; // Move past the closing quote + buffer_fast_strcat(buffer, start, xml - start); + return xml; + } + + return append_the_rest(buffer, start, end); +} + +// Parse a field name and return the next position to parse +const char *parse_field(BUFFER *buffer, const char *xml, const char *end) { + while(isspace((uint8_t)*xml) && xml < end) xml++; + + const char *start = xml; + + while (*xml != '=' && xml < end) + xml++; + + // Append the field name + buffer_fast_strcat(buffer, start, xml - start); + + if(xml < end && *xml == '=') { + xml++; + + buffer_putc(buffer, '='); + + if(xml < end && (*xml == '"' || *xml == '\'')) + xml = parse_field_value(buffer, xml, end); + + return xml; // Return the next character to parse + } + + return append_the_rest(buffer, start, end); +} + +// Parse a node (handles fields and subnodes) and return the next position to parse +static inline const char *parse_node(BUFFER *buffer, const char *xml, const char *end, int level) { + if(*xml != '<') + return append_the_rest(buffer, xml, end); + + const char *start = xml++; // skip the < + + buffer_putc(buffer, '\n'); + buffer_add_xml_indent(buffer, level); + + // skip spaces before the tag name + while(xml < end && isspace((uint8_t)*xml)) xml++; + + // Parse the tag name +// const char *tag_start = xml, *tag_end = NULL; + while (xml < end && *xml != '>' && *xml != '/') { + xml++; + + if(xml < end && isspace((uint8_t)*xml)) { + xml++; +// tag_end = xml; + + while(xml < end && isspace((uint8_t)*xml)) + xml++; + + if(xml < end && *xml == '/') { + // an opening tag that is self-closing + xml++; + if(xml < end && *xml == '>') { + xml++; + buffer_fast_strcat(buffer, start, xml - start); + return xml; + } + else + return append_the_rest(buffer, start, end); + } + else if(xml < end && *xml == '>') { + // the end of an opening tag + xml++; + buffer_fast_strcat(buffer, start, xml - start); + return parse_value_and_closing_tag(buffer, xml, end, level); + } + else { + buffer_fast_strcat(buffer, start, xml - start); + xml = start = parse_field(buffer, xml, end); + while(xml < end && isspace((uint8_t)*xml)) + xml++; + } + } + } + + bool self_closing_tag = false; + if(xml < end && *xml == '/') { + self_closing_tag = true; + xml++; + } + + if(xml < end && *xml == '>') { + xml++; + buffer_fast_strcat(buffer, start, xml - start); + + if(self_closing_tag) + return xml; + + return parse_value_and_closing_tag(buffer, xml, end, level); + } + + return append_the_rest(buffer, start, end); +} + +static inline void buffer_pretty_print_xml_object(BUFFER *buffer, const char *xml, const char *end) { + while(xml < end) { + while(xml < end && isspace((uint8_t)*xml)) + xml++; + + if(xml < end && *xml == '<') + xml = parse_node(buffer, xml, end, 1); + else { + append_the_rest(buffer, xml, end); + return; + } + } +} + +void buffer_pretty_print_xml(BUFFER *buffer, const char *xml, size_t xml_len) { + const char *end = xml + xml_len; + buffer_pretty_print_xml_object(buffer, xml, end); +} + +// -------------------------------------------------------------------------------------------------------------------- + +bool buffer_extract_and_print_xml_with_cb(BUFFER *buffer, const char *xml, size_t xml_len, const char *prefix, const char *keys[], + void (*cb)(BUFFER *, const char *, const char *, const char *)) { + if(!keys || !*keys[0]) { + buffer_pretty_print_xml(buffer, xml, xml_len); + return true; + } + + const char *start = xml, *end = NULL; + for(size_t k = 0; keys[k] ; k++) { + if(!*keys[k]) continue; + + size_t klen = strlen(keys[k]); + char tag_open[klen + 2]; + tag_open[0] = '<'; + strcpy(&tag_open[1], keys[k]); + tag_open[klen + 1] = '\0'; + + const char *new_start = strstr(start, tag_open); + if(!new_start) + return false; + + start = new_start + klen + 1; + + if(*start != '>' && !isspace((uint8_t)*start)) + return false; + + if(*start != '>') { + start = strchr(start, '>'); + if(!start) return false; + } + start++; // skip the > + + char tag_close[klen + 4]; + tag_close[0] = '<'; + tag_close[1] = '/'; + strcpy(&tag_close[2], keys[k]); + tag_close[klen + 2] = '>'; + tag_close[klen + 3] = '\0'; + + const char *new_end = strstr(start, tag_close); + if(!new_end || (end && new_end > end)) + return false; + + end = new_end; + } + + if(!start || !end || start == end) + return false; + + cb(buffer, prefix, start, end); + return true; +} + +static void print_xml_cb(BUFFER *buffer, const char *prefix, const char *start, const char *end) { + if(prefix) + buffer_strcat(buffer, prefix); + + buffer_pretty_print_xml_object(buffer, start, end); +} + +bool buffer_extract_and_print_xml(BUFFER *buffer, const char *xml, size_t xml_len, const char *prefix, const char *keys[]) { + return buffer_extract_and_print_xml_with_cb( + buffer, xml, xml_len, + prefix, keys, + print_xml_cb); +} + +static void print_value_cb(BUFFER *buffer, const char *prefix, const char *start, const char *end) { + if(prefix) + buffer_strcat(buffer, prefix); + + buffer_need_bytes(buffer, end - start + 1); + + char *started = &buffer->buffer[buffer->len]; + char *d = started; + const char *s = start; + + while(s < end && s) { + if(*s == '&' && s + 3 < end) { + if(*(s + 1) == '#') { + if(s + 4 < end && *(s + 2) == '1' && *(s + 4) == ';') { + if (*(s + 3) == '0') { + s += 5; + *d++ = '\n'; + continue; + } else if (*(s + 3) == '3') { + s += 5; + // *d++ = '\r'; + continue; + } + } else if (*(s + 2) == '9' && *(s + 3) == ';') { + s += 4; + *d++ = '\t'; + continue; + } + } + else if(s + 3 < end && *(s + 2) == 't' && *(s + 3) == ';') { + if(*(s + 1) == 'l') { + s += 4; + *d++ = '<'; + continue; + } + else if(*(s + 1) == 'g') { + s += 4; + *d++ = '>'; + continue; + } + } + } + *d++ = *s++; + } + *d = '\0'; + buffer->len += d - started; +} + +bool buffer_xml_extract_and_print_value(BUFFER *buffer, const char *xml, size_t xml_len, const char *prefix, const char *keys[]) { + return buffer_extract_and_print_xml_with_cb( + buffer, xml, xml_len, + prefix, keys, + print_value_cb); +} diff --git a/src/collectors/windows-events.plugin/windows-events-xml.h b/src/collectors/windows-events.plugin/windows-events-xml.h new file mode 100644 index 000000000..78d2f686e --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events-xml.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef WINDOWS_EVENTS_XML_H +#define WINDOWS_EVENTS_XML_H + +#include "libnetdata/libnetdata.h" + +void buffer_pretty_print_xml(BUFFER *buffer, const char *xml, size_t xml_len); +bool buffer_extract_and_print_xml(BUFFER *buffer, const char *xml, size_t xml_len, const char *prefix, const char *keys[]); +bool buffer_xml_extract_and_print_value(BUFFER *buffer, const char *xml, size_t xml_len, const char *prefix, const char *keys[]); + +#endif //WINDOWS_EVENTS_XML_H diff --git a/src/collectors/windows-events.plugin/windows-events.c b/src/collectors/windows-events.plugin/windows-events.c new file mode 100644 index 000000000..09ce558ae --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events.c @@ -0,0 +1,1402 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "libnetdata/libnetdata.h" +#include "libnetdata/required_dummies.h" + +#include "windows-events.h" + +netdata_mutex_t stdout_mutex = NETDATA_MUTEX_INITIALIZER; +static bool plugin_should_exit = false; + +#define WEVT_ALWAYS_VISIBLE_KEYS NULL + +#define WEVT_KEYS_EXCLUDED_FROM_FACETS \ + "|" WEVT_FIELD_MESSAGE \ + "|" WEVT_FIELD_XML \ + "" + +#define WEVT_KEYS_INCLUDED_IN_FACETS \ + "|" WEVT_FIELD_COMPUTER \ + "|" WEVT_FIELD_PROVIDER \ + "|" WEVT_FIELD_LEVEL \ + "|" WEVT_FIELD_KEYWORDS \ + "|" WEVT_FIELD_OPCODE \ + "|" WEVT_FIELD_TASK \ + "|" WEVT_FIELD_ACCOUNT \ + "|" WEVT_FIELD_DOMAIN \ + "|" WEVT_FIELD_SID \ + "" + +#define query_has_fts(lqs) ((lqs)->rq.query != NULL) + +static inline WEVT_QUERY_STATUS check_stop(const bool *cancelled, const usec_t *stop_monotonic_ut) { + if(cancelled && __atomic_load_n(cancelled, __ATOMIC_RELAXED)) { + nd_log(NDLS_COLLECTORS, NDLP_INFO, "Function has been cancelled"); + return WEVT_CANCELLED; + } + + if(now_monotonic_usec() > __atomic_load_n(stop_monotonic_ut, __ATOMIC_RELAXED)) { + internal_error(true, "Function timed out"); + return WEVT_TIMED_OUT; + } + + return WEVT_OK; +} + +FACET_ROW_SEVERITY wevt_levelid_to_facet_severity(FACETS *facets __maybe_unused, FACET_ROW *row, void *data __maybe_unused) { + FACET_ROW_KEY_VALUE *levelid_rkv = dictionary_get(row->dict, WEVT_FIELD_LEVEL "ID"); + if(!levelid_rkv || levelid_rkv->empty) + return FACET_ROW_SEVERITY_NORMAL; + + int windows_event_level = str2i(buffer_tostring(levelid_rkv->wb)); + + switch (windows_event_level) { + case WEVT_LEVEL_VERBOSE: + return FACET_ROW_SEVERITY_DEBUG; + + default: + case WEVT_LEVEL_INFORMATION: + return FACET_ROW_SEVERITY_NORMAL; + + case WEVT_LEVEL_WARNING: + return FACET_ROW_SEVERITY_WARNING; + + case WEVT_LEVEL_ERROR: + case WEVT_LEVEL_CRITICAL: + return FACET_ROW_SEVERITY_CRITICAL; + } +} + +struct wevt_bin_data { + bool rendered; + WEVT_EVENT ev; + WEVT_LOG *log; + EVT_HANDLE hEvent; + PROVIDER_META_HANDLE *provider; +}; + +static void wevt_cleanup_bin_data(void *data) { + struct wevt_bin_data *d = data; + + if(d->hEvent) + EvtClose(d->hEvent); + + provider_release(d->provider); + freez(d); +} + +static inline void wevt_facets_register_bin_data(WEVT_LOG *log, FACETS *facets, WEVT_EVENT *ev) { + struct wevt_bin_data *d = mallocz(sizeof(struct wevt_bin_data)); + +#ifdef NETDATA_INTERNAL_CHECKS + internal_fatal(strcmp(log->ops.provider.data, provider_get_name(log->provider)) != 0, + "Provider name mismatch in data!"); + + internal_fatal(!UUIDeq(ev->provider, provider_get_uuid(log->provider)), + "Provider UUID mismatch in data!"); +#endif + + d->ev = *ev; + d->log = log; + d->rendered = false; + + // take the bookmark + d->hEvent = log->hEvent; log->hEvent = NULL; + + // dup the provider + d->provider = provider_dup(log->provider); + + facets_row_bin_data_set(facets, wevt_cleanup_bin_data, d); +} + +static void wevt_lazy_loading_event_and_xml(struct wevt_bin_data *d, FACET_ROW *row __maybe_unused) { + if(d->rendered) return; + +#ifdef NETDATA_INTERNAL_CHECKS + const FACET_ROW_KEY_VALUE *provider_rkv = dictionary_get(row->dict, WEVT_FIELD_PROVIDER); + internal_fatal(!provider_rkv || strcmp(buffer_tostring(provider_rkv->wb), provider_get_name(d->provider)) != 0, + "Provider of row does not match the bin data associated with it"); + + uint64_t event_record_id = UINT64_MAX; + const FACET_ROW_KEY_VALUE *event_record_id_rkv = dictionary_get(row->dict, WEVT_FIELD_EVENTRECORDID); + if(event_record_id_rkv) + event_record_id = str2uint64_t(buffer_tostring(event_record_id_rkv->wb), NULL); + internal_fatal(event_record_id != d->ev.id, + "Event Record ID of row does not match the bin data associated with it"); +#endif + + // the message needs the xml + EvtFormatMessage_Xml_utf8(&d->log->ops.unicode, d->provider, d->hEvent, &d->log->ops.xml); + EvtFormatMessage_Event_utf8(&d->log->ops.unicode, d->provider, d->hEvent, &d->log->ops.event); + d->rendered = true; +} + +static void wevt_lazy_load_xml( + FACETS *facets, + BUFFER *json_array, + FACET_ROW_KEY_VALUE *rkv __maybe_unused, + FACET_ROW *row, + void *data __maybe_unused) { + + struct wevt_bin_data *d = facets_row_bin_data_get(facets, row); + if(!d) { + buffer_json_add_array_item_string(json_array, "Failed to get row BIN DATA from facets"); + return; + } + + wevt_lazy_loading_event_and_xml(d, row); + buffer_json_add_array_item_string(json_array, d->log->ops.xml.data); +} + +static void wevt_lazy_load_message( + FACETS *facets, + BUFFER *json_array, + FACET_ROW_KEY_VALUE *rkv __maybe_unused, + FACET_ROW *row, + void *data __maybe_unused) { + + struct wevt_bin_data *d = facets_row_bin_data_get(facets, row); + if(!d) { + buffer_json_add_array_item_string(json_array, "Failed to get row BIN DATA from facets"); + return; + } + + wevt_lazy_loading_event_and_xml(d, row); + + if(d->log->ops.event.used <= 1) { + TXT_UTF8 *xml = &d->log->ops.xml; + + buffer_flush(rkv->wb); + + bool added_message = false; + if(xml->used > 1) { + const char *message_path[] = { + "RenderingInfo", + "Message", + NULL}; + + added_message = buffer_xml_extract_and_print_value( + rkv->wb, + xml->data, xml->used - 1, + NULL, + message_path); + } + + if(!added_message) { + const FACET_ROW_KEY_VALUE *event_id_rkv = dictionary_get(row->dict, WEVT_FIELD_EVENTID); + if (event_id_rkv && buffer_strlen(event_id_rkv->wb)) { + buffer_fast_strcat(rkv->wb, "Event ", 6); + buffer_fast_strcat(rkv->wb, buffer_tostring(event_id_rkv->wb), buffer_strlen(event_id_rkv->wb)); + } else + buffer_strcat(rkv->wb, "Unknown Event "); + + const FACET_ROW_KEY_VALUE *provider_rkv = dictionary_get(row->dict, WEVT_FIELD_PROVIDER); + if (provider_rkv && buffer_strlen(provider_rkv->wb)) { + buffer_fast_strcat(rkv->wb, " of ", 4); + buffer_fast_strcat(rkv->wb, buffer_tostring(provider_rkv->wb), buffer_strlen(provider_rkv->wb)); + buffer_putc(rkv->wb, '.'); + } else + buffer_strcat(rkv->wb, "of unknown Provider."); + } + + if(xml->used > 1) { + const char *event_path[] = { + "EventData", + NULL + }; + bool added_event_data = buffer_extract_and_print_xml( + rkv->wb, + xml->data, xml->used - 1, + "\n\nRelated event data:\n", + event_path); + + const char *user_path[] = { + "UserData", + NULL + }; + bool added_user_data = buffer_extract_and_print_xml( + rkv->wb, + xml->data, xml->used - 1, + "\n\nRelated user data:\n", + user_path); + + if(!added_event_data && !added_user_data) + buffer_strcat(rkv->wb, " Without any related data."); + } + + buffer_json_add_array_item_string(json_array, buffer_tostring(rkv->wb)); + } + else + buffer_json_add_array_item_string(json_array, d->log->ops.event.data); +} + +static void wevt_register_fields(LOGS_QUERY_STATUS *lqs) { + // the order of the fields here, controls the order of the fields at the table presented + + FACETS *facets = lqs->facets; + LOGS_QUERY_REQUEST *rq = &lqs->rq; + + facets_register_row_severity(facets, wevt_levelid_to_facet_severity, NULL); + + facets_register_key_name( + facets, WEVT_FIELD_COMPUTER, + rq->default_facet | FACET_KEY_OPTION_VISIBLE); + + facets_register_key_name( + facets, WEVT_FIELD_CHANNEL, + rq->default_facet | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_PROVIDER, + rq->default_facet | FACET_KEY_OPTION_VISIBLE | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_ACCOUNT, + rq->default_facet | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_DOMAIN, + rq->default_facet | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_SID, + rq->default_facet | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_EVENTID, + rq->default_facet | + FACET_KEY_OPTION_VISIBLE | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_EVENTS_API, + rq->default_facet | + FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_LEVEL, + rq->default_facet | FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_EXPANDED_FILTER); + + facets_register_key_name( + facets, WEVT_FIELD_LEVEL "ID", + FACET_KEY_OPTION_NONE); + + facets_register_key_name( + facets, WEVT_FIELD_PROCESSID, + FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_THREADID, + FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_TASK, + rq->default_facet | FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_VISIBLE); + + facets_register_key_name( + facets, WEVT_FIELD_TASK "ID", + FACET_KEY_OPTION_NONE); + + facets_register_key_name( + facets, WEVT_FIELD_OPCODE, + rq->default_facet | FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_VISIBLE); + + facets_register_key_name( + facets, WEVT_FIELD_OPCODE "ID", + FACET_KEY_OPTION_NONE); + + facets_register_key_name( + facets, WEVT_FIELD_KEYWORDS, + rq->default_facet | FACET_KEY_OPTION_FTS); + + facets_register_key_name( + facets, WEVT_FIELD_KEYWORDS "ID", + FACET_KEY_OPTION_NONE); + + facets_register_dynamic_key_name( + facets, + WEVT_FIELD_MESSAGE, + FACET_KEY_OPTION_NEVER_FACET | FACET_KEY_OPTION_MAIN_TEXT | FACET_KEY_OPTION_VISIBLE, + wevt_lazy_load_message, + NULL); + + facets_register_dynamic_key_name( + facets, + WEVT_FIELD_XML, + FACET_KEY_OPTION_NEVER_FACET | FACET_KEY_OPTION_PRETTY_XML, + wevt_lazy_load_xml, + NULL); + + if(query_has_fts(lqs)) { + facets_register_key_name( + facets, WEVT_FIELD_EVENT_MESSAGE_HIDDEN, + FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_HIDDEN | FACET_KEY_OPTION_NEVER_FACET); + + facets_register_key_name( + facets, WEVT_FIELD_EVENT_XML_HIDDEN, + FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_HIDDEN | FACET_KEY_OPTION_NEVER_FACET); + + facets_register_key_name( + facets, WEVT_FIELD_EVENT_DATA_HIDDEN, + FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_HIDDEN | FACET_KEY_OPTION_NEVER_FACET); + } + +#ifdef NETDATA_INTERNAL_CHECKS + facets_register_key_name( + facets, "z_level_source", + rq->default_facet); + + facets_register_key_name( + facets, "z_keywords_source", + rq->default_facet); + + facets_register_key_name( + facets, "z_opcode_source", + rq->default_facet); + + facets_register_key_name( + facets, "z_task_source", + rq->default_facet); +#endif +} + +#ifdef NETDATA_INTERNAL_CHECKS +static const char *source_to_str(TXT_UTF8 *txt) { + switch(txt->src) { + default: + case TXT_SOURCE_UNKNOWN: + return "unknown"; + + case TXT_SOURCE_EVENT_LOG: + return "event-log"; + + case TXT_SOURCE_PROVIDER: + return "provider"; + + case TXT_SOURCE_FIELD_CACHE: + return "fields-cache"; + + case TXT_SOURCE_HARDCODED: + return "hardcoded"; + } +} +#endif + +static const char *events_api_to_str(WEVT_PROVIDER_PLATFORM platform) { + switch(platform) { + case WEVT_PLATFORM_WEL: + return "Windows Event Log"; + + case WEVT_PLATFORM_ETW: + return "Event Tracing for Windows"; + + case WEVT_PLATFORM_TL: + return "TraceLogging"; + + default: + return "Unknown"; + } +} + +static inline size_t wevt_process_event(WEVT_LOG *log, FACETS *facets, LOGS_QUERY_SOURCE *src, usec_t *msg_ut __maybe_unused, WEVT_EVENT *ev) { + static __thread char uuid_str[UUID_STR_LEN]; + + size_t len, bytes = log->ops.raw.system.used + log->ops.raw.user.used; + + if(!UUIDiszero(ev->provider)) { + uuid_unparse_lower(ev->provider.uuid, uuid_str); + facets_add_key_value_length( + facets, WEVT_FIELD_PROVIDER_GUID, sizeof(WEVT_FIELD_PROVIDER_GUID) - 1, + uuid_str, sizeof(uuid_str) - 1); + } + + if(!UUIDiszero(ev->activity_id)) { + uuid_unparse_lower(ev->activity_id.uuid, uuid_str); + facets_add_key_value_length( + facets, WEVT_FIELD_ACTIVITY_ID, sizeof(WEVT_FIELD_ACTIVITY_ID) - 1, + uuid_str, sizeof(uuid_str) - 1); + } + + if(!UUIDiszero(ev->related_activity_id)) { + uuid_unparse_lower(ev->related_activity_id.uuid, uuid_str); + facets_add_key_value_length( + facets, WEVT_FIELD_RELATED_ACTIVITY_ID, sizeof(WEVT_FIELD_RELATED_ACTIVITY_ID) - 1, + uuid_str, sizeof(uuid_str) - 1); + } + + if(ev->qualifiers) { + static __thread char qualifiers[UINT64_HEX_MAX_LENGTH]; + len = print_uint64_hex(qualifiers, ev->qualifiers); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_QUALIFIERS, sizeof(WEVT_FIELD_QUALIFIERS) - 1, + qualifiers, len); + } + + { + static __thread char event_record_id_str[UINT64_MAX_LENGTH]; + len = print_uint64(event_record_id_str, ev->id); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_EVENTRECORDID, sizeof(WEVT_FIELD_EVENTRECORDID) - 1, + event_record_id_str, len); + } + + if(ev->version) { + static __thread char version[UINT64_MAX_LENGTH]; + len = print_uint64(version, ev->version); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_VERSION, sizeof(WEVT_FIELD_VERSION) - 1, + version, len); + } + + if(log->ops.provider.used > 1) { + bytes += log->ops.provider.used * 2; // unicode is double + facets_add_key_value_length( + facets, WEVT_FIELD_PROVIDER, sizeof(WEVT_FIELD_PROVIDER) - 1, + log->ops.provider.data, log->ops.provider.used - 1); + } + + if(log->ops.channel.used > 1) { + bytes += log->ops.channel.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_CHANNEL, sizeof(WEVT_FIELD_CHANNEL) - 1, + log->ops.channel.data, log->ops.channel.used - 1); + } + else { + bytes += src->fullname_len * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_CHANNEL, sizeof(WEVT_FIELD_CHANNEL) - 1, + src->fullname, src->fullname_len); + } + + if(log->ops.level.used > 1) { + bytes += log->ops.level.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_LEVEL, sizeof(WEVT_FIELD_LEVEL) - 1, + log->ops.level.data, log->ops.level.used - 1); + } + + if(log->ops.computer.used > 1) { + bytes += log->ops.computer.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_COMPUTER, sizeof(WEVT_FIELD_COMPUTER) - 1, + log->ops.computer.data, log->ops.computer.used - 1); + } + + if(log->ops.opcode.used > 1) { + bytes += log->ops.opcode.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_OPCODE, sizeof(WEVT_FIELD_OPCODE) - 1, + log->ops.opcode.data, log->ops.opcode.used - 1); + } + + if(log->ops.keywords.used > 1) { + bytes += log->ops.keywords.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_KEYWORDS, sizeof(WEVT_FIELD_KEYWORDS) - 1, + log->ops.keywords.data, log->ops.keywords.used - 1); + } + + if(log->ops.task.used > 1) { + bytes += log->ops.task.used * 2; + facets_add_key_value_length( + facets, WEVT_FIELD_TASK, sizeof(WEVT_FIELD_TASK) - 1, + log->ops.task.data, log->ops.task.used - 1); + } + + if(log->ops.account.used > 1) { + bytes += log->ops.account.used * 2; + facets_add_key_value_length( + facets, + WEVT_FIELD_ACCOUNT, sizeof(WEVT_FIELD_ACCOUNT) - 1, + log->ops.account.data, log->ops.account.used - 1); + } + + if(log->ops.domain.used > 1) { + bytes += log->ops.domain.used * 2; + facets_add_key_value_length( + facets, + WEVT_FIELD_DOMAIN, sizeof(WEVT_FIELD_DOMAIN) - 1, + log->ops.domain.data, log->ops.domain.used - 1); + } + + if(log->ops.sid.used > 1) { + bytes += log->ops.sid.used * 2; + facets_add_key_value_length( + facets, + WEVT_FIELD_SID, sizeof(WEVT_FIELD_SID) - 1, + log->ops.sid.data, log->ops.sid.used - 1); + } + + { + static __thread char event_id_str[UINT64_MAX_LENGTH]; + len = print_uint64(event_id_str, ev->event_id); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_EVENTID, sizeof(WEVT_FIELD_EVENTID) - 1, + event_id_str, len); + } + + { + const char *s = events_api_to_str(ev->platform); + facets_add_key_value_length( + facets, WEVT_FIELD_EVENTS_API, sizeof(WEVT_FIELD_EVENTS_API) - 1, s, strlen(s)); + } + + if(ev->process_id) { + static __thread char process_id_str[UINT64_MAX_LENGTH]; + len = print_uint64(process_id_str, ev->process_id); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_PROCESSID, sizeof(WEVT_FIELD_PROCESSID) - 1, + process_id_str, len); + } + + if(ev->thread_id) { + static __thread char thread_id_str[UINT64_MAX_LENGTH]; + len = print_uint64(thread_id_str, ev->thread_id); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_THREADID, sizeof(WEVT_FIELD_THREADID) - 1, + thread_id_str, len); + } + + { + static __thread char str[UINT64_MAX_LENGTH]; + len = print_uint64(str, ev->level); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_LEVEL "ID", sizeof(WEVT_FIELD_LEVEL) + 2 - 1, str, len); + } + + { + static __thread char str[UINT64_HEX_MAX_LENGTH]; + len = print_uint64_hex_full(str, ev->keywords); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_KEYWORDS "ID", sizeof(WEVT_FIELD_KEYWORDS) + 2 - 1, str, len); + } + + { + static __thread char str[UINT64_MAX_LENGTH]; + len = print_uint64(str, ev->opcode); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_OPCODE "ID", sizeof(WEVT_FIELD_OPCODE) + 2 - 1, str, len); + } + + { + static __thread char str[UINT64_MAX_LENGTH]; + len = print_uint64(str, ev->task); + bytes += len; + facets_add_key_value_length( + facets, WEVT_FIELD_TASK "ID", sizeof(WEVT_FIELD_TASK) + 2 - 1, str, len); + } + + if(log->type & WEVT_QUERY_EVENT_DATA) { + // the query has full text-search + if(log->ops.event.used > 1) { + bytes += log->ops.event.used; + facets_add_key_value_length( + facets, WEVT_FIELD_EVENT_MESSAGE_HIDDEN, sizeof(WEVT_FIELD_EVENT_MESSAGE_HIDDEN) - 1, + log->ops.event.data, log->ops.event.used - 1); + } + + if(log->ops.xml.used > 1) { + bytes += log->ops.xml.used; + facets_add_key_value_length( + facets, WEVT_FIELD_EVENT_XML_HIDDEN, sizeof(WEVT_FIELD_EVENT_XML_HIDDEN) - 1, + log->ops.xml.data, log->ops.xml.used - 1); + } + + if(log->ops.event_data->len) { + bytes += log->ops.event_data->len; + facets_add_key_value_length( + facets, WEVT_FIELD_EVENT_DATA_HIDDEN, sizeof(WEVT_FIELD_EVENT_DATA_HIDDEN) - 1, + buffer_tostring(log->ops.event_data), buffer_strlen(log->ops.event_data)); + } + } + + wevt_facets_register_bin_data(log, facets, ev); + +#ifdef NETDATA_INTERNAL_CHECKS + facets_add_key_value(facets, "z_level_source", source_to_str(&log->ops.level)); + facets_add_key_value(facets, "z_keywords_source", source_to_str(&log->ops.keywords)); + facets_add_key_value(facets, "z_opcode_source", source_to_str(&log->ops.opcode)); + facets_add_key_value(facets, "z_task_source", source_to_str(&log->ops.task)); +#endif + + return bytes; +} + +static void send_progress_update(LOGS_QUERY_STATUS *lqs, size_t current_row_counter, bool flush_current_file) { + usec_t now_ut = now_monotonic_usec(); + + if(current_row_counter > lqs->c.progress.entries.current_query_total) { + lqs->c.progress.entries.total += current_row_counter - lqs->c.progress.entries.current_query_total; + lqs->c.progress.entries.current_query_total = current_row_counter; + } + + if(flush_current_file) { + lqs->c.progress.entries.total += current_row_counter; + lqs->c.progress.entries.total -= lqs->c.progress.entries.current_query_total; + lqs->c.progress.entries.completed += current_row_counter; + lqs->c.progress.entries.current_query_total = 0; + } + + size_t completed = lqs->c.progress.entries.completed + current_row_counter; + if(completed > lqs->c.progress.entries.total) + lqs->c.progress.entries.total = completed; + + usec_t progress_duration_ut = now_ut - lqs->c.progress.last_ut; + if(progress_duration_ut >= WINDOWS_EVENTS_PROGRESS_EVERY_UT) { + lqs->c.progress.last_ut = now_ut; + + netdata_mutex_lock(&stdout_mutex); + pluginsd_function_progress_to_stdout(lqs->rq.transaction, completed, lqs->c.progress.entries.total); + netdata_mutex_unlock(&stdout_mutex); + } +} + +static WEVT_QUERY_STATUS wevt_query_backward( + WEVT_LOG *log, BUFFER *wb __maybe_unused, FACETS *facets, + LOGS_QUERY_SOURCE *src, + LOGS_QUERY_STATUS *lqs) +{ + usec_t start_ut = lqs->query.start_ut; + usec_t stop_ut = lqs->query.stop_ut; + bool stop_when_full = lqs->query.stop_when_full; + +// lqs->c.query_file.start_ut = start_ut; +// lqs->c.query_file.stop_ut = stop_ut; + + if(!wevt_query(log, channel2unicode(src->fullname), lqs->c.query, EvtQueryReverseDirection)) + return WEVT_FAILED_TO_SEEK; + + size_t errors_no_timestamp = 0; + usec_t latest_msg_ut = 0; // the biggest timestamp we have seen so far + usec_t first_msg_ut = 0; // the first message we got from the db + size_t row_counter = 0, last_row_counter = 0, rows_useful = 0; + size_t bytes = 0, last_bytes = 0; + + usec_t last_usec_from = 0; + usec_t last_usec_to = 0; + + WEVT_QUERY_STATUS status = WEVT_OK; + + facets_rows_begin(facets); + WEVT_EVENT e; + while (status == WEVT_OK && wevt_get_next_event(log, &e)) { + usec_t msg_ut = e.created_ns / NSEC_PER_USEC; + + if(unlikely(!msg_ut)) { + errors_no_timestamp++; + continue; + } + + if (unlikely(msg_ut > start_ut)) + continue; + + if (unlikely(msg_ut < stop_ut)) + break; + + if(unlikely(msg_ut > latest_msg_ut)) + latest_msg_ut = msg_ut; + + if(unlikely(!first_msg_ut)) { + first_msg_ut = msg_ut; + // lqs->c.query_file.first_msg_ut = msg_ut; + } + +// sampling_t sample = is_row_in_sample(log, lqs, src, msg_ut, +// FACETS_ANCHOR_DIRECTION_BACKWARD, +// facets_row_candidate_to_keep(facets, msg_ut)); +// +// if(sample == SAMPLING_FULL) { + bytes += wevt_process_event(log, facets, src, &msg_ut, &e); + + // make sure each line gets a unique timestamp + if(unlikely(msg_ut >= last_usec_from && msg_ut <= last_usec_to)) + msg_ut = --last_usec_from; + else + last_usec_from = last_usec_to = msg_ut; + + if(facets_row_finished(facets, msg_ut)) + rows_useful++; + + row_counter++; + if(unlikely((row_counter % FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS) == 0 && + stop_when_full && + facets_rows(facets) >= lqs->rq.entries)) { + // stop the data only query + usec_t oldest = facets_row_oldest_ut(facets); + if(oldest && msg_ut < (oldest - lqs->anchor.delta_ut)) + break; + } + + if(unlikely(row_counter % FUNCTION_PROGRESS_EVERY_ROWS == 0)) { + status = check_stop(lqs->cancelled, lqs->stop_monotonic_ut); + + if(status == WEVT_OK) { + lqs->c.rows_read += row_counter - last_row_counter; + last_row_counter = row_counter; + + lqs->c.bytes_read += bytes - last_bytes; + last_bytes = bytes; + + send_progress_update(lqs, row_counter, false); + } + } +// } +// else if(sample == SAMPLING_SKIP_FIELDS) +// facets_row_finished_unsampled(facets, msg_ut); +// else { +// sampling_update_running_query_file_estimates(facets, log, lqs, src, msg_ut, FACETS_ANCHOR_DIRECTION_BACKWARD); +// break; +// } + } + + send_progress_update(lqs, row_counter, true); + lqs->c.rows_read += row_counter - last_row_counter; + lqs->c.bytes_read += bytes - last_bytes; + lqs->c.rows_useful += rows_useful; + + if(errors_no_timestamp) + netdata_log_error("WINDOWS-EVENTS: %zu events did not have timestamps", errors_no_timestamp); + + if(latest_msg_ut > lqs->last_modified) + lqs->last_modified = latest_msg_ut; + + wevt_query_done(log); + + return status; +} + +static WEVT_QUERY_STATUS wevt_query_forward( + WEVT_LOG *log, BUFFER *wb __maybe_unused, FACETS *facets, + LOGS_QUERY_SOURCE *src, + LOGS_QUERY_STATUS *lqs) +{ + usec_t start_ut = lqs->query.start_ut; + usec_t stop_ut = lqs->query.stop_ut; + bool stop_when_full = lqs->query.stop_when_full; + +// lqs->c.query_file.start_ut = start_ut; +// lqs->c.query_file.stop_ut = stop_ut; + + if(!wevt_query(log, channel2unicode(src->fullname), lqs->c.query, EvtQueryForwardDirection)) + return WEVT_FAILED_TO_SEEK; + + size_t errors_no_timestamp = 0; + usec_t latest_msg_ut = 0; // the biggest timestamp we have seen so far + usec_t first_msg_ut = 0; // the first message we got from the db + size_t row_counter = 0, last_row_counter = 0, rows_useful = 0; + size_t bytes = 0, last_bytes = 0; + + usec_t last_usec_from = 0; + usec_t last_usec_to = 0; + + WEVT_QUERY_STATUS status = WEVT_OK; + + facets_rows_begin(facets); + WEVT_EVENT e; + while (status == WEVT_OK && wevt_get_next_event(log, &e)) { + usec_t msg_ut = e.created_ns / NSEC_PER_USEC; + + if(unlikely(!msg_ut)) { + errors_no_timestamp++; + continue; + } + + if (unlikely(msg_ut < start_ut)) + continue; + + if (unlikely(msg_ut > stop_ut)) + break; + + if(likely(msg_ut > latest_msg_ut)) + latest_msg_ut = msg_ut; + + if(unlikely(!first_msg_ut)) { + first_msg_ut = msg_ut; + // lqs->c.query_file.first_msg_ut = msg_ut; + } + +// sampling_t sample = is_row_in_sample(log, lqs, src, msg_ut, +// FACETS_ANCHOR_DIRECTION_FORWARD, +// facets_row_candidate_to_keep(facets, msg_ut)); +// +// if(sample == SAMPLING_FULL) { + bytes += wevt_process_event(log, facets, src, &msg_ut, &e); + + // make sure each line gets a unique timestamp + if(unlikely(msg_ut >= last_usec_from && msg_ut <= last_usec_to)) + msg_ut = ++last_usec_to; + else + last_usec_from = last_usec_to = msg_ut; + + if(facets_row_finished(facets, msg_ut)) + rows_useful++; + + row_counter++; + if(unlikely((row_counter % FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS) == 0 && + stop_when_full && + facets_rows(facets) >= lqs->rq.entries)) { + // stop the data only query + usec_t newest = facets_row_newest_ut(facets); + if(newest && msg_ut > (newest + lqs->anchor.delta_ut)) + break; + } + + if(unlikely(row_counter % FUNCTION_PROGRESS_EVERY_ROWS == 0)) { + status = check_stop(lqs->cancelled, lqs->stop_monotonic_ut); + + if(status == WEVT_OK) { + lqs->c.rows_read += row_counter - last_row_counter; + last_row_counter = row_counter; + + lqs->c.bytes_read += bytes - last_bytes; + last_bytes = bytes; + + send_progress_update(lqs, row_counter, false); + } + } +// } +// else if(sample == SAMPLING_SKIP_FIELDS) +// facets_row_finished_unsampled(facets, msg_ut); +// else { +// sampling_update_running_query_file_estimates(facets, log, lqs, src, msg_ut, FACETS_ANCHOR_DIRECTION_FORWARD); +// break; +// } + } + + send_progress_update(lqs, row_counter, true); + lqs->c.rows_read += row_counter - last_row_counter; + lqs->c.bytes_read += bytes - last_bytes; + lqs->c.rows_useful += rows_useful; + + if(errors_no_timestamp) + netdata_log_error("WINDOWS-EVENTS: %zu events did not have timestamps", errors_no_timestamp); + + if(latest_msg_ut > lqs->last_modified) + lqs->last_modified = latest_msg_ut; + + wevt_query_done(log); + + return status; +} + +static WEVT_QUERY_STATUS wevt_query_one_channel( + WEVT_LOG *log, + BUFFER *wb, FACETS *facets, + LOGS_QUERY_SOURCE *src, + LOGS_QUERY_STATUS *lqs) { + + errno_clear(); + + WEVT_QUERY_STATUS status; + if(lqs->rq.direction == FACETS_ANCHOR_DIRECTION_FORWARD) + status = wevt_query_forward(log, wb, facets, src, lqs); + else + status = wevt_query_backward(log, wb, facets, src, lqs); + + return status; +} + +static bool source_is_mine(LOGS_QUERY_SOURCE *src, LOGS_QUERY_STATUS *lqs) { + if( + // no source is requested + (lqs->rq.source_type == WEVTS_NONE && !lqs->rq.sources) || + + // matches our internal source types + (src->source_type & lqs->rq.source_type) || + + // matches the source name + (lqs->rq.sources && src->source && simple_pattern_matches(lqs->rq.sources, string2str(src->source))) || + + // matches the provider (providers start with a special prefix to avoid mix and match) + (lqs->rq.sources && src->provider && simple_pattern_matches(lqs->rq.sources, string2str(src->provider))) + + ) { + + if(!src->msg_last_ut) + // the file is not scanned yet, or the timestamps have not been updated, + // so we don't know if it can contribute or not - let's add it. + return true; + + usec_t anchor_delta = ANCHOR_DELTA_UT; + usec_t first_ut = src->msg_first_ut - anchor_delta; + usec_t last_ut = src->msg_last_ut + anchor_delta; + + if(last_ut >= lqs->rq.after_ut && first_ut <= lqs->rq.before_ut) + return true; + } + + return false; +} + +static int wevt_master_query(BUFFER *wb __maybe_unused, LOGS_QUERY_STATUS *lqs __maybe_unused) { + // make sure the sources list is updated + wevt_sources_scan(); + + lqs->c.query = wevt_generate_query_no_xpath(lqs, wb); + if(!lqs->c.query) + return rrd_call_function_error(wb, "failed to generate query", HTTP_RESP_INTERNAL_SERVER_ERROR); + + FACETS *facets = lqs->facets; + + WEVT_QUERY_STATUS status = WEVT_NO_CHANNEL_MATCHED; + + lqs->c.files_matched = 0; + lqs->c.file_working = 0; + lqs->c.rows_useful = 0; + lqs->c.rows_read = 0; + lqs->c.bytes_read = 0; + + size_t files_used = 0; + size_t files_max = dictionary_entries(wevt_sources); + const DICTIONARY_ITEM *file_items[files_max]; + + // count the files + bool files_are_newer = false; + LOGS_QUERY_SOURCE *src; + dfe_start_read(wevt_sources, src) { + if(!source_is_mine(src, lqs)) + continue; + + file_items[files_used++] = dictionary_acquired_item_dup(wevt_sources, src_dfe.item); + + if(src->msg_last_ut > lqs->rq.if_modified_since) + files_are_newer = true; + + lqs->c.progress.entries.total += src->entries; + } + dfe_done(jf); + + lqs->c.files_matched = files_used; + + if(lqs->rq.if_modified_since && !files_are_newer) { + // release the files + for(size_t f = 0; f < files_used ;f++) + dictionary_acquired_item_release(wevt_sources, file_items[f]); + + return rrd_call_function_error(wb, "not modified", HTTP_RESP_NOT_MODIFIED); + } + + // sort the files, so that they are optimal for facets + if(files_used >= 2) { + if (lqs->rq.direction == FACETS_ANCHOR_DIRECTION_BACKWARD) + qsort(file_items, files_used, sizeof(const DICTIONARY_ITEM *), + wevt_sources_dict_items_backward_compar); + else + qsort(file_items, files_used, sizeof(const DICTIONARY_ITEM *), + wevt_sources_dict_items_forward_compar); + } + + bool partial = false; + usec_t query_started_ut = now_monotonic_usec(); + usec_t started_ut = query_started_ut; + usec_t ended_ut = started_ut; + usec_t duration_ut, max_duration_ut = 0; + + WEVT_LOG *log = wevt_openlog6(query_has_fts(lqs) ? WEVT_QUERY_FTS : WEVT_QUERY_NORMAL); + if(!log) { + // release the files + for(size_t f = 0; f < files_used ;f++) + dictionary_acquired_item_release(wevt_sources, file_items[f]); + + netdata_log_error("WINDOWS EVENTS: cannot open windows event log"); + return rrd_call_function_error(wb, "cannot open windows events log", HTTP_RESP_INTERNAL_SERVER_ERROR); + } + + // sampling_query_init(lqs, facets); + + buffer_json_member_add_array(wb, "_channels"); + for(size_t f = 0; f < files_used ;f++) { + const char *fullname = dictionary_acquired_item_name(file_items[f]); + src = dictionary_acquired_item_value(file_items[f]); + + if(!source_is_mine(src, lqs)) + continue; + + started_ut = ended_ut; + + // do not even try to do the query if we expect it to pass the timeout + if(ended_ut + max_duration_ut * 3 >= *lqs->stop_monotonic_ut) { + partial = true; + status = WEVT_TIMED_OUT; + break; + } + + lqs->c.file_working++; + + size_t rows_useful = lqs->c.rows_useful; + size_t rows_read = lqs->c.rows_read; + size_t bytes_read = lqs->c.bytes_read; + size_t matches_setup_ut = lqs->c.matches_setup_ut; + + // sampling_file_init(lqs, src); + + lqs->c.progress.entries.current_query_total = src->entries; + WEVT_QUERY_STATUS tmp_status = wevt_query_one_channel(log, wb, facets, src, lqs); + + rows_useful = lqs->c.rows_useful - rows_useful; + rows_read = lqs->c.rows_read - rows_read; + bytes_read = lqs->c.bytes_read - bytes_read; + matches_setup_ut = lqs->c.matches_setup_ut - matches_setup_ut; + + ended_ut = now_monotonic_usec(); + duration_ut = ended_ut - started_ut; + + if(duration_ut > max_duration_ut) + max_duration_ut = duration_ut; + + buffer_json_add_array_item_object(wb); // channel source + { + // information about the file + buffer_json_member_add_string(wb, "_name", fullname); + buffer_json_member_add_uint64(wb, "_source_type", src->source_type); + buffer_json_member_add_string(wb, "_source", string2str(src->source)); + buffer_json_member_add_uint64(wb, "_msg_first_ut", src->msg_first_ut); + buffer_json_member_add_uint64(wb, "_msg_last_ut", src->msg_last_ut); + + // information about the current use of the file + buffer_json_member_add_uint64(wb, "duration_ut", ended_ut - started_ut); + buffer_json_member_add_uint64(wb, "rows_read", rows_read); + buffer_json_member_add_uint64(wb, "rows_useful", rows_useful); + buffer_json_member_add_double(wb, "rows_per_second", (double) rows_read / (double) duration_ut * (double) USEC_PER_SEC); + buffer_json_member_add_uint64(wb, "bytes_read", bytes_read); + buffer_json_member_add_double(wb, "bytes_per_second", (double) bytes_read / (double) duration_ut * (double) USEC_PER_SEC); + buffer_json_member_add_uint64(wb, "duration_matches_ut", matches_setup_ut); + + // if(lqs->rq.sampling) { + // buffer_json_member_add_object(wb, "_sampling"); + // { + // buffer_json_member_add_uint64(wb, "sampled", lqs->c.samples_per_file.sampled); + // buffer_json_member_add_uint64(wb, "unsampled", lqs->c.samples_per_file.unsampled); + // buffer_json_member_add_uint64(wb, "estimated", lqs->c.samples_per_file.estimated); + // } + // buffer_json_object_close(wb); // _sampling + // } + } + buffer_json_object_close(wb); // channel source + + bool stop = false; + switch(tmp_status) { + case WEVT_OK: + case WEVT_NO_CHANNEL_MATCHED: + status = (status == WEVT_OK) ? WEVT_OK : tmp_status; + break; + + case WEVT_FAILED_TO_OPEN: + case WEVT_FAILED_TO_SEEK: + partial = true; + if(status == WEVT_NO_CHANNEL_MATCHED) + status = tmp_status; + break; + + case WEVT_CANCELLED: + case WEVT_TIMED_OUT: + partial = true; + stop = true; + status = tmp_status; + break; + + case WEVT_NOT_MODIFIED: + internal_fatal(true, "this should never be returned here"); + break; + } + + if(stop) + break; + } + buffer_json_array_close(wb); // _channels + + // release the files + for(size_t f = 0; f < files_used ;f++) + dictionary_acquired_item_release(wevt_sources, file_items[f]); + + switch (status) { + case WEVT_OK: + if(lqs->rq.if_modified_since && !lqs->c.rows_useful) + return rrd_call_function_error(wb, "no useful logs, not modified", HTTP_RESP_NOT_MODIFIED); + break; + + case WEVT_TIMED_OUT: + case WEVT_NO_CHANNEL_MATCHED: + break; + + case WEVT_CANCELLED: + return rrd_call_function_error(wb, "client closed connection", HTTP_RESP_CLIENT_CLOSED_REQUEST); + + case WEVT_NOT_MODIFIED: + return rrd_call_function_error(wb, "not modified", HTTP_RESP_NOT_MODIFIED); + + case WEVT_FAILED_TO_OPEN: + return rrd_call_function_error(wb, "failed to open event log", HTTP_RESP_INTERNAL_SERVER_ERROR); + + case WEVT_FAILED_TO_SEEK: + return rrd_call_function_error(wb, "failed to execute event log query", HTTP_RESP_INTERNAL_SERVER_ERROR); + + default: + return rrd_call_function_error(wb, "unknown status", HTTP_RESP_INTERNAL_SERVER_ERROR); + } + + buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK); + buffer_json_member_add_boolean(wb, "partial", partial); + buffer_json_member_add_string(wb, "type", "table"); + + // build a message for the query + if(!lqs->rq.data_only) { + CLEAN_BUFFER *msg = buffer_create(0, NULL); + CLEAN_BUFFER *msg_description = buffer_create(0, NULL); + ND_LOG_FIELD_PRIORITY msg_priority = NDLP_INFO; + + // if(!journal_files_completed_once()) { + // buffer_strcat(msg, "Journals are still being scanned. "); + // buffer_strcat(msg_description + // , "LIBRARY SCAN: The journal files are still being scanned, you are probably viewing incomplete data. "); + // msg_priority = NDLP_WARNING; + // } + + if(partial) { + buffer_strcat(msg, "Query timed-out, incomplete data. "); + buffer_strcat(msg_description + , "QUERY TIMEOUT: The query timed out and may not include all the data of the selected window. "); + msg_priority = NDLP_WARNING; + } + + // if(lqs->c.samples.estimated || lqs->c.samples.unsampled) { + // double percent = (double) (lqs->c.samples.sampled * 100.0 / + // (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled)); + // buffer_sprintf(msg, "%.2f%% real data", percent); + // buffer_sprintf(msg_description, "ACTUAL DATA: The filters counters reflect %0.2f%% of the data. ", percent); + // msg_priority = MIN(msg_priority, NDLP_NOTICE); + // } + // + // if(lqs->c.samples.unsampled) { + // double percent = (double) (lqs->c.samples.unsampled * 100.0 / + // (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled)); + // buffer_sprintf(msg, ", %.2f%% unsampled", percent); + // buffer_sprintf(msg_description + // , "UNSAMPLED DATA: %0.2f%% of the events exist and have been counted, but their values have not been evaluated, so they are not included in the filters counters. " + // , percent); + // msg_priority = MIN(msg_priority, NDLP_NOTICE); + // } + // + // if(lqs->c.samples.estimated) { + // double percent = (double) (lqs->c.samples.estimated * 100.0 / + // (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled)); + // buffer_sprintf(msg, ", %.2f%% estimated", percent); + // buffer_sprintf(msg_description + // , "ESTIMATED DATA: The query selected a large amount of data, so to avoid delaying too much, the presented data are estimated by %0.2f%%. " + // , percent); + // msg_priority = MIN(msg_priority, NDLP_NOTICE); + // } + + buffer_json_member_add_object(wb, "message"); + if(buffer_tostring(msg)) { + buffer_json_member_add_string(wb, "title", buffer_tostring(msg)); + buffer_json_member_add_string(wb, "description", buffer_tostring(msg_description)); + buffer_json_member_add_string(wb, "status", nd_log_id2priority(msg_priority)); + } + // else send an empty object if there is nothing to tell + buffer_json_object_close(wb); // message + } + + if(!lqs->rq.data_only) { + buffer_json_member_add_time_t(wb, "update_every", 1); + buffer_json_member_add_string(wb, "help", WEVT_FUNCTION_DESCRIPTION); + } + + if(!lqs->rq.data_only || lqs->rq.tail) + buffer_json_member_add_uint64(wb, "last_modified", lqs->last_modified); + + facets_sort_and_reorder_keys(facets); + facets_report(facets, wb, used_hashes_registry); + + wb->expires = now_realtime_sec() + (lqs->rq.data_only ? 3600 : 0); + buffer_json_member_add_time_t(wb, "expires", wb->expires); + + // if(lqs->rq.sampling) { + // buffer_json_member_add_object(wb, "_sampling"); + // { + // buffer_json_member_add_uint64(wb, "sampled", lqs->c.samples.sampled); + // buffer_json_member_add_uint64(wb, "unsampled", lqs->c.samples.unsampled); + // buffer_json_member_add_uint64(wb, "estimated", lqs->c.samples.estimated); + // } + // buffer_json_object_close(wb); // _sampling + // } + + wevt_closelog6(log); + + wb->content_type = CT_APPLICATION_JSON; + wb->response_code = HTTP_RESP_OK; + return wb->response_code; +} + +void function_windows_events(const char *transaction, char *function, usec_t *stop_monotonic_ut, bool *cancelled, + BUFFER *payload, HTTP_ACCESS access __maybe_unused, + const char *source __maybe_unused, void *data __maybe_unused) { + bool have_slice = LQS_DEFAULT_SLICE_MODE; + + LOGS_QUERY_STATUS tmp_fqs = { + .facets = lqs_facets_create( + LQS_DEFAULT_ITEMS_PER_QUERY, + FACETS_OPTION_ALL_KEYS_FTS | FACETS_OPTION_HASH_IDS, + WEVT_ALWAYS_VISIBLE_KEYS, + WEVT_KEYS_INCLUDED_IN_FACETS, + WEVT_KEYS_EXCLUDED_FROM_FACETS, + have_slice), + + .rq = LOGS_QUERY_REQUEST_DEFAULTS(transaction, have_slice, FACETS_ANCHOR_DIRECTION_BACKWARD), + + .cancelled = cancelled, + .stop_monotonic_ut = stop_monotonic_ut, + }; + LOGS_QUERY_STATUS *lqs = &tmp_fqs; + + CLEAN_BUFFER *wb = lqs_create_output_buffer(); + + // ------------------------------------------------------------------------ + // parse the parameters + + if(lqs_request_parse_and_validate(lqs, wb, function, payload, have_slice, WEVT_FIELD_LEVEL)) { + wevt_register_fields(lqs); + + // ------------------------------------------------------------------------ + // add versions to the response + + buffer_json_wevt_versions(wb); + + // ------------------------------------------------------------------------ + // run the request + + if (lqs->rq.info) + lqs_info_response(wb, lqs->facets); + else { + wevt_master_query(wb, lqs); + if (wb->response_code == HTTP_RESP_OK) + buffer_json_finalize(wb); + } + } + + netdata_mutex_lock(&stdout_mutex); + pluginsd_function_result_to_stdout(transaction, wb); + netdata_mutex_unlock(&stdout_mutex); + + lqs_cleanup(lqs); +} + +int main(int argc __maybe_unused, char **argv __maybe_unused) { + nd_thread_tag_set("wevt.plugin"); + nd_log_initialize_for_external_plugins("windows-events.plugin"); + + // ------------------------------------------------------------------------ + // initialization + + wevt_sources_init(); + provider_cache_init(); + cached_sid_username_init(); + field_cache_init(); + + if(!EnableWindowsPrivilege(SE_SECURITY_NAME)) + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_SECURITY_NAME); + + if(!EnableWindowsPrivilege(SE_BACKUP_NAME)) + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_BACKUP_NAME); + + if(!EnableWindowsPrivilege(SE_AUDIT_NAME)) + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_AUDIT_NAME); + + // ------------------------------------------------------------------------ + // debug + + if(argc >= 2 && strcmp(argv[argc - 1], "debug") == 0) { + wevt_sources_scan(); + + struct { + const char *func; + } array[] = { + { "windows-events after:-8640000 before:0 last:200 source:All" }, + //{ "windows-events after:-86400 before:0 direction:backward last:200 facets:HdUoSYab5wV,Cq2r7mRUv4a,LAnVlsIQfeD,BnPLNbA5VWT,KeCITtVD5AD,HytMJ9kj82B,JM3OPW3kHn6,H106l8MXSSr,HREiMN.4Ahu,ClaDGnYSQE7,ApYltST_icg,PtkRm91M0En data_only:false slice:true source:All" }, + //{ "windows-events after:1726055370 before:1726056270 direction:backward last:200 facets:HdUoSYab5wV,Cq2r7mRUv4a,LAnVlsIQfeD,BnPLNbA5VWT,KeCITtVD5AD,HytMJ9kj82B,LT.Xp9I9tiP,No4kPTQbS.g,LQ2LQzfE8EG,PtkRm91M0En,JM3OPW3kHn6,ClaDGnYSQE7,H106l8MXSSr,HREiMN.4Ahu data_only:false source:All HytMJ9kj82B:BlC24d5JBBV,PtVoyIuX.MU,HMj1B38kHTv KeCITtVD5AD:PY1JtCeWwSe,O9kz5J37nNl,JZoJURadhDb" }, + // { "windows-events after:1725636012 before:1726240812 direction:backward last:200 facets:HdUoSYab5wV,Cq2r7mRUv4a,LAnVlsIQfeD,BnPLNbA5VWT,KeCITtVD5AD,HytMJ9kj82B,JM3OPW3kHn6,H106l8MXSSr,HREiMN.4Ahu,ClaDGnYSQE7,ApYltST_icg,PtkRm91M0En data_only:false source:All PtkRm91M0En:LDzHbP5libb" }, + //{ "windows-events after:1725650386 before:1725736786 anchor:1725652420809461 direction:forward last:200 facets:HWNGeY7tg6c,LAnVlsIQfeD,BnPLNbA5VWT,Cq2r7mRUv4a,KeCITtVD5AD,I_Amz_APBm3,HytMJ9kj82B,LT.Xp9I9tiP,No4kPTQbS.g,LQ2LQzfE8EG,PtkRm91M0En,JM3OPW3kHn6 if_modified_since:1725736649011085 data_only:true delta:true tail:true source:all Cq2r7mRUv4a:PPc9fUy.q6o No4kPTQbS.g:Dwo9PhK27v3 HytMJ9kj82B:KbbznGjt_9r LAnVlsIQfeD:OfU1t5cpjgG JM3OPW3kHn6:CS_0g5AEpy2" }, + //{ "windows-events info after:1725650420 before:1725736820" }, + //{ "windows-events after:1725650420 before:1725736820 last:200 facets:HWNGeY7tg6c,LAnVlsIQfeD,BnPLNbA5VWT,Cq2r7mRUv4a,KeCITtVD5AD,I_Amz_APBm3,HytMJ9kj82B,LT.Xp9I9tiP,No4kPTQbS.g,LQ2LQzfE8EG,PtkRm91M0En,JM3OPW3kHn6 source:all Cq2r7mRUv4a:PPc9fUy.q6o No4kPTQbS.g:Dwo9PhK27v3 HytMJ9kj82B:KbbznGjt_9r LAnVlsIQfeD:OfU1t5cpjgG JM3OPW3kHn6:CS_0g5AEpy2" }, + //{ "windows-events after:1725650430 before:1725736830 last:200 facets:HWNGeY7tg6c,LAnVlsIQfeD,BnPLNbA5VWT,Cq2r7mRUv4a,KeCITtVD5AD,I_Amz_APBm3,HytMJ9kj82B,LT.Xp9I9tiP,No4kPTQbS.g,LQ2LQzfE8EG,PtkRm91M0En,JM3OPW3kHn6 source:all Cq2r7mRUv4a:PPc9fUy.q6o No4kPTQbS.g:Dwo9PhK27v3 HytMJ9kj82B:KbbznGjt_9r LAnVlsIQfeD:OfU1t5cpjgG JM3OPW3kHn6:CS_0g5AEpy2" }, + { NULL }, + }; + + for(int i = 0; array[i].func ;i++) { + bool cancelled = false; + usec_t stop_monotonic_ut = now_monotonic_usec() + 600 * USEC_PER_SEC; + //char buf[] = "windows-events after:-86400 before:0 direction:backward last:200 data_only:false slice:true source:all"; + function_windows_events("123", (char *)array[i].func, &stop_monotonic_ut, &cancelled, NULL, HTTP_ACCESS_ALL, NULL, NULL); + } + printf("\n\nAll done!\n\n"); + fflush(stdout); + exit(1); + } + + // ------------------------------------------------------------------------ + // the event loop for functions + + struct functions_evloop_globals *wg = + functions_evloop_init(WINDOWS_EVENTS_WORKER_THREADS, "WEVT", &stdout_mutex, &plugin_should_exit); + + functions_evloop_add_function(wg, + WEVT_FUNCTION_NAME, + function_windows_events, + WINDOWS_EVENTS_DEFAULT_TIMEOUT, + NULL); + + // ------------------------------------------------------------------------ + // register functions to netdata + + netdata_mutex_lock(&stdout_mutex); + + fprintf(stdout, PLUGINSD_KEYWORD_FUNCTION " GLOBAL \"%s\" %d \"%s\" \"logs\" "HTTP_ACCESS_FORMAT" %d\n", + WEVT_FUNCTION_NAME, WINDOWS_EVENTS_DEFAULT_TIMEOUT, WEVT_FUNCTION_DESCRIPTION, + (HTTP_ACCESS_FORMAT_CAST)(HTTP_ACCESS_SIGNED_ID | HTTP_ACCESS_SAME_SPACE | HTTP_ACCESS_SENSITIVE_DATA), + RRDFUNCTIONS_PRIORITY_DEFAULT); + + fflush(stdout); + netdata_mutex_unlock(&stdout_mutex); + + // ------------------------------------------------------------------------ + + usec_t send_newline_ut = 0; + usec_t since_last_scan_ut = WINDOWS_EVENTS_SCAN_EVERY_USEC * 2; // something big to trigger scanning at start + usec_t since_last_providers_release_ut = 0; + const bool tty = isatty(fileno(stdout)) == 1; + + heartbeat_t hb; + heartbeat_init(&hb, USEC_PER_SEC); + while(!plugin_should_exit) { + + if(since_last_scan_ut > WINDOWS_EVENTS_SCAN_EVERY_USEC) { + wevt_sources_scan(); + since_last_scan_ut = 0; + } + + if(since_last_providers_release_ut > WINDOWS_EVENTS_RELEASE_PROVIDERS_HANDLES_EVERY_UT) { + providers_release_unused_handles(); + since_last_providers_release_ut = 0; + } + + usec_t dt_ut = heartbeat_next(&hb); + since_last_providers_release_ut += dt_ut; + since_last_scan_ut += dt_ut; + send_newline_ut += dt_ut; + + if(!tty && send_newline_ut > USEC_PER_SEC) { + send_newline_and_flush(&stdout_mutex); + send_newline_ut = 0; + } + } + + exit(0); +} diff --git a/src/collectors/windows-events.plugin/windows-events.h b/src/collectors/windows-events.plugin/windows-events.h new file mode 100644 index 000000000..34d600a98 --- /dev/null +++ b/src/collectors/windows-events.plugin/windows-events.h @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WINDOWS_EVENTS_H +#define NETDATA_WINDOWS_EVENTS_H + +#include "libnetdata/libnetdata.h" +#include "collectors/all.h" + +typedef enum { + WEVT_NO_CHANNEL_MATCHED, + WEVT_FAILED_TO_OPEN, + WEVT_FAILED_TO_SEEK, + WEVT_TIMED_OUT, + WEVT_OK, + WEVT_NOT_MODIFIED, + WEVT_CANCELLED, +} WEVT_QUERY_STATUS; + +#define WEVT_CHANNEL_CLASSIC_TRACE 0x0 +#define WEVT_CHANNEL_GLOBAL_SYSTEM 0x8 +#define WEVT_CHANNEL_GLOBAL_APPLICATION 0x9 +#define WEVT_CHANNEL_GLOBAL_SECURITY 0xa + +#define WEVT_LEVEL_NONE 0x0 +#define WEVT_LEVEL_CRITICAL 0x1 +#define WEVT_LEVEL_ERROR 0x2 +#define WEVT_LEVEL_WARNING 0x3 +#define WEVT_LEVEL_INFORMATION 0x4 +#define WEVT_LEVEL_VERBOSE 0x5 +#define WEVT_LEVEL_RESERVED_6 0x6 +#define WEVT_LEVEL_RESERVED_7 0x7 +#define WEVT_LEVEL_RESERVED_8 0x8 +#define WEVT_LEVEL_RESERVED_9 0x9 +#define WEVT_LEVEL_RESERVED_10 0xa +#define WEVT_LEVEL_RESERVED_11 0xb +#define WEVT_LEVEL_RESERVED_12 0xc +#define WEVT_LEVEL_RESERVED_13 0xd +#define WEVT_LEVEL_RESERVED_14 0xe +#define WEVT_LEVEL_RESERVED_15 0xf + +#define WEVT_OPCODE_INFO 0x0 +#define WEVT_OPCODE_START 0x1 +#define WEVT_OPCODE_STOP 0x2 +#define WEVT_OPCODE_DC_START 0x3 +#define WEVT_OPCODE_DC_STOP 0x4 +#define WEVT_OPCODE_EXTENSION 0x5 +#define WEVT_OPCODE_REPLY 0x6 +#define WEVT_OPCODE_RESUME 0x7 +#define WEVT_OPCODE_SUSPEND 0x8 +#define WEVT_OPCODE_SEND 0x9 +#define WEVT_OPCODE_RECEIVE 0xf0 +#define WEVT_OPCODE_RESERVED_241 0xf1 +#define WEVT_OPCODE_RESERVED_242 0xf2 +#define WEVT_OPCODE_RESERVED_243 0xf3 +#define WEVT_OPCODE_RESERVED_244 0xf4 +#define WEVT_OPCODE_RESERVED_245 0xf5 +#define WEVT_OPCODE_RESERVED_246 0xf6 +#define WEVT_OPCODE_RESERVED_247 0xf7 +#define WEVT_OPCODE_RESERVED_248 0xf8 +#define WEVT_OPCODE_RESERVED_249 0xf9 +#define WEVT_OPCODE_RESERVED_250 0xfa +#define WEVT_OPCODE_RESERVED_251 0xfb +#define WEVT_OPCODE_RESERVED_252 0xfc +#define WEVT_OPCODE_RESERVED_253 0xfd +#define WEVT_OPCODE_RESERVED_254 0xfe +#define WEVT_OPCODE_RESERVED_255 0xff + +#define WEVT_TASK_NONE 0x0 + +#define WEVT_KEYWORD_NONE 0x0 +#define WEVT_KEYWORD_RESPONSE_TIME 0x0001000000000000 +#define WEVT_KEYWORD_WDI_CONTEXT 0x0002000000000000 +#define WEVT_KEYWORD_WDI_DIAG 0x0004000000000000 +#define WEVT_KEYWORD_SQM 0x0008000000000000 +#define WEVT_KEYWORD_AUDIT_FAILURE 0x0010000000000000 +#define WEVT_KEYWORD_AUDIT_SUCCESS 0x0020000000000000 +#define WEVT_KEYWORD_CORRELATION_HINT 0x0040000000000000 +#define WEVT_KEYWORD_EVENTLOG_CLASSIC 0x0080000000000000 +#define WEVT_KEYWORD_RESERVED_56 0x0100000000000000 +#define WEVT_KEYWORD_RESERVED_57 0x0200000000000000 +#define WEVT_KEYWORD_RESERVED_58 0x0400000000000000 +#define WEVT_KEYWORD_RESERVED_59 0x0800000000000000 +#define WEVT_KEYWORDE_RESERVED_60 0x1000000000000000 +#define WEVT_KEYWORD_RESERVED_61 0x2000000000000000 +#define WEVT_KEYWORD_RESERVED_62 0x4000000000000000 +#define WEVT_KEYWORD_RESERVED_63 0x8000000000000000 + +#define WEVT_LEVEL_NAME_NONE "None" +#define WEVT_LEVEL_NAME_CRITICAL "Critical" +#define WEVT_LEVEL_NAME_ERROR "Error" +#define WEVT_LEVEL_NAME_WARNING "Warning" +#define WEVT_LEVEL_NAME_INFORMATION "Information" +#define WEVT_LEVEL_NAME_VERBOSE "Verbose" + +#define WEVT_OPCODE_NAME_INFO "Info" +#define WEVT_OPCODE_NAME_START "Start" +#define WEVT_OPCODE_NAME_STOP "Stop" +#define WEVT_OPCODE_NAME_DC_START "DC Start" +#define WEVT_OPCODE_NAME_DC_STOP "DC Stop" +#define WEVT_OPCODE_NAME_EXTENSION "Extension" +#define WEVT_OPCODE_NAME_REPLY "Reply" +#define WEVT_OPCODE_NAME_RESUME "Resume" +#define WEVT_OPCODE_NAME_SUSPEND "Suspend" +#define WEVT_OPCODE_NAME_SEND "Send" +#define WEVT_OPCODE_NAME_RECEIVE "Receive" + +#define WEVT_TASK_NAME_NONE "None" + +#define WEVT_KEYWORD_NAME_NONE "None" +#define WEVT_KEYWORD_NAME_RESPONSE_TIME "Response Time" +#define WEVT_KEYWORD_NAME_WDI_CONTEXT "WDI Context" +#define WEVT_KEYWORD_NAME_WDI_DIAG "WDI Diagnostics" +#define WEVT_KEYWORD_NAME_SQM "SQM (Software Quality Metrics)" +#define WEVT_KEYWORD_NAME_AUDIT_FAILURE "Audit Failure" +#define WEVT_KEYWORD_NAME_AUDIT_SUCCESS "Audit Success" +#define WEVT_KEYWORD_NAME_CORRELATION_HINT "Correlation Hint" +#define WEVT_KEYWORD_NAME_EVENTLOG_CLASSIC "Event Log Classic" + +#define WEVT_PREFIX_LEVEL "Level " // the space at the end is needed +#define WEVT_PREFIX_KEYWORDS "Keywords " // the space at the end is needed +#define WEVT_PREFIX_OPCODE "Opcode " // the space at the end is needed +#define WEVT_PREFIX_TASK "Task " // the space at the end is needed + +#include "windows-events-sources.h" +#include "windows-events-unicode.h" +#include "windows-events-xml.h" +#include "windows-events-providers.h" +#include "windows-events-fields-cache.h" +#include "windows-events-query.h" + +// enable or disable preloading on full-text-search +#define ON_FTS_PRELOAD_MESSAGE 1 +#define ON_FTS_PRELOAD_XML 0 +#define ON_FTS_PRELOAD_EVENT_DATA 1 + +#define WEVT_FUNCTION_DESCRIPTION "View, search and analyze the Microsoft Windows Events log." +#define WEVT_FUNCTION_NAME "windows-events" + +#define WINDOWS_EVENTS_WORKER_THREADS 5 +#define WINDOWS_EVENTS_DEFAULT_TIMEOUT 600 +#define WINDOWS_EVENTS_SCAN_EVERY_USEC (5 * 60 * USEC_PER_SEC) +#define WINDOWS_EVENTS_PROGRESS_EVERY_UT (250 * USEC_PER_MS) +#define FUNCTION_PROGRESS_EVERY_ROWS (2000) +#define FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS (1000) +#define ANCHOR_DELTA_UT (10 * USEC_PER_SEC) + +// run providers release every 5 mins +#define WINDOWS_EVENTS_RELEASE_PROVIDERS_HANDLES_EVERY_UT (5 * 60 * USEC_PER_SEC) +// release idle handles that are older than 5 mins +#define WINDOWS_EVENTS_RELEASE_IDLE_PROVIDER_HANDLES_TIME_UT (5 * 60 * USEC_PER_SEC) + +#define WEVT_FIELD_COMPUTER "Computer" +#define WEVT_FIELD_CHANNEL "Channel" +#define WEVT_FIELD_PROVIDER "Provider" +#define WEVT_FIELD_PROVIDER_GUID "ProviderGUID" +#define WEVT_FIELD_EVENTRECORDID "EventRecordID" +#define WEVT_FIELD_VERSION "Version" +#define WEVT_FIELD_QUALIFIERS "Qualifiers" +#define WEVT_FIELD_EVENTID "EventID" +#define WEVT_FIELD_LEVEL "Level" +#define WEVT_FIELD_KEYWORDS "Keywords" +#define WEVT_FIELD_OPCODE "Opcode" +#define WEVT_FIELD_ACCOUNT "UserAccount" +#define WEVT_FIELD_DOMAIN "UserDomain" +#define WEVT_FIELD_SID "UserSID" +#define WEVT_FIELD_TASK "Task" +#define WEVT_FIELD_PROCESSID "ProcessID" +#define WEVT_FIELD_THREADID "ThreadID" +#define WEVT_FIELD_ACTIVITY_ID "ActivityID" +#define WEVT_FIELD_RELATED_ACTIVITY_ID "RelatedActivityID" +#define WEVT_FIELD_XML "XML" +#define WEVT_FIELD_MESSAGE "Message" +#define WEVT_FIELD_EVENTS_API "EventsAPI" +#define WEVT_FIELD_EVENT_DATA_HIDDEN "__HIDDEN__EVENT__DATA__" +#define WEVT_FIELD_EVENT_MESSAGE_HIDDEN "__HIDDEN__MESSAGE__DATA__" +#define WEVT_FIELD_EVENT_XML_HIDDEN "__HIDDEN__XML__DATA__" + +// functions needed by LQS + +// structures needed by LQS +struct lqs_extension { + wchar_t *query; + + struct { + struct { + size_t completed; + size_t total; + } queries; + + struct { + size_t current_query_total; + size_t completed; + size_t total; + } entries; + + usec_t last_ut; + } progress; + + // struct { + // usec_t start_ut; + // usec_t stop_ut; + // usec_t first_msg_ut; + // + // uint64_t first_msg_seqnum; + // } query_file; + + // struct { + // uint32_t enable_after_samples; + // uint32_t slots; + // uint32_t sampled; + // uint32_t unsampled; + // uint32_t estimated; + // } samples; + + // struct { + // uint32_t enable_after_samples; + // uint32_t every; + // uint32_t skipped; + // uint32_t recalibrate; + // uint32_t sampled; + // uint32_t unsampled; + // uint32_t estimated; + // } samples_per_file; + + // struct { + // usec_t start_ut; + // usec_t end_ut; + // usec_t step_ut; + // uint32_t enable_after_samples; + // uint32_t sampled[SYSTEMD_JOURNAL_SAMPLING_SLOTS]; + // uint32_t unsampled[SYSTEMD_JOURNAL_SAMPLING_SLOTS]; + // } samples_per_time_slot; + + // per file progress info + // size_t cached_count; + + // progress statistics + usec_t matches_setup_ut; + size_t rows_useful; + size_t rows_read; + size_t bytes_read; + size_t files_matched; + size_t file_working; +}; + +// prepare LQS +#define LQS_DEFAULT_SLICE_MODE 0 +#define LQS_FUNCTION_NAME WEVT_FUNCTION_NAME +#define LQS_FUNCTION_DESCRIPTION WEVT_FUNCTION_DESCRIPTION +#define LQS_DEFAULT_ITEMS_PER_QUERY 200 +#define LQS_DEFAULT_ITEMS_SAMPLING 1000000 +#define LQS_SOURCE_TYPE WEVT_SOURCE_TYPE +#define LQS_SOURCE_TYPE_ALL WEVTS_ALL +#define LQS_SOURCE_TYPE_NONE WEVTS_NONE +#define LQS_PARAMETER_SOURCE_NAME "Event Channels" // this is how it is shown to users +#define LQS_FUNCTION_GET_INTERNAL_SOURCE_TYPE(value) WEVT_SOURCE_TYPE_2id_one(value) +#define LQS_FUNCTION_SOURCE_TO_JSON_ARRAY(wb) wevt_sources_to_json_array(wb) +#include "libnetdata/facets/logs_query_status.h" + +#include "windows-events-query-builder.h" // needs the LQS definition, so it has to be last + +#endif //NETDATA_WINDOWS_EVENTS_H |