diff options
Diffstat (limited to 'toolkit/components/glean/docs/dev')
| -rw-r--r-- | toolkit/components/glean/docs/dev/builtin_pings.md | 13 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/code_organization.md | 56 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/images/fog-modules.svg | 6 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/index.md | 15 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/ipc.md | 183 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/jog.md | 91 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/local_glean.md | 100 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/new_metric_types.md | 282 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/preferences.md | 105 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/storage.md | 14 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/style_guide.md | 41 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/testing.md | 240 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/updating_parser.md | 52 | ||||
| -rw-r--r-- | toolkit/components/glean/docs/dev/updating_sdk.md | 48 |
14 files changed, 1246 insertions, 0 deletions
diff --git a/toolkit/components/glean/docs/dev/builtin_pings.md b/toolkit/components/glean/docs/dev/builtin_pings.md new file mode 100644 index 0000000000..cbfff8c559 --- /dev/null +++ b/toolkit/components/glean/docs/dev/builtin_pings.md @@ -0,0 +1,13 @@ +# Built-in Pings + +FOG embeds the Glean SDK so +[its documentation on pings is authoritative](https://mozilla.github.io/glean/book/user/pings/index.html). +The only detail FOG adds is to clarify +[the "baseline" ping's schedule](https://mozilla.github.io/glean/book/user/pings/baseline.html#scheduling). +Specifically, in Firefox Desktop, the application is considered +* "active" when started, + or when a user interacts with Firefox after a 20min period of inactivity, +* "inactive" after the user stops interacting with Firefox after 2min of activity. + +For more details about why, see the bug tree around +[bug 1635242](https://bugzilla.mozilla.org/show_bug.cgi?id=1635242). diff --git a/toolkit/components/glean/docs/dev/code_organization.md b/toolkit/components/glean/docs/dev/code_organization.md new file mode 100644 index 0000000000..e943dbfd50 --- /dev/null +++ b/toolkit/components/glean/docs/dev/code_organization.md @@ -0,0 +1,56 @@ +# FOG code organization + + + +The diagram shows the different modules of Project FOG. + +## FOG control + +This module is the glue between Firefox and Glean. + +* The code lives in `toolkit/components/glean/src`. +* It is written in Rust. +* The crate is named `fog_control`. +* It is not published to crates.io. +* It is not consumed by other Rust crates inside mozilla-central. + +This module is responsible for + +* collecting and assembling the [client information](https://mozilla.github.io/glean/book/user/pings/index.html#the-client_info-section) +* configuring the Glean SDK via the Rust Language Binding +* watching the Firefox Telemetry data upload preference (`datareporting.healthreport.uploadEnabled`) +* scheduling builtin pings +* controling ping upload workers +* passing IPC buffers + +It calls into `glean` (the Glean SDK Rust Language Binding) to: + +* configure and initialize Glean +* toggle `upload_enabled` +* get upload tasks + +It calls into `fog` to: + +* pass IPC buffers +* record to its own metrics + +## FOG API + +This module provides the user-facing API for Glean inside mozilla-central. + +* The code lives in `toolkit/components/glean/api`. +* It is written in Rust. +* The crate is named `fog`. +* It is not published to crates.io. +* It can be consumed by other Rust crates inside mozilla-central for their Glean usage. + +This module is responsible for + +* exposing a specific metric API in Rust +* wrapping metric implementations for handling IPC +* exposing FFI functionality to implement other language APIs on top. + See also [Adding a New Metric Type](new_metric_types.md). + +It calls into `glean` (the Glean SDK Rust Language Binding) for: + +* metric types (including pings) diff --git a/toolkit/components/glean/docs/dev/images/fog-modules.svg b/toolkit/components/glean/docs/dev/images/fog-modules.svg new file mode 100644 index 0000000000..c0271e9f6e --- /dev/null +++ b/toolkit/components/glean/docs/dev/images/fog-modules.svg @@ -0,0 +1,6 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- This Source Code Form is subject to the terms of the Mozilla Public + - License, v. 2.0. If a copy of the MPL was not distributed with this + - file, You can obtain one at http://mozilla.org/MPL/2.0/. --> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" style="background-color: rgb(255, 255, 255);" version="1.1" width="601px" height="511px" viewBox="-0.5 -0.5 601 511" content="<mxfile host="app.diagrams.net" modified="2020-03-27T09:36:49.445Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0" etag="_te5ruI7nKv5hlBG0WM7" version="12.9.3" type="device"><diagram id="02tFoor6QwIjLFNeosYU" name="Page-1">7Vtbd6M2EP41fkyPQYDxo+NcmnbbpOvu2eZRBtmmxYgVsmP311eAZC6SDU4AX5ok5wSNLsA334xGI9ED4+XmkcBw8Rt2kd/T++6mB+56uq4NLYP9iyVbLtHsYSqZE8/lskww8f5FXNjn0pXnoqjQkGLsUy8sCh0cBMihBRkkBL8Vm82wX7xrCOdIEkwc6MvS755LF6nUNvuZ/GfkzRfizlqf1yyhaMwF0QK6+C0nAvc9MCYY0/RquRkjP0ZP4JL2e9hTu3swggJap0P/YUNGP0L/5QvY+N8XzwT/Y93wUdbQX/EX5g9LtwIBgleBi+JB+j1w+7bwKJqE0Ilr35jSmWxBlz4raexy5vn+GPuYJH3BbDbTHYfJI8puh3I1rjW1TIvVyK8hngkRijY5EX+tR4SXiJItayJqBcScZBYvvmUKs4RaFnlliX6Qk2S+GzrDkV1wKI+AVW8XVtdEtmuoYLX1KbBaglWza+IKQFu4AgWuls9ue+t66/iGvjcPkgrrxyo2rFuSPNauyK7m/H/SbYYZPHm9iIZxxU2UuKMRa6BZ4UYe5Q6tkY9DRGLPAh0vmItx2dulQxdvx8TJgwppiRNMMbSo+KKCAxygEhu4SLx4+rrgNtayx7zYiMuXnuvGd1ESrUjFBphj2pUGqRkq4rTFG6OCN0frv2umPQUUkQD60SfBEoINq12TBrpkmGZKYCOXRRK8iAld4DlmGrzPpCVcsjZfMA65fv5GlG55WARXFBe1hzYe/Svu/pPJS6+5mrsNHzkpbEUhYO+b6xQXX/N1WbektK1SW4RXxEEHsLF43AbJHPGuT9/MP23zj8lo8818CbSvweDGFsFIjNtBEhDkQ+qtixGaSqNJ1xEhcJtrEGIvoFFu5JdYkHELDErcGpbiqlJ73TjYnl2kT5Bxa/cq76ebJTk0Ns5t8tdk+GY7SB2+TW3TMBsyZqAVARwqbFlX2LLVli0PJXB/gWs4cYgX0qNdah6sffAK5+qg2M2f3Lsa9ukUonQMxqdv3Tvv6BflXM3DzvI8nKvAtJ4DOMrBXrh1quGyPs1zLzYiZ5UzT3VDcBb2qfdL9jaosM/D7VuyT1XWKrecSxc/X1cRrbks+lAuBiJ7poyRLMdG01lDuRjj3IxeTsU8PD8ywejlieEMlzFqwTQK08Vpqdy/AhXs1pInC4tqZBk7cMTvd44CrErnqDcevHwsu9vfw/0xZtMG9q+A3IZxagejy07+0UcwYKLJ3a8XCLE+PDuIZQeSTpoXh+1uG6IKy/Z2fOT58HIzMlYpqqs707WHrimh+/Qyzr3mg4zyAi+nq6iTJVk5g7UDIgeYrcDLbg2vQY3QIHBH8QY5Kzk+jCLPKS2w3rlYes/C7AMxhNl0DJHTmanQmZAdtw6TFk5GycTMspdP35z3OrACM8zSQFppoBQZaaCmFmO63STVCkTLeFdBNS1HtIx2TVNNLHmqqWZ+Uq0VqsmJ+df7icS2j29ynkkeHpTw3k0zOd6ottFbm4ZFHJBTwO/PV4x/KWLXFRF7t/i3fVzLjH+VwXry00tPK+Tk6U87iW1V2KmrNvV3ScjmAa+TYmk7juogihKoVk5toG6a+n8ytR29/1V2KVYH+XJwmg2azjdbQN0ADRg1WdxNPhHIuQNKYBDNkoOFKHCwmxwtvLi0jAlKbB/IHr3bRAKwT2IKF7FXKayi2nzqrm86Mh/5dGeaHIrY1NxsfNRJ9q1sNaqwv1urMcCn1VQZQ7XVDM/LavalVAly1tdgNapD591aTY3V2lnvzzbP2D3Bdyl9bgy6Cb6NisMt+56r3WBdTnLlgkEXXU0waJx8z86QV/epC/ThNkb7WvNaVo2TLIOWFHHobKriPFftr2Y01VczsS7T80jqb2YubpLbqUBorr2MMCtm36amzi37xBfc/wc=</diagram></mxfile>"><defs/><g><rect x="0" y="60" width="600" height="120" fill="#fff2cc" stroke="#d6b656" pointer-events="all"/><rect x="0" y="180" width="600" height="330" fill="#d5e8d4" stroke="#82b366" pointer-events="all"/><rect x="460" y="60" width="140" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe flex-end; width: 138px; height: 1px; padding-top: 75px; margin-left: 460px;"><div style="box-sizing: border-box; font-size: 0; text-align: right; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div align="right"><font style="font-size: 16px">Developer facing</font></div></div></div></div></foreignObject><text x="598" y="79" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="end">Developer facing</text></switch></g><rect x="470" y="180" width="130" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe flex-end; width: 128px; height: 1px; padding-top: 195px; margin-left: 470px;"><div style="box-sizing: border-box; font-size: 0; text-align: right; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div style="font-size: 16px" align="right"><font style="font-size: 16px">Internals</font></div></div></div></div></foreignObject><text x="598" y="199" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="end">Internals</text></switch></g><path d="M 250 150 L 250 190 L 120 190 L 120 203.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 120 208.88 L 116.5 201.88 L 120 203.63 L 123.5 201.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="190" y="90" width="120" height="60" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 191px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">C++</div></div></div></foreignObject><text x="250" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">C++</text></switch></g><rect x="360" y="90" width="120" height="60" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 361px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">JavaScript</div></div></div></foreignObject><text x="420" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">JavaScript</text></switch></g><path d="M 420 150 L 420 190 L 120 190 L 120 203.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 120 208.88 L 116.5 201.88 L 120 203.63 L 123.5 201.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="360" y="90" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 361px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">JavaScript</div></div></div></foreignObject><text x="420" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">JavaScript</text></switch></g><path d="M 80 150 L 80 170 L 80 223.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 80 228.88 L 76.5 221.88 L 80 223.63 L 83.5 221.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="20" y="90" width="120" height="60" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 21px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Rust</div></div></div></div></foreignObject><text x="80" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Rust</text></switch></g><rect x="20" y="230" width="120" height="60" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 260px; margin-left: 21px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">FOG API </div></div></div></foreignObject><text x="80" y="264" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">FOG API </text></switch></g><path d="M 140 470 L 160 470 L 150 470 L 163.63 470" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 168.88 470 L 161.88 473.5 L 163.63 470 L 161.88 466.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="20" y="440" width="120" height="60" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 470px; margin-left: 21px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">FOG Control</div></div></div></foreignObject><text x="80" y="474" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">FOG Control</text></switch></g><rect x="170" y="440" width="120" height="60" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 470px; margin-left: 171px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Glean SDK</div></div></div></foreignObject><text x="230" y="474" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Glean SDK</text></switch></g><rect x="480" y="0" width="120" height="20" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 10px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Rust</div></div></div></foreignObject><text x="540" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Rust</text></switch></g><rect x="480" y="30" width="120" height="20" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 40px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">C++</div></div></div></foreignObject><text x="540" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">C++</text></switch></g><path d="M 230 220 L 270 260 L 230 300 L 190 260 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 260px; margin-left: 191px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">IPC parent?</div></div></div></foreignObject><text x="230" y="264" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">IPC parent?</text></switch></g><path d="M 230 300 L 230 433.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 230 438.88 L 226.5 431.88 L 230 433.63 L 233.5 431.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 140 260 L 183.63 260" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 188.88 260 L 181.88 263.5 L 183.63 260 L 181.88 256.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="230" y="310" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 320px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">YES</div></div></div></foreignObject><text x="250" y="324" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">YES</text></switch></g><rect x="270" y="240" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 250px; margin-left: 271px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NO</div></div></div></foreignObject><text x="290" y="254" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NO</text></switch></g><rect x="360" y="230" width="230" height="200" fill="#f5f5f5" stroke="#666666" pointer-events="all"/><path d="M 270 260 L 460 260 Q 470 260 470 261.82 L 470 263.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 470 268.88 L 466.5 261.88 L 470 263.63 L 473.5 261.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 470 290 L 470 310 L 470 290 L 470 303.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 470 308.88 L 466.5 301.88 L 470 303.63 L 473.5 301.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="410" y="270" width="120" height="20" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 411px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">transfer encoding</div></div></div></foreignObject><text x="470" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">transfer encoding</text></switch></g><path d="M 470 330 L 470 353.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 470 358.88 L 466.5 351.88 L 470 353.63 L 473.5 351.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="410" y="310" width="120" height="20" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 320px; margin-left: 411px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">IPC send</div></div></div></foreignObject><text x="470" y="324" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">IPC send</text></switch></g><path d="M 470 380 L 470 393.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 470 398.88 L 466.5 391.88 L 470 393.63 L 473.5 391.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="410" y="360" width="120" height="20" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 370px; margin-left: 411px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">IPC recv</div></div></div></foreignObject><text x="470" y="374" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">IPC recv</text></switch></g><path d="M 470 420 L 470 470 L 296.37 470" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 291.12 470 L 298.12 466.5 L 296.37 470 L 298.12 473.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="410" y="400" width="120" height="20" fill="#dae8fc" stroke="#6c8ebf" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 410px; margin-left: 411px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">transfer decoding</div></div></div></foreignObject><text x="470" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">transfer decoding</text></switch></g><rect x="520" y="230" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 68px; height: 1px; padding-top: 240px; margin-left: 521px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">IPC layer</div></div></div></foreignObject><text x="555" y="244" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">IPC layer</text></switch></g><rect x="100" y="210" width="40" height="20" fill="#f8cecc" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 220px; margin-left: 101px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><font style="font-size: 11px">C API</font></div></div></div></foreignObject><text x="120" y="224" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">C API</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/toolkit/components/glean/docs/dev/index.md b/toolkit/components/glean/docs/dev/index.md new file mode 100644 index 0000000000..334f040ed9 --- /dev/null +++ b/toolkit/components/glean/docs/dev/index.md @@ -0,0 +1,15 @@ +# Developing Firefox on Glean + +This section of docs is designed to be helpful to people developing FOG. +If you're not touching code, tests, or documentation in `toolkit/components/glean`, +and you're not interested in implementation details, you probably want +[the user docs instead](../user/index). + +```{toctree} +:titlesonly: +:maxdepth: 1 +:glob: + +* +Glean SDK Source <https://github.com/mozilla/glean/> +``` diff --git a/toolkit/components/glean/docs/dev/ipc.md b/toolkit/components/glean/docs/dev/ipc.md new file mode 100644 index 0000000000..e008235c05 --- /dev/null +++ b/toolkit/components/glean/docs/dev/ipc.md @@ -0,0 +1,183 @@ +# Inter-process Communication (IPC) + +Firefox Desktop is a multi-process desktop application. +Code requiring instrumentation may be on any of its processes, +so FOG provide facilities to do just that. + +## Design + +The IPC Design of FOG was worked out in +[bug 1618253](https://bugzilla.mozilla.org/show_bug.cgi?id=1618253). + +It centred around a few specific concepts: + +### Forbidding Non-Commutative Operations + +Because we cannot nicely impose a canonical ordering of metric operations across all processes, +FOG forbids non-[commutative](https://en.wikipedia.org/wiki/Commutative_property) +metric operations in some circumstances. + +For example, +`Add()`-ing to a Counter metric works from multiple processes because the order doesn't matter. +However, given a String metric being `Set()` from multiple processes simultaneously, +which value should it take? + +This ambiguity is not a good foundation to build trust on, +so we forbid setting a String metric from multiple processes. + +#### List of Forbidden Operations + +* Boolean's `set` (this is the metric type's only operation) +* Labeled Boolean's `set` (this is the metric type's only operation) +* String's `set` (this is the metric type's only operation) +* Labeled String's `set` (this is the metric type's only operation) +* String List's `set` + * `add` is permitted (order and uniqueness are not guaranteed) +* Timespan's `start`, `stop`, and `cancel` (these are the metric type's only operations) +* UUID's `set` and `generateAndSet` (these are the metric type's only operations) +* Datetime's `set` (this is the metric type's only operation) +* Quantity's `set` (this is the metric type's only operation) + +This list may grow over time as new metric types are added. +If there's an operation/metric type on this list that you need to use in a non-parent process, +please reach out +[on the #glean channel](https://chat.mozilla.org/#/room/#glean:mozilla.org) +and we'll help you out. + +### Process Agnosticism + +For metric types that can be used cross-process, +FOG provides no facility for identifying which process the instrumentation is on. + +What this means is that if you accumulate to a +[Timing Distribution](https://mozilla.github.io/glean/book/user/metrics/timing_distribution.html) +in multiple processes, +all the samples from all the processes will be combined in the same metric. + +If you wish to distinguish samples from different process types, +you will need multiple metrics and inline code to select the proper one for the given process. +For example: + +```C++ +if (XRE_GetProcessType() == GeckoProcessType_Default) { + mozilla::glean::performance::cache_size.Accumulate(numBytes / 1024); +} else { + mozilla::glean::performance::non_main_process_cache_size.Accumulate(numBytes / 1024); +} +``` + +### Scheduling + +FOG makes no guarantee about when non-main-process metric values are sent across IPC. +FOG will try its best to schedule opportunistically in idle moments, +and during orderly shutdowns. + +There are a few cases where we provide more firm guarantees: + +#### Tests + +There are test-only APIs in Rust, C++, and Javascript. +These do not await a flush of child process metric values. +You can use the test-only method `testFlushAllChildren` on the `FOG` +XPCOM component to await child data's arrival: +```js +await Services.fog.testFlushAllChildren(); +``` +See [the test documentation](testing.md) for more details on testing FOG. +For writing tests about instrumentation, see +[the instrumentation test documentation](../user/instrumentation_tests). + +#### Pings + +We do not guarantee that non-main-process data has made it into a specific ping. + +[Built-in pings](https://mozilla.github.io/glean/book/user/pings/index.html) +are submitted by the Rust Glean SDK at times FOG doesn't directly control, +so there may be data not present in the parent process when a built-in ping is submitted. +We don't anticipate this causing a problem since child-process data that +"misses" a given ping will be included in the next one. + +At this time, +[Custom Pings](https://mozilla.github.io/glean/book/user/pings/custom.html) +must be sent in the parent process and have no mechanism +to schedule their submission for after child-process data arrives in the parent process. +[bug 1732118](https://bugzilla.mozilla.org/show_bug.cgi?id=1732118) +tracks the addition of such a mechanism or guarantee. + +#### Shutdown + +We will make a best effort during an orderly shutdown to flush all pending data in child processes. +This means a disorderly shutdown (usually a crash) +may result in child process data being lost. + +#### Size + +We don't measure or keep an up-to-date calculation of the size of the IPC Payload. +We do, however, keep a count of the number of times the IPC Payload has been accessed. +This is used as a (very) conservative estimate of the size of the IPC Payload so we do not exceed the +[IPC message size limit](https://searchfox.org/mozilla-central/search?q=kMaximumMessageSize). + +See [bug 1745660](https://bugzilla.mozilla.org/show_bug.cgi?id=1745660). + +### Mechanics + +The rough design is that the parent process can request an immediate flush of pending data, +and each child process can decide to flush its pending data whenever it wishes. +The former is via `FlushFOGData() returns (ByteBuf)` and the latter via `FOGData(ByteBuf)`. + +Pending Data is a buffer of bytes generated by `bincode` in Rust in the Child, +handed off to C++, passed over IPC, +then given back to `bincode` in Rust on the Parent. + +Rust is then responsible for turning the pending data into +[metrics API][glean-metrics] calls on the metrics in the parent process. + +#### Supported Process Types + +FOG supports messaging between the following types of child process and the parent process: +* content children (via `PContent` + (for now. See [bug 1641989](https://bugzilla.mozilla.org/show_bug.cgi?id=1641989)) +* gmp children (via `PGMP`) +* gpu children (via `PGPU`) +* rdd children (via `PRDD`) +* socket children (via `PSocketProcess`) + +See +[the process model docs](/dom/ipc/process_model.rst) +for more information about what that means. + +### Adding Support for a new Process Type + +Adding support for a new process type is a matter of extending the two messages +mentioned above in "Mechanics" to another process type's protocol (ipdl file). + +1. Add two messages to the appropriate sections in `P<ProcessType>.ipdl` + * (( **Note:** `PGPU` _should_ be the only ipdl where `parent` + means the non-parent/-main/-UI process, + but double-check that you get this correct.)) + * Add `async FOGData(ByteBuf&& aBuf);` to the parent/main/UI process side of things + (most often `parent:`). + * Add `async FlushFOGData() returns (ByteBuf buf);` to the non-parent/-main/-UI side + (most often `child:`). +2. Implement the protocol endpoints in `P<ProcessType>{Child|Parent}.{h|cpp}` + * The message added to the `parent: ` section goes in + `P<ProcessType>Parent.{h|cpp}` and vice versa. +3. Add to `FOGIPC.cpp`'s `FlushAllChildData` code that + 1. Enumerates all processes of the newly-supported type (there may only be one), + 2. Calls `SendFlushFOGData on each, and + 3. Adds the resulting promise to the array. +4. Add to `FOGIPC.cpp`'s `SendFOGData` the correct `GeckoProcessType_*` + enum value, and appropriate code for getting the parent process singleton and calling + `SendFOGData` on it. +5. Add to the fog crate's `register_process_shutdown` function + handling for at-shutdown flushing of IPC data. + If this isn't added, we will log (but not panic) + on the first use of Glean APIs on an unsupported process type. + * "Handling" might be an empty block with a comment explaining where to find it + (like how `PROCESS_TYPE_DEFAULT` is handled) + * Or it might be custom code + (like `PROCESS_TYPE_CONTENT`'s) +6. Add to the documented [list of supported process types](#supported-process-types) + the process type you added support for. + +[glean-metrics]: https://mozilla.github.io/glean/book/reference/metrics/index.html diff --git a/toolkit/components/glean/docs/dev/jog.md b/toolkit/components/glean/docs/dev/jog.md new file mode 100644 index 0000000000..c8d0549cd4 --- /dev/null +++ b/toolkit/components/glean/docs/dev/jog.md @@ -0,0 +1,91 @@ +# Runtime Metric Definition Subsystem: JOG + +```{admonition} I'm Sorry +Why is it caled JOG? Because it's concerned with... run... time. +``` + +The normal mechanism for registering metrics, +for reasons as varied from ease-of-impl to performance, +happens at compile time. +However, this doesn't support use cases like +* [Artifact Builds][artifact-build] + (Where only the JavaScript of Firefox Desktop is repackaged at build time, + so there is no compile environment) +* Dynamic Telemetry + (A theorized system for instrumenting Firefox Desktop without shipping code) +* Web Extensions + (Or at least the kind that can't or won't use + [the Glean JS SDK][glean-js]) + +Thus we need a subsystem that supports the runtime registration of metrics. +We call it JOG and it was implemented in [bug 1698184][impl-bug]. + +## JavaScript Only + +Metrics in C++ and Rust are identified by identifiers which we can't swap out at runtime. +Thus, in order for changes to metrics to be visible to instrumented systems in C++ or Rust, you must compile. + +JavaScript, on the other hand, we supply instances to on-demand. +It not only supports the specific use cases driving this project, +it's the only environment that can benefit from runtime metric definition in Firefox Desktop. + +## Design + +The original design was done as part of +[bug 1662863][design-bug]. +Things have mostly just been refined from there. + +## Architecture + +We silo as much of the subsystem as we can into the +`jog` module located in `toolkit/components/glean/bindings/jog/`. +This includes the metrics construction factory and storage for metrics instances and their names and ids. + +Unfortunately, so that the metrics instances can be accessed by FFI, +the Rust metrics instances created by the `jog` crate are stored within the `fog` crate. + +Speaking of FFI, the `jog` crate is using cbindgen to be accessible to C++. + +If necessary or pleasant, it is probably possible to do away with the C++ storage, +moving the category set and metrics id map to Rust and moving information over FFI as needed. + +Test methods are run from `nsIFOG` (so we can use them in JS in xpcshell) +to static `JOG::` functions. + +### Build Integration + +If JOG detects we're an artifact build (by checking `MOZ_ARTIFACT_BUILDS`), +it generates `jogfile.json` and ensures it is placed in `GreD` +(next to the `firefox` binary). + +`jogfile.json` includes only the metric and ping information necessary to register them at runtime. +(It doesn't know about tags or descriptions, just the shapes and names of things) + +This file is read the first time JS tries to get a metric category from the +`Glean` global or a ping from the `GleanPings` global. + +Yes, this is on the main thread. Yes, this is synchronous. Yes, this is file I/O. + +Since this is a developer build, we think this is worth it to support Artifact Builds. + +If we're wrong about this and there are additional conditions we should place JOG under, +please [contact us][glean-channel]. + +#### If things get weird, delete `objdir/dist/bin/jogfile.json` + +Sometimes, metrics or pings you've added may not appear when you run Firefox. +For these and other odd cases, the solution is the same: +delete `jogfile.json` from the `dist/bin` directory of your objdir, then try again. + +This shouldn't happen if you keep your artifact and non-artifact objdirs segregated +(as is good practice). + +If, despite doing things properly you still see this or something else odd, then that's a bug. +Please [file it in Toolkit :: Telemetry][file-bug] + +[artifact-build]: https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html +[glean-js]: https://mozilla.github.io/glean/book/user/adding-glean-to-your-project/javascript.html +[impl-bug]: https://bugzilla.mozilla.org/show_bug.cgi?id=1698184 +[design-bug]: https://bugzilla.mozilla.org/show_bug.cgi?id=1662863 +[glean-channel]: https://chat.mozilla.org/#/room/#glean:mozilla.org +[file-bug]: https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=--&bug_status=NEW&bug_type=defect&cf_a11y_review_project_flag=---&cf_fx_iteration=---&cf_fx_points=---&cf_performance_impact=---&cf_status_firefox106=---&cf_status_firefox107=---&cf_status_firefox108=---&cf_status_firefox_esr102=---&cf_status_thunderbird_esr102=---&cf_status_thunderbird_esr91=---&cf_tracking_firefox106=---&cf_tracking_firefox107=---&cf_tracking_firefox108=---&cf_tracking_firefox_esr102=---&cf_tracking_firefox_relnote=---&cf_tracking_thunderbird_esr102=---&cf_tracking_thunderbird_esr91=---&cf_webcompat_priority=---&component=Telemetry&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&filed_via=standard_form&flag_type-203=X&flag_type-37=X&flag_type-41=X&flag_type-607=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-799=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-855=X&flag_type-864=X&flag_type-930=X&flag_type-936=X&flag_type-937=X&flag_type-952=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Toolkit&rep_platform=Unspecified&short_desc=Problem%20with%20JOG%3A%20&target_milestone=---&version=unspecified diff --git a/toolkit/components/glean/docs/dev/local_glean.md b/toolkit/components/glean/docs/dev/local_glean.md new file mode 100644 index 0000000000..046cf6e79c --- /dev/null +++ b/toolkit/components/glean/docs/dev/local_glean.md @@ -0,0 +1,100 @@ +# Developing with a local Glean build + +FOG uses a release version of Glean, as published on [crates.io][cratesio-glean]. + +For local development and try runs you can replace this Glean implementation with a local or remote version. + +1. To tell `mach` where to find your Glean, patch the [top-level `Cargo.toml`][cargo-toml]. E.g. like this: + + ```toml + [patches.crates-io] + glean = { git = "https://github.com/myfork/glean", branch = "my-feature-branch" } + glean-core = { git = "https://github.com/myfork/glean", branch = "my-feature-branch" } + ``` + + Both crates are required to ensure they are in sync. + + You can specify the exact code to use by `branch`, `tag` or `rev` (Git commit). + See the [cargo documentation for details][cargo-doc]. + + You can also use a path dependency: + + ```toml + [patches.crates-io] + glean = { path = "../glean/glean-core/rlb" } + glean-core = { path = "../glean/glean-core" } + ``` + +2. If the crate version in the patched repository is not + [semver]-compatible with the version required by the + `fog` and `fog_control` crates, + you need to change the version in the following files to match the ones in your + `glean` repo: + + ``` + toolkit/components/glean/Cargo.toml + toolkit/components/glean/api/Cargo.toml + ``` + + This tells FOG's crates that it needs your local Glean's version. + +3. Update the Cargo lockfile: + + ``` + cargo update -p glean + ``` + +4. Mozilla's supply-chain management policy requires that third-party software + (which includes the Glean SDK because it is distributed as though it is third-party) + be audited and certified as safe. + Your local Glean SDK probably hasn't been vetted. If you try to vendor right now, + `./mach vendor rust` will complain something like: + + ``` + Vet error: There are some issues with your policy.audit-as-crates-io entries + ``` + + This is because your local Glean SDK is neither of a version nor is from a source that has been vetted. + To allow your local Glean crates to be treated as crates.io-sourced crates for vetting, + add the following sections to the top of `supply-chain/config.toml`: + + ```toml + [policy.glean] + audit-as-crates-io = true + + [policy.glean-core] + audit-as-crates-io = true + ``` + + If your local Glean is of a non-vetted version, you can update `glean` and + `glean-core`'s entries in `supply-chain/audits.toml` to the version you're using. + If you don't, `./mach vendor rust` will complain and not complete. + + **Note:** Do not attempt to check these changes in. + These changes bypass supply chain defenses. + `@supply-chain-reviewers` may become cross as they `r-` your patch. + +5. Vendor the changed crates: + + ``` + ./mach vendor rust + ``` + + **Note:** If you're using a path dependency, `mach vendor rust` doesn't actually change files. + Instead it pulls the files directly from the location on disk you specify. + +6. Finally, build Firefox: + + ``` + ./mach build + ``` + +A remote reference works for try runs as well, +but a path dependency will not. + +Please ensure to not land a non-release version of Glean. + +[cratesio-glean]: https://crates.io/crates/glean +[cargo-toml]: https://searchfox.org/mozilla-central/rev/f07a609a76136ef779c65185165ff5ac513cc172/Cargo.toml#76 +[cargo-doc]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories +[semver]: https://semver.org/ diff --git a/toolkit/components/glean/docs/dev/new_metric_types.md b/toolkit/components/glean/docs/dev/new_metric_types.md new file mode 100644 index 0000000000..151b41dc34 --- /dev/null +++ b/toolkit/components/glean/docs/dev/new_metric_types.md @@ -0,0 +1,282 @@ +# Adding a New Metric Type + +This document covers how to add a new metric type to FOG. +You should only have to do this if a new metric type is added to the +[Glean SDK](https://mozilla.github.io/glean/book/user/metrics/index.html) +and it is needed in Firefox Desktop. + +## IPC + +For detailed information about the IPC design, +including a list of forbidden operations, +please consult +[the FOG IPC documentation](ipc.md). + +When adding a new metric type, the main IPC considerations are: +* Which operations are forbidden because they are not commutative? + * Most `set`-style operations cannot be reconciled sensibly across multiple processes. +* If there are non-forbidden operations, +what partial representation will this metric have in non-main processes? +Put another way, what shape of storage will this take up in the +[IPC Payload](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/api/src/ipc.rs)? + * For example, Counters can aggregate all partial counts together to a single + "partial sum". So + [its representation](https://searchfox.org/mozilla-central/rev/803b368879fa332e8e2c1840bf1ec164f7ed2c32/toolkit/components/glean/api/src/ipc.rs#45) + in the IPC Payload is just a single number per Counter. + * In contrast, Timing Distributions' bucket arrangements are known only to the core, + so it can't combine sample counts in child processes. + Instead we record durations in the highest resolution (nanos), + and send a stream of high-precision samples across IPC. + +To implement IPC support in a metric type, +we split the metric into three pieces: +1. An umbrella `enum` with the name `MetricTypeMetric`. + * It has a `Child` and a `Parent` variant. + * It is IPC-aware and is responsible for + * If on a non-parent-process, + either storing partial representations in the IPC Payload, + or logging errors if forbidden non-test APIs are called. + (Or panicking if test APIs are called.) + * If on the parent process, dispatching API calls on its inner Rust Language Binding metric. +2. The parent-process implementation is supplied by + [the RLB](https://crates.io/crates/glean/). + * For testing, it stores the `MetricId` that identifies this particular metric in a cross-process fashion. + * For testing, it exposes a `child_metric()` function to create its `Child` equivalent. + * For testing and if it supports operations in a non-parent-process, + it exposes a `metric_id()` function to access the stored `MetricId`. +3. The `MetricTypeIpc` is the non-parent-process implementation. + * If it does support operations in non-parent processes it stores the + `MetricId` that identifies this particular metric in a cross-process fashion. + +## Mirrors + +FOG can mirror Glean metrics to Telemetry probes via the +[Glean Interface For Firefox Telemetry](../user/gifft.md). + +Can this metric type be mirrored? +Should it be mirrored? + +If so, add an appropriate Telemetry probe for it to mirror to, +documenting the compatibility in +[the GIFFT docs](../user/gifft.md). + +### GIFFT Tests + +If you add a GIFFT mirror, don't forget to test that the mirror works. +You should be able to do this by adding a task to +[`toolkit/components/glean/tests/xpcshell/test_GIFFT.js`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/xpcshell/test_GIFFT.js). + +### GIFFT C++ State: Typical Locking and Shutdown + +Some metric types (`labeled_*`, `timespan`, `timing_distribution`) +require holding state in C++ to make GIFFT work. +Pings also hold state to support `testBeforeNextSubmit()`. +If your new metric type requires state in C++, +the current state-of-the-art is a `StaticDataMutex`-locked `UniquePtr` to a `nsTHashTable`. +Access to the inner map is guarded by the lock and is controlled and lazily-instantiated through a single access function. +[See Ping's `GetCallbackMapLock()`](https://searchfox.org/mozilla-central/source/toolkit/components/glean/bindings/private/Ping.cpp) +for example. + +It is important to clear this state to avoid leaks. +(See [bug 1752417](https://bugzilla.mozilla.org/show_bug.cgi?id=1752417).) +However, instrumentation may call metrics APIs at any time. + +Therefore, GIFFT explicitly stops supporting these state-requiring operations after the +[`AppShutdownTelemetry` shutdown phase](https://searchfox.org/mozilla-central/source/xpcom/base/ShutdownPhase.h). +This is because during the next phase (`XPCOMWillShutdown`) we clear the state. + +## Rust + +FOG uses the Rust Language Binding APIs (the `glean` crate) with a layer of IPC on top. + +The IPC additions and glean-core trait implementations are in the +[`private` module of the `fog` crate](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/api/src/metrics). + +Each metric type gets its own file, mimicking the structure in +[`glean_core`](https://github.com/mozilla/glean/tree/main/glean-core/src/metrics) +and [`glean`](https://github.com/mozilla/glean/tree/main/glean-core/rlb/src/private). +Unless, of course, that metric is a labeled metric type. +Then the sub metric type gets its own file, +and you need to add "Labeledness" to it by implementing +`Sealed` for your new type following the pattern in +[`api/src/private/labeled.rs`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/api/src/private/labeled.rs). + +Every method on the metric type is public for now, +including test methods, +and is at least all the methods exposed via the +[metric traits](https://github.com/mozilla/glean/tree/main/glean-core/src/traits). + +To support IPC and the MLA FFI (see below) +we identify metric instances by MetricId and store them in maps in +[the `__glean_metric_maps` mod of `metrics.rs`](https://hg.mozilla.org/mozilla-central/toolkit/components/glean/api/src/metrics.rs). +This work is done by the `rust.py` and `rust(_pings).jinja2` extensions to `glean_parser` found +[in the `build_scripts/glean_parser_ext/` folder](https://searchfox.org/mozilla-central/source/toolkit/components/glean/build_scripts/glean_parser_ext). + +You shouldn't have to edit these files for new metric types, +as the original modifications to `glean_parser` for this type should already be generating correct code. + +### Rust Tests + +You should be able to smoke test the basic functionality in Rust unit tests. +You can do this within the metric type implementation file directly. + +## C++ and JS + +The C++ and JS APIs are implemented [atop the Rust API](code_organization.md). +We treat them both together since, though they're different languages, +they're both implemented in C++ and share much of their implementation. + +The overall design is to build the C++ API atop the Multi-Language Architecture's +(MLA's) FFI, then build the JS API atop the C++ API. +This allows features like the +[Glean Interface For Firefox Telemetry (GIFFT)](../user/gifft.md) +that target only C++ and JS to be more simply implemented in the C++ layer. +Exceptions to this (where the JS uses the FFI directly) are discouraged. + +Each metric type has six pieces you'll need to cover: + +### 1. MLA FFI + +- Using our convenient macros, + define the metric type's Multi-Language Architecture FFI layer above the Rust API in + [`api/src/ffi/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/api/src/ffi/). + +### 2. C++ Impl + +- Implement a type called `XMetric` (e.g. `CounterMetric`) in `mozilla::glean::impl` in + [`bindings/private/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/bindings/private/). + - Its methods should be named the same as the ones in the Rust API, transformed to `CamelCase`. + - They should all be public. + - Multiplex the FFI's `test_have` and `test_get` functions into a single + `TestGetValue` function that returns a + `mozilla::Maybe` wrapping the C++ type that best fits the metric type. +- Include the new metric type in + [`bindings/MetricTypes.h`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/bindings/MetricTypes.h). +- Include the new files in + [`moz.build`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/moz.build). + The header file should be added to `EXPORTS.mozilla.glean.bindings` and the + `.cpp` file should be added to `UNIFIED_SOURCES`. + +### 3. IDL + +- Duplicate the public API (including its docs) to + [`xpcom/nsIGleanMetrics.idl`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/xpcom/nsIGleanMetrics.idl) + with the name `nsIGleanX` (e.g. `nsIGleanCounter`). + - Inherit from `nsISupports`. + - The naming style for members here is `lowerCamelCase`. + You'll need a + [GUID](https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Generating_GUIDs) + because this is XPCOM, but you'll only need the canonical form since we're only exposing to JS. + - The `testGetValue` method will return a + `jsval` to permit it to return `undefined` when there is no value. + +### 4. JS Impl + +- Add an `nsIGleanX`-deriving, `XMetric`-owning type called + `GleanX` (e.g. `GleanCounter`) in the same header and `.cpp` as `XMetric` in + [`bindings/private/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/bindings/private/). + - Don't declare any methods beyond a ctor + (takes a `uint32_t` metric id, init-constructs a `impl::XMetric` member) + and dtor (`default`): the IDL will do the rest so long as you remember to add + `NS_DECL_ISUPPORTS` and `NS_DECL_NSIGLEANX`. + - In the definition of `GleanX`, member identifiers are back to + `CamelCase` and need macros like `NS_IMETHODIMP`. + Delegate operations to the owned `XMetric`, returning + `NS_OK` no matter what in non-test methods. + - Test-only methods can return `NS_ERROR` codes on failures, + but mostly return `NS_OK` and use `undefined` in the + `JS::MutableHandleValue` result to signal no value. + +### 6. Tests + +Two languages means two test suites. + +- Add a never-expiring test-only metric of your type to + [`test_metrics.yaml`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/test_metrics.yaml). + - Feel free to be clever with the name, + but be sure to make clear that it is test-only. +- **C++ Tests (GTest)** - Add a small test case to + [`gtest/TestFog.cpp`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/gtest/TestFog.cpp). + - For more details, peruse the [testing docs](testing.md). +- **JS Tests (xpcshell)** - Add a small test case to + [`xpcshell/test_Glean.js`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/xpcshell/test_Glean.js) + and + [`xpcshell/test_JOG.js`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/xpcshell/test_JOG.js). + If your metric type has supported IPC operations, also add cases to the `IPC` variants of these test files. + - For more details, peruse the [testing docs](testing.md). + +### 7. API Documentation + +Metric API Documentation is centralized in +[the Glean SDK Book](https://mozilla.github.io/glean/book/user/metrics/index.html). + +You will need to craft a Pull Request against +[the SDK](https://github.com/mozilla/glean/) +adding a C++ and JS example to the specific metric type's API docs. + +Add a notice at the top of both examples that these APIs are only available in Firefox Desktop: +````md +<div data-lang="C++" class="tab"> + +> **Note**: C++ APIs are only available in Firefox Desktop. + +```c++ +#include "mozilla/glean/GleanMetrics.h" + +mozilla::glean::category_name::metric_name.Api(args); +``` + +There are test APIs available too: + +```c++ +#include "mozilla/glean/GleanMetrics.h" + +ASSERT_EQ(value, mozilla::glean::category_name::metric_name.TestGetValue().ref()); +``` +</div> + +// and again for <div data-lang="JS"> +```` + +If you're lucky, the Rust API will have already been added. +Otherwise you'll need to write an example for that one too. + +### 8. Labeled metrics (if necessary) + +If your new metric type is Labeled, you have more work to do. +I'm assuming you've already implemented the non-labeled sub metric type following the steps above. +Now you must add "Labeledness" to it. + +There are four pieces to this: + +#### FFI + +- To add the writeable storage Rust will use to store the dynamically-generated sub metric instances, + add your sub metric type's map as a list item in the `submetric_maps` `mod` of + [`rust.jinja2`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/build_scripts/glean_parser_ext/templates/rust.jinja2). +- Following the pattern of the others, add a `fog_{your labeled metric name here}_get()` FFI API to + `api/src/ffi/mod.rs`. + This is what C++ and JS will use to allocate and retrieve sub metric instances by id. + +#### C++ + +- Following the pattern of the others, add a template specialiation for `Labeled<YourSubMetric>::Get` to + [`bindings/private/Labeled.cpp`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/bindings/private/Labeled.cpp). + This will ensure C++ consumers can fetch or create sub metric instances. + +#### JS + +- Already handled for you since the JS types all inherit from `nsISupports` + and the JS template knows to add your new type to `NewSubMetricFromIds(...)` + (see `GleanLabeled::NamedGetter` if you're curious). + +#### Tests + +- The labeled variant will need tests the same as Step #6. + A tip: be sure to test two labels with different values. + +## Python Tests + +We have a suite of tests for ensuring code generation generates appropriate code. +You should add a metric to [that suite](testing.md) for your new metric type. +You will need to regenerate the expected files. diff --git a/toolkit/components/glean/docs/dev/preferences.md b/toolkit/components/glean/docs/dev/preferences.md new file mode 100644 index 0000000000..732e25c5c9 --- /dev/null +++ b/toolkit/components/glean/docs/dev/preferences.md @@ -0,0 +1,105 @@ +# Preferences and Defines + +## User Preferences + +`datareporting.healthreport.uploadEnabled` + +This determines whether the Glean SDK is enabled. +It can be controlled by users via `about:preferences#privacy`. +If this is set to false from true, we send a +["deletion-request" ping](https://mozilla.github.io/glean/book/user/pings/deletion_request.html) +and no data collections will be persisted or reported from that point. + +## Test-only Preferences + +`telemetry.fog.test.localhost_port` + +If set to a value `port` which is greater than 0, pings will be sent to +`http://localhost:port` instead of `https://incoming.telemetry.mozilla.org`. +If set to a value `port` which is less than 0, FOG will: +1) Tell Glean that upload is enabled, even if it isn't. +2) Take all pings scheduled for upload and drop them on the floor, + telling the Glean SDK that it was sent successfully. + +This is how you emulate "recording enabled but upload disabled" +like developer builds have in Firefox Telemetry. +When switching from `port < 0` to `port >= 0`, +Glean will be told (if just temporarily) that upload is disabled. +This clears the stores of recorded-but-not-reported data. +Defaults to 0. + +`telemetry.fog.test.activity_limit` +`telemetry.fog.test.inactivity_limit` + +This pair of prefs control the length of time of activity before inactivity +(or vice versa) +needed before FOG informs the SDK's Client Activity API that the client was (in)active. +Present to allow testing without figuring out how to mock Rust's clock. +Their values are integer seconds. +Defaults to 120 (activity), 1200 (inactivity). + +## Internal Preferences + +`telemetry.fog.artifact_build` + +Read-only. This pref is `true` only if `MOZ_ARTIFACT_BUILDS` was set during configure. +If true, [JOG](./jog) is enabled so that artifact builds will exhibit changes to their Glean metrics. + +## Defines + +`MOZ_AUTOMATION` + +If set, and `GLEAN_SOURCE_TAGS` isn't set, FOG will set a +[Glean source tag](https://mozilla.github.io/glean/book/reference/debug/sourceTags.html) +of `automation`. + +If `GLEAN_SOURCE_TAGS` is set, the `automation` source tag will not be added automatically. + +If not set, any tags set by `GLEAN_SOURCE_TAGS` will be present. + +`MOZ_GLEAN_ANDROID` + +If set, the Glean SDK is assumed to be managed by something other than FOG, meaning: +* [GIFFT][gifft] is disabled. +* FOG doesn't initialize Glean +* FOG doesn't relay (in)activity or experiment annotations to Glean + +`MOZILLA_OFFICIAL` + +If unset, we set a `glean_disable_upload` Rust feature in +`gkrust` and `gkrust-shared` which is forwarded to `fog_control` as `disable_upload`. +This feature defaults FOG to an "upload disabled" +mode where collection on the client proceeds as normal but no ping is sent. +This mode can be overridden at runtime in two ways: +* If the ping has a + [Debug Tag](https://mozilla.github.io/glean/book/user/debugging/index.html) + then it is sent so that it may be inspected in the + [Glean Debug Ping Viewer](https://debug-ping-preview.firebaseapp.com/). +* If the preference `telemetry.fog.test.localhost_port` is set to a value greater than 0, + then pings are sent to a server operating locally at that port + (even if the ping has a Debug Tag), to enable testing. + +`MOZILLA_OFFICIAL` tends to be set on most builds released to users, +including builds distributed by Linux distributions. +It tends to not be set on local developer builds. +See [bug 1680025](https://bugzilla.mozilla.org/show_bug.cgi?id=1680025) for details. + +`MOZ_ARTIFACT_BUILDS` + +If `MOZ_ARTIFACT_BUILDS` is set in the build config, +[JOG](./jog) will generate a file for the runtime-registration of metrics and pings. +This is to support [Artifact Builds](/contributing/build/artifact_builds). + +See also `telemetry.fog.artifact_build`. + +`OS_TARGET` + +If not set to `'Android'` we set a `glean_million_queue` Rust feature +([see gkrust-features.mozbuild][gkrust-features]) +which, when passed to the Glean SDK, +opts us into a preinit queue that doesn't discard tasks until there are 10^6 of them. + +See [bug 1797494](https://bugzilla.mozilla.org/show_bug.cgi?id=1797494) for details. + +[gkrust-features]: https://searchfox.org/mozilla-central/source/toolkit/library/rust/gkrust-features.mozbuild +[gifft]: ../user/gifft diff --git a/toolkit/components/glean/docs/dev/storage.md b/toolkit/components/glean/docs/dev/storage.md new file mode 100644 index 0000000000..00464c6173 --- /dev/null +++ b/toolkit/components/glean/docs/dev/storage.md @@ -0,0 +1,14 @@ +# Storage + +Both FOG and the Glean SDK require some storage in the +[Firefox Profile Directory](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Multiple_profiles). + +## FOG + +At present FOG's storage is limited to its [preferences](preferences.md). + +## Glean SDK + +The Glean SDK stores things in the +[Glean Data Directory](https://mozilla.github.io/glean/book/dev/core/internal/directory-structure.html) +which can be found at `<profile_dir>/datareporting/glean`. diff --git a/toolkit/components/glean/docs/dev/style_guide.md b/toolkit/components/glean/docs/dev/style_guide.md new file mode 100644 index 0000000000..c289919f93 --- /dev/null +++ b/toolkit/components/glean/docs/dev/style_guide.md @@ -0,0 +1,41 @@ +# FOG Documentation Style Guide + +FOG's Documentation is written in Markdown. +You can find its source at `toolkit/components/glean/docs`. + +## Line breaks + +We will use [semantic linefeeds]: +* Break anywhere before 80-100 characters +* Break after any punctuation when it makes sense +* Break before or after any markdown when it makes sense + +**Tip:** To keep lines narrow, use markdown's [reference link] +feature whenever it makes sense (or all the time. Up to you.). + +## Linking to other documentation + +Linking to other external documentation is [easy][reference link]. +Linking to other pieces of documentation in the source docs requires a +link to the source file in the sphinx tree. + +Links can be relative e.g. to link to the [preferences] docs: + +```md +[preferences](preferences.md) +``` + +Or they can be absolute e.g. to link to the [Telemetry] docs: +```md +[Telemetry](/toolkit/components/telemetry/index.rst) +``` + +Sphinx will automagically transform that to an +appropriately-base-url'd url with a `.html` suffix. + + +[semantic linefeeds]: https://rhodesmill.org/brandon/2012/one-sentence-per-line/ +[reference link]: https://spec.commonmark.org/0.29/#reference-link +[Telemetry]: /toolkit/components/telemetry/index.rst +[#firefox-source-docs:mozilla.org]: https://chat.mozilla.org/#/room/#firefox-source-docs:mozilla.org +[bug 1621950]: https://bugzilla.mozilla.org/show_bug.cgi?id=1621950 diff --git a/toolkit/components/glean/docs/dev/testing.md b/toolkit/components/glean/docs/dev/testing.md new file mode 100644 index 0000000000..fe6c3e0fe7 --- /dev/null +++ b/toolkit/components/glean/docs/dev/testing.md @@ -0,0 +1,240 @@ +# Testing + +```{admonition} This documentation is about testing FOG itself +This document contains information about how FOG tests itself, +how to add new tests, how and what to log, and stuff like that. +If you're interested in learning how to test instrumentation you added, +you'll want to read +[the instrumetnation testing docs](../user/instrumentation_tests) instead. +``` + +Given the multiple API languages, processes, and dependencies, +testing FOG is a matter of choosing the right tool for the situation. + +## One Big Command + +To run all the things, here's the tl;dr: + +`./mach build && ./mach lint -W -w -o --fix +&& ./mach rusttests && ./mach gtest "FOG*" +&& python3 ./mach python-test toolkit/components/glean/tests/pytest +&& ./mach test toolkit/components/glean/tests/xpcshell +&& ./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/test_fog* --gecko-log "-" +&& ./mach test toolkit/components/glean/tests/browser +` + +## Logging + +An often-overlooked first line of testing is "what do the logs say?". +To turn on logging for FOG, use any of the following: +* Run Firefox with `RUST_LOG="fog_control,fog,glean_core"`. + * On some platforms this will use terminal colours to indicate log level. +* Run Firefox with `MOZ_LOG="timestamp,sync,glean::*:5,fog::*:5,fog_control::*:5,glean_core::*:5"`. +* Set the following prefs: + * `logging.config.timestamp` to `true` + * `logging.config.sync` to `true` + * `logging.fog_control::*` to `5` + * `logging.fog::*` to `5` + * `logging.glean::*` to `5` + * `logging.glean_core::*` to `5` + * `logging.config.clear_on_startup` to `false` (or all these prefs will be cleared on startup) + +For more information on logging in Gecko, see the +[Gecko Logging docs](/xpcom/logging). + +User-destined logs (of the "You did something wrong" variety) might output to the +[Browser Console](/devtools-user/browser_console/index) +if they originate in JS land. Open via +<kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>J</kbd>, or +<kbd>Cmd</kbd>+<kbd>Shift</kbd>+<kbd>J</kbd>. + +```{admonition} Note +At present, Rust logging in non-main processes just doesn't work. +``` + +### What to log, and to where? + +FOG covers a lot a ground (languages, layers): +where you are determines what logging you have available. + +Here are some common situtations for logging: + +#### JS to C++ + +If your logging is aimed at the user using the JS API +(e.g. because the type provided isn't convertable to the necessary C++ type) +then use the Browser Console via +[FOG's Common's `LogToBrowserConsole`](https://searchfox.org/mozilla-central/rev/d107bc8aeadcc816ba85cb21c1a6a1aac1d4ef9f/toolkit/components/glean/bindings/private/Common.cpp#19). + +#### C++ + +If you are in C++ and didn't come from JS, use `MOZ_LOG` with module `fog`. + +#### Rust + +Use the logging macros from `log`, e.g. `log::info!` or `log::error!`. +Remember that, no matter the log level, `log::debug!` and `log::trace!` +[will not appear in non-debug builds](/testing-rust-code/index.md#gecko-logging) + +If you are logging due to a situation caused by and fixable by a developer using the API, +use `log::error!(...)`. Otherwise, use a quieter level. + +## `about:glean` + +`about:glean` is a page in a running Firefox that allows you to +[debug the Glean SDK](https://mozilla.github.io/glean/book/user/debugging/index.html) +in Firefox Desktop. +It does this through the displayed user interface (just follow the instructions). + +## Linting + +To keep in accordance with Mozilla's various and several Coding Styles, +we rely on `mach lint`. + +To lint the code in the "usual" way, automatically fixing where possible, run: +`./mach lint -W -w -o --fix` + +This should keep you from checking in code that will automatically be backed out. + +## Rust + +Not all of our Rust code can be tested in a single fashion, unfortunately. + +### Using `rusttests` (Treeherder symbol `Br` (a build task)) + +If the crate you're testing has no Gecko symbols you can write standard +[Rust tests](https://doc.rust-lang.org/book/ch11-01-writing-tests.html). + +This supports both unit tests +(inline in the file under test) and integration tests +(in the `tests/` folder in the crate root). +Metric type tests are currently written as unit tests inline in the file, +as they require access to the metric ID, which should only be exposed in tests. + +To run FOG's `rusttests` suite use `mach rusttests` + +If the crate uses only a few Gecko symbols, they may use the +`with_gecko` feature to conditionally use them. +This allows the crate to test its non-Gecko-adjacent code using Rust tests. +(You will need to cover the Gecko-adjacent code via another means.) + +**Note:** Some FOG rusttests panic on purpose. They print stack traces to stdout. +If the rusttests fail and you see a stack trace, +double-check it isn't from a purposefully-panicking test. + +**Note:** If a test fails, it is very likely they'll poison the test lock. +This will cause all subsequent tests that attempt to take the test lock +(which is all of them) +to also fail due to `PoisonError`s. They can be safely ignored. + +### Using `gtest` (Treeherder symbol `GTest` (a build task)) + +Because Gecko symbols aren't built for the +`rusttests` build, +any test that is written for code that uses Gecko symbols should be written as a +[`gtest`](https://github.com/google/googletest) +in `toolkit/components/glean/tests/gtest/`. +You can write the actual test code in Rust. +It needs to be accompanied by a C++ GTest that calls a C FFI-exported Rust function. +See [Testing & Debugging Rust Code](/testing-rust-code/index.md) for more. +See [`toolkit/components/glean/tests/gtest/TestFog.cpp`](https://searchfox.org/mozilla-central/source/toolkit/components/glean/tests/gtest/TestFog.cpp) +and [`toolkit/components/glean/tests/gtest/test.rs`](https://searchfox.org/mozilla-central/source/toolkit/components/glean/tests/gtest/test.rs) +for an example. + +By necessity these can only be integration tests against the compiled crate. + +**Note:** When adding a new test file, don't forget to add it to +`toolkit/components/glean/tests/gtest/moz.build` and use the +`FOG` prefix in your test names +(e.g. `TEST(FOG, YourTestName) { ... }`). + +To run FOG's Rust `gtest` suite use `mach gtest FOG.*` + +## Python (Treeherder symbol `py3(fp)` aka `source-test-python-fog`) + +The [Glean Parser](https://github.com/mozilla/glean_parser/) +has been augmented to generate FOG-specific APIs for Glean metrics. +This augmentation is tested by running: + +`mach test toolkit/components/glean/tests/pytest` + +These tests require Python 3+. +If your default Python is Python 2, you may need to instead run: + +`python3 mach python-test toolkit/components/glean/tests/pytest` + +These tests check the code generator output against known good file contents. +If you change the code generator the files will need an update. +Run the test suite with the `UPDATE_EXPECT` environment variable set to do that automatically: + +`UPDATE_EXPECT=1 mach test toolkit/components/glean/tests/pytest` + +## C++ (Treeherder symbol `GTest` (a build task)) + +To test the C++ parts of FOG's implementation +(like metric types) +you should use `gtest`. +FOG's `gtest` tests are in +[`gtest/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/gtest/). + +You can either add a test case to an existing file or add a new file. +If you add a new file, remember to add it to the +[`moz.build`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/gtest/moz.build)) +or the test runner won't be able to find it. + +All tests should start with `FOG` so that all tests are run with +`./mach gtest FOG*`. + +## JS (Treeherder symbol `X(Xn)` for some number `n`) + +To test the JS parts of FOG's implementation +(like metric types) +you should use `xpcshell`. +FOG's `xpcshell` tests are in +[`xpcshell/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/xpcshell). + +You can either add a test case to an existing file or add a new file. +If you add a new file, remember to add it to the +[`xpcshell.ini`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/xpcshell/xpcshell.ini) +or the test runner will not be able to find it. + +To run FOG's JS tests, run: +`./mach test toolkit/components/glean/tests/xpcshell` + +## Non-content-process multiprocess (Browser Chrome Mochitests with Treeherder symbol `M(bcN)` for some number `N`) + +To test e.g. the GPU process support you need a full Firefox browser: +xpcshell doesn't have the flexibility. +To test that and have access to privileged JS (i.e. `Glean` and `FOG` APIs), +we use browser-chrome-flavoured mochitests you can find in +[`browser/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/browser). + +If you need to add a new test file, remember to add it to the +[`browser.ini`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/tests/browser/browser.ini) +manifest, or the test runner will not be able to find it. + +To run FOG's browser chrome tests, run: +`./mach test toolkit/components/glean/tests/browser` + +## Integration (Marionette, borrowing `telemetry-tests-client` Treeherder symbol `tt(c)`) + +To test pings (See [bug 1681742](https://bugzilla.mozilla.org/show_bug.cgi?id=1681742)) +or anything that requires one or more full browsers running, +we use the `telemetry-tests-client` suite in +[`toolkit/components/telemetry/tests/marionette/tests/client/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/tests/marionette/tests/client/). + +For more information on this suite, look to +[Firefox Telemetry's Test Documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/internals/tests.html#integration-tests-telemetry-tests-client-and-telemetry-integration-tests) +and +[Marionette's Documentation](/testing/marionette/Testing.md). + +To run these integration tests, run: +`./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/` + +To capture the Firefox under test's logs, use the `--gecko-log` parameter. +For example, to echo to stdout: +`./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/test_fog* --gecko-log "-"` + +**Note:** Running the `tt(c)` suite in this way ignored skip directives in the manifest. +This means that you might run tests that are not expected to succeed on your platform. +Check `toolkit/components/telemetry/tests/marionette/tests/client/manifest.ini` for details. diff --git a/toolkit/components/glean/docs/dev/updating_parser.md b/toolkit/components/glean/docs/dev/updating_parser.md new file mode 100644 index 0000000000..4609f76a80 --- /dev/null +++ b/toolkit/components/glean/docs/dev/updating_parser.md @@ -0,0 +1,52 @@ +# Updating glean_parser + +Project FOG uses the `glean_parser` to generate code from metric definitions. +It depends on [glean-parser] from pypi.org + +[glean-parser]: https://pypi.org/project/glean-parser/ + +To update the in-tree glean-parser change the version in `third_party/python/requirements.in`, +then run + +``` +./mach vendor python +``` + +```{note} +**Important**: the glean_parser and all of its dependencies must support Python 3.5, as discussed here. +This is the minimum version supported by mach and installed on the CI images for running tests. +This is enforced by the version ranges declared in the Python installation manifest. +``` + +## Version mismatch of the Python dependencies + +The logic for handling version mismatches is very similar to the one for the Rust crates. +See [Updating the Glean SDK](updating_sdk.md) for details. +However, updating Python packages also requires to think about Python 3.5 (and Python 2, still) compatibility. + +## Keeping versions in sync + +The Glean SDK and `glean_parser` are currently released as separate projects. +However each Glean SDK release requires a specific `glean_parser` version. +When updating one or the other ensure versions stay compatible. +You can find the currently used `glean_parser` version in the Glean SDK source tree, e.g. in [sdk_generator.sh]. + +[sdk_generator.sh]: https://github.com/mozilla/glean/blob/main/glean-core/ios/sdk_generator.sh#L28 + +## Using a local `glean_parser` development version + +To test out a new `glean_parser` in mozilla-central follow these steps: + +1. Remove `glean_parser` from the user-wide virtual environment. + This can be found in a path like `~/.mozbuild/srcdirs/gecko-f5e3b9c6ded5/_virtualenvs/mach/lib/python3.10/site-packages/glean_parser` + Note that the `gecko-f5e3b9c6ded5` part will be different depending on your local checkout. + Remove all directories and files mentioning `glean_parser` +2. Remove `glean_parser` from the build virtual enviromment. + This can be found in `$MOZ_OBJDIR/_virtualenvs/common/lib/python3.6/site-packages/glean_parser`. + Note that `$MOZ_OBJDIR` depends on your local mozconfig configuration. + Remove all directories and files mentioning `glean_parser` +3. Copy the local `glean_parser` checkout into `third_party/python/glean_parser`. + E.g. `cp ~/code/glean_parser $GECKO/third_party/python/glean_parser`. + +You should now be able to build `mozilla-central` and it will use the modified `glean_parser`. +You can make further edits in `$GECKO/third_party/python/glean_parser`. diff --git a/toolkit/components/glean/docs/dev/updating_sdk.md b/toolkit/components/glean/docs/dev/updating_sdk.md new file mode 100644 index 0000000000..fafec08d39 --- /dev/null +++ b/toolkit/components/glean/docs/dev/updating_sdk.md @@ -0,0 +1,48 @@ +# Updating the Glean SDK + +Project FOG uses the Glean SDK published as the [`glean`][glean-crate] +and [`glean-core`][glean-core] crates on crates.io. + +[glean-crate]: https://crates.io/crates/glean +[glean-core]: https://crates.io/crates/glean-core + +These two crates' versions are included in several places in mozilla-central. +To update them all, you should use the command +`mach update-glean <version, like "54.1.0">`. + +This is a semi-manual process. +Please pay attention to the output of `mach update-glean` for instructions, +and follow them closely. + +## Version mismatches of Rust dependencies + +Other crates that are already vendored might require a different version of the same dependencies that the Glean SDK requires. +The general strategy for Rust dependencies is to keep one single version of the dependency in-tree +(see [comment #8 in bug 1591555](https://bugzilla.mozilla.org/show_bug.cgi?id=1591555#c8)). +This might be hard to do in reality since some dependencies might require tweaks in order to work. +The following strategy can be followed to decide on version mismatches: + +* If the versions only **differ by the patch version**, Cargo will keep the vendored version, + unless some other dependency pinned specific patch versions; + assuming it doesn’t break the Glean SDK; + if it does, follow the next steps; +* If the version of the **vendored dependency is newer** (greater major or minor version) than the version required by the Glean SDK, + [file a bug in the Glean SDK component][glean-bug] to get Glean to require the same version; + * You will have to abandon updating the Glean SDK to this version. + You will have to wait for Glean SDK to update its dependency and for a new Glean SDK release. + Then you will have to update to that new Glean SDK version. +* If the version of the **vendored dependency is older** (lower major or minor version), consider updating the vendored version to the newer one; + seek review from the person who vendored that dependency in the first place; + if that is not possible or breaks mozilla-central build, then consider keeping both versions vendored in-tree; please note that this option will probably only be approved for small crates, + and will require updating the `TOLERATED_DUPES` list in `mach vendor` + (instructions are provided as you go). + +## Keeping versions in sync + +The Glean SDK and `glean_parser` are currently released as separate projects. +However each Glean SDK release requires a specific `glean_parser` version. +When updating one or the other ensure versions stay compatible. +You can find the currently used `glean_parser` version in the Glean SDK source tree, e.g. in [sdk_generator.sh]. + +[sdk_generator.sh]: https://github.com/mozilla/glean/blob/main/glean-core/ios/sdk_generator.sh#L28 +[glean-bug]: https://bugzilla.mozilla.org/enter_bug.cgi?product=Data+Platform+and+Tools&component=Glean%3A+SDK&priority=P3&status_whiteboard=%5Btelemetry%3Aglean-rs%3Am%3F%5D |
