diff options
Diffstat (limited to 'mobile/android/docs/geckoview/design/sharing-rust-libraries.rst')
| -rw-r--r-- | mobile/android/docs/geckoview/design/sharing-rust-libraries.rst | 279 |
1 files changed, 279 insertions, 0 deletions
diff --git a/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst b/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst new file mode 100644 index 0000000000..1aa6657b1f --- /dev/null +++ b/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst @@ -0,0 +1,279 @@ +Sharing rust libraries across the Firefox (for Android) stack +============================================================= + +`Agi Sferro <agi@sferro.dev>` + +March 20th, 2021 + +The problem +----------- + +We don’t have a good story for integrating a rust library so that it’s +available to use in Gecko, GeckoView, AC and Fenix and also in a way that rust +can call rust directly avoiding a C FFI layer. + +Goals +----- + +- Being able to integrate a rust library that can be called from Gecko, + GeckoView, AC, Fenix, including having singleton-like instances that are + shared across the stack, per-process. +- The rust library should be able to call and be called by other rust libraries + or rust code in Gecko directly (i.e. without a C FFI layer) +- A build-time assurance that all components in the stack compile against the + same version of the rust library +- Painless, quick and automated updates. Should be able to produce chemspill + updates for the rust library in under 24h with little manual intervention + (besides security checks / code review / QA). +- Support for non-Gecko consumers of the rust library is essential. I.e. + providing a version of Gecko that does not include any of the libraries +- (optional?) Provide an easy way to create bundles of rust libraries depending + on consumers needs. + +Proposal +-------- + +1. Rename libmegazord.so to librustcomponents.so to clarify what the purpose of + this artifact is. +2. Every rust library that wants to be called or wants to call rust code + directly will be included in libxul.so (which contains most of Gecko native + code), and vendored in mozilla-central. This includes, among others, Glean and + Nimbus. +3. libxul.so will expose the necessary FFI symbols for the Kotlin wrappers + needed by the libraries vendored in mozilla-central in step (2). +4. At every nightly / beta / release build of Gecko, we will generate an (or + possibly many) additional librustcomponents.so artifacts that will be published + as an AAR in maven.mozilla.org. This will also publish all the vendored + libraries in mozilla-central to maven, which will have a dependency on the + librustcomponents.so produced as part of this step. Doing this will ensure that + both libxul.so and librustcomponents.so contain the exact same code and can be + swapped freely in the dependency graph. +5. Provide a new GeckoView build with artifactId geckoview-omni which will + depend on all the rust libraries. The existing geckoview will not have such + dependency and will be kept for third-party users of GeckoView. +6. GeckoView will depend on the Kotlin wrappers of all the libraries that + depend on librustcomponents.so built in step (4) in the .pom file. For example + + .. code:: xml + + <dependency> + <groupId>org.mozilla.telemetry</groupId> + <artifactId>glean</artifactId> + <version>33.1.2</version> + <scope>compile</scope> + </dependency> + + It will also exclude the org.mozilla.telemetry.glean dependency to + librustcomponents.so, as the native code is now included in libxul.so as part + of step (2). Presumably Glean will discover where its native code lives by + either trying librustcomponents.so or libxul.so (or some other better methods, + suggestions welcome). + +7. Android Components and Fenix will remove their explicit dependency on Glean, + Nimbus and all other libraries provided by GeckoView, and instead consume the + one provided by GeckoView (this step is optional, note that any version + conflict would cause a build error). + + +The good +-------- + +- We get automated integration with AC for free. When an update for a library + is pushed to mozilla-central, a new nightly build for GeckoView will be + produced which is already consumed by AC automatically (and indirectly into + Fenix). +- Publishing infrastructure to maven is already figured out, and we can reuse + the existing process for GeckoView to publish all the dependencies. +- If a consumer (say AC) uses a mismatched version for a dependency, a + compile-time error will be thrown. +- All consumers of the rust libraries packaged this way are on the same version + (provided they stay up to date with releases) +- Non-Mozilla consumers get first-class visibility into what is packaged into + GeckoView, and can independently discover Glean, Nimbus, etc, since we define + our dependencies in the pom file. +- Gecko Desktop and Gecko Mobile consumer Glean and other libraries in the same + way, removing unnecessary deviation. + +Worth Noting +------------ + +- Uplifts to beta / release versions of Fenix will involve more checks as they + impact Gecko too. + +The Bad +------- + +- Libraries need to be vendored in mozilla-central. Dependencies will follow + the Gecko train which might not be right for them, as some dependencies don’t + really have a nightly vs stable version. - This could change in the future, as + the integration gets deeper and updates to the library become more frequent / + at every commit. +- Locally testing a change in a rust library involves rebuilding all of Gecko. + This is a side effect of statically linking rust libraries to Gecko. +- All rust libraries that are both used by Android and Gecko will need to be + updated together, and we cannot have separate versions on Desktop/Mobile. + Although this can be mitigated by providing flexible dependency on the library + side (e.g. nimbus doesn’t need to depend on a specific version of - Glean and + can accept whatever is in Gecko) +- Code that doesn’t natively live in mozilla-central has double the work to get + into a product - first a release process is needed from the native repo, then + a phabricator process for the vendoring. + +Alternatives Considered +----------------------- + +Telemetry delegate +^^^^^^^^^^^^^^^^^^ + +GeckoView provides a Java Telemetry delegate interface that Glean can implement +on the AC layer to provide Glean functionality to consumers. Glean would offer +a rust wrapper to the Java delegate API to transparently call either the +delegate (when built for mobile) or the Glean instance directly (when built for +Desktop). + +Drawbacks +""""""""" + +- This involves a lot of work on the Glean side to build and maintain the + delegate +- A large section of the Glean API is embedded in the GeckoView API without a + direct dependency +- We don’t expect the telemetry delegate to have other implementations other + than Glean itself, despite the apparent generic nature of the telemetry + delegate +- Glean and GeckoView engineers need to coordinate for every API update, as an + update to the Glean API likely triggers an update to the GV API. +- Gecko Desktop and Gecko Mobile use Glean a meaningfully different way +- Doesn’t solve the dependency problem: even though in theory this would allow + Gecko to work with multiple Glean versions, in practice the GV Telemetry + delegate is going to track Glean so closely that it will inevitably require + pretty specific Glean versions to work. + +Advantages +"""""""""" + +- Explicit code dependency, an uninformed observer can understand how telemetry + is extracted from GeckoView by just looking at the API +- No hard Glean version requirement, AC can be (in theory) built with a + different Glean version than Gecko and things would still work + +Why we decided against +"""""""""""""""""""""" + +The amount of ongoing maintenance work involved on the Glean side far outweighs +the small advantages, namely to not tie AC to a specific Glean version. +Significantly complicates the stack. + +Dynamic Discovery +^^^^^^^^^^^^^^^^^ + +Gecko discovers when it’s being loaded as part of Fenix (or some other +Gecko-powered browser) by calling dlsym on the Glean library. When the +discovery is successful, and the Glean version matches, Gecko will directly use +the Glean provided by Fenix. + +Drawbacks +""""""""" + +- Non standard, non-Mozilla apps will not expect this to work the way it does +- “Magic”: there’s no way to know that the dyscovery is happening (or what + version of Glean is provided with Gecko) unless you know it’s there. +- The standard failure mode is at runtime, as there’s no built-in way to check + that the version provided by Gecko is the same as the one provided by Fenix + at build time. +- Doesn’t solve the synchronization problem: Gecko and Fenix will have to be on + the same Glean version for this to work. +- Gecko Mobile deviates meaningfully from Desktop in the way it uses Glean for + no intrinsic reason + +Advantages +"""""""""" + +- This system is transparent to Consuming apps, e.g. Nimbus can use Glean as + is, with no significant modifications needed. + +Why we decided against +"""""""""""""""""""""" + +- This alternative does not provide substantial benefits over the proposal + outlined in this doc and has significant drawbacks like the runtime failure + case and the non-standard linking process. + +Hybrid Dynamic Discovery +^^^^^^^^^^^^^^^^^^^^^^^^ + +This is a variation of the Dynamic Discovery where Gecko and GeckoView include +Glean directly and consumers get Glean from Gecko dynamically (i.e. they dlsym +libxul.so). + +Drawbacks +""""""""" + +- Glean still needs to build a wrapper for libraries not included in Gecko + (like Nimbus) that want to call Glean directly. + +Advantages +"""""""""" + +- The dependency to Glean is explicit and clear from an uninformed observer + point of view. +- Smaller scope, only Glean would need to be moved to mozilla-central + +Why we decided against +"""""""""""""""""""""" + +Not enough advantages over the proposal, significant ongoing maintenance work +required from the Glean side. + +Open Questions +-------------- + +- How does iOS consume megazord today? Do they have a maven-like dependency + system we can use to publish the iOS megazord? +- How do we deal with licenses in about:license? Application-services has a + build step that extracts rust dependencies and puts them in the pom file +- What would be the process for coordinating a-c breaking changes? +- Would the desire to vendor apply even if this were not Rust code? + +Common Questions +---------------- + +- **How do we make sure GV/AC/Gecko consume the same version of the native + libraries?** The pom dependency in GeckoView ensures that any GeckoView + consumers depend on the same version of a given library, this includes AC and + Fenix. +- **What happens to non-Gecko consumers of megazord?** This plan is transparent + to a non-Gecko consumer of megazord, as they will still consume the native + libraries through the megazord dependency in Glean/Nimbus/etc. With the added + benefit that, if the consumer stays up to date with the megazord dependency, + they will use the same version that Gecko uses. +- **What’s the process to publish an update to the megazord?** When a team + wants to publish an update to the megazord it will need to commit the update + to mozilla-central. A new build will be generated in the next nightly cycle, + producing an updated version of the megazord. My understanding is that current + megazord releases are stable (and don’t have beta/nightly cycles) so for + external consumers, consuming the nightly build could be adequate, and provide + the fastest turnaround on updates. For Gecko consumers the turnaround will be + the same to Firefox Desktop (i.e. roughly 6-8 weeks from commit to release + build). +- **How do we handle security uplifts?** If you have a security release one + rust library you would need to request uplift to beta/release branches + (depending on impact) like all other Gecko changes. The process in itself can + be expedited and have a fast turnaround when needed (below 24h). We have been + using this process for all Gecko changes so I would not expect particular + problems with it. +- **What about OOP cases? E.g. GeckoView as a service?** We briefly discussed + this in the email chain, there are ways we could make that work (e.g. + providing a IPC shim). The details are fuzzy but since we don’t have any + immediate need for such support knowing that it’s doable with a reasonable + amount of work is enough for now. +- **Vendoring in mozilla-central seems excessive.** I agree. This is an + unfortunate requirement stemming from a few assumptions (which could be + challenged! We are choosing not to): + + - Gecko wants to vendor whatever it consumes for rust + - We want rust to call rust directly (without a C FFI layer) + - We want adding new libraries to be a painless experience + + Because of the above, vendoring in mozilla-central seems to be the best if not + the only way to achieve our goals. |