summaryrefslogtreecommitdiffstats
path: root/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--mobile/android/docs/geckoview/design/sharing-rust-libraries.rst279
1 files changed, 279 insertions, 0 deletions
diff --git a/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst b/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst
new file mode 100644
index 0000000000..1aa6657b1f
--- /dev/null
+++ b/mobile/android/docs/geckoview/design/sharing-rust-libraries.rst
@@ -0,0 +1,279 @@
+Sharing rust libraries across the Firefox (for Android) stack
+=============================================================
+
+`Agi Sferro <agi@sferro.dev>`
+
+March 20th, 2021
+
+The problem
+-----------
+
+We don’t have a good story for integrating a rust library so that it’s
+available to use in Gecko, GeckoView, AC and Fenix and also in a way that rust
+can call rust directly avoiding a C FFI layer.
+
+Goals
+-----
+
+- Being able to integrate a rust library that can be called from Gecko,
+ GeckoView, AC, Fenix, including having singleton-like instances that are
+ shared across the stack, per-process.
+- The rust library should be able to call and be called by other rust libraries
+ or rust code in Gecko directly (i.e. without a C FFI layer)
+- A build-time assurance that all components in the stack compile against the
+ same version of the rust library
+- Painless, quick and automated updates. Should be able to produce chemspill
+ updates for the rust library in under 24h with little manual intervention
+ (besides security checks / code review / QA).
+- Support for non-Gecko consumers of the rust library is essential. I.e.
+ providing a version of Gecko that does not include any of the libraries
+- (optional?) Provide an easy way to create bundles of rust libraries depending
+ on consumers needs.
+
+Proposal
+--------
+
+1. Rename libmegazord.so to librustcomponents.so to clarify what the purpose of
+ this artifact is.
+2. Every rust library that wants to be called or wants to call rust code
+ directly will be included in libxul.so (which contains most of Gecko native
+ code), and vendored in mozilla-central. This includes, among others, Glean and
+ Nimbus.
+3. libxul.so will expose the necessary FFI symbols for the Kotlin wrappers
+ needed by the libraries vendored in mozilla-central in step (2).
+4. At every nightly / beta / release build of Gecko, we will generate an (or
+ possibly many) additional librustcomponents.so artifacts that will be published
+ as an AAR in maven.mozilla.org. This will also publish all the vendored
+ libraries in mozilla-central to maven, which will have a dependency on the
+ librustcomponents.so produced as part of this step. Doing this will ensure that
+ both libxul.so and librustcomponents.so contain the exact same code and can be
+ swapped freely in the dependency graph.
+5. Provide a new GeckoView build with artifactId geckoview-omni which will
+ depend on all the rust libraries. The existing geckoview will not have such
+ dependency and will be kept for third-party users of GeckoView.
+6. GeckoView will depend on the Kotlin wrappers of all the libraries that
+ depend on librustcomponents.so built in step (4) in the .pom file. For example
+
+ .. code:: xml
+
+ <dependency>
+ <groupId>org.mozilla.telemetry</groupId>
+ <artifactId>glean</artifactId>
+ <version>33.1.2</version>
+ <scope>compile</scope>
+ </dependency>
+
+ It will also exclude the org.mozilla.telemetry.glean dependency to
+ librustcomponents.so, as the native code is now included in libxul.so as part
+ of step (2). Presumably Glean will discover where its native code lives by
+ either trying librustcomponents.so or libxul.so (or some other better methods,
+ suggestions welcome).
+
+7. Android Components and Fenix will remove their explicit dependency on Glean,
+ Nimbus and all other libraries provided by GeckoView, and instead consume the
+ one provided by GeckoView (this step is optional, note that any version
+ conflict would cause a build error).
+
+
+The good
+--------
+
+- We get automated integration with AC for free. When an update for a library
+ is pushed to mozilla-central, a new nightly build for GeckoView will be
+ produced which is already consumed by AC automatically (and indirectly into
+ Fenix).
+- Publishing infrastructure to maven is already figured out, and we can reuse
+ the existing process for GeckoView to publish all the dependencies.
+- If a consumer (say AC) uses a mismatched version for a dependency, a
+ compile-time error will be thrown.
+- All consumers of the rust libraries packaged this way are on the same version
+ (provided they stay up to date with releases)
+- Non-Mozilla consumers get first-class visibility into what is packaged into
+ GeckoView, and can independently discover Glean, Nimbus, etc, since we define
+ our dependencies in the pom file.
+- Gecko Desktop and Gecko Mobile consumer Glean and other libraries in the same
+ way, removing unnecessary deviation.
+
+Worth Noting
+------------
+
+- Uplifts to beta / release versions of Fenix will involve more checks as they
+ impact Gecko too.
+
+The Bad
+-------
+
+- Libraries need to be vendored in mozilla-central. Dependencies will follow
+ the Gecko train which might not be right for them, as some dependencies don’t
+ really have a nightly vs stable version. - This could change in the future, as
+ the integration gets deeper and updates to the library become more frequent /
+ at every commit.
+- Locally testing a change in a rust library involves rebuilding all of Gecko.
+ This is a side effect of statically linking rust libraries to Gecko.
+- All rust libraries that are both used by Android and Gecko will need to be
+ updated together, and we cannot have separate versions on Desktop/Mobile.
+ Although this can be mitigated by providing flexible dependency on the library
+ side (e.g. nimbus doesn’t need to depend on a specific version of - Glean and
+ can accept whatever is in Gecko)
+- Code that doesn’t natively live in mozilla-central has double the work to get
+ into a product - first a release process is needed from the native repo, then
+ a phabricator process for the vendoring.
+
+Alternatives Considered
+-----------------------
+
+Telemetry delegate
+^^^^^^^^^^^^^^^^^^
+
+GeckoView provides a Java Telemetry delegate interface that Glean can implement
+on the AC layer to provide Glean functionality to consumers. Glean would offer
+a rust wrapper to the Java delegate API to transparently call either the
+delegate (when built for mobile) or the Glean instance directly (when built for
+Desktop).
+
+Drawbacks
+"""""""""
+
+- This involves a lot of work on the Glean side to build and maintain the
+ delegate
+- A large section of the Glean API is embedded in the GeckoView API without a
+ direct dependency
+- We don’t expect the telemetry delegate to have other implementations other
+ than Glean itself, despite the apparent generic nature of the telemetry
+ delegate
+- Glean and GeckoView engineers need to coordinate for every API update, as an
+ update to the Glean API likely triggers an update to the GV API.
+- Gecko Desktop and Gecko Mobile use Glean a meaningfully different way
+- Doesn’t solve the dependency problem: even though in theory this would allow
+ Gecko to work with multiple Glean versions, in practice the GV Telemetry
+ delegate is going to track Glean so closely that it will inevitably require
+ pretty specific Glean versions to work.
+
+Advantages
+""""""""""
+
+- Explicit code dependency, an uninformed observer can understand how telemetry
+ is extracted from GeckoView by just looking at the API
+- No hard Glean version requirement, AC can be (in theory) built with a
+ different Glean version than Gecko and things would still work
+
+Why we decided against
+""""""""""""""""""""""
+
+The amount of ongoing maintenance work involved on the Glean side far outweighs
+the small advantages, namely to not tie AC to a specific Glean version.
+Significantly complicates the stack.
+
+Dynamic Discovery
+^^^^^^^^^^^^^^^^^
+
+Gecko discovers when it’s being loaded as part of Fenix (or some other
+Gecko-powered browser) by calling dlsym on the Glean library. When the
+discovery is successful, and the Glean version matches, Gecko will directly use
+the Glean provided by Fenix.
+
+Drawbacks
+"""""""""
+
+- Non standard, non-Mozilla apps will not expect this to work the way it does
+- “Magic”: there’s no way to know that the dyscovery is happening (or what
+ version of Glean is provided with Gecko) unless you know it’s there.
+- The standard failure mode is at runtime, as there’s no built-in way to check
+ that the version provided by Gecko is the same as the one provided by Fenix
+ at build time.
+- Doesn’t solve the synchronization problem: Gecko and Fenix will have to be on
+ the same Glean version for this to work.
+- Gecko Mobile deviates meaningfully from Desktop in the way it uses Glean for
+ no intrinsic reason
+
+Advantages
+""""""""""
+
+- This system is transparent to Consuming apps, e.g. Nimbus can use Glean as
+ is, with no significant modifications needed.
+
+Why we decided against
+""""""""""""""""""""""
+
+- This alternative does not provide substantial benefits over the proposal
+ outlined in this doc and has significant drawbacks like the runtime failure
+ case and the non-standard linking process.
+
+Hybrid Dynamic Discovery
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is a variation of the Dynamic Discovery where Gecko and GeckoView include
+Glean directly and consumers get Glean from Gecko dynamically (i.e. they dlsym
+libxul.so).
+
+Drawbacks
+"""""""""
+
+- Glean still needs to build a wrapper for libraries not included in Gecko
+ (like Nimbus) that want to call Glean directly.
+
+Advantages
+""""""""""
+
+- The dependency to Glean is explicit and clear from an uninformed observer
+ point of view.
+- Smaller scope, only Glean would need to be moved to mozilla-central
+
+Why we decided against
+""""""""""""""""""""""
+
+Not enough advantages over the proposal, significant ongoing maintenance work
+required from the Glean side.
+
+Open Questions
+--------------
+
+- How does iOS consume megazord today? Do they have a maven-like dependency
+ system we can use to publish the iOS megazord?
+- How do we deal with licenses in about:license? Application-services has a
+ build step that extracts rust dependencies and puts them in the pom file
+- What would be the process for coordinating a-c breaking changes?
+- Would the desire to vendor apply even if this were not Rust code?
+
+Common Questions
+----------------
+
+- **How do we make sure GV/AC/Gecko consume the same version of the native
+ libraries?** The pom dependency in GeckoView ensures that any GeckoView
+ consumers depend on the same version of a given library, this includes AC and
+ Fenix.
+- **What happens to non-Gecko consumers of megazord?** This plan is transparent
+ to a non-Gecko consumer of megazord, as they will still consume the native
+ libraries through the megazord dependency in Glean/Nimbus/etc. With the added
+ benefit that, if the consumer stays up to date with the megazord dependency,
+ they will use the same version that Gecko uses.
+- **What’s the process to publish an update to the megazord?** When a team
+ wants to publish an update to the megazord it will need to commit the update
+ to mozilla-central. A new build will be generated in the next nightly cycle,
+ producing an updated version of the megazord. My understanding is that current
+ megazord releases are stable (and don’t have beta/nightly cycles) so for
+ external consumers, consuming the nightly build could be adequate, and provide
+ the fastest turnaround on updates. For Gecko consumers the turnaround will be
+ the same to Firefox Desktop (i.e. roughly 6-8 weeks from commit to release
+ build).
+- **How do we handle security uplifts?** If you have a security release one
+ rust library you would need to request uplift to beta/release branches
+ (depending on impact) like all other Gecko changes. The process in itself can
+ be expedited and have a fast turnaround when needed (below 24h). We have been
+ using this process for all Gecko changes so I would not expect particular
+ problems with it.
+- **What about OOP cases? E.g. GeckoView as a service?** We briefly discussed
+ this in the email chain, there are ways we could make that work (e.g.
+ providing a IPC shim). The details are fuzzy but since we don’t have any
+ immediate need for such support knowing that it’s doable with a reasonable
+ amount of work is enough for now.
+- **Vendoring in mozilla-central seems excessive.** I agree. This is an
+ unfortunate requirement stemming from a few assumptions (which could be
+ challenged! We are choosing not to):
+
+ - Gecko wants to vendor whatever it consumes for rust
+ - We want rust to call rust directly (without a C FFI layer)
+ - We want adding new libraries to be a painless experience
+
+ Because of the above, vendoring in mozilla-central seems to be the best if not
+ the only way to achieve our goals.