diff options
Diffstat (limited to 'toolkit/components/search/docs')
| -rw-r--r-- | toolkit/components/search/docs/DefaultSearchEngines.rst | 102 | ||||
| -rw-r--r-- | toolkit/components/search/docs/Preferences.rst | 46 | ||||
| -rw-r--r-- | toolkit/components/search/docs/SearchConfigurationSchema.rst | 586 | ||||
| -rw-r--r-- | toolkit/components/search/docs/SearchEngineConfiguration.rst | 72 | ||||
| -rw-r--r-- | toolkit/components/search/docs/SearchEngines.rst | 197 | ||||
| -rw-r--r-- | toolkit/components/search/docs/SearchService.rst | 6 | ||||
| -rw-r--r-- | toolkit/components/search/docs/SearchServiceHighlevelOverview.rst | 85 | ||||
| -rw-r--r-- | toolkit/components/search/docs/Telemetry.rst | 94 | ||||
| -rw-r--r-- | toolkit/components/search/docs/assets/bugzilla-open-search1.png | bin | 0 -> 179185 bytes | |||
| -rw-r--r-- | toolkit/components/search/docs/assets/bugzilla-open-search2.png | bin | 0 -> 211617 bytes | |||
| -rw-r--r-- | toolkit/components/search/docs/assets/search-service-diagram.png | bin | 0 -> 134442 bytes | |||
| -rw-r--r-- | toolkit/components/search/docs/index.rst | 40 | ||||
| -rw-r--r-- | toolkit/components/search/docs/search-add-on-prompts-flow.png | bin | 0 -> 79061 bytes |
13 files changed, 1228 insertions, 0 deletions
diff --git a/toolkit/components/search/docs/DefaultSearchEngines.rst b/toolkit/components/search/docs/DefaultSearchEngines.rst new file mode 100644 index 0000000000..3dfe68abb1 --- /dev/null +++ b/toolkit/components/search/docs/DefaultSearchEngines.rst @@ -0,0 +1,102 @@ +====================== +Default Search Engines +====================== + +Default Engine +============== + +The search service specifies default search engines via the `configuration +schema`_. + +Changing Defaults +================= + +The default engine may change when: + +* The user has the default engine set and the configuration for the locale/region + changes. +* The user has the default engine set and their locale/region changes to one + which has a different default. +* The user chooses to set a different engine via preferences. +* The user installs an add-on which sets its default as one of the application + provided engines. +* The user installs an add-on which supplies a different engine and the user allows + the different engine to be set as default. +* The user or Firefox (e.g. via blocklist) causes the default engine to be removed. + +Add-ons and Prompting for Default +--------------------------------- + +The prompt for selecting a search engine from an add-on as default is shown to +the user on installation of the add-on. It may also be shown if an add-on is +re-enabled, if the default engine was not changed in the meantime. + +The following diagram shows the full flow for search engines from add-ons: + +.. image:: ./search-add-on-prompts-flow.png + :align: center + :alt: Flowchart for prompting for default engine for Search Engines related to add-ons. + +When the Default Engine is Removed +================================== + +If the default engine is removed by the user, or by Firefox in the case of a +blocklist or for some other region, the new default engine is chosen by the +following process. + +* If the default engine specified by the configuration for the user's region and locale + is visible, then it will be selected as default. +* If there is another engine visible, fall back to the first engine identified + as a general search engine (see below). +* If there are no other visible engines, unhide the region/locale default engine + from the configuration and set it as default if it is not the one being removed. +* Otherwise, unhide the first general search engine, or the first visible engine. + +A general search engine is defined as one that returns general search results, +for example Google or DuckDuckGo. A non-general search engine returns results +for a specific area, e.g. shopping, books, dictionaries. + +Add-ons and App-provided Engines +================================ + +An add-on may set the name of the search provider in the manifest.json to be +the name of an app-provided engine. In this case: + +* If the add-on is a non-authorised partner, then we set the user's default + engine to be the name of the app-provided engine. +* If the add-on is from an authorised partner, then we set the users' default + engine to be the same as the app-provided engine, and we allow the + app-provided urls to be overridden with those of the add-on. + +If the specified engine is already default, then the add-on does +not override the app-provided engine, and it's settings are ignored and no +new engine is added. + +The list of authorised add-ons is stored in `remote settings`_ in the +`search-default-override-allowlist bucket`_. The list +includes records containing: + +* Third-party Add-on Id: The identifier of the third party add-on which will + override the app provided one. +* Add-on Id to Override: The identifier of the app-provided add-on to be + overridden. +* a list of the url / params that are authorised to be replaced. + +When an authorised add-on overrides the default, we record the add-on's id +with the app-provided engine in the ``overriddenBy`` field. This is used +when the engine is loaded on startup to known that it should load the parameters +from that add-on. + +The ``overriddenBy`` annotation may be removed when: + +* The associated authorised add-on is removed, disabled or can no longer be found. +* The user changes their default to another engine. + +If the ``overriddenBy`` annotation is present, but the add-on is not authorised, +then the annotation will be maintained in case the add-on is later re-authorised. +For example, a url is updated, but the update is performed before the allow list +is updated. + +.. _configuration schema: SearchConfigurationSchema.html +.. _remote settings: /services/settings/index.html +.. _search-default-override-allowlist bucket: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-default-override-allowlist/records diff --git a/toolkit/components/search/docs/Preferences.rst b/toolkit/components/search/docs/Preferences.rst new file mode 100644 index 0000000000..2fdf4699cb --- /dev/null +++ b/toolkit/components/search/docs/Preferences.rst @@ -0,0 +1,46 @@ +Preferences +=========== + +This document describes Preferences affecting the toolkit Search Service. +Preferences that are generated and updated by code won't be described here. + +User Exposed +------------ +These preferences are exposed through the Firefox UI. + +browser.search.suggest.enabled (default: true) + Whether search suggestions are enabled in the search bar. Turning this off + also currently prevents search suggestions in the address bar. + +Hidden +------ +These preferences are normally hidden, and should not be used unless you really +know what you are doing. + +browser.search.log (boolean, default: false) + Whether search service console logging is enabled or not. + +browser.search.suggest.timeout (number, default: 500) + When requesting suggestions from a search server, it has this long to respond. + +browser.search.update (boolean, default: true) + Whether updates are enabled for OpenSearch based engines. This only applies + to OpenSearch engines where their definition supplies an update url. + Note that this does not affect any of: application updates, add-on updates, + remote settings updates. + +Experimental +------------ +These preferences are experimental and not officially supported. They could be +removed at any time. + +browser.search.separatePrivateDefault.ui.enabled (boolean, default: false) + Whether the UI is enabled for having a separate default search engine in + private browsing mode. + +browser.search.separatePrivateDefault (boolean, default: false) + Whether the user has selected to have a separate default search engine in + private browsing mode. + +browser.search.suggest.enabled.private (boolean, default: false) + Whether search suggestions are enabled in a private browsing window. diff --git a/toolkit/components/search/docs/SearchConfigurationSchema.rst b/toolkit/components/search/docs/SearchConfigurationSchema.rst new file mode 100644 index 0000000000..b57f69fe13 --- /dev/null +++ b/toolkit/components/search/docs/SearchConfigurationSchema.rst @@ -0,0 +1,586 @@ +=========================== +Search Configuration Schema +=========================== + +This document outlines the details of the schema and how the various sub-parts +interact. For the full fields and descriptions, please see the `schema itself`_. + +.. note:: + In the examples, only relevant properties are displayed. + +Overview +======== + +The configuration is a JSON blob which is object with a `data` property which +is an array of engines: + +.. code-block:: js + + { + data: [ + { + // engine 1 details + }, + { + // engine 2 details + } + ] + } + +Engine Objects +============== + +An engine's details are located in the properties of the object associated with it. +An engine that is deployed globally could be listed simply as: + +.. code-block:: js + + { + "default": "no", + "telemetryId": "engine1-telem", + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + } + }] + } + +The ``appliesTo`` section is an array of objects. At least one object is required +to specify which regions/locales the engine is included within. If an +``appliesTo`` object lists additional attributes then these will override any +attributes at the top-level. + +For example, a more complex engine definition may be available only to users +located specific regions or with certain locales. For example: + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "region": "us" + }, + "webExtension": { + "id": "web-us@ext" + } + }, { + "included": { + "region": "gb" + }, + "webExtension": { + "id": "web-gb@ext" + } + }] + } + +In this case users identified as being in the US region would use the WebExtension +with identifier ``web-us@ext``. GB region users would get +``web-gb@ext``, and all other users would get ``web@ext``. + +To direct search engines to pull ``_locale`` data from a specific locale +directory, you can use ``webExtension.locales``. + +For example, in this code block: + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [ + { + "included": { + "locales": "en-US" + }, + "webExtension": { + "locales": [ + "us" + ] + } + }, { + "included": { + "locales": "en-GB" + }, + "webExtension": { + "locales": [ + "uk" + ] + } + } + ] + } + +There should exist a ``us`` and ``uk`` folder in the ``locales`` directory +of the extension, ``web``. + +If a locale is not provided, ``webExtension.locales`` is set to +``SearchUtils.DEFAULT_TAG``. + +`Search Extensions directory <https://searchfox.org/mozilla-central/source/browser/components/search/extensions>`__ + +`Example of a locales directory <https://searchfox.org/mozilla-central/source/browser/components/search/extensions/wikipedia/_locales>`__ + + +Special Attributes +================== + +$USER_LOCALE +------------ + +If a ``webExtension.locales`` property contains an element with the value +``"$USER_LOCALE"`` then the special value will be replaced in the +configuration object with the users locale. For example: + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "locales": { + "matches": [ + "en-US", + "en-GB" + ] + } + }, + "webExtension": { + "locales": ["$USER_LOCALE"] + } + }] + } + +Will report either ``[en-US]`` or ``[en-GB]`` as the ``webExtension.locales`` +property depending on the user's locale. + +Since the special string is replaced, custom folder names can be searched for +by adding the keyword in between a consistent prefix/suffix. + +For example, if ``webExtension.locales`` was ``["example-$USER_LOCALE"]``, +the locale generator will generate locale names in the form of ``example-en-US`` +and ``example-en-GB``. + +Note: Prior to Firefox 100.0, $USER_LOCALE used an exact match. +In Firefox 100.0 the replacement was updated to use a standard string replacement. + +From Firefox 98.0.1 and 97.7.1esr, ``"$USER_LOCALE"`` may also be used in the +``telemetryId`` field. + +$USER_REGION +------------ + +This can be used in the same situations as ``"$USER_LOCALE"``, instead +replacing ``webExtension.locale`` with a string that uses the users region. + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + }, + "webExtension": { + "locales": ["foo-$USER_REGION"] + } + }] + } + +In this example, if the user's region is ``fr``, the ``webExtension.locale`` +will be ``foo-fr``, and the code will look for the ``messages.json`` in +the ``foo-fr`` folder of the ``_locales`` folder for this extension. + +Note: ``"$USER_REGION"`` was added in Firefox 98.0.1 and 97.7.1esr and used an exact match. +In Firefox 100.0 the replacement was updated to use a standard string replacement. + +"default" +--------- + +You can specify ``"default"`` as a region in the configuration if +the engine is to be included when we do not know the user's region. + +"override" +---------- + +The ``"override"`` field can be set to true if you want a section to +only override otherwise included engines. ``"override"`` will only work for +sections which apply to distributions or experiments. The experiment case was +added in Firefox 81. + +Starting with Firefox 96, ``"override"`` sections may include ``included`` and +``excluded`` information which will be applied accordingly. If they are not +supplied, then the override section will be applied to everywhere. + +Example: + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + // Complicated and lengthy inclusion rules + }, { + "override": true, + "application": { "distributions": ["mydistrocode"]}, + "params": { + "searchUrlGetParams": [ + { "name": "custom", "value": "foobar" } + ] + } + }] + } + +Application Scoping +=================== + +An engine configuration may be scoped to a particular application. + +Name +---- + +One or more application names may be specified. Currently the only application +type supported is ``firefox``. If an application name is specified, then it +must be matched for the section to apply. If there are no application names +specified, then the section will match any consumer of the configuration. + +In the following example, ``web@ext`` would be included on any consumer +of the configuration, but ``web1@ext`` would only be included on Firefox desktop. + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "application": { + "name": [] + } + } + ]} + }, + { + "webExtension": { + "id": "web1@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "application": { + "name": ["firefox"] + } + } + ]} + } + +Channel +------- + +One or more channels may be specified in an array to restrict a configuration +to just those channels. The current known channels are: + + - default: Self-builds of Firefox, or possibly some self-distributed versions. + - nightly: Firefox Nightly builds. + - aurora: Firefox Developer Edition + - beta: Firefox Beta + - release: The main Firefox release channel. + - esr: The ESR Channel. This will also match versions of Firefox where the + displayed version number includes ``esr``. We do this to include Linux + distributions and other manual builds of ESR. + +In the following example, ``web@ext`` would be set as default on the default +channel only, whereas ``web1@ext`` would be set as default on release and esr +channels. + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "default": "yes", + "application": { + "channel": ["default"] + } + } + ]} + }, + { + "webExtension": { + "id": "web1@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "default": "yes", + "application": { + "channel": ["release", "esr"] + } + } + ]} + } + +Distributions +------------- + +Distributions may be specified to be included or excluded in an ``appliesTo`` +section. The ``distributions`` field in the ``application`` section is an array +of distribution identifiers. The identifiers match those supplied by the +``distribution.id`` preference. + +In the following, ``web@ext`` would be included in only the ``cake`` +distribution. ``web1@ext`` would be excluded from the ``apples`` distribution +but included in the main desktop application, and all other distributions. + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "application": { + "distributions": ["cake"] + } + } + ]} + }, + { + "webExtension": { + "id": "web1@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "application": { + "excludedDistributions": ["apples"] + } + } + ]} + } + +Version +------- + +Minimum and Maximum versions may be specified to restrict a configuration to +specific ranges. These may be open-ended. Version comparison is performed +using `the version comparator`_. + +Note: comparison against ``maxVersion`` is a less-than comparison. The +``maxVersion`` won't be matched directly. + +In the following example, ``web@ext`` would be included for any version after +72.0a1, whereas ``web1@ext`` would be included only between 68.0a1 and 71.x +version. + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "application": { + "minVersion": "72.0a1" + } + } + ]} + }, + { + "webExtension": { + "id": "web1@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + "default": "yes", + "application": { + "minVersion": "68.0a1" + "maxVersion": "72.0a1" + } + } + ]} + } + +Experiments +=========== + +We can run experiments by giving sections within ``appliesTo`` a +``experiment`` value, the Search Service can then optionally pass in a +matching ``experiment`` value to match those sections. + +Sections which have a ``experiment`` will not be used unless a matching +``experiment`` has been passed in, for example: + +.. code-block:: js + + { + "webExtension": { + "id": "web@ext" + }, + "appliesTo": [{ + "included": { + "everywhere": true + }, + "experiment": "nov-16", + "webExtension": { + "id": "web-experimental@ext" + } + }, { + "included": { + "everywhere": true + }, + "webExtension": { + "id": "web-gb@ext" + } + }] + } + +Engine Defaults +=============== + +An engine may be specified as the default for one of two purposes: + +#. normal browsing mode, +#. private browsing mode. + +If there is no engine specified for private browsing mode for a particular region/locale +pair, then the normal mode engine is used. + +If the instance of the application does not support a separate private browsing mode engine, +then it will only use the normal mode engine. + +An engine may or may not be default for particular regions/locales. The ``default`` +property is a tri-state value with states of ``yes``, ``yes-if-no-other`` and +``no``. Here's an example of how they apply: + +.. code-block:: js + + { + "webExtension": { + "id": "engine1@ext" + }, + "appliesTo": [{ + "included": { + "region": "us" + }, + "default": "yes" + }, { + "excluded": { + "region": "us" + }, + "default": "yes-if-no-other" + }] + }, + { + "webExtension": { + "id": "engine2@ext" + }, + "appliesTo": [{ + "included": { + "region": "gb" + }, + "default": "yes" + }] + }, + "webExtension": { + "id": "engine3@ext" + }, + "default": "no" + "appliesTo": [{ + "included": { + "everywhere": true + }, + }] + }, + { + "webExtension": { + "id": "engine4@ext" + }, + "defaultPrivate": "yes", + "appliesTo": [{ + "included": { + "region": "fr" + } + }] + } + +In this example, for normal mode: + + - engine1@ext is default in the US region, and all other regions except for GB + - engine2@ext is default in only the GB region + - engine3@ext and engine4 are never default anywhere + +In private browsing mode: + + - engine1@ext is default in the US region, and all other regions except for GB and FR + - engine2@ext is default in only the GB region + - engine3@ext is never default anywhere + - engine4@ext is default in the FR region. + +Engine Ordering +=============== + +The ``orderHint`` field indicates the suggested ordering of an engine relative to +other engines when displayed to the user, unless the user has customized their +ordering. + +The default ordering of engines is based on a combination of if the engine is +default, and the ``orderHint`` fields. The ordering is structured as follows: + +#. Default engine in normal mode +#. Default engine in private browsing mode (if different from the normal mode engine) +#. Other engines in order from the highest ``orderHint`` to the lowest. + +Example: + +.. code-block:: js + + { + "webExtension": { + "id": "engine1@ext" + }, + "orderHint": 2000, + "default": "no", + }, + { + "webExtension": { + "id": "engine2@ext" + }, + "orderHint": 1000, + "default": "yes" + }, + { + "webExtension": { + "id": "engine3@ext" + }, + "orderHint": 500, + "default": "no" + } + +This would result in the order: ``engine2@ext, engine1@ext, engine3@ext``. + +.. _schema itself: https://searchfox.org/mozilla-central/source/toolkit/components/search/schema/ +.. _the version comparator: https://developer.mozilla.org/en-US/docs/Mozilla/Toolkit_version_format diff --git a/toolkit/components/search/docs/SearchEngineConfiguration.rst b/toolkit/components/search/docs/SearchEngineConfiguration.rst new file mode 100644 index 0000000000..c782f9f7c3 --- /dev/null +++ b/toolkit/components/search/docs/SearchEngineConfiguration.rst @@ -0,0 +1,72 @@ +=========================== +Search Engine Configuration +=========================== + +The search engine configuration is a mapping that is used to determine the +list of search engines for each user. The mapping is primarily based on the +user's region and locale. + +Configuration Management +======================== + +The application stores a dump of the configuration that is used for first +initialisation. Subsequent updates to the configuration are either updates to the +static dump, or they may be served via remote servers. + +The mechanism of delivering the settings dumps to the Search Service is +`the remote settings`_. + +Remote settings +--------------- + +The remote settings bucket for the search engine configuration list is +``search-config``. The version that is currently being delivered +to clients can be `viewed live`_. + +Configuration Schema +==================== + +The configuration format is defined via a `JSON schema`_. The search engine +configuration schema is `stored in mozilla-central`_ and is uploaded to the +Remote Settings server at convenient times after it changes. + +An outline of the schema may be found on the `Search Configuration Schema`_ page. + +Updating Search Engine WebExtensions +==================================== + +Updates for application provided search engine WebExtensions are provided via +`Normandy`_. + +It is likely that updates for search engine WebExtensions will be +received separately to configuration updates which may or may not be directly +related. As a result several situations may occur: + + - The updated WebExtension is for an app-provided engine already in-use by + the user. + + - In this case, the search service will apply the changes to the + app-provided engine's data. + + - A WebExtension addition/update is for an app-provided engine that is not + in-use by the user, or not in the configuration. + + - In this case, the search service will ignore the WebExtension. + - If the configuration (search or user) is updated later and the + new engine is added, then the Search Service will start to use the + new engine. + + - A configuration update is received that needs a WebExtension that is + not found locally. + + - In this case, the search service will ignore the missing engine and + continue without it. + - When the WebExtension is delivered, the search engine will then be + installed and added. + +.. _the remote settings: /services/settings/index.html +.. _JSON schema: https://json-schema.org/ +.. _stored in mozilla-central: https://searchfox.org/mozilla-central/source/toolkit/components/search/schema/ +.. _Search Configuration Schema: SearchConfigurationSchema.html +.. _viewed live: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-config/records +.. _Normandy: /toolkit/components/normandy/normandy/services.html diff --git a/toolkit/components/search/docs/SearchEngines.rst b/toolkit/components/search/docs/SearchEngines.rst new file mode 100644 index 0000000000..9993abf8c6 --- /dev/null +++ b/toolkit/components/search/docs/SearchEngines.rst @@ -0,0 +1,197 @@ +============== +Search Engines +============== +This document describes the three main ways Firefox serves search engines to the +user, enabling users to search the internet with different search providers. +The three main ways Firefox serves search engines to the users is through: + +- Add-on Search Engines +- OpenSearch Engines +- Enterprise Policy Engines + +An example of a search provider is Google, which is one of the Add-on Search +Engines described in the first section below. Another example of a search +provider is Bugzilla, which is an OpenSearch Engine described in the second +section below. Lastly, there are Enterprise Policy Search Engines, +which will be the third section described in this documentation. + +Add-on Search Engines +===================== +Add-ons are additional functionality that third-party developers provide for +users to install into Firefox. The add-on mechanism is also used by Firefox to +ship the search engines provided by the application. To define Add-on Search +Engines, developers use the `WebExtensions API`_. Since the WebExtensions API +technology is used, developers interchangeably used the term WebExtension Search +Engines when referring to Add-ons Search Engines. + +.. _WebExtensions API: + https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions + +The list of Add-on Search Engines provided by Firefox and their extension files +can be found in `mozilla-central/browser/components/search/extensions +<https://searchfox.org/mozilla-central/source/browser/components/search/extensions>`__. +Within each Add-on Search Engine folder, there is a manifest.json file. One of the +keys in that manifest.json file is `chrome_settings_overrides`, whose value is an object +that describes how to construct the url, images, strings, icon, etc. Here’s an example of +how the search provider is set within `chrome_settings_overrides`: + +.. code-block:: js + + "chrome_settings_overrides": { + "search_provider": { + "name": "Discogs", + "search_url": "https://www.discogs.com/search/?q={searchTerms}", + "keyword": "disc", + "favicon_url": "https://www.discogs.com/favicon.ico" + } + } + + +To see more details on the syntax and properties, visit the `chrome settings +overrides MDN documentation. +<https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/ +manifest.json/chrome_settings_overrides>`__ + +In Practice +----------- +All versions of Firefox support add-ons. Firefox switched over from OpenSearch +to Add-on Search Engines internally in Firefox version 78. Add-on Search engines +allows Firefox developers to have more flexibility and control in the +modification of formatting search engines as we support different search +providers. + +We maintain these Add-on Search Engines through a search configuration file that +is bundled and configured via Remote Settings. As of this writing, June 2022, we +use Remote Settings for managing search engines only for Firefox Desktop but not +outside of Firefox Desktop. + +OpenSearch Engines +=================== +OpenSearch is a plugin, software installed on Firefox to enhance capabilities +for searching. OpenSearch has a collection of formats that describe how to +construct the url, images, strings, icon, etc. These formats provided by +OpenSearch allow Firefox to make a search over the internet with a specific +search provider that is not an application provided search engine on Firefox. +The purpose of OpenSearch is to provide more convenient ways of searching and +different ways of searching. + +OpenSearch allows users to search with a vast variety of search providers which +do not come installed with Firefox out of the box. The main benefit of OpenSearch +is it allows site owners to easily provide users with a way to search a site. + +History +------- +Prior to OpenSearch, search plugins were first created by the `Mycroft Project +<https://mycroftproject.com/>`__ and based off of `Sherlock +<https://en.wikipedia.org/wiki/Sherlock_(software)>`__, a file and web search +tool created by Apple. + +The OpenSearch Protocol was created and launched by A9.com in 2005. OpenSearch +was added to Firefox version 2 in the year 2006. As of today in 2022, OpenSearch +is a collection of formats for sharing of search results. The code is stable but +unchanged for many years. + +See the `OpenSearch Documentation <https://github.com/dewitt/opensearch>`__ for +more information on the OpenSearch formats. + +Autodiscovery +------------- +Autodiscovery is a feature on Firefox which automatically notifies the user when +the webpage they visited has a search plugin. + +Here is an example of Autodiscovery from Bugzilla. You can visit +https://bugzilla.mozilla.org and Firefox will automatically detect that the +website has a provided search plugin. In the results dropdown, you can look at +the search engine shortcuts section at the bottom and it will show a green plus +sign over the Bugzilla search icon. The green plus sign indicates that the user +can add Bugzilla as an OpenSearch Engine. After the user adds Bugzilla as an +OpenSearch Engine, the green plus icon disappears. The user can now click the +Bugzilla icon to make a search directly on bugzilla.mozilla.org. + +.. figure:: assets/bugzilla-open-search1.png + :alt: Image of the address bar input showing a URL + :scale: 28% + :align: center + +.. figure:: assets/bugzilla-open-search2.png + :alt: Image of the address bar input showing a URL + :scale: 28% + :align: center + +See the `Autodiscovery MDN Documentation <https://developer.mozilla.org/en-US/ +docs/Web/OpenSearch#autodiscovery_of_search_plugins>`__ for more information on +Autodiscovery. + +Enterprise Policy Engines +========================= +Enterprise Policies are customizable configurations for the Firefox browser set +by enterprises or companies who want to distribute configuration for their +users. The idea of Enterprise Policies is to allow companies to customize Firefox +and how their users can or cannot change the usage of Firefox based on predefined +configuration that was set in place. + +Enterprise Policy Engines are search engines that a company has added as search +engines on Firefox for their users by setting the Enterprise Policy. In this +`Enterprise Policy Documentation +<https://mozilla.github.io/policy-templates/#searchengines +-this-policy-is-only-available-on-the-esr>`__, +it outlines the different options that are available for enterprises using +Firefox ESR (Extended Support Release) and what’s available in terms of adding, +updating, or removing search engines. The company can use the policy to define +which search engines are available on their Firefox ESR. + +See the `policy-templates +<https://mozilla.github.io/policy-templates/>`__ for more +information on Enterprise Policy templates and the different configuration rules +available. + +Configuration +------------- +In practice, there are different ways for a company to specify their policy, +depending on which operating system their machines are on. The admin can +configure the policy on a server and when the user logs in, those configurations +are automatically pushed to the user’s Firefox. + +For Windows, the `GPO (Group Policy Object) +<https://github.com/mozilla/policy-templates/tree/master/windows>`__ or `Intune +(Microsoft Endpoint Manager) <https://support.mozilla.org/en-US/kb/managing-firefox-intune>`__ is +used to set the policy. For macOS, `configuration profiles +<https://github.com/mozilla/policy-templates/tree/master/mac>`__ are created. +For the generic case, there is a JSON file to describe the policy. + +When these configurations are set, Firefox takes the configuration as inputs and +turns them into settings that Firefox can consume. + +A Hypothetical Use of Enterprise Policy +--------------------------------------- +A company that is in the banking industry and requires tighter security over +their users may not want their users to do something on Firefox without the +company's knowledge. It may make sense for the company to disable private +browsing for Firefox. + +Within a specific company, the employees of the finance department could use the +Firefox ESR version. In this situation, we think of the finance department as +the Firefox user rather than the individual employees as Firefox users. The +department makes choices for the individuals that use the Firefox browser +through the Enterprise Policy. + +Features On Enterprise Policy +----------------------------- +All Firefox versions have to honor the Enterprise Policy, but the Enterprise +Policy may not have effect on an individual who is not using Firefox ESR at a +company. There are features that are enterprise specific that are only available +in ESR. These features allow search engines to be configured, allowing for +unsigned extensions, installing search engines, and setting a default search +engine. + +How To Set Up and Use an Enterprise Policy for Firefox +------------------------------------------------------ +Install the ESR version of Firefox since Enterprise Policies are not supported on +rapid release. Then, create the JSON file that is located in the README.md within +https://github.com/mozilla/policy-templates. There are instructions there on how +to configure and use the policy. Once the JSON is created with the appropriate +settings, drop the JSON file in the directory outlined by the README.md and +Firefox will find it and Firefox will open and run with the policy. + +Common formatting mistakes are often made when creating the JSON file. The JSON +file can be validated using a JSON validator such as https://jsonlint.com/. diff --git a/toolkit/components/search/docs/SearchService.rst b/toolkit/components/search/docs/SearchService.rst new file mode 100644 index 0000000000..4f332ecedf --- /dev/null +++ b/toolkit/components/search/docs/SearchService.rst @@ -0,0 +1,6 @@ +SearchService Reference +======================= + +.. js:autoclass:: SearchService + :members: + :private-members: diff --git a/toolkit/components/search/docs/SearchServiceHighlevelOverview.rst b/toolkit/components/search/docs/SearchServiceHighlevelOverview.rst new file mode 100644 index 0000000000..1701f3a52b --- /dev/null +++ b/toolkit/components/search/docs/SearchServiceHighlevelOverview.rst @@ -0,0 +1,85 @@ +================================= +SearchService High-level Overview +================================= +``SearchService`` is the core component that manages Search Engines on the +Firefox browser. + +The following diagram is a high level systems context diagram of the +``SearchService``. The diagram illustrates which systems interface with the +``SearchService`` so that it can do its job of managing the Search Engines. + +The diagram and description is an over-simplification of the ``SearchService's`` +responsibilities. It specifically highlights how the ``SearchService`` serves +search engines to the browser on startup. However, the ``SearchService`` has +many other responsibilities that are not outlined in the diagram such as +maintaining and managing the Search Engines when various items change, e.g. the +user's region or locale, configuration changes received from remote settings, +updates received to search engine data. Nonetheless, the diagram gives a broad +overview of the main components the ``SearchService`` interacts with most often. + +Diagram +======= +.. figure:: assets/search-service-diagram.png + :scale: 85% + :align: center + +Description of the Diagram +========================== +These steps follow the same number on the diagram. Number 1 on the diagram is +described by number 1 below. + +1. When the user opens the Firefox Browser, the code starts to build the browser + UI components. During this startup phase, we have various systems making + calls to the ``SearchService``. E.g. `browser.js <https://searchfox.org/mozilla-central/rev/cb6f8d7b1f1782b9d4b2ee7312de1dcc284aaf06/browser/base/content/browser.js#3797>`_ + calls ``Services.search.getDefault`` to fetch the default Search Engine. + +2. The ``SearchService`` needs information from ``System Add-ons``, + ``SearchSettings``, and ``Remote Settings`` to build the correct engines in + the right order and to return the list of engines to its callers. + + a) First, the ``SearchService`` makes a request for the search configuration. + ``SearchService`` calls `SearchEngineSelector.fetchEngineConfiguration <https://searchfox.org/mozilla-central/rev/cb6f8d7b1f1782b9d4b2ee7312de1dcc284aaf06/toolkit/components/search/SearchService.sys.mjs#2247>`_ + which makes a call to `Remote Settings <https://searchfox.org/mozilla-central/rev/cb6f8d7b1f1782b9d4b2ee7312de1dcc284aaf06/toolkit/components/search/SearchEngineSelector.sys.mjs#129>`_ + for the search configuration. Remote Settings does not fetch the search + configuration from the remote database on startup. Instead Remote Settings + tries to load the :searchfox:`search configuration dump file <services/settings/dumps/main/search-config.json>` + from its local database and if that is empty, it will load the dump file into + its local database. Only after startup will Remote Settings connect to the + remote database when necessary. By connecting after startup, it avoids + a potential network request that could delay startup. + + b) Second, the ``SearchService`` `fetches a JSON file <https://searchfox.org/mozilla-central/rev/cb6f8d7b1f1782b9d4b2ee7312de1dcc284aaf06/toolkit/components/search/SearchService.sys.mjs#1296-1297>`_ + from the `SearchSettings <https://searchfox.org/mozilla-central/source/toolkit/components/search/SearchSettings.sys.mjs>`_. + This JSON file contains Search Engine metadata that is saved on the user's + computer. It's information that helps the ``SearchService`` remember the + user's custom settings for the Search Engines. + + c) Third, the `System Add-ons <https://searchfox.org/mozilla-central/rev/cb6f8d7b1f1782b9d4b2ee7312de1dcc284aaf06/browser/components/extensions/parent/ext-chrome-settings-overrides.js#536>`_ + passes the extension data to the ``SearchService``. At this point, the + ``SearchService`` only installs user installed search extensions. For the + Application Provided engines we create those when ``SearchService`` calls `_makeEngineFromConfig <https://searchfox.org/mozilla-central/rev/3002762e41363de8ee9ca80196d55e79651bcb6b/toolkit/components/search/SearchService.sys.mjs#3421-3440>`_. + Then ``_makeEngineFromConfig`` will create a new ``AddonSearchEngine``. + When the `AddonSearchEngine.init <https://searchfox.org/mozilla-central/rev/3002762e41363de8ee9ca80196d55e79651bcb6b/toolkit/components/search/AddonSearchEngine.sys.mjs#83-87,89>`_ + method is called, it combines both the extension and search configuration + data to create the correct engine for the user based on locale, region and + other information. + + After steps 2a, 2b, and 2c the ``SearchService`` has finished gathering + search engine data from ``System Add-ons``, ``SearchSettings``, and + ``Remote Settings``. Now the ``SearchService`` can build the different + types of Search Engines. + +3. The ``SearchService`` creates new instances of :searchfox:`SearchEngines <toolkit/components/search/SearchEngine.sys.mjs>` + by making an `Add-on, Open Search, or Enterprise Policy Search Engine <https://firefox-source-docs.mozilla.org/toolkit/search/SearchEngines.html>`_. + +4. The ``SearchService`` returns the engines to the caller that requested it. + E.g. the ``SearchService`` passes the default Search Engine back to + ``browser.js``, the system that initially requested it. + +Updates +======= +This page is up to date as of March 10, 2023. If the diagram or description +becomes out of date, please find access to the +``Search Service Diagram`` through the ``Firefox Search > Search Service +Documentation`` folder in the shared drive or through this link `here <https://drive.google.com/file/d/1vKRRK87kIGt6xamHJuclkC04EKrS69Qw/view?usp=sharing>`_. +Contributions are welcomed to keep this page up to date. diff --git a/toolkit/components/search/docs/Telemetry.rst b/toolkit/components/search/docs/Telemetry.rst new file mode 100644 index 0000000000..75f24b8d7e --- /dev/null +++ b/toolkit/components/search/docs/Telemetry.rst @@ -0,0 +1,94 @@ +Telemetry +========= + +This document describes search telemetry recorded by Toolkit such as search +service telemetry and telemetry related to fetching search suggestions. + +Other important search-related telemetry is recorded by Firefox and is +documented in :doc:`/browser/search/telemetry` in the Firefox documentation. + +Legacy Telemetry +---------------- + +Scalars +------- + +browser.searchinit.secure_opensearch_engine_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Records the number of secure (i.e., using https) OpenSearch search + engines a given user has installed. + +browser.searchinit.insecure_opensearch_engine_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Records the number of insecure (i.e., using http) OpenSearch search + engines a given user has installed. + +browser.searchinit.secure_opensearch_update_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Records the number of OpenSearch search engines with secure updates + enabled (i.e., using https) a given user has installed. + +browser.searchinit.insecure_opensearch_update_count +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Records the number of OpenSearch search engines with insecure updates + enabled (i.e., using http) a given user has installed. + +Keyed Scalars +------------- + +browser.searchinit.engine_invalid_webextension +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + Records the WebExtension ID of a search engine where the saved search engine + settings do not match the WebExtension. + + The keys are the WebExtension IDs. The values are integers: + + 1. Associated WebExtension is not installed. + 2. Associated WebExtension is disabled. + 3. The submission URL of the associated WebExtension is different to that of the saved settings. + +Histograms +---------- + +SEARCH_SUGGESTIONS_LATENCY_MS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + This histogram records the latency in milliseconds of fetches to the + suggestions endpoints of search engines, or in other words, the time from + Firefox's request to a suggestions endpoint to the time Firefox receives a + response. It is a keyed exponential histogram with 50 buckets and values + between 0 and 30000 (0s and 30s). Keys in this histogram are search engine IDs + for built-in search engines and 'other' for non-built-in search engines. + +Default Search Engine +~~~~~~~~~~~~~~~~~~~~~ +Telemetry for the user's default search engine is currently reported via two +systems: + + 1. Legacy telemetry: + `Fields are reported within the legacy telemetry environment <https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/environment.html#defaultsearchengine>`__ + 2. Glean: + `Fields are documented in the Glean dictionary <https://dictionary.telemetry.mozilla.org/apps/firefox_desktop?search=search.engine>`__. + +Glean Telemetry +--------------- +`These search service fields are documented via Glean dictionary <https://dictionary.telemetry.mozilla.org/apps/firefox_desktop?page=1&search=search.service>`__. + +search.service.startup_time +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + The time duration it takes for the search service to start up. + +search.service.initializaitonStatus +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + A labeled counter for the type of initialization statuses that can occur on + start up. Labels include: ``failedSettings``, ``failedFetchEngines``, + ``failedLoadEngines``, ``success``. + + A counter for initialization successes on start up. diff --git a/toolkit/components/search/docs/assets/bugzilla-open-search1.png b/toolkit/components/search/docs/assets/bugzilla-open-search1.png Binary files differnew file mode 100644 index 0000000000..de265e5223 --- /dev/null +++ b/toolkit/components/search/docs/assets/bugzilla-open-search1.png diff --git a/toolkit/components/search/docs/assets/bugzilla-open-search2.png b/toolkit/components/search/docs/assets/bugzilla-open-search2.png Binary files differnew file mode 100644 index 0000000000..d30260d7f7 --- /dev/null +++ b/toolkit/components/search/docs/assets/bugzilla-open-search2.png diff --git a/toolkit/components/search/docs/assets/search-service-diagram.png b/toolkit/components/search/docs/assets/search-service-diagram.png Binary files differnew file mode 100644 index 0000000000..cdd244b8f4 --- /dev/null +++ b/toolkit/components/search/docs/assets/search-service-diagram.png diff --git a/toolkit/components/search/docs/index.rst b/toolkit/components/search/docs/index.rst new file mode 100644 index 0000000000..d8d4bc8d8f --- /dev/null +++ b/toolkit/components/search/docs/index.rst @@ -0,0 +1,40 @@ +============== +Search Service +============== + +This is documentation for the Search Service. + +Definitions +=========== + +* Application-provided engines (aka app-provided): These are engines provided + by the application to the user as part of the configurations for the user's + locale/region. +* Application default engine: The engine automatically selected by the + application as default, in the absence of user settings. +* Default engine: This is the engine that is the one used by default when + doing searches from the address bar, search bar and other places. This may be + the application default engine or an user selected engine. +* Default private engine: Same as for the default engine, but this is used by + default when in private browsing mode. + +Contents +======== + +.. toctree:: + :maxdepth: 2 + + SearchServiceHighlevelOverview + SearchEngineConfiguration + SearchConfigurationSchema + SearchEngines + DefaultSearchEngines + Preferences + Telemetry + +API Reference +------------- + +.. toctree:: + + SearchService diff --git a/toolkit/components/search/docs/search-add-on-prompts-flow.png b/toolkit/components/search/docs/search-add-on-prompts-flow.png Binary files differnew file mode 100644 index 0000000000..ac2eeefadb --- /dev/null +++ b/toolkit/components/search/docs/search-add-on-prompts-flow.png |
