diff options
Diffstat (limited to 'intl/l10n/docs/migrations')
| -rw-r--r-- | intl/l10n/docs/migrations/fluent.rst | 153 | ||||
| -rw-r--r-- | intl/l10n/docs/migrations/index.rst | 53 | ||||
| -rw-r--r-- | intl/l10n/docs/migrations/legacy.rst | 642 | ||||
| -rw-r--r-- | intl/l10n/docs/migrations/localizations.rst | 42 | ||||
| -rw-r--r-- | intl/l10n/docs/migrations/overview.rst | 136 | ||||
| -rw-r--r-- | intl/l10n/docs/migrations/testing.rst | 58 |
6 files changed, 1084 insertions, 0 deletions
diff --git a/intl/l10n/docs/migrations/fluent.rst b/intl/l10n/docs/migrations/fluent.rst new file mode 100644 index 0000000000..bc14293ed7 --- /dev/null +++ b/intl/l10n/docs/migrations/fluent.rst @@ -0,0 +1,153 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + + +=========================== +Fluent to Fluent Migrations +=========================== + +When migrating existing Fluent messages, +it's possible to copy a source directly with :python:`COPY_PATTERN`, +or to apply string replacements and other changes +by extending the :python:`TransformPattern` visitor class. + +These transforms work with individual Fluent patterns, +i.e. the body of a Fluent message or one of its attributes. + +Copying Fluent Patterns +----------------------- + +Consider for example a patch modifying an existing message to move the original +value to a :js:`alt` attribute. + +Original message: + + +.. code-block:: fluent + + about-logins-icon = Warning icon + .title = Breached website + + +New message: + + +.. code-block:: fluent + + about-logins-breach-icon = + .alt = Warning icon + .title = Breached website + + +This type of changes requires a new message identifier, which in turn causes +existing translations to be lost. It’s possible to migrate the existing +translated content with: + + +.. code-block:: python + + from fluent.migrate import COPY_PATTERN + + ctx.add_transforms( + "browser/browser/aboutLogins.ftl", + "browser/browser/aboutLogins.ftl", + transforms_from( + """ + about-logins-breach-icon = + .alt = {COPY_PATTERN(from_path, "about-logins-icon")} + .title = {COPY_PATTERN(from_path, "about-logins-icon.title")} + """,from_path="browser/browser/aboutLogins.ftl"), + ) + + +In this specific case, the destination and source files are the same. The dot +notation is used to access attributes: :js:`about-logins-icon.title` matches +the :js:`title` attribute of the message with identifier +:js:`about-logins-icon`, while :js:`about-logins-icon` alone matches the value +of the message. + + +.. warning:: + + The second argument of :python:`COPY_PATTERN` and :python:`TransformPattern` + identifies a pattern, so using the message identifier will not + migrate the message as a whole, with all its attributes, only its value. + +Transforming Fluent Patterns +---------------------------- + +To apply changes to Fluent messages, you may extend the +:python:`TransformPattern` class to create your transformation. +This is a powerful general-purpose tool, of which :python:`COPY_PATTERN` is the +simplest extension that applies no transformation to the source. + +Consider for example a patch copying an existing message to strip out its HTML +content to use as an ARIA value. + +Original message: + + +.. code-block:: fluent + + videocontrols-label = + { $position }<span data-l10n-name="duration"> / { $duration }</span> + + +New message: + + +.. code-block:: fluent + + videocontrols-scrubber = + .aria-valuetext = { $position } / { $duration } + + +A migration may be applied to create this new message with: + + +.. code-block:: python + + from fluent.migrate.transforms import TransformPattern + import fluent.syntax.ast as FTL + + class STRIP_SPAN(TransformPattern): + def visit_TextElement(self, node): + node.value = re.sub("</?span[^>]*>", "", node.value) + return node + + def migrate(ctx): + path = "toolkit/toolkit/global/videocontrols.ftl" + ctx.add_transforms( + path, + path, + [ + FTL.Message( + id=FTL.Identifier("videocontrols-scrubber"), + attributes=[ + FTL.Attribute( + id=FTL.Identifier("aria-valuetext"), + value=STRIP_SPAN(path, "videocontrols-label"), + ), + ], + ), + ], + ) + + +Note that a custom extension such as :python:`STRIP_SPAN` is not supported by +the :python:`transforms_from` utility, so the list of transforms needs to be +defined explicitly. + +Internally, :python:`TransformPattern` extends the `fluent.syntax`__ +:python:`Transformer`, which defines the :python:`FTL` AST used here. +As a specific convenience, pattern element visitors such as +:python:`visit_TextElement` are allowed to return a :python:`FTL.Pattern` +to replace themselves with more than one node. + +__ https://projectfluent.org/python-fluent/fluent.syntax/stable/ diff --git a/intl/l10n/docs/migrations/index.rst b/intl/l10n/docs/migrations/index.rst new file mode 100644 index 0000000000..e9ed12aa22 --- /dev/null +++ b/intl/l10n/docs/migrations/index.rst @@ -0,0 +1,53 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + +============================================= +Migrating Strings From Legacy or Fluent Files +============================================= + +Firefox is a project localized in over 100 languages. As the code for existing +features moves away from the old localization systems and starts using +`Fluent`_, we need to ensure that we don’t lose existing translations, which +would have the adverse effect of forcing contributors to localize hundreds of +strings from scratch. + +`Fluent Migration`_ is a Python library designed to solve this specific problem: +it allows to migrate translations from `.properties` and other legacy file formats, +not only moving strings and transforming them as needed to adapt to the `FTL` +syntax, but also replicating "blame" for each string in VCS. + +The library also includes basic support for migrating existing Fluent messages +without interpolations (e.g. variable replacements). The typical use cases +would be messages moving as-is to a different file, or changes to the +morphology of existing messages (e.g move content from an attribute to the +value of the message). + +.. toctree:: + :maxdepth: 2 + + overview + legacy + fluent + testing + localizations + +How to Get Help +=============== + +Writing migration recipes can be challenging for non trivial cases, and it can +require extensive l10n knowledge to avoid localizability issues. + +Don’t hesitate to reach out to the l10n-drivers for feedback, help to test or +write the migration recipes: + + - Francesco Lodolo (:flod) + - Eemeli Aro (:eemeli) + +.. _Fluent: http://projectfluent.org/ +.. _Fluent Migration: https://hg.mozilla.org/l10n/fluent-migration/ diff --git a/intl/l10n/docs/migrations/legacy.rst b/intl/l10n/docs/migrations/legacy.rst new file mode 100644 index 0000000000..b1287ba4d4 --- /dev/null +++ b/intl/l10n/docs/migrations/legacy.rst @@ -0,0 +1,642 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + +======================== +Migrating Legacy Formats +======================== + +Migrating from legacy formats (.dtd, .properties) is different from migrating +Fluent to Fluent. When migrating legacy code paths, you'll need to adjust the +Fluent strings for the quirks Mozilla uses in the legacy code paths. You'll +find a number of specialized functionalities here. + +Legacy Migration Tools +---------------------- + +To assist with legacy format migrations, some scripting tools are provided: + + - `XUL+DTD to Fluent`_ + - `.properties to Fluent`_ + +When creating a migration, one or both of these tools may provide a good +starting point for manual work by automating at least a part of the migration, +including recipe generation and refactoring the calling code. + +.. _XUL+DTD to Fluent: https://github.com/zbraniecki/convert_xul_to_fluent +.. _.properties to Fluent: https://github.com/mozilla/properties-to-ftl + +Basic Migration +--------------- + +Let’s consider a basic example: one string needs to be migrated, without +any further change, from a DTD file to Fluent. + +The legacy string is stored in :bash:`toolkit/locales/en-US/chrome/global/findbar.dtd`: + + +.. code-block:: dtd + + <!ENTITY next.tooltip "Find the next occurrence of the phrase"> + + +The new Fluent string is stored in :bash:`toolkit/locales/en-US/toolkit/main-window/findbar.ftl`: + + +.. code-block:: properties + + findbar-next = + .tooltiptext = Find the next occurrence of the phrase + + +This is how the migration recipe looks: + + +.. code-block:: python + + # Any copyright is dedicated to the Public Domain. + # http://creativecommons.org/publicdomain/zero/1.0/ + + from __future__ import absolute_import + import fluent.syntax.ast as FTL + from fluent.migrate.helpers import transforms_from + + def migrate(ctx): + """Bug 1411707 - Migrate the findbar XBL binding to a Custom Element, part {index}.""" + + ctx.add_transforms( + "toolkit/toolkit/main-window/findbar.ftl", + "toolkit/toolkit/main-window/findbar.ftl", + transforms_from( + """ + findbar-next = + .tooltiptext = { COPY(from_path, "next.tooltip") } + """, from_path="toolkit/chrome/global/findbar.dtd")) + + +The first important thing to notice is that the migration recipe needs file +paths relative to a localization repository, losing :bash:`locales/en-US/`: + + - :bash:`toolkit/locales/en-US/chrome/global/findbar.dtd` becomes + :bash:`toolkit/chrome/global/findbar.dtd`. + - :bash:`toolkit/locales/en-US/toolkit/main-window/findbar.ftl` becomes + :bash:`toolkit/toolkit/main-window/findbar.ftl`. + +The :python:`context.add_transforms` function takes 3 arguments: + + - Path to the target l10n file. + - Path to the reference (en-US) file. + - An array of Transforms. Transforms are AST nodes which describe how legacy + translations should be migrated. + +.. note:: + + For migrations of Firefox localizations, the target and reference path + are the same. This isn't true for all projects that use Fluent, so both + arguments are required. + +In this case there is only one Transform that migrates the string with ID +:js:`next.tooltip` from :bash:`toolkit/chrome/global/findbar.dtd`, and injects +it in the FTL fragment. The :python:`COPY` Transform allows to copy the string +from an existing file as is, while :python:`from_path` is used to avoid +repeating the same path multiple times, making the recipe more readable. Without +:python:`from_path`, this could be written as: + + +.. code-block:: python + + ctx.add_transforms( + "toolkit/toolkit/main-window/findbar.ftl", + "toolkit/toolkit/main-window/findbar.ftl", + transforms_from( + """ + findbar-next = + .tooltiptext = { COPY("toolkit/chrome/global/findbar.dtd", "next.tooltip") } + """)) + + +This method of writing migration recipes allows to take the original FTL +strings, and simply replace the value of each message with a :python:`COPY` +Transform. :python:`transforms_from` takes care of converting the FTL syntax +into an array of Transforms describing how the legacy translations should be +migrated. This manner of defining migrations is only suitable to simple strings +where a copy operation is sufficient. For more complex use-cases which require +some additional logic in Python, it’s necessary to resort to the raw AST. + + +The example above is equivalent to the following syntax, which exposes +the underlying AST structure: + + +.. code-block:: python + + ctx.add_transforms( + "toolkit/toolkit/main-window/findbar.ftl", + "toolkit/toolkit/main-window/findbar.ftl", + [ + FTL.Message( + id=FTL.Identifier("findbar-next"), + attributes=[ + FTL.Attribute( + id=FTL.Identifier("tooltiptext"), + value=COPY( + "toolkit/chrome/global/findbar.dtd", + "next.tooltip" + ) + ) + ] + ) + ] + ) + +This creates a :python:`Message`, taking the value from the legacy string +:js:`findbar-next`. A message can have an array of attributes, each with an ID +and a value: in this case there is only one attribute, with ID :js:`tooltiptext` +and :js:`value` copied from the legacy string. + +Notice how both the ID of the message and the ID of the attribute are +defined as an :python:`FTL.Identifier`, not simply as a string. + + +.. tip:: + + It’s possible to concatenate arrays of Transforms defined manually, like in + the last example, with those coming from :python:`transforms_from`, by using + the :python:`+` operator. Alternatively, it’s possible to use multiple + :python:`add_transforms`. + + The order of Transforms provided in the recipe is not relevant, the reference + file is used for ordering messages. + + +Replacing Content in Legacy Strings +----------------------------------- + +While :python:`COPY` allows to copy a legacy string as is, :python:`REPLACE` +(from `fluent.migrate`) allows to replace content while performing the +migration. This is necessary, for example, when migrating strings that include +placeholders or entities that need to be replaced to adapt to Fluent syntax. + +Consider for example the following string: + + +.. code-block:: DTD + + <!ENTITY aboutSupport.featuresTitle "&brandShortName; Features"> + + +Which needs to be migrated to: + + +.. code-block:: fluent + + features-title = { -brand-short-name } Features + + +The entity :js:`&brandShortName;` needs to be replaced with a term reference: + + +.. code-block:: python + + FTL.Message( + id=FTL.Identifier("features-title"), + value=REPLACE( + "toolkit/chrome/global/aboutSupport.dtd", + "aboutSupport.featuresTitle", + { + "&brandShortName;": TERM_REFERENCE("brand-short-name"), + }, + ) + ), + + +This creates an :python:`FTL.Message`, taking the value from the legacy string +:js:`aboutSupport.featuresTitle`, but replacing the specified text with a +Fluent term reference. + +.. note:: + :python:`REPLACE` replaces all occurrences of the specified text. + + +It’s also possible to replace content with a specific text: in that case, it +needs to be defined as a :python:`TextElement`. For example, to replace +:js:`example.com` with HTML markup: + + +.. code-block:: python + + value=REPLACE( + "browser/chrome/browser/preferences/preferences.properties", + "searchResults.sorryMessageWin", + { + "example.com": FTL.TextElement('<span data-l10n-name="example"></span>') + } + ) + + +The situation is more complex when a migration recipe needs to replace +:js:`printf` arguments like :js:`%S`. In fact, the format used for localized +and source strings doesn’t need to match, and the two following strings using +unordered and ordered argument are perfectly equivalent: + + +.. code-block:: properties + + btn-quit = Quit %S + btn-quit = Quit %1$S + + +In this scenario, replacing :js:`%S` would work on the first version, but not +on the second, and there’s no guarantee that the localized string uses the +same format as the source string. + +Consider also the following string that uses :js:`%S` for two different +variables, implicitly relying on the order in which the arguments appear: + + +.. code-block:: properties + + updateFullName = %S (%S) + + +And the target Fluent string: + + +.. code-block:: fluent + + update-full-name = { $name } ({ $buildID }) + + +As indicated, :python:`REPLACE` would replace all occurrences of :js:`%S`, so +only one variable could be set. The string needs to be normalized and treated +like: + + +.. code-block:: properties + + updateFullName = %1$S (%2$S) + + +This can be obtained by calling :python:`REPLACE` with +:python:`normalize_printf=True`: + + +.. code-block:: python + + FTL.Message( + id=FTL.Identifier("update-full-name"), + value=REPLACE( + "toolkit/chrome/mozapps/update/updates.properties", + "updateFullName", + { + "%1$S": VARIABLE_REFERENCE("name"), + "%2$S": VARIABLE_REFERENCE("buildID"), + }, + normalize_printf=True + ) + ) + + +.. attention:: + + To avoid any issues :python:`normalize_printf=True` should always be used when + replacing :js:`printf` arguments. This is the default behaviour when working + with .properties files. + +.. note:: + + :python:`VARIABLE_REFERENCE`, :python:`MESSAGE_REFERENCE`, and + :python:`TERM_REFERENCE` are helper Transforms which can be used to save + keystrokes in common cases where using the raw AST is too verbose. + + :python:`VARIABLE_REFERENCE` is used to create a reference to a variable, e.g. + :js:`{ $variable }`. + + :python:`MESSAGE_REFERENCE` is used to create a reference to another message, + e.g. :js:`{ another-string }`. + + :python:`TERM_REFERENCE` is used to create a reference to a `term`__, + e.g. :js:`{ -brand-short-name }`. + + Both Transforms need to be imported at the beginning of the recipe, e.g. + :python:`from fluent.migrate.helpers import VARIABLE_REFERENCE` + + __ https://projectfluent.org/fluent/guide/terms.html + + +Trimming Unnecessary Whitespaces in Translations +------------------------------------------------ + +.. note:: + + This section was updated in May 2020 to reflect the change to the default + behavior: legacy translations are now trimmed, unless the :python:`trim` + parameter is set explicitly. + +It’s not uncommon to have strings with unnecessary leading or trailing spaces +in legacy translations. These are not meaningful, don’t have practical results +on the way the string is displayed in products, and are added mostly for +formatting reasons. For example, consider this DTD string: + + +.. code-block:: DTD + + <!ENTITY aboutAbout.note "This is a list of “about” pages for your convenience.<br/> + Some of them might be confusing. Some are for diagnostic purposes only.<br/> + And some are omitted because they require query strings."> + + +By default, the :python:`COPY`, :python:`REPLACE`, and :python:`PLURALS` +transforms will strip the leading and trailing whitespace from each line of the +translation, as well as the empty leading and trailing lines. The above string +will be migrated as the following Fluent message, despite copious indentation +on the second and the third line in the original: + + +.. code-block:: fluent + + about-about-note = + This is a list of “about” pages for your convenience.<br/> + Some of them might be confusing. Some are for diagnostic purposes only.<br/> + And some are omitted because they require query strings. + + +To disable the default trimming behavior, set :python:`trim:"False"` or +:python:`trim=False`, depending on the context: + + +.. code-block:: python + + transforms_from( + """ + about-about-note = { COPY("toolkit/chrome/global/aboutAbout.dtd", "aboutAbout.note", trim:"False") } + """) + + FTL.Message( + id=FTL.Identifier("discover-description"), + value=REPLACE( + "toolkit/chrome/mozapps/extensions/extensions.dtd", + "discover.description2", + { + "&brandShortName;": TERM_REFERENCE("-brand-short-name") + }, + trim=False + ) + ), + + +Concatenating Strings +--------------------- + +It's best practice to only expose complete phrases to localization, and to avoid +stitching localized strings together in code. With `DTD` and `properties`, +there were few options. So when migrating to Fluent, you'll find +it quite common to concatenate multiple strings coming from `DTD` and +`properties`, for example to create sentences with HTML markup. It’s possible to +concatenate strings and text elements in a migration recipe using the +:python:`CONCAT` Transform. + +Note that in case of simple migrations using :python:`transforms_from`, the +concatenation is carried out implicitly by using the Fluent syntax interleaved +with :python:`COPY()` transform calls to define the migration recipe. + +Consider the following example: + + +.. code-block:: properties + + # %S is replaced by a link, using searchResults.needHelpSupportLink as text + searchResults.needHelp = Need help? Visit %S + + # %S is replaced by "Firefox" + searchResults.needHelpSupportLink = %S Support + + +In Fluent: + + +.. code-block:: fluent + + search-results-need-help-support-link = Need help? Visit <a data-l10n-name="url">{ -brand-short-name } Support</a> + + +This is quite a complex migration: it requires to take 2 legacy strings, and +concatenate their values with HTML markup. Here’s how the Transform is defined: + + +.. code-block:: python + + FTL.Message( + id=FTL.Identifier("search-results-help-link"), + value=REPLACE( + "browser/chrome/browser/preferences/preferences.properties", + "searchResults.needHelp", + { + "%S": CONCAT( + FTL.TextElement('<a data-l10n-name="url">'), + REPLACE( + "browser/chrome/browser/preferences/preferences.properties", + "searchResults.needHelpSupportLink", + { + "%1$S": TERM_REFERENCE("brand-short-name"), + }, + normalize_printf=True + ), + FTL.TextElement("</a>") + ) + } + ) + ), + + +:js:`%S` in :js:`searchResults.needHelpSupportLink` is replaced by a reference +to the term :js:`-brand-short-name`, migrating from :js:`%S Support` to :js:`{ +-brand-short-name } Support`. The result of this operation is then inserted +between two text elements to create the anchor markup. The resulting text is +finally used to replace :js:`%S` in :js:`searchResults.needHelp`, and used as +value for the FTL message. + + +.. important:: + + When concatenating existing strings, avoid introducing changes to the original + text, for example adding spaces or punctuation. Each language has its own + rules, and this might result in poor migrated strings. In case of doubt, + always ask for feedback. + + +When more than 1 element is passed in to concatenate, :python:`CONCAT` +disables whitespace trimming described in the section above on all legacy +Transforms passed into it: :python:`COPY`, :python:`REPLACE`, and +:python:`PLURALS`, unless the :python:`trim` parameters has been set +explicitly on them. This helps ensure that spaces around segments are not +lost during the concatenation. + +When only a single element is passed into :python:`CONCAT`, however, the +trimming behavior is not altered, and follows the rules described in the +previous section. This is meant to make :python:`CONCAT(COPY())` equivalent +to a bare :python:`COPY()`. + + +Plural Strings +-------------- + +Migrating plural strings from `.properties` files usually involves two +Transforms from :python:`fluent.migrate.transforms`: the +:python:`REPLACE_IN_TEXT` Transform takes TextElements as input, making it +possible to pass it as the foreach function of the :python:`PLURALS` Transform. + +Consider the following legacy string: + + +.. code-block:: properties + + # LOCALIZATION NOTE (disableContainersOkButton): Semi-colon list of plural forms. + # See: http://developer.mozilla.org/en/docs/Localization_and_Plurals + # #1 is the number of container tabs + disableContainersOkButton = Close #1 Container Tab;Close #1 Container Tabs + + +In Fluent: + + +.. code-block:: fluent + + containers-disable-alert-ok-button = + { $tabCount -> + [one] Close { $tabCount } Container Tab + *[other] Close { $tabCount } Container Tabs + } + + +This is how the Transform for this string is defined: + + +.. code-block:: python + + FTL.Message( + id=FTL.Identifier("containers-disable-alert-ok-button"), + value=PLURALS( + "browser/chrome/browser/preferences/preferences.properties", + "disableContainersOkButton", + VARIABLE_REFERENCE("tabCount"), + lambda text: REPLACE_IN_TEXT( + text, + { + "#1": VARIABLE_REFERENCE("tabCount") + } + ) + ) + ) + + +The `PLURALS` Transform will take care of creating the correct number of plural +categories for each language. Notice how `#1` is replaced for each of these +variants with :js:`{ $tabCount }`, using :python:`REPLACE_IN_TEXT` and +:python:`VARIABLE_REFERENCE("tabCount")`. + +In this case it’s not possible to use :python:`REPLACE` because it takes a file +path and a message ID as arguments, whereas here the recipe needs to operate on +regular text. The replacement is performed on each plural form of the original +string, where plural forms are separated by a semicolon. + +Explicit Variants +----------------- + +Explicitly creating variants of a string is useful for platform-dependent +terminology, but also in cases where you want a one-vs-many split of a string. +It’s always possible to migrate strings by manually creating the underlying AST +structure. Consider the following complex Fluent string: + + +.. code-block:: fluent + + use-current-pages = + .label = + { $tabCount -> + [1] Use Current Page + *[other] Use Current Pages + } + .accesskey = C + + +The migration for this string is quite complex: the :js:`label` attribute is +created from 2 different legacy strings, and it’s not a proper plural form. +Notice how the first string is associated to the :js:`1` case, not the :js:`one` +category used in plural forms. For these reasons, it’s not possible to use +:python:`PLURALS`, the Transform needs to be crafted recreating the AST. + + +.. code-block:: python + + + FTL.Message( + id=FTL.Identifier("use-current-pages"), + attributes=[ + FTL.Attribute( + id=FTL.Identifier("label"), + value=FTL.Pattern( + elements=[ + FTL.Placeable( + expression=FTL.SelectExpression( + selector=VARIABLE_REFERENCE("tabCount"), + variants=[ + FTL.Variant( + key=FTL.NumberLiteral("1"), + default=False, + value=COPY( + "browser/chrome/browser/preferences/main.dtd", + "useCurrentPage.label", + ) + ), + FTL.Variant( + key=FTL.Identifier("other"), + default=True, + value=COPY( + "browser/chrome/browser/preferences/main.dtd", + "useMultiple.label", + ) + ) + ] + ) + ) + ] + ) + ), + FTL.Attribute( + id=FTL.Identifier("accesskey"), + value=COPY( + "browser/chrome/browser/preferences/main.dtd", + "useCurrentPage.accesskey", + ) + ), + ], + ), + + +This Transform uses several concepts already described in this document. Notable +is the :python:`SelectExpression` inside a :python:`Placeable`, with an array +of :python:`Variant` objects. Exactly one of those variants needs to have +``default=True``. + +This example can still use :py:func:`transforms_from()``, since existing strings +are copied without interpolation. + +.. code-block:: python + + transforms_from( + """ + use-current-pages = + .label = + { $tabCount -> + [1] { COPY(main_dtd, "useCurrentPage.label") } + *[other] { COPY(main_dtd, "useMultiple.label") } + } + .accesskey = { COPY(main_dtd, "useCurrentPage.accesskey") } + """, main_dtd="browser/chrome/browser/preferences/main.dtd" + ) diff --git a/intl/l10n/docs/migrations/localizations.rst b/intl/l10n/docs/migrations/localizations.rst new file mode 100644 index 0000000000..0861d6e52b --- /dev/null +++ b/intl/l10n/docs/migrations/localizations.rst @@ -0,0 +1,42 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + +=========================================== +How Migrations Are Run on l10n Repositories +=========================================== + +Once a patch including new FTL strings and a migration recipe lands in +mozilla-central, l10n-drivers will perform a series of actions to migrate +strings in all 100+ localization repositories: + + - New Fluent strings land in `mozilla-central`, together with a migration + recipe. + - New strings are added to `gecko-strings-quarantine`_, a unified repository + including strings for all shipping versions of Firefox, and used as a buffer + before exposing strings to localizers. + - Migration recipes are run against all l10n repositories, migrating strings + from old to new files, and storing them in VCS. + - New en-US strings are pushed to the official `gecko-strings`_ repository + used by localization tools, and exposed to all localizers. + +Migration recipes could be run again within a release cycle, in order to migrate +translations for legacy strings added after the first run. They’re usually +removed from `mozilla-central` within 2 cycles, e.g. a migration recipe created +for Firefox 59 would be removed when Firefox 61 is available in Nightly. + + +.. tip:: + + A script to run migrations on all l10n repositories is available in `this + repository`__, automating part of the steps described for manual testing, and + it could be adapted to local testing. + + __ https://github.com/flodolo/fluent-migrations +.. _gecko-strings-quarantine: https://hg.mozilla.org/l10n/gecko-strings-quarantine/ +.. _gecko-strings: https://hg.mozilla.org/l10n/gecko-strings diff --git a/intl/l10n/docs/migrations/overview.rst b/intl/l10n/docs/migrations/overview.rst new file mode 100644 index 0000000000..dc9c128fb9 --- /dev/null +++ b/intl/l10n/docs/migrations/overview.rst @@ -0,0 +1,136 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + +===================================== +Migration Recipes and Their Lifecycle +===================================== + +The actual migrations are performed running Python modules called **migration +recipes**, which contain directives on how to migrate strings, which files are +involved, transformations to apply, etc. These recipes are stored in +`mozilla-central`__. + +__ https://hg.mozilla.org/mozilla-central/file/default/python/l10n/fluent_migrations + +When part of Firefox’s UI is migrated to Fluent, a migration recipe should be +attached to the same patch that adds new strings to `.ftl` files. + +Migration recipes can quickly become obsolete, because the referenced strings +and files are removed from repositories as part of ongoing development. +For these reasons, l10n-drivers periodically clean up the `fluent_migrations` +folder in mozilla-central, keeping only recipes for 2 +shipping versions (Nightly and Beta). + + +.. hint:: + + As a developer you don’t need to bother about updating migration recipes + already in `mozilla-central`: if a new patch removes a string or file that is + used in a migration recipe, simply ignore it, since the entire recipe will be + removed within a couple of cycles. + + +How to Write Migration Recipes +============================== + +The migration recipe’s filename should start with a reference to the associated +bug number, and include a brief description of the bug, e.g. +:bash:`bug_1451992_preferences_applicationManager.py` is the migration recipe +used to migrate the Application Manager window in preferences. It’s also +possible to look at existing recipes in `mozilla-central`__ for inspiration. + +__ https://hg.mozilla.org/mozilla-central/file/default/python/l10n/fluent_migrations + + +General Recipe Structure +======================== + +A migration recipe is a Python module, implementing the :py:func:`migrate` +function, which takes a :py:class:`MigrationContext` as input. The API provided +by the context is + +.. code-block:: python + + class MigrationContext: + def add_transforms(self, target, reference, transforms): + """Define transforms for target using reference as template. + + `target` is a path of the destination FTL file relative to the + localization directory. `reference` is a path to the template FTL + file relative to the reference directory. + + Each transform is an extended FTL node with `Transform` nodes as some + values. + + For transforms that merely copy legacy messages or Fluent patterns, + using `fluent.migrate.helpers.transforms_from` is recommended. + """ + +The skeleton of a migration recipe just implements the :py:func:`migrate` +function calling into :py:func:`ctx.add_transforms`, and looks like + +.. code-block:: python + + # coding=utf8 + + # Any copyright is dedicated to the Public Domain. + # http://creativecommons.org/publicdomain/zero/1.0/ + + from __future__ import absolute_import + + + def migrate(ctx): + """Bug 1552333 - Migrate feature to Fluent, part {index}""" + target = 'browser/browser/feature.ftl' + reference = 'browser/browser/feature.ftl' + ctx.add_transforms( + target, + reference, + [], # Actual transforms go here. + ) + +One can call into :py:func:`ctx.add_transforms` multiple times. In particular, one +can create migrated content in multiple files as part of a single migration +recipe by calling :py:func:`ctx.add_transforms` with different target-reference +pairs. + +The *docstring* for this function will be used +as a commit message in VCS, that’s why it’s important to make sure the bug +reference is correct, and to keep the `part {index}` section: multiple strings +could have multiple authors, and would be migrated in distinct commits (part 1, +part 2, etc.). + +Transforms +========== + +The work of the migrations is done by the transforms that are passed as +last argument to :py:func:`ctx.add_transforms`. They're instances of either Fluent +:py:class:`fluent.syntax.ast.Message` or :py:class:`Term`, and their content +can depend on existing translation sources. The skeleton of a Message looks like + +.. code-block:: python + + FTL.Message( + id=FTL.Identifier( + name="msg", + ), + value=FTL.Pattern( + elements=[ + FTL.TextElement( + value="A string", + ), + ], + ), + ) + +When migrating existing legacy translations, you'll replace an +``FTL.TextElement`` with a ``COPY(legacy_path, "old_id")``, or one of its +variations we detail :doc:`next <legacy>`. When migrating existing Fluent +translations, an ``FTL.Pattern`` is replaced with a +``COPY_PATTERN(old_path, "old-id")``. diff --git a/intl/l10n/docs/migrations/testing.rst b/intl/l10n/docs/migrations/testing.rst new file mode 100644 index 0000000000..aa9b9747f6 --- /dev/null +++ b/intl/l10n/docs/migrations/testing.rst @@ -0,0 +1,58 @@ +.. role:: bash(code) + :language: bash + +.. role:: js(code) + :language: javascript + +.. role:: python(code) + :language: python + +============================= +How to Test Migration Recipes +============================= + +To test migration recipes, use the following mach command: + +.. code-block:: bash + + ./mach fluent-migration-test python/l10n/fluent_migrations/bug_1485002_newtab.py + +This will analyze your migration recipe to check that the :python:`migrate` +function exists, and interacts correctly with the migration context. Once that +passes, it clones :bash:`gecko-strings` into :bash:`$OBJDIR/python/l10n`, creates a +reference localization by adding your local Fluent strings to the ones in +:bash:`gecko-strings`. It then runs the migration recipe, both as dry run and +as actual migration. Finally it analyzes the commits, and checks if any +migrations were actually run and the bug number in the commit message matches +the migration name. + +It will also show the diff between the migrated files and the reference, ignoring +blank lines. + +You can inspect the generated repository further by looking at + +.. code-block:: bash + + ls $OBJDIR/python/l10n/bug_1485002_newtab/en-US + +Caveats +------- + +Be aware of hard-coded English context in migration. Consider for example: + + +.. code-block:: python + + ctx.add_transforms( + "browser/browser/preferences/siteDataSettings.ftl", + "browser/browser/preferences/siteDataSettings.ftl", + transforms_from( + """ + site-usage-persistent = { site-usage-pattern } (Persistent) + """) + ) + + +This Transform will pass a manual comparison, since the two files are identical, +but will result in :js:`(Persistent)` being hard-coded in English for all +languages. |