diff options
Diffstat (limited to 'devtools/client/debugger/src/utils/pause/mapScopes')
12 files changed, 1577 insertions, 0 deletions
diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/README.md b/devtools/client/debugger/src/utils/pause/mapScopes/README.md new file mode 100644 index 0000000000..2f65b8e847 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/README.md @@ -0,0 +1,191 @@ +# mapScopes + +The files in this directory manage the Devtools logic for taking the original +JS file from the sourcemap and using the content, combined with source +mappings, to provide an enhanced user experience for debugging the code. + +In this document, we'll refer to the files as either: + +* `original` - The code that the user wrote originally. +* `generated` - The code actually executing in the engine, which may have been + transformed from the `original`, and if a bundler was used, may contain the + final output for several different `original` files. + +The enhancements implemented here break down into as two primary improvements: + +* Rendering the scopes as the user expects them, using the position in the + original file, rather than the content of the generated file. +* Allowing users to interact with the code in the console as if it were the + original code, rather than the generated file. + + +## Overall Approach + +The core goal of scope mapping is to parse the original and generated files, +and then infer a correlation between the bindings in the original file, +and the bindings in the generated file. This is correlation is done via the +source mappings provided alongside the original code in the sourcemap. + +The overall steps break down into: + + +### 1. Parsing + +First the generated and original files are parsed into ASTs, and then the ASTs +are each traversed in order to generate a full picture of the scopes that are +present in the file. This covers information for each binding in each scope, +including all metadata about the declarations themselves, and all references +to each of the bindings. The exact location of each binding reference is the +most important factor because these will be used when working with sourcemaps. + +Importantly, this scope tree's structure also mirrors the structure of the +scope data that the engine itself returns when paused. + + +### 2. Generated Binding -> Grip Correlation + +When the engine pauses, we get back a full description of the current scope, +as well as `grip` objects that can be used as proxy-like objects in order to +access the actual values in the engine itself. + +With this data, along with the location where the engine is paused, we can +use the AST-based scope information from step 1 to attach a line/column +range to each `grip`. Using those mappings we have enough information to get +the real engine value of any variable based on its position in the generated +code, as long as it is in scope at the paused location in the generated file. + +The generated and engine scopes are correlated depth-first, because in some +cases the scope data from the engine will be deeper that the data from the +parsed code, meaning there are additional unknown parent scopes. This can +happen, for instance, if the executing code is running inside of an `eval` +since the parser only knows about the scopes inside of `eval`, and cannot +know what context it is executed inside of. + + +### 3. Original Binding -> Generated Binding Correlation + +Now that we can get the value of a generated location, we need to decide which +generated location correlates with which original location. For each of +the original bindings in each original scope, we iterate through the +individual references to the binding in the file and: + +1. Use the sourcemap to convert the original location into a range on the + generated code. +2. Filter the available bindings in the generated file down to those that + overlap with this generated range. +3. Perform heuristics to decide if any of the bindings in the range appear + to be a valid and safe mapping. + +These steps allow us to build a datastructure that describes the scope +as it is declared in the original file, while rendering the value using the +`grip` objects, as returned from the engine itself. + +There is additional complexity here in the exact details of the heuristics +mentioned above. See the later Heuristics sections diving into these details. + +During this phase, one additional task is performed, which is the construction +of expressions that relate back to the original binding. After matching each +binding in a range, we know the name of the binding in the original scope, +and we _also_ know the name of the generated binding, or more generally the +expression to evaluate in order to get the value of the original binding. + +These expression values are important for ensuring that the developer console +is able to allow users to interact with the original code. If a user types +a variable into the console, it can be translated into the expression +correlated with that original variable, allowing the code to behave as it +would have, were it actually part of the original file. + + +### 4. Generate a Scope Tree + +The structure generated in step 3 is converted into a structure compatible +with the standard scope data returned from the engine itself in order to allow +for consistent usage of scope data, irrespective of the scope data source. +This stage also re-attaches the global scope data, which is otherwise ignored +during the correlation process. + + +### 5. Validation + +As a simple form of validation, we ensure that a large percentage of bindings +were actually resolved to a `grip` object. This offers some amount of +protection, if the sourcemap itself turned out to be somewhat low-quality. + +This happens often with tooling that generates maps that simply map an original +line to a generated line. While line-to-line mappings still enable step +debugging, they do not provide enough information for original-scope mapping. + +On the other hand, generally line-to-line mappings are usually generated by +tooling that performs minimal transformations on their own, so it is often +acceptable to fall back to the engine-only generated-file scope data in this +case. + + +## Range Mapping Heuristics + +Since we know lots of information about the original bindings when performing +matches, it is possible to make educated guesses about how their generated +code will have likely been transformed. This allows us to perform more +aggressive matching logic what would normally be too false-positive-heavy +for generic use, when it is deemed safe enough. + + +### Standard Matching + +In the general case, we iterate through the bindings that were found in the +generated range, and consider it a match if the binding overlaps with the +start of the range. One extra case here is that ranges at the start of +a line often point to the first binding in a range and do not overlap the +first binding, so there is special-casing for the first binding in each range. + + +### ES6 Import Reference Special Case + +References to ES6 imports are often transformed into property accesses on objects +in order to emulate the "live-binding" behavior defined by ES6. The standard +logic here would end up always resolving the imported value to be the import +namespace object itself, rather than reading the value of the property. + +To support this, specifically for ES6 imports, we walk outward from the matched +binding itself, taking property accesses into account, as long as the +property access itself is also within the mapped range. + +While decently effective, there are currently two downsides to this approach: + +* The "live" behavior of imports is often implemented using accessor + properties, which as of the time of this writing, cannot be evaluated to + retrieve their real value. +* The "live" behavior of imports is sometimes implemented with function calls, + which also also cannot be evaluated, causing their value to be + unknown. + + +### ES6 Import Declaration Special Case + +If there are no references to an imported value, or matching based on the +reference failed, we fall back to a second case. + +ES6 import declarations themselves often map back to the location of the +declaration of the imported module's namespace object. By getting the range for +the import declaration itself, we can infer which generated binding is the +namespace object. Alongside that, we already know the name of the property on +the namespace itself because it is statically knowable by analyzing the +import declaration. + +By combining both of those pieces of information, we can access the namespace's +property to get the imported value. + + +### Typescript Classes + +Typescript have several non-ideal ways in which they output source maps, most +of which center around classes that are either exported, or decorated. These +issues are currently tracked in [Typescript issue #22833][1]. + +To work around this issue, we use a method similar to the import declaration +case above. While the class name itself often maps to unhelpful locations, +the class declaration itself generally maps to the class's transformed binding, +so we make use of the class declaration location instead of the location of +the class's declared name in these cases. + + [1]: https://github.com/Microsoft/TypeScript/issues/22833 diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/buildGeneratedBindingList.js b/devtools/client/debugger/src/utils/pause/mapScopes/buildGeneratedBindingList.js new file mode 100644 index 0000000000..5f90138013 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/buildGeneratedBindingList.js @@ -0,0 +1,141 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { clientCommands } from "../../../client/firefox"; + +import { locColumn } from "./locColumn"; +import { getOptimizedOutGrip } from "./optimizedOut"; + +export function buildGeneratedBindingList( + scopes, + generatedAstScopes, + thisBinding +) { + // The server's binding data doesn't include general 'this' binding + // information, so we manually inject the one 'this' binding we have into + // the normal binding data we are working with. + const frameThisOwner = generatedAstScopes.find( + generated => "this" in generated.bindings + ); + + let globalScope = null; + const clientScopes = []; + for (let s = scopes; s; s = s.parent) { + const bindings = s.bindings + ? Object.assign({}, ...s.bindings.arguments, s.bindings.variables) + : {}; + + clientScopes.push(bindings); + globalScope = s; + } + + const generatedMainScopes = generatedAstScopes.slice(0, -2); + const generatedGlobalScopes = generatedAstScopes.slice(-2); + + const clientMainScopes = clientScopes.slice(0, generatedMainScopes.length); + const clientGlobalScopes = clientScopes.slice(generatedMainScopes.length); + + // Map the main parsed script body using the nesting hierarchy of the + // generated and client scopes. + const generatedBindings = generatedMainScopes.reduce((acc, generated, i) => { + const bindings = clientMainScopes[i]; + + if (generated === frameThisOwner && thisBinding) { + bindings.this = { + value: thisBinding, + }; + } + + for (const name of Object.keys(generated.bindings)) { + // If there is no 'this' value, we exclude the binding entirely. + // Otherwise it would pass through as found, but "(unscoped)", causing + // the search logic to stop with a match. + if (name === "this" && !bindings[name]) { + continue; + } + + const { refs } = generated.bindings[name]; + for (const loc of refs) { + acc.push({ + name, + loc, + desc: () => Promise.resolve(bindings[name] || null), + }); + } + } + return acc; + }, []); + + // Bindings in the global/lexical global of the generated code may or + // may not be the real global if the generated code is running inside + // of an evaled context. To handle this, we just look up the client scope + // hierarchy to find the closest binding with that name. + for (const generated of generatedGlobalScopes) { + for (const name of Object.keys(generated.bindings)) { + const { refs } = generated.bindings[name]; + const bindings = clientGlobalScopes.find(b => name in b); + + for (const loc of refs) { + if (bindings) { + generatedBindings.push({ + name, + loc, + desc: () => Promise.resolve(bindings[name]), + }); + } else { + const globalGrip = globalScope?.object; + if (globalGrip) { + // Should always exist, just checking to keep Flow happy. + + generatedBindings.push({ + name, + loc, + desc: async () => { + const objectFront = + clientCommands.createObjectFront(globalGrip); + return (await objectFront.getProperty(name)).descriptor; + }, + }); + } + } + } + } + } + + // Sort so we can binary-search. + return sortBindings(generatedBindings); +} + +export function buildFakeBindingList(generatedAstScopes) { + // TODO if possible, inject real bindings for the global scope + const generatedBindings = generatedAstScopes.reduce((acc, generated) => { + for (const name of Object.keys(generated.bindings)) { + if (name === "this") { + continue; + } + const { refs } = generated.bindings[name]; + for (const loc of refs) { + acc.push({ + name, + loc, + desc: () => Promise.resolve(getOptimizedOutGrip()), + }); + } + } + return acc; + }, []); + return sortBindings(generatedBindings); +} + +function sortBindings(generatedBindings) { + return generatedBindings.sort((a, b) => { + const aStart = a.loc.start; + const bStart = b.loc.start; + + if (aStart.line === bStart.line) { + return locColumn(aStart) - locColumn(bStart); + } + return aStart.line - bStart.line; + }); +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/filtering.js b/devtools/client/debugger/src/utils/pause/mapScopes/filtering.js new file mode 100644 index 0000000000..dec3772e82 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/filtering.js @@ -0,0 +1,45 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +function findInsertionLocation(array, callback) { + let left = 0; + let right = array.length; + while (left < right) { + const mid = Math.floor((left + right) / 2); + const item = array[mid]; + + const result = callback(item); + if (result === 0) { + left = mid; + break; + } + if (result >= 0) { + right = mid; + } else { + left = mid + 1; + } + } + + // Ensure the value is the start of any set of matches. + let i = left; + if (i < array.length) { + while (i >= 0 && callback(array[i]) >= 0) { + i--; + } + return i + 1; + } + + return i; +} + +export function filterSortedArray(array, callback) { + const start = findInsertionLocation(array, callback); + + const results = []; + for (let i = start; i < array.length && callback(array[i]) === 0; i++) { + results.push(array[i]); + } + + return results; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/findGeneratedBindingFromPosition.js b/devtools/client/debugger/src/utils/pause/mapScopes/findGeneratedBindingFromPosition.js new file mode 100644 index 0000000000..6e833959f4 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/findGeneratedBindingFromPosition.js @@ -0,0 +1,305 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { locColumn } from "./locColumn"; +import { mappingContains } from "./mappingContains"; + +// eslint-disable-next-line max-len + +import { clientCommands } from "../../../client/firefox"; + +/** + * Given a mapped range over the generated source, attempt to resolve a real + * binding descriptor that can be used to access the value. + */ +export async function findGeneratedReference(applicableBindings) { + // We can adjust this number as we go, but these are a decent start as a + // general heuristic to assume the bindings were bad or just map a chunk of + // whole line or something. + if (applicableBindings.length > 4) { + // Babel's for..of generates at least 3 bindings inside one range for + // block-scoped loop variables, so we shouldn't go below that. + applicableBindings = []; + } + + for (const applicable of applicableBindings) { + const result = await mapBindingReferenceToDescriptor(applicable); + if (result) { + return result; + } + } + return null; +} + +export async function findGeneratedImportReference(applicableBindings) { + // When wrapped, for instance as `Object(ns.default)`, the `Object` binding + // will be the first in the list. To avoid resolving `Object` as the + // value of the import itself, we potentially skip the first binding. + applicableBindings = applicableBindings.filter((applicable, i) => { + if ( + !applicable.firstInRange || + applicable.binding.loc.type !== "ref" || + applicable.binding.loc.meta + ) { + return true; + } + + const next = + i + 1 < applicableBindings.length ? applicableBindings[i + 1] : null; + + return !next || next.binding.loc.type !== "ref" || !next.binding.loc.meta; + }); + + // We can adjust this number as we go, but these are a decent start as a + // general heuristic to assume the bindings were bad or just map a chunk of + // whole line or something. + if (applicableBindings.length > 2) { + // Babel's for..of generates at least 3 bindings inside one range for + // block-scoped loop variables, so we shouldn't go below that. + applicableBindings = []; + } + + for (const applicable of applicableBindings) { + const result = await mapImportReferenceToDescriptor(applicable); + if (result) { + return result; + } + } + + return null; +} + +/** + * Given a mapped range over the generated source and the name of the imported + * value that is referenced, attempt to resolve a binding descriptor for + * the import's value. + */ +export async function findGeneratedImportDeclaration( + applicableBindings, + importName +) { + // We can adjust this number as we go, but these are a decent start as a + // general heuristic to assume the bindings were bad or just map a chunk of + // whole line or something. + if (applicableBindings.length > 10) { + // Import declarations tend to have a large number of bindings for + // for things like 'require' and 'interop', so this number is larger + // than other binding count checks. + applicableBindings = []; + } + + let result = null; + + for (const { binding } of applicableBindings) { + if (binding.loc.type === "ref") { + continue; + } + + const namespaceDesc = await binding.desc(); + if (isPrimitiveValue(namespaceDesc)) { + continue; + } + if (!isObjectValue(namespaceDesc)) { + // We want to handle cases like + // + // var _mod = require(...); + // var _mod2 = _interopRequire(_mod); + // + // where "_mod" is optimized out because it is only referenced once. To + // allow that, we track the optimized-out value as a possible result, + // but allow later binding values to overwrite the result. + result = { + name: binding.name, + desc: namespaceDesc, + expression: binding.name, + }; + continue; + } + + const desc = await readDescriptorProperty(namespaceDesc, importName); + const expression = `${binding.name}.${importName}`; + + if (desc) { + result = { + name: binding.name, + desc, + expression, + }; + break; + } + } + + return result; +} + +/** + * Given a generated binding, and a range over the generated code, statically + * check if the given binding matches the range. + */ +async function mapBindingReferenceToDescriptor({ + binding, + range, + firstInRange, + firstOnLine, +}) { + // Allow the mapping to point anywhere within the generated binding + // location to allow for less than perfect sourcemaps. Since you also + // need at least one character between identifiers, we also give one + // characters of space at the front the generated binding in order + // to increase the probability of finding the right mapping. + if ( + range.start.line === binding.loc.start.line && + // If a binding is the first on a line, Babel will extend the mapping to + // include the whitespace between the newline and the binding. To handle + // that, we skip the range requirement for starting location. + (firstInRange || + firstOnLine || + locColumn(range.start) >= locColumn(binding.loc.start)) && + locColumn(range.start) <= locColumn(binding.loc.end) + ) { + return { + name: binding.name, + desc: await binding.desc(), + expression: binding.name, + }; + } + + return null; +} + +/** + * Given an generated binding, and a range over the generated code, statically + * evaluate accessed properties within the mapped range to resolve the actual + * imported value. + */ +async function mapImportReferenceToDescriptor({ binding, range }) { + if (binding.loc.type !== "ref") { + return null; + } + + // Expression matches require broader searching because sourcemaps usage + // varies in how they map certain things. For instance given + // + // import { bar } from "mod"; + // bar(); + // + // The "bar()" expression is generally expanded into one of two possibly + // forms, both of which map the "bar" identifier in different ways. See + // the "^^" markers below for the ranges. + // + // (0, foo.bar)() // Babel + // ^^^^^^^ // mapping + // ^^^ // binding + // vs + // + // __webpack_require__.i(foo.bar)() // Webpack 2 + // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ // mapping + // ^^^ // binding + // vs + // + // Object(foo.bar)() // Webpack >= 3 + // ^^^^^^^^^^^^^^^ // mapping + // ^^^ // binding + // + // Unfortunately, Webpack also has a tendancy to over-map past the call + // expression to the start of the next line, at least when there isn't + // anything else on that line that is mapped, e.g. + // + // Object(foo.bar)() + // ^^^^^^^^^^^^^^^^^ + // ^ // wrapped to column 0 of next line + + if (!mappingContains(range, binding.loc)) { + return null; + } + + // Webpack 2's import declarations wrap calls with an identity fn, so we + // need to make sure to skip that binding because it is mapped to the + // location of the original binding usage. + if ( + binding.name === "__webpack_require__" && + binding.loc.meta && + binding.loc.meta.type === "member" && + binding.loc.meta.property === "i" + ) { + return null; + } + + let expression = binding.name; + let desc = await binding.desc(); + + if (binding.loc.type === "ref") { + const { meta } = binding.loc; + + // Limit to 2 simple property or inherits operartions, since it would + // just be more work to search more and it is very unlikely that + // bindings would be mapped to more than a single member + inherits + // wrapper. + for ( + let op = meta, index = 0; + op && mappingContains(range, op) && desc && index < 2; + index++, op = op?.parent + ) { + // Calling could potentially trigger side-effects, which would not + // be ideal for this case. + if (op.type === "call") { + return null; + } + + if (op.type === "inherit") { + continue; + } + + desc = await readDescriptorProperty(desc, op.property); + expression += `.${op.property}`; + } + } + + return desc + ? { + name: binding.name, + desc, + expression, + } + : null; +} + +function isPrimitiveValue(desc) { + return desc && (!desc.value || typeof desc.value !== "object"); +} +function isObjectValue(desc) { + return ( + desc && + !isPrimitiveValue(desc) && + desc.value.type === "object" && + // Note: The check for `.type` might already cover the optimizedOut case + // but not 100% sure, so just being cautious. + !desc.value.optimizedOut + ); +} + +async function readDescriptorProperty(desc, property) { + if (!desc) { + return null; + } + + if (typeof desc.value !== "object" || !desc.value) { + // If accessing a property on a primitive type, just return 'undefined' + // as the value. + return { + value: { + type: "undefined", + }, + }; + } + + if (!isObjectValue(desc)) { + // If we got a non-primitive descriptor but it isn't an object, then + // it's definitely not the namespace and it is probably an error. + return desc; + } + + const objectFront = clientCommands.createObjectFront(desc.value); + return (await objectFront.getProperty(property)).descriptor; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/getApplicableBindingsForOriginalPosition.js b/devtools/client/debugger/src/utils/pause/mapScopes/getApplicableBindingsForOriginalPosition.js new file mode 100644 index 0000000000..c1a64ddfa0 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/getApplicableBindingsForOriginalPosition.js @@ -0,0 +1,112 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { positionCmp } from "./positionCmp"; +import { filterSortedArray } from "./filtering"; +import { mappingContains } from "./mappingContains"; +import { getGeneratedLocation } from "../../source-maps"; + +export async function originalRangeStartsInside({ start, end }, thunkArgs) { + const endPosition = await getGeneratedLocation(end, thunkArgs); + const startPosition = await getGeneratedLocation(start, thunkArgs); + + // If the start and end positions collapse into eachother, it means that + // the range in the original content didn't _start_ at the start position. + // Since this likely means that the range doesn't logically apply to this + // binding location, we skip it. + return positionCmp(startPosition, endPosition) !== 0; +} + +export async function getApplicableBindingsForOriginalPosition( + generatedAstBindings, + source, + { start, end }, + bindingType, + locationType, + thunkArgs +) { + const { sourceMapLoader } = thunkArgs; + const ranges = await sourceMapLoader.getGeneratedRanges(start); + + const resultRanges = ranges.map(mapRange => ({ + start: { + line: mapRange.line, + column: mapRange.columnStart, + }, + end: { + line: mapRange.line, + // SourceMapConsumer's 'lastColumn' is inclusive, so we add 1 to make + // it exclusive like all other locations. + column: mapRange.columnEnd + 1, + }, + })); + + // When searching for imports, we expand the range to up to the next available + // mapping to allow for import declarations that are composed of multiple + // variable statements, where the later ones are entirely unmapped. + // Babel 6 produces imports in this style, e.g. + // + // var _mod = require("mod"); // mapped from import statement + // var _mod2 = interop(_mod); // entirely unmapped + if (bindingType === "import" && locationType !== "ref") { + const endPosition = await getGeneratedLocation(end, thunkArgs); + const startPosition = await getGeneratedLocation(start, thunkArgs); + + for (const range of resultRanges) { + if ( + mappingContains(range, { start: startPosition, end: startPosition }) && + positionCmp(range.end, endPosition) < 0 + ) { + range.end = { + line: endPosition.line, + column: endPosition.column, + }; + break; + } + } + } + + return filterApplicableBindings(generatedAstBindings, resultRanges); +} + +function filterApplicableBindings(bindings, ranges) { + const result = []; + for (const range of ranges) { + // Any binding overlapping a part of the mapping range. + const filteredBindings = filterSortedArray(bindings, binding => { + if (positionCmp(binding.loc.end, range.start) <= 0) { + return -1; + } + if (positionCmp(binding.loc.start, range.end) >= 0) { + return 1; + } + + return 0; + }); + + let firstInRange = true; + let firstOnLine = true; + let line = -1; + + for (const binding of filteredBindings) { + if (binding.loc.start.line === line) { + firstOnLine = false; + } else { + line = binding.loc.start.line; + firstOnLine = true; + } + + result.push({ + binding, + range, + firstOnLine, + firstInRange, + }); + + firstInRange = false; + } + } + + return result; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/index.js b/devtools/client/debugger/src/utils/pause/mapScopes/index.js new file mode 100644 index 0000000000..8736c42218 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/index.js @@ -0,0 +1,583 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { + debuggerToSourceMapLocation, + sourceMapToDebuggerLocation, +} from "../../location"; +import { locColumn } from "./locColumn"; +import { loadRangeMetadata, findMatchingRange } from "./rangeMetadata"; + +// eslint-disable-next-line max-len +import { + findGeneratedReference, + findGeneratedImportReference, + findGeneratedImportDeclaration, +} from "./findGeneratedBindingFromPosition"; +import { + buildGeneratedBindingList, + buildFakeBindingList, +} from "./buildGeneratedBindingList"; +import { + originalRangeStartsInside, + getApplicableBindingsForOriginalPosition, +} from "./getApplicableBindingsForOriginalPosition"; +import { getOptimizedOutGrip } from "./optimizedOut"; + +import { log } from "../../log"; + +// Create real location objects for all location start and end. +// +// Parser worker returns scopes with location having a sourceId +// instead of a source object as it doesn't know about main thread source objects. +function updateLocationsInScopes(state, scopes) { + for (const item of scopes) { + for (const name of Object.keys(item.bindings)) { + for (const ref of item.bindings[name].refs) { + const locs = [ref]; + if (ref.type !== "ref") { + locs.push(ref.declaration); + } + for (const loc of locs) { + loc.start = sourceMapToDebuggerLocation(state, loc.start); + loc.end = sourceMapToDebuggerLocation(state, loc.end); + } + } + } + } +} + +export async function buildMappedScopes( + source, + content, + frame, + scopes, + thunkArgs +) { + const { getState, parserWorker } = thunkArgs; + if (!parserWorker.isLocationSupported(frame.location)) { + return null; + } + const originalAstScopes = await parserWorker.getScopes(frame.location); + updateLocationsInScopes(getState(), originalAstScopes); + const generatedAstScopes = await parserWorker.getScopes( + frame.generatedLocation + ); + updateLocationsInScopes(getState(), generatedAstScopes); + + if (!originalAstScopes || !generatedAstScopes) { + return null; + } + + const originalRanges = await loadRangeMetadata( + frame.location, + originalAstScopes, + thunkArgs + ); + + if (hasLineMappings(originalRanges)) { + return null; + } + + let generatedAstBindings; + if (scopes) { + generatedAstBindings = buildGeneratedBindingList( + scopes, + generatedAstScopes, + frame.this + ); + } else { + generatedAstBindings = buildFakeBindingList(generatedAstScopes); + } + + const { mappedOriginalScopes, expressionLookup } = + await mapOriginalBindingsToGenerated( + source, + content, + originalRanges, + originalAstScopes, + generatedAstBindings, + thunkArgs + ); + + const globalLexicalScope = scopes + ? getGlobalFromScope(scopes) + : generateGlobalFromAst(generatedAstScopes); + const mappedGeneratedScopes = generateClientScope( + globalLexicalScope, + mappedOriginalScopes + ); + + return isReliableScope(mappedGeneratedScopes) + ? { mappings: expressionLookup, scope: mappedGeneratedScopes } + : null; +} + +async function mapOriginalBindingsToGenerated( + source, + content, + originalRanges, + originalAstScopes, + generatedAstBindings, + thunkArgs +) { + const expressionLookup = {}; + const mappedOriginalScopes = []; + + const cachedSourceMaps = batchScopeMappings( + originalAstScopes, + source, + thunkArgs + ); + // Override sourceMapLoader attribute with the special cached SourceMapLoader instance + // in order to make it used by all functions used in this method. + thunkArgs = { ...thunkArgs, sourceMapLoader: cachedSourceMaps }; + + for (const item of originalAstScopes) { + const generatedBindings = {}; + + for (const name of Object.keys(item.bindings)) { + const binding = item.bindings[name]; + + const result = await findGeneratedBinding( + source, + content, + name, + binding, + originalRanges, + generatedAstBindings, + thunkArgs + ); + + if (result) { + generatedBindings[name] = result.grip; + + if ( + binding.refs.length !== 0 && + // These are assigned depth-first, so we don't want shadowed + // bindings in parent scopes overwriting the expression. + !Object.prototype.hasOwnProperty.call(expressionLookup, name) + ) { + expressionLookup[name] = result.expression; + } + } + } + + mappedOriginalScopes.push({ + ...item, + generatedBindings, + }); + } + + return { + mappedOriginalScopes, + expressionLookup, + }; +} + +/** + * Consider a scope and its parents reliable if the vast majority of its + * bindings were successfully mapped to generated scope bindings. + */ +function isReliableScope(scope) { + let totalBindings = 0; + let unknownBindings = 0; + + for (let s = scope; s; s = s.parent) { + const vars = s.bindings?.variables || {}; + for (const key of Object.keys(vars)) { + const binding = vars[key]; + + totalBindings += 1; + if ( + binding.value && + typeof binding.value === "object" && + (binding.value.type === "unscoped" || binding.value.type === "unmapped") + ) { + unknownBindings += 1; + } + } + } + + // As determined by fair dice roll. + return totalBindings === 0 || unknownBindings / totalBindings < 0.25; +} + +function hasLineMappings(ranges) { + return ranges.every( + range => range.columnStart === 0 && range.columnEnd === Infinity + ); +} + +/** + * Build a special SourceMapLoader instance, based on the one passed in thunkArgs, + * which will both: + * - preload generated ranges/locations for original locations mentioned + * in originalAstScopes + * - cache the requests to fetch these genereated ranges/locations + */ +function batchScopeMappings(originalAstScopes, source, thunkArgs) { + const { sourceMapLoader } = thunkArgs; + const precalculatedRanges = new Map(); + const precalculatedLocations = new Map(); + + // Explicitly dispatch all of the sourcemap requests synchronously up front so + // that they will be batched into a single request for the worker to process. + for (const item of originalAstScopes) { + for (const name of Object.keys(item.bindings)) { + for (const ref of item.bindings[name].refs) { + const locs = [ref]; + if (ref.type !== "ref") { + locs.push(ref.declaration); + } + + for (const loc of locs) { + precalculatedRanges.set( + buildLocationKey(loc.start), + sourceMapLoader.getGeneratedRanges( + debuggerToSourceMapLocation(loc.start) + ) + ); + precalculatedLocations.set( + buildLocationKey(loc.start), + sourceMapLoader.getGeneratedLocation( + debuggerToSourceMapLocation(loc.start) + ) + ); + precalculatedLocations.set( + buildLocationKey(loc.end), + sourceMapLoader.getGeneratedLocation( + debuggerToSourceMapLocation(loc.end) + ) + ); + } + } + } + } + + return { + async getGeneratedRanges(pos) { + const key = buildLocationKey(pos); + + if (!precalculatedRanges.has(key)) { + log("Bad precalculated mapping"); + return sourceMapLoader.getGeneratedRanges( + debuggerToSourceMapLocation(pos) + ); + } + return precalculatedRanges.get(key); + }, + + async getGeneratedLocation(pos) { + const key = buildLocationKey(pos); + + if (!precalculatedLocations.has(key)) { + log("Bad precalculated mapping"); + return sourceMapLoader.getGeneratedLocation( + debuggerToSourceMapLocation(pos) + ); + } + return precalculatedLocations.get(key); + }, + }; +} +function buildLocationKey(loc) { + return `${loc.line}:${locColumn(loc)}`; +} + +function generateClientScope(globalLexicalScope, originalScopes) { + // Build a structure similar to the client's linked scope object using + // the original AST scopes, but pulling in the generated bindings + // linked to each scope. + const result = originalScopes + .slice(0, -2) + .reverse() + .reduce((acc, orig, i) => { + const { + // The 'this' binding data we have is handled independently, so + // the binding data is not included here. + // eslint-disable-next-line no-unused-vars + this: _this, + ...variables + } = orig.generatedBindings; + + return { + parent: acc, + actor: `originalActor${i}`, + type: orig.type, + scopeKind: orig.scopeKind, + bindings: { + arguments: [], + variables, + }, + ...(orig.type === "function" + ? { + function: { + displayName: orig.displayName, + }, + } + : null), + ...(orig.type === "block" + ? { + block: { + displayName: orig.displayName, + }, + } + : null), + }; + }, globalLexicalScope); + + // The rendering logic in getScope 'this' bindings only runs on the current + // selected frame scope, so we pluck out the 'this' binding that was mapped, + // and put it in a special location + const thisScope = originalScopes.find(scope => scope.bindings.this); + if (result.bindings && thisScope) { + result.bindings.this = thisScope.generatedBindings.this || null; + } + + return result; +} + +function getGlobalFromScope(scopes) { + // Pull the root object scope and root lexical scope to reuse them in + // our mapped scopes. This assumes that file being processed is + // a CommonJS or ES6 module, which might not be ideal. Potentially + // should add some logic to try to detect those cases? + let globalLexicalScope = null; + for (let s = scopes; s.parent; s = s.parent) { + globalLexicalScope = s; + } + if (!globalLexicalScope) { + throw new Error("Assertion failure - there should always be a scope"); + } + return globalLexicalScope; +} + +function generateGlobalFromAst(generatedScopes) { + const globalLexicalAst = generatedScopes[generatedScopes.length - 2]; + if (!globalLexicalAst) { + throw new Error("Assertion failure - there should always be a scope"); + } + return { + actor: "generatedActor1", + type: "block", + scopeKind: "", + bindings: { + arguments: [], + variables: Object.fromEntries( + Object.keys(globalLexicalAst).map(key => [key, getOptimizedOutGrip()]) + ), + }, + parent: { + actor: "generatedActor0", + object: getOptimizedOutGrip(), + scopeKind: "", + type: "object", + }, + }; +} + +function hasValidIdent(range, pos) { + return ( + range.type === "match" || + // For declarations, we allow the range on the identifier to be a + // more general "contains" to increase the chances of a match. + (pos.type !== "ref" && range.type === "contains") + ); +} + +// eslint-disable-next-line complexity +async function findGeneratedBinding( + source, + content, + name, + originalBinding, + originalRanges, + generatedAstBindings, + thunkArgs +) { + // If there are no references to the implicits, then we have no way to + // even attempt to map it back to the original since there is no location + // data to use. Bail out instead of just showing it as unmapped. + if ( + originalBinding.type === "implicit" && + !originalBinding.refs.some(item => item.type === "ref") + ) { + return null; + } + + const loadApplicableBindings = async (pos, locationType) => { + let applicableBindings = await getApplicableBindingsForOriginalPosition( + generatedAstBindings, + source, + pos, + originalBinding.type, + locationType, + thunkArgs + ); + if (applicableBindings.length) { + hadApplicableBindings = true; + } + if (locationType === "ref") { + // Some tooling creates ranges that map a line as a whole, which is useful + // for step-debugging, but can easily lead to finding the wrong binding. + // To avoid these false-positives, we entirely ignore bindings matched + // by ranges that cover full lines. + applicableBindings = applicableBindings.filter( + ({ range }) => + !(range.start.column === 0 && range.end.column === Infinity) + ); + } + if ( + locationType !== "ref" && + !(await originalRangeStartsInside(pos, thunkArgs)) + ) { + applicableBindings = []; + } + return applicableBindings; + }; + + const { refs } = originalBinding; + + let hadApplicableBindings = false; + let genContent = null; + for (const pos of refs) { + const applicableBindings = await loadApplicableBindings(pos, pos.type); + + const range = findMatchingRange(originalRanges, pos); + if (range && hasValidIdent(range, pos)) { + if (originalBinding.type === "import") { + genContent = await findGeneratedImportReference(applicableBindings); + } else { + genContent = await findGeneratedReference(applicableBindings); + } + } + + if ( + (pos.type === "class-decl" || pos.type === "class-inner") && + content.contentType && + content.contentType.match(/\/typescript/) + ) { + const declRange = findMatchingRange(originalRanges, pos.declaration); + if (declRange && declRange.type !== "multiple") { + const applicableDeclBindings = await loadApplicableBindings( + pos.declaration, + pos.type + ); + + // Resolve to first binding in the range + const declContent = await findGeneratedReference( + applicableDeclBindings + ); + + if (declContent) { + // Prefer the declaration mapping in this case because TS sometimes + // maps class declaration names to "export.Foo = Foo;" or to + // the decorator logic itself + genContent = declContent; + } + } + } + + if ( + !genContent && + pos.type === "import-decl" && + typeof pos.importName === "string" + ) { + const { importName } = pos; + const declRange = findMatchingRange(originalRanges, pos.declaration); + + // The import declaration should have an original position mapping, + // but otherwise we don't really have preferences on the range type + // because it can have multiple bindings, but we do want to make sure + // that all of the bindings that match the range are part of the same + // import declaration. + if (declRange?.singleDeclaration) { + const applicableDeclBindings = await loadApplicableBindings( + pos.declaration, + pos.type + ); + + // match the import declaration location + genContent = await findGeneratedImportDeclaration( + applicableDeclBindings, + importName + ); + } + } + + if (genContent) { + break; + } + } + + if (genContent && genContent.desc) { + return { + grip: genContent.desc, + expression: genContent.expression, + }; + } else if (genContent) { + // If there is no descriptor for 'this', then this is not the top-level + // 'this' that the server gave us a binding for, and we can just ignore it. + if (name === "this") { + return null; + } + + // If the location is found but the descriptor is not, then it + // means that the server scope information didn't match the scope + // information from the DevTools parsed scopes. + return { + grip: { + configurable: false, + enumerable: true, + writable: false, + value: { + type: "unscoped", + unscoped: true, + + // HACK: Until support for "unscoped" lands in devtools-reps, + // this will make these show as (unavailable). + missingArguments: true, + }, + }, + expression: null, + }; + } else if (!hadApplicableBindings && name !== "this") { + // If there were no applicable bindings to consider while searching for + // matching bindings, then the source map for this file didn't make any + // attempt to map the binding, and that most likely means that the + // code was entirely emitted from the output code. + return { + grip: getOptimizedOutGrip(), + expression: ` + (() => { + throw new Error('"' + ${JSON.stringify( + name + )} + '" has been optimized out.'); + })() + `, + }; + } + + // If no location mapping is found, then the map is bad, or + // the map is okay but it original location is inside + // of some scope, but the generated location is outside, leading + // us to search for bindings that don't technically exist. + return { + grip: { + configurable: false, + enumerable: true, + writable: false, + value: { + type: "unmapped", + unmapped: true, + + // HACK: Until support for "unmapped" lands in devtools-reps, + // this will make these show as (unavailable). + missingArguments: true, + }, + }, + expression: null, + }; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/locColumn.js b/devtools/client/debugger/src/utils/pause/mapScopes/locColumn.js new file mode 100644 index 0000000000..075e902299 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/locColumn.js @@ -0,0 +1,13 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +export function locColumn(loc) { + if (typeof loc.column !== "number") { + // This shouldn't really happen with locations from the AST, but + // the datatype we are using allows null/undefined column. + return 0; + } + + return loc.column; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/mappingContains.js b/devtools/client/debugger/src/utils/pause/mapScopes/mappingContains.js new file mode 100644 index 0000000000..ca82fe42ec --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/mappingContains.js @@ -0,0 +1,12 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { positionCmp } from "./positionCmp"; + +export function mappingContains(mapped, item) { + return ( + positionCmp(item.start, mapped.start) >= 0 && + positionCmp(item.end, mapped.end) <= 0 + ); +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/moz.build b/devtools/client/debugger/src/utils/pause/mapScopes/moz.build new file mode 100644 index 0000000000..05f2b7e3d8 --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/moz.build @@ -0,0 +1,19 @@ +# vim: set filetype=python: +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. + +DIRS += [] + +CompiledModules( + "buildGeneratedBindingList.js", + "filtering.js", + "findGeneratedBindingFromPosition.js", + "getApplicableBindingsForOriginalPosition.js", + "index.js", + "locColumn.js", + "mappingContains.js", + "optimizedOut.js", + "positionCmp.js", + "rangeMetadata.js", +) diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/optimizedOut.js b/devtools/client/debugger/src/utils/pause/mapScopes/optimizedOut.js new file mode 100644 index 0000000000..755b308a2d --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/optimizedOut.js @@ -0,0 +1,15 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +export function getOptimizedOutGrip() { + return { + configurable: false, + enumerable: true, + writable: false, + value: { + type: "null", + optimizedOut: true, + }, + }; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/positionCmp.js b/devtools/client/debugger/src/utils/pause/mapScopes/positionCmp.js new file mode 100644 index 0000000000..5d53fb933e --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/positionCmp.js @@ -0,0 +1,24 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { locColumn } from "./locColumn"; + +/** + * * === 0 - Positions are equal. + * * < 0 - first position before second position + * * > 0 - first position after second position + */ +export function positionCmp(p1, p2) { + if (p1.line === p2.line) { + const l1 = locColumn(p1); + const l2 = locColumn(p2); + + if (l1 === l2) { + return 0; + } + return l1 < l2 ? -1 : 1; + } + + return p1.line < p2.line ? -1 : 1; +} diff --git a/devtools/client/debugger/src/utils/pause/mapScopes/rangeMetadata.js b/devtools/client/debugger/src/utils/pause/mapScopes/rangeMetadata.js new file mode 100644 index 0000000000..7a22313aca --- /dev/null +++ b/devtools/client/debugger/src/utils/pause/mapScopes/rangeMetadata.js @@ -0,0 +1,117 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at <http://mozilla.org/MPL/2.0/>. */ + +import { locColumn } from "./locColumn"; +import { positionCmp } from "./positionCmp"; +import { filterSortedArray } from "./filtering"; + +// * match - Range contains a single identifier with matching start location +// * contains - Range contains a single identifier with non-matching start +// * multiple - Range contains multiple identifiers +// * empty - Range contains no identifiers + +export async function loadRangeMetadata( + location, + originalAstScopes, + { sourceMapLoader } +) { + const originalRanges = await sourceMapLoader.getOriginalRanges( + location.sourceId + ); + + const sortedOriginalAstBindings = []; + for (const item of originalAstScopes) { + for (const name of Object.keys(item.bindings)) { + for (const ref of item.bindings[name].refs) { + sortedOriginalAstBindings.push(ref); + } + } + } + sortedOriginalAstBindings.sort((a, b) => positionCmp(a.start, b.start)); + + let i = 0; + + return originalRanges.map(range => { + const bindings = []; + + while ( + i < sortedOriginalAstBindings.length && + (sortedOriginalAstBindings[i].start.line < range.line || + (sortedOriginalAstBindings[i].start.line === range.line && + locColumn(sortedOriginalAstBindings[i].start) < range.columnStart)) + ) { + i++; + } + + while ( + i < sortedOriginalAstBindings.length && + sortedOriginalAstBindings[i].start.line === range.line && + locColumn(sortedOriginalAstBindings[i].start) >= range.columnStart && + locColumn(sortedOriginalAstBindings[i].start) < range.columnEnd + ) { + const lastBinding = bindings[bindings.length - 1]; + // Only add bindings when they're in new positions + if ( + !lastBinding || + positionCmp(lastBinding.start, sortedOriginalAstBindings[i].start) !== 0 + ) { + bindings.push(sortedOriginalAstBindings[i]); + } + i++; + } + + let type = "empty"; + let singleDeclaration = true; + if (bindings.length === 1) { + const binding = bindings[0]; + if ( + binding.start.line === range.line && + binding.start.column === range.columnStart + ) { + type = "match"; + } else { + type = "contains"; + } + } else if (bindings.length > 1) { + type = "multiple"; + const binding = bindings[0]; + const declStart = + binding.type !== "ref" ? binding.declaration.start : null; + + singleDeclaration = bindings.every(b => { + return ( + declStart && + b.type !== "ref" && + positionCmp(declStart, b.declaration.start) === 0 + ); + }); + } + + return { + type, + singleDeclaration, + ...range, + }; + }); +} + +export function findMatchingRange(sortedOriginalRanges, bindingRange) { + return filterSortedArray(sortedOriginalRanges, range => { + if (range.line < bindingRange.start.line) { + return -1; + } + if (range.line > bindingRange.start.line) { + return 1; + } + + if (range.columnEnd <= locColumn(bindingRange.start)) { + return -1; + } + if (range.columnStart > locColumn(bindingRange.start)) { + return 1; + } + + return 0; + }).pop(); +} |