diff options
Diffstat (limited to 'gfx/docs/GraphicsOverview.rst')
| -rw-r--r-- | gfx/docs/GraphicsOverview.rst | 149 |
1 files changed, 149 insertions, 0 deletions
diff --git a/gfx/docs/GraphicsOverview.rst b/gfx/docs/GraphicsOverview.rst new file mode 100644 index 0000000000..77b0379743 --- /dev/null +++ b/gfx/docs/GraphicsOverview.rst @@ -0,0 +1,149 @@ +Graphics Overview +========================= + +Work in progress. Possibly incorrect or incomplete. +--------------------------------------------------- + +Jargon +------ + +There's a lot of jargon in the graphics stack. We try to maintain a list +of common words and acronyms `here <https://wiki.mozilla.org/Platform/GFX/Jargon>`__. + +Overview +-------- + +The graphics systems is responsible for rendering (painting, drawing) +the frame tree (rendering tree) elements as created by the layout +system. Each leaf in the tree has content, either bounded by a rectangle +(or perhaps another shape, in the case of SVG.) + +The simple approach for producing the result would thus involve +traversing the frame tree, in a correct order, drawing each frame into +the resulting buffer and displaying (printing non-withstanding) that +buffer when the traversal is done. It is worth spending some time on the +“correct order” note above. If there are no overlapping frames, this is +fairly simple - any order will do, as long as there is no background. If +there is background, we just have to worry about drawing that first. +Since we do not control the content, chances are the page is more +complicated. There are overlapping frames, likely with transparency, so +we need to make sure the elements are draw “back to front”, in layers, +so to speak. Layers are an important concept, and we will revisit them +shortly, as they are central to fixing a major issue with the above +simple approach. + +While the above simple approach will work, the performance will suffer. +Each time anything changes in any of the frames, the complete process +needs to be repeated, everything needs to be redrawn. Further, there is +very little space to take advantage of the modern graphics (GPU) +hardware, or multi-core computers. If you recall from the previous +sections, the frame tree is only accessible from the UI thread, so while +we’re doing all this work, the UI is basically blocked. + +(Retained) Layers +~~~~~~~~~~~~~~~~~ + +Layers framework was introduced to address the above performance issues, +by having a part of the design address each item. At the high level: + +1. We create a layer tree. The leaf elements of the tree contain all + frames (possibly multiple frames per leaf). +2. We render each layer tree element and cache (retain) the result. +3. We composite (combine) all the leaf elements into the final result. + +Let’s examine each of these steps, in reverse order. + +Compositing +~~~~~~~~~~~ + +We use the term composite as it implies that the order is important. If +the elements being composited overlap, whether there is transparency +involved or not, the order in which they are combined will effect the +result. Compositing is where we can use some of the power of the modern +graphics hardware. It is optimal for doing this job. In the scenarios +where only the position of individual frames changes, without the +content inside them changing, we see why caching each layer would be +advantageous - we only need to repeat the final compositing step, +completely skipping the layer tree creation and the rendering of each +leaf, thus speeding up the process considerably. + +Another benefit is equally apparent in the context of the stated +deficiencies of the simple approach. We can use the available graphics +hardware accelerated APIs to do the compositing step. Direct3D, OpenGL +can be used on different platforms and are well suited to accelerate +this step. + +Finally, we can now envision performing the compositing step on a +separate thread, unblocking the UI thread for other work, and doing more +work in parallel. More on this below. + +It is important to note that the number of operations in this step is +proportional to the number of layer tree (leaf) elements, so there is +additional work and complexity involved, when the layer tree is large. + +Render and retain layer elements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As we saw, the compositing step benefits from caching the intermediate +result. This does result in the extra memory usage, so needs to be +considered during the layer tree creation. Beyond the caching, we can +accelerate the rendering of each element by (indirectly) using the +available platform APIs (e.g., Direct2D, CoreGraphics, even some of the +3D APIs like OpenGL or Direct3D) as available. This is actually done +through a platform independent API (see Moz2D) below, but is important +to realize it does get accelerated appropriately. + +Creating the layer tree +~~~~~~~~~~~~~~~~~~~~~~~ + +We need to create a layer tree (from the frames tree), which will give +us the correct result while striking the right balance between a layer +per frame element and a single layer for the complete frames tree. As +was mentioned above, there is an overhead in traversing the whole tree +and caching each of the elements, balanced by the performance +improvements. Some of the performance improvements are only noticed when +something changes (e.g., one element is moving, we only need to redo the +compositing step). + +Refresh Driver +~~~~~~~~~~~~~~ + +Layers +~~~~~~ + +Rendering each layer +~~~~~~~~~~~~~~~~~~~~ + +Tiling vs. Buffer Rotation vs. Full paint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Compositing for the final result +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Graphics API +~~~~~~~~~~~~ + +Compositing +~~~~~~~~~~~ + +Image Decoding +~~~~~~~~~~~~~~ + +Image Animation +~~~~~~~~~~~~~~~ + +`Historical Documents <http://www.youtube.com/watch?v=lLZQz26-kms>`__ +--------------------------------------------------------------------- + +A number of posts and blogs that will give you more details or more +background, or reasoning that led to different solutions and approaches. + +- 2010-01 `Layers: Cross Platform Acceleration <http://www.basschouten.com/blog1.php/layers-cross-platform-acceleration>`__ +- 2010-04 `Layers <http://robert.ocallahan.org/2010/04/layers_01.html>`__ +- 2010-07 `Retained Layers <http://robert.ocallahan.org/2010/07/retained-layers_16.html>`__ +- 2011-04 `Introduction <https://web.archive.org/web/20140604005804/https://blog.mozilla.org/joe/2011/04/26/introducing-the-azure-project/>`__ +- 2011-07 `Layers <http://chrislord.net/index.php/2011/07/25/shadow-layers-and-learning-by-failing/%20Shadow>`__ +- 2011-09 `Graphics API Design <http://robert.ocallahan.org/2011/09/graphics-api-design.html>`__ +- 2012-04 `Moz2D Canvas on OSX <http://muizelaar.blogspot.ca/2012/04/azure-canvas-on-os-x.html>`__ +- 2012-05 `Mask Layers <http://featherweightmusings.blogspot.co.uk/2012/05/mask-layers_26.html>`__ +- 2013-07 `Graphics related <http://www.basschouten.com/blog1.php>`__ |