Scribd
Upload
145 views

Life of A Pixel

This document provides an overview of the life of a pixel, beginning with rendering web content into pixels. It discusses how content such as HTML, CSS, JavaScript, and images are parsed and used to build DOM and style tree data structures. Styles are calculated and layout is performed to place elements spatially before pixels are generated. It describes stages like parsing, DOM construction, styling, layout, and rendering in 3 sentences or less.

Uploaded by

sriharsh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
145 views

Life of A Pixel

This document provides an overview of the life of a pixel, beginning with rendering web content into pixels. It discusses how content such as HTML, CSS, JavaScript, and images are parsed and used to build DOM and style tree data structures. Styles are calculated and layout is performed to place elements spatially before pixels are generated. It describes stages like parsing, DOM construction, styling, layout, and rendering in 3 sentences or less.

Uploaded by

sriharsh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 80

LIFE OF A

Steve Kobes
[email protected]

Nov 2020

slides: bit.ly/lifeofapixel

with special thanks to past presenters "The unexamined pixel is not worth rendering."
dknox@, chrishtr@, pdr@ — Socrates
LIFE OF A

88.0.4307.0
(Nov 2020)
LIFE OF A

content
rendering pixels
content
NOT "content"

content::WebContents

sandboxed renderer process

Blink
cc
content

● HTML (Hyper-Text Markup Language) <p> hello </p>

● CSS (Cascading Style Sheets) p { color: red }

● JS (JavaScript) p.innerHTML = "goodbye";

● images <img src="kitten.gif">

(other kinds of rendered content: <video>, <canvas>, WebAssembly, WebGL, WebVR, PDF, ...)
content
https://www.nytimes.com
pixels
( )
( )
Operating System
( ?)
Device Driver
#include <GL/gl.h>

● textures
● shaders
● vertex buffers
● …
goals

1. render content into pixels

2. build data structures to enable updating the rendering efficiently

● JavaScript
up

up

up
d

d
● user input
ate

ate

ate

● asynchronous loading
initial render
● animations
● scrolling
● zooming
time
stages

lifecycle
pixels
content
<html>...</html> stage ? stage ? stage

intermediate data structures


parsing
HTMLDocumentParser HTMLTreeBuilder

Document (always the root)


<div>
HTML
<p> hello </p> (these tags are
<p> world </p> BODY optional)
</div>

bu
DIV

il
ds
P
● HTML tags can nest Text

P
● an object model reflects the
containment relationships Text
DOM

Document This is the


HTML
Document
Object
BODY Model.
DIV
The DOM
P
is a tree!
Text

P ● parents
● children
Text
● siblings
DOM
The DOM is both
Document
● Chrome's internal representation
HTML
AND
BODY ● the API exposed to JavaScript
DIV
var div = document.body.firstChild;
P
var p2 = div.childNodes[1];
Text p2.appendChild(document.createElement("span"));
p2.foo = {arbitrary: "state"};
P
[ JS ]
Text
[bindings] V8
DOM
<my-widget> customElements.define('my-widget',
<b>inside</b> class extends HTMLElement { constructor() { super();
</my-widget> this.attachShadow({mode: 'open'}).innerHTML =
"<wrapper><slot /></wrapper>"; }});

<my-widget> ShadowRoot
ShadowRoot()
<wrapper>

<b> <slot>
A custom element
AssignedNodes()
host tree has a shadow tree.
shadow tree
DOM

A flat tree traversal presents a composed view.

FlatTreeTraversal
<my-widget> ShadowRoot
ShadowRoot()
<wrapper> <my-widget>

<b> <slot> <wrapper>


AssignedNodes()
host tree <b>
shadow tree
style

Document /* every <p> has red text */


HTML p { color: red }
BODY

DIV

ts
P
ec
sel hello
Text world

Text
style
font-weight: bold; hello

margin-left: 2cm; hello

outline: dashed blue; hello

transform: rotate(20deg); hello

background: url(kitten.jpg); hello


style

/* every other <p> in any <div> without class="foo" */


div:not(.foo) > p:nth-of-type(2n) {
color: red !important;
}

p { ● selectors can be complex!


color: blue;
} ● declarations may conflict!
style
css_properties.json5
div:not(.foo) > p {
CSSParser color: red !important;
} make_css_property_subclasses.py
builds p {
margin: 0;
border-left-color: blue; border_left_color.cc
...
StyleSheetContents }
...
StyleRule

CSSPropertyValueSet
BorderLeftColor
CSSSelectorList
CSSPropertyValue CSSColorValue
CSSSelector Property()
Value() 0x0000FFFF
style
StyleSheetContents
StyleSheetContents
StyleSheetContents
Document::UpdateStyle

Document ComputedStyle StyleResolver::StyleForElement

HTML ComputedStyle

BODY ComputedStyle
ComputedStyle {
fontWeight = ...
marginLeft = ...
DIV ComputedStyle
outline = ...
transform = ...
P ComputedStyle background = ...
...
Text ComputedStyle
}
P ComputedStyle
Style recalc: for every node in the DOM tree,
Text ComputedStyle
compute the value of every style property.
style

getComputedStyle(element)["padding-top"] [ JS ]

(But: getComputedStyle returns some layout data also.)


layout

DIV

LayoutRect
x = 183
y = 148
width = 402
height = 116
layout

block flow
Simple "block" layout objects
are placed one after another,
flowing down the page.
layout

Text and "inline" elements


inline flow flow left-to-right, and are
broken into lines.

<span class="bigger">
#text
Lorem ipsum dolor sit amet, line break
‫اﻟﻌﺮﺑﯿﺔ‬
consectetur adipiscing elit. ‫ִעב ְִרית‬
(Some languages flow right-to-left.)
layout
Layout measures runs of text in fonts.

"fire"
Font

HarfBuzzShaper ShapeResult

HarfBuzz
fire
Shaping selects the glyphs and
computes their placement.
layout

The contents of a layout object


CSS border box rect
can overflow its border box.

IS layout overflow rect

AWESOME

Overflow can be visible,


hidden, or scrollable.
layout
Other kinds of layout are even more complex.

● <table>
● float: left
● column-count: 3
● display: flex
● writing-mode: vertical-lr
● ...
layout
DOM tree layout tree StyleEngine::RebuildLayoutTree

Document LayoutView LocalFrameView::UpdateLayout


HTML LayoutBlockFlow

BODY LayoutBlockFlow
Each layout pass
DIV LayoutBlockFlow
walks the layout tree,
P LayoutBlockFlow computing the visual
Text LayoutText geometry of each
LayoutObject.
P LayoutBlockFlow
layout
DOM nodes are not 1:1 with layout objects. <div>
ComputedStyle {
display: none
<div> block </div> LayoutBlock DIV }
<span> inline </span>
LayoutBlock (anonymous)
no LayoutObject!
LayoutInline SPAN
no DOM node!
<my-widget>

LayoutBlock MY-WIDGET ShadowRoot


The layout tree also
LayoutBlock DIV <div> crosses shadow roots.
layout
LayoutObject The layout tree has legacy and
NG layout objects.
LayoutNGMixin

LayoutNGMixin

LayoutObject “next generation”


layout engine!
LayoutObject
layout
LayoutObject 🔍

LayoutNGMixin 🔍

LayoutNGMixin

LayoutObject �� A legacy layout object holds inputs,


outputs, and algorithms of layout.
UpdateLayout()
It can see the state of the whole tree.

LayoutObject 🔍
layout
LayoutNG separates the
LayoutObject inputs and outputs of layout. NGConstraintSpace

LayoutNGMixin NGLayoutInputNode
✓
✗
UpdateLayout()

LayoutNGMixin NGLayoutInputNode
NGLayoutAlgorithm

LayoutObject
NGLayoutResult

LayoutObject immutable!
layout
LayoutObject

LayoutNGMixin NGPhysicalFragment
UpdateLayout() NGLayoutResult NGPhysicalFragment

NGPhysicalFragment
LayoutNGMixin

LayoutObject

A layout result has a tree of fragments.


LayoutObject
layout — example
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
layout — example
<div style="max-width: 100px">
<div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
<div style="margin: -60px 0 0 80px">jumps</div>
</div>
layout — example
DIV <div style="max-width: 100px">
DIV <div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
"F" <div style="margin: -60px 0 0 80px">jumps</div>
BR </div>
"The "
B
"quick brown"
" fox"
DIV
"jumps" DOM tree
layout — example
LayoutNGBlockFlow {DIV} <div style="max-width: 100px">
LayoutNGBlockFlow (anonymous) <div style="float: left; padding: 1ex">F</div>
<br>The <b>quick brown</b> fox
LayoutNGBlockFlow (floating) <div style="margin: -60px 0 0 80px">jumps</div>
LayoutText "F" </div>
LayoutBR
LayoutText "The "
LayoutInline {B}
LayoutText "quick brown"
LayoutText " fox"
LayoutNGBlockFlow {DIV}
layout tree
LayoutText "jumps"
layout — example
Box (block-flow) at 0,0 100x12 fragment tree
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
layout — example
Box (block-flow) at 0,0 100x12 fragment tree
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
layout — example
Box (block-flow) at 0,0 100x12 fragment tree
Box (block-flow children-inline) at 0,0 100x54
LineBox at 24.9,0 0x18
Box (floating block-flow children-inline) at 0,0 24.9x34
LineBox at 8,8 8.9x18
Text 'F' at 8,8 8.9x17
Text '\n' at 24.9,0 0x17
LineBox at 24.9,18 67.1x18
Text 'The ' at 24.9,18 28.9x17
Text 'quick' at 53.8,18 38.25x17
LineBox at 0,36 69.5x18
Text 'brown' at 0,36 44.2x17
Text ' fox' at 44.2,36 25.3x17
Box (block-flow children-inline) at 80,-6 20x18
LineBox at 0,0 39.125x18
Text 'jumps' at 0,0 39.125x17
paint
LocalFrameView::PaintTree Paint builds a list of paint ops.

DIV LayoutObject

Paint() PaintCanvas::drawRect

PaintArtifact
DisplayItem paint op buffer
PO DrawRectOp PO PO … display
- rect {x, y, w, h} item
- PaintFlags {color, …} list
paint
Paint uses stacking order, BODY

not DOM order. DIV .yellow

DIV .green

DOM tree
<div class="yellow"></div>
<div class="green"></div>
<style>
.yellow { z-index: 2; ... }
.green { z-index: 1; ... }
</style> yellow paints last
paint
Each paint phase is a separate
<div id="green">
traversal of a stacking context.
green's text
</div>
paint phases <div id="blue"></div>
(simplified) backgrounds

floats

foregrounds

outlines
blue after green, but
foregrounds after backgrounds
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
paint — example
<style> #p {
position: absolute; padding: 2px;
width: 50px; height: 20px;
left: 25px; top: 25px;
border: 4px solid purple;
background-color: lightgrey;
} </style>
<div id=p> pixels </div>
paint — example
<style> #p {
DrawingDocumentBackground
position: absolute; padding: 2px;
DrawRectOp width: 50px; height: 20px;
{0, 0, 610, 316} kFill_Style left: 25px; top: 25px;
border: 4px solid purple;
DrawingBoxDecorationBackground background-color: lightgrey;
DrawRectOp } </style>
{25, 25, 87, 57} kFill_Style <div id=p> pixels </div>

DrawRectOp
{27, 27, 85, 55} kStroke_Style (w=4)

DrawingPaintPhaseForeground
DrawTextBlobOp
{31, 45} "pixels" display items
paint — example
DrawingPaintPhaseForeground <style> #p {
position: absolute; padding: 2px;
DrawTextBlobOp width: 50px; height: 20px;
{31, 45} left: 25px; top: 25px;
border: 4px solid purple;
SkTextBlob background-color: lightgrey;
} </style>
53 4C 5B 48 4F 56 <div id=p> pixels </div>
0.0 8.0 12.4 20.4 27.5 32.0
raster
cc::ResourcePool::InUsePoolResource
cc::DisplayItemList
PaintOp …
raster

Rasterization turns (part of) a display bitmap


item list into a bitmap of color values.

FFFF FFFF 0000


FFFF FFFF 00FF
FFFF 0000 FFFF (red, green,
FFFF 00FF 00FF blue, alpha)
0000 FFFF FFFF
00FF 00FF 00FF
raster
<img src="kitten.jpeg"> Raster includes image decoding.

paint

DrawImageOp raster DecodedDrawImage

PaintImage
ImageDecodeCache
raster
InUsePoolResource
cc::DisplayItemList
PaintOp … GpuRasterBacking

GL texture ID
Raster can be accelerated by the GPU.
GPU memory
raster
Raster uses the Skia
DrawRectOp graphics library to
Raster() DrawRectOp::RasterWithFlags
{rect, flags}
issue GL calls.
SkCanvas::drawRect

GrRenderTargetContext::addDrawOp

GrOp FillRectOp … another op list!


SkSurface::flush

GrOp::execute GrGLOpsRenderPass::onDraw glDrawArrays(...)


gpu

Raster runs in the GPU process.


renderer process

RasterTaskImpl
GPU process

RasterInterface PaintOp::Raster

PaintOp PaintOp … RasterDecoderImpl


gpu
The request is sent through a command buffer.

renderer process
GPU process
GpuChannelHost
GpuChannel
RasterInterface
RasterDecoderImpl

GPU RasterCHROMIUM
command
buffer PaintOp PaintOp …
gpu
GPU process In the GPU process, the GL functions
link dynamically to native OpenGL.
GLApi::glDrawArraysFn
On Windows, we translate to DirectX.

dlopen("libGLESv2.so") ANGLE
dlsym("glDrawArrays")

LoadLibrary("libglesv2.dll")
GetProcAddress("glDrawArrays")
change

DOM
We now have a complete pipeline.

style But what if state can change?


layout
● scrolling
paint ● zooming
raster ● animations
● incremental “Be the change you wish
gpu
loading to see in the pixels.”
● JavaScript
● … —not Gandhi

(in memory)
frames
The renderer produces animation frames.

Below 60 frames per second,


scrolling and animations look "janky".

time

VSync signal
= time spent to render the frame
invalidation
Each pipeline stage tracks granular asynchronous invalidations.

Node::SetNeedsStyleRecalc() style

LayoutObject::SetNeedsLayout() layout

PaintInvalidator::InvalidatePaint() paint

RasterInvalidator::Generate() raster

Outputs are reused from previous frames when possible.


repaint
Paint + raster remain expensive if a large region is transformed…

scroll

all the pixels changed!


jank
… and anything on the main thread competes with JavaScript.

main

time

<script>
mineSomeBitcoins(); // why not?
</script>
enter: compositing
● Decompose the page into layers

lay
ers
which raster independently.

in
De
● Combine the layers on another thread.

ve
lop
er
To
o
main build layers

ls
commit
impl* draw layers

* ("impl" = compositor thread) ¯\_(ツ)_/¯


layers
<div>
AAA
<p class="wobble"> BBB </p>
</div>
CCC

BODY

DIV #text

P #text

#text layer A simple layer captures


a subtree of content.
layers
<div class="wobble">
AAA
<p> BBB </p>
</div>
CCC

BODY

DIV #text

P #text

#text layer A simple layer captures


a subtree of content.
compositing
Pinch Zoom:
a layer scales

Animation:
a layer moves

Scrolling:
a layer moves;
another clips
threaded input
renderer process

main busy

impl scroll scroll scroll

input from The compositor thread


browser process can handle input events.
layer promotion
󰣻
LayerTreeHost

LayoutView PaintLayer cc::Layer


LayoutBlockFlow

LayoutBlockFlow transform: ... PaintLayer cc::Layer

LayoutText PaintLayer is a
compositing layer tree list
LayoutBlockFlow "candidate".

layout tree
Layers are created by "promoting"
elements with compositing triggers.
scrolling layers
LayoutBlockFlow PaintLayer main layer

overflow: scroll CompositedLayerMapping scrolling contents layer

horz. scrollbar layer

Scroll containers vert. scrollbar layer


create a set of
scroll corner layer
special layers.
scrolling layers

LayoutBlockFlow Android? ChromeOS? CompositedLayerMapping


overflow: scroll
high DPI display (DSF >= 1.5)? ✓

solid background?

user disabled LCD font smoothing?

no non-composited
scroller

Compositing a transparent scroller


disables subpixel antialiasing.
compositing assignments

compositing
DOM style layout paint
assignments
main

GraphicsLayer::Paint

PaintLayerCompositor::UpdateAssignmentsIfNeeded
cc::Layer

GraphicsLayer cc::Layer DI DI DI
bu
ild

GraphicsLayer cc::Layer
s

GraphicsLayer cc::Layer Each layer is painted separately.


property trees

transform clip effect scroll


tree tree tree tree

The compositor can apply some paint properties to a layer.


property trees

... layout comp. assignments prepaint paint


main

PrePaintTreeWalk::Walk

PaintPropertyTreeBuilder

The prepaint stage builds the property trees.


composite after paint (CAP)

PaintArtifact
PaintArtifactCompositor cc::Layer
PaintOp …

compositing
... prepaint paint composite
assignments

main

In the future, layers will be created after paint.


commit

main … paint
(blocked) cc::Layer

copy layers and


ready to properties
commit
commit complete

impl commit …
LayerImpl
tiling
impl prepare tiles rastered
tile
viewport

visible
tiles
raster raster task

raster

CategorizedWorkerPool

layer Layers are broken into tiles for raster.


draw layers
Tiles are "drawn" as quads.
impl (raster complete) draw

CompositorFrame

PictureLayerImpl::AppendQuads DrawQuad
DrawQuaDra
DrawQuad
wQuad GPU
d rect = … process
Tile GPU memory
TileDrawInfo
resource_

raster output CompositorFrame is the "output"


of the renderer process.
activation

commit
Drawing can continue while
a new commit is rastered.
pending
tree impl
LayerImpl raster activation

active
tree
LayerImpl LayerImpl

draw draw
display (viz)
The display compositor
combines frames for multiple surfaces.
browser process
browser UI
ui::Compositor GPU process
CompositorFrame viz

display
renderer process CompositorFrame compositor

SurfaceAggregator
renderer process CompositorFrame
display (viz)

GPU process
Display::DrawAndSwap CompositorFrame

DrawQuad
DrawQuad
GLRenderer::DrawTileQuad

viz GLES2Implementation

command
buffer

gpu
Viz draws quads
GLES2DecoderImpl to the back buffer.
display (viz)

GPU process
Display::DrawAndSwap CompositorFrame

DrawQuad
DrawQuad
SkiaRenderer::DrawTileDrawQuad

viz

SkDeferredDisplayList

gpu SkSurface::draw
SkiaOutputSurfaceImplOnGpu
display (viz)

GPU process Finally, we swap the buffers.


Display::DrawAndSwap

viz GLRenderer::SwapBuffers

The user can see the pixels.


gpu GLSurface::SwapBuffers

eglSwapBuffers(...)
review content
<html>...</html>

Blink renderer process

main DOM style layout comp. assign prepaint paint

impl activate draw


commit tiling

GPU process
raster display
end LIFE OF A

slides: bit.ly/lifeofapixel

feedback: [email protected]

You might also like