Jetpack Compose Semantics

Bryan Herbst (@bryancherbst)

Jetpack Compose Semantics
1. Why do I need semantics?
2. How do I use semantics?
3. Deep Dives:
• Accessibility
• Testing
What do accessibility services
and testing have in common?
Accessibility & Testing
• Need to find UI elements
• Need to act on UI elements
• Need UI element metadata (e.g. content
description, available interactions)
Composables emit UI
(they don’t return anything!)
The Composition is a state-
aware tree of Composables
How do we support these
features in Compose?
Semantics to the rescue!
Semantics Tree
• Parallel tree to the Composition
• Describes the semantic meaning of
Composables
Basic Semantics
Image requires a
contentDescription
@Composable
fun Image(
painter: Painter,
contentDescription: String?
)
contentDescription
• Text used by accessibility services to describe
content
• Same as View world
Not limited to images!
Box(
Modifier.semantics {
contentDescription = ”I’m a box”
}
)
What else can semantics do?
SemanticsPropertyReceiver

Modifier.semantics { this

}
SemanticsPropertyReceiver
• disabled()
• heading()
• selectableGroup()
• collapse()/expand()
Text(
modifier = Modifier.semantics {
heading()
},
text = ”I am a header”
)
SemanticsPropertyReceiver
• selected
• stateDescription
• contentDescription
• role
Text(
modifier = Modifier.semantics {
selected = true
},
text = ”I am selected”
)
Shortcut Modifiers:
clickable(), toggleable(), etc.
What is a Button?
Button
Surface(
Modifier.clickable {}
)
Modifier.clickable()
Modifier.semantics(
mergeDescendants = true
) { … }
Modifier.clickable()
Modifier.semantics(
mergeDescendants = true
) {
onClick(action = { onClick(); })
}
Modifier.clickable()
Modifier.semantics(
mergeDescendants = true
) {
onClick(action = { onClick(); })

if (!enabled) {
disabled()
}
}
Modifier.toggleable()
Row {
Checkbox()
Text("Get milk")
}
Row(
Modifier.toggleable(
value = checked,
onValueChange = { checked = it }
)
) { … }
Row(…) {
Checkbox(
checked = checked,
onCheckedChange = null
)
}
Semantics and Accessibility
AndroidComposeView
AccessibilityDelegateCompat
ACVADC maps semantics to
AccessibilityNodeInfo
TalkBack has a View problem
Wait, we don’t have View
classes!
ACVADC
val className = when (it) {
Role.Button -> "android.widget.Button”
Role.Checkbox -> "android.widget.CheckBox”
}
Roles to the rescue!
Modifier.clickable(
role = Role.RadioButton
)
Roles
• Button
• Checkbox
• Switch
• RadioButton
• Tab
• Image
Other semantics may imply
other class names
More common accessibility
features
What does clicking do?
Modifier.clickable(
onClickLabel = "Create new folder"
)
Merging Composables
Column {
Text("100")
Text("steps")
}
Column(
Modifier.semantics(
mergeDescendants = true
) {}
) { … }
Heading
Text(
modifier = Modifier.semantics {
heading()
},
text = "Title”
)
Selectable Group
Selectable Group
Column(
Modifier.selectableGroup()
)
Column(
Modifier.selectableGroup()
)
State
Description
State
Description
Modifier.semantics {
stateDescription = ”Subscribed”
}
Modifier.semantics {
stateDescription = ”Subscribed”
}
Custom Actions
Modifier.semantics {
customActions = listOf(
CustomAccessibilityAction(
"delete email”
),
)
}
What’s Missing?
Announcements
view.announceForAccessibility(
"Something changed!”
)
“This is not a recommend
method in View system.”

Source: https://issuetracker.google.com/issues/172590945
One option: Live Region
var text by remember {
mutableStateOf("Hello")
}
var text by remember {
mutableStateOf("Hello")
}

LaunchedEffect(true) {
delay(2000)
text = "world!”
}
Modifier.semantics {
liveRegion = LiveRegionMode.Polite
}
Semantics for Testing
Ye olde days
Ye olde days (probably today)
Espresso
onView(withId(R.id.some_view))
.makeAnAssertion()
Espresso
onView(withId(R.id.some_view))
.makeAnAssertion()
Espresso
onView(withId(R.id.some_view))
.makeAnAssertion()

onView(withContentDescription("text"))
Espresso
onView(withId(R.id.some_view))
.makeAnAssertion()

onView(withContentDescription("text"))

onView(withText("text"))
Semantics power Compose
finders & matchers
composeRule
.onNodeWithContentDescription("…")
.assertExists()
composeRule
.onNodeWithText("Some Text")
.assertExists()
composeRule
.onNode(hasText("Some Text"))
.assertExists()
fun hasText(
text: String
): SemanticsMatcher {
// …
}
Semantics & Matchers
• text -> hasText()
Semantics & Matchers
• text -> hasText()
• contentDescription -> hasContentDescription()
Semantics & Matchers
• text -> hasText()
• contentDescription -> hasContentDescription()
• stateDescription -> hasStateDescription()
Semantics & Matchers
• text -> hasText()
• contentDescription -> hasContentDescription()
• stateDescription -> hasStateDescription()
• selected -> isSelected()
Modifier of Last Resort:
testTag()
Bonus: Testable generally
implies accessible!
Merged/Unmerged Tree
Modifier.semantics(
mergeDescendants = true
)
Or clickable(), selectable(),
etc.
Column(
Modifier.clickable()
) {
Text("Jane Doe")
Text("555-555-5555")
}
Column(
Modifier.clickable()
) {
Text("Jane Doe")
Text("555-555-5555")
}
composeTestRule
.onNodeWithText("Jane Doe")
.assertTextEquals("Jane Doe")
composeTestRule
.onNodeWithText("Jane Doe")
.assertTextEquals("Jane Doe")

❌ Test failed
composeTestRule
.onRoot()
.printToLog("whereIsJane")
com.example D/whereIsJane: printToLog:
Printing with useUnmergedTree = 'false'
Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
|-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
Text = '[Jane Doe, 555-555-5555]'
Actions = [OnClick, GetTextLayoutResult]
MergeDescendants = 'true'
com.example D/whereIsJane: printToLog:
Printing with useUnmergedTree = 'false'
Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
|-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
Text = '[Jane Doe, 555-555-5555]'
Actions = [OnClick, GetTextLayoutResult]
MergeDescendants = 'true'
com.example D/whereIsJane: printToLog:
Printing with useUnmergedTree = 'false'
Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
|-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
Text = '[Jane Doe, 555-555-5555]'
Actions = [OnClick, GetTextLayoutResult]
MergeDescendants = 'true'
composeTestRule.onNodeWithText(
text = "Jane Doe",
useUnmergedTree = true
).assertTextEquals("Jane Doe")
com.example D/whereIsJane: printToLog:
Printing with useUnmergedTree = 'true'
Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
|-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
Actions = [OnClick]
MergeDescendants = 'true'
|-Node #3 at (l=0.0, t=145.0, r=159.0, b=197.0)px
| Text = '[Jane Doe]'
| Actions = [GetTextLayoutResult]
|-Node #5 at (l=0.0, t=197.0, r=232.0, b=249.0)px
Text = '[555-555-5555]'
Actions = [GetTextLayoutResult]
composeTestRule.onNodeWithText(
text = "Jane Doe",
useUnmergedTree = true
).assertTextEquals("Jane Doe")

✅ Test passed!
Custom Semantics
From Crane:
How can we assert on the
state of each day?
enum class DayStatus {
NoSelected,
Selected,
FirstDay,
LastDay,
}
val DayKey =
SemanticsPropertyKey<DayStatus>("DayKey")
Modifier.semantics {

}
SemanticsPropertyReceiver

Modifier.semantics { this

}
val DayKey =
SemanticsPropertyKey<DayStatus>("DayKey")

var SemanticsPropertyReceiver.dayStatus
by DayKey
Modifier.semantics {
dayStatus = DayStatus.FirstDay
}
composeTestRule.onNode(
SemanticsMatcher.expectValue(
key = DayKey,
expectedValue = DayStatus.FirstDay
)
)
private fun ComposeTestRule.onDateNode(
status: DaySelectedStatus
) = onNode(
SemanticsMatcher.expectValue(…)
)
Use lightly, avoid writing test-
specific code!
Thanks!