DevOps Practices for Cloud Native

Author: Yao Dong

Department: Application Platform Service Product Dept, Cloud BU
Sep 20, 2019

Security Level:
Yao Dong

HUAWEI CLOUD
DevCloud

• Chief technology evangelist

• Principal DevOps, Lean, and Agile expert
• HUAWEI CLOUD expert
• Core organizer for the China DevOps Community
• Core organizer for the China DevOpsDays
Community
Background — HUAWEI CLOUD DevCloud's Journey

HUAWEI CONNECT 2018

October • Huawei Mirrors release
2018 • HiChat release

• College & university solution release

• API test release
February • 2018 Huawei Code Craft
2018
A team that has grown from a couple
of members to hundreds of members HUAWEI CONNECT 2017
September DevCloud 2.0 release
2017 • MobileAppTest release
• Wiki release
• Doc management feature release

January
2017 • CloudIDE release
• CloudPipeline release
December
2016 DevCloud 1.0 release
September
2016 • CodeCheck
July 2015 • CloudDeploy
• CloudRelease
April 2015
HUAWEI CONNECT 2016
DevCloud OBT DevCloud official release
Start • ProjectMan • ProjectMan
• CodeHub • CodeHub
• CloudBuild
3
 Background — Continuous Improvement

Advanced industry ideas • Cloud-based code and job management

Scrum, Kanban, DevOps, and • Compilation of 100 million lines of code: 75 min
Cloud computing
large-scale agility reduced to 20 min
• Version compilation: 94 min reduced to 31 min
Huawei R&D experience Microservices
• Build success rate: up from 40% to over 95%
Over 30 years of experience in
continuous R&D transformation + Container
• System regression testing: 2 days reduced to 8
DevCloud hours
E2E toolchain • R&D time: reduced by 50%
Full R&D lifecycle, from management to Pipeline • Resource reuse rate: up by 2.5x
engineering

2018
2017 Microservice-based, independent
release
2016 Pipeline-based, independent
release 10–20 releases per day, < 30 min
Automated deployment and per release
2015 5 releases a day, 30 min to 1 hour
testing, decoupled systems
R&D, testing, and O&M worked per release
Release took 3–6 hours over 2
in silos
weeks
DevCloud initial release took
weeks

4
4
Cloud Native — New Requirements of Traditional and Internet Applications

Traditional Internet VUCA

applications applications applications

Vague requirements
Constant requirements Changing requirements

Seen as a project that only requires Seen as a product that continuously Seen as a service in constant operation
O&M once finished develops
Requires service agility
Predictable and constant user requests Unpredictable and constantly
growing user requests Continuous delivery (CD)
Concurrent access for 10,000+ users
Concurrent access for millions of users Massive-scale concurrency
Allowing for temporary interruption of Service continuity, dark launch,
offline services for events such as Requires 24/7 online services with no
rollback, and online upgrades
midnight downtime and system O&M excuse

Microservices framework
Cloud-native application
Cloud-native applications architecture
5
Cloud Native — Architecture Evolution

Enterprise application architectures

Monolithic Vertical SOA Microservice

Enterprise integration architectures

Monolithic Mesh ESB Hybrid

MES SRM Public
ERP PLM MES SRM CRM SaaS cloud PaaS/EI
PLM CRM CRM VMALL Mobile

ERP OA

ESB HIP
Customer HR
service

... Devices Partners

Marketing Customer
Marketing Finance HR OA
Finance Budgeting service Private
cloud
ERP PLM MES

6
Cloud Native — A Triangle of Value

Cloud Native is a new paradigm for building, running, and managing software in the cloud environment by using the cloud infrastructure and
platform services. It has the following architectural features: (micro) service construction, auto scaling, distribution, high availability, multi-tenancy,
and automation. The best practices for supporting the Cloud Native architecture include setting up cross-functional teams, developing
organizations where full-stack engineers highly collaborate with each other, and using DevOps and automation tools to implement microservice CD.

Architecture

Auto scaling Distributed High availability

Multi-tenancy Automated O&M Self-service

Microservices architecture

Cloud infrastructure and platforms

Value Organization
Project
delivery
Lean and agile Speed Scale Reliability Flexibility Efficiency
Cross-functional teams

CD Full-stack engineers

Automation capability Small and autonomous

7
 Cloud Native — Capability Building

Service/Microservices architecture–based decoupling Self-service, agile cloud infrastructure Sharing underlying capabilities through APIs

 A cloud-native application is a collection of small, independent, and

autonomous (micro) services, each with a distinct function and clear Reliance on underlying compute, storage, and  Leveraging cloud-native underlying
boundaries. network infrastructure capabilities, such as microservices, lifecycle
Monitoring basic resources and service management, database, message queuing, and
 System architecture evolves based on (micro) services.
operation, and triggering of O&M events cache services
 Each (micro) service runs more than one stateless process to achieve accordingly achieves auto-scaling and self-  Automating deployment and provisioning
high availability and load balancing. healing through application and resource orchestration
 Service data is distributed across (micro) services for vertical sharding.

Decoupling systems from environments, processes, and Architecture

configurations
Cross-functional team
 Development, testing, and production parity
 Design, development, testing, release,
 Immutable infrastructure
deployment, and O&M roles within one
 Automated unit, integration, and mock testing team
 Standardized and agile R&D processes  Trusted collaboration between developers,
 Organization release engineers, IT, and O&M
Configurations separated from code and stored in environment Engineering
variables
 Strict separation between the build, release, and run stages
Cloud-based O&M
Shorten TTM through fast iteration, CD, and
quick response to service changes
 A dedicated team is responsible for
DevOps ensuring the system availability and
 (Micro) service CD pipeline (release according to uninterrupted service provisioning during
business needs) upgrade or rollback using monitoring and
 Development and O&M combined, with shared responsibilities alarm cloud platform functionality.
 Automated unit, integration, and mock testing
 Automated monitoring with continuous visualized feedback for
developers  Real-time deployment and updates with hot
reloading
8
 Architecture — Dividing a System into DevOps Units

Web UI
Web UI Web UI

 Divide services vertically MQ

MQ
LB
 Adopt microservices architecture for LB LB

new independent services Cache

Cache
 Start from common and core services
 Start from easily identifiable and clear- Backend
Backend
SSO
Backend SSO service
cut services service service

 Adopt the strangler pattern

DB Table 1 Table 2 DB DB

Web UI
Web UI
MQ
MQ
API Gateway
API Gateway
Cache
Cache
Details page Order placement

Legacy Shopping
Order Inventory Legacy
system cart Order Inventory Price
system

DB DB DB DB
DB DB DB DB

9
 Architecture — Self-Service DevOps Units

Services can be discovered, obtained, used, measured, and managed by other applications and developers.

Discover

Highly autonomous

Obtain Manage
User-friendly web UI, CLI, SDK, etc.
Service
Service

Enable efficient and easy user of services by applications and developers

Use Measure

10
 Architecture — Ensuring Agility with Architecture Decoupling and MVPs

1. Component- and microservice-based architecture decoupled from system

Architecture is decoupled from the system for concurrent development, build, testing, deployment,
and operation of MVPs.

2. Requirement breakdown principles:

Features evolve gradually from minimal to comprehensive functionality.

Architecture Evolution

Monolithic: monthly release Servitization: bi-weekly release Cloud-native microservices:

daily releases
Physical machines VMs All-cloud
Service 1 Service 2
Service 1 Frontend Service 2
Frontend Service 2 Service 3
Service 3 Service n services
Service 3 Service n Service n
Service 1
Microservices
framework
Control center ... 2015– Hub 2017–
Backend
2016 Data
2018 Data
Cache
DC Cache Others service
service Cache service Others
service

Stage 1: Traditional architecture with the Stage 2: Servitized cloud product delivery Stage 3: Independent microservice releases + E2E
waterfall model >> A release takes 1 month. >> A release takes 1–2 weeks. delivery pipeline >> Multiple daily releases

• Horizontally layered, large-scale system  Vertically decoupled, small-scale systems

• Different teams develop and deliver feature microservices separately. 11
 Cloud Native — 12-Factor App Methodology
Value and Methods

I. Codebase Fast delivery: Set clear boundaries; maintain healthy software lifecycle

II. Dependencies Efficient development: Standardize development to avoid risks

Software release management: Automate continuous integration

V. Build, release, run (CI) and CD using pipelines
Software release management: Store configurations in the
III. Config
environment variables

XI. Logs Real-time monitoring: Use a log management system

IX. Disposability Auto-scaling: Turn slow processes into backend services

IV. Backing services Elasticity/Agility: Use circuit breakers and relaxed binding

X. Dev/Prod parity Reliability: Achieve parity using cloud platforms

Reliability: Turn admin into backend services and expose them

XII. Admin processes using REST APIs
Colors indicate the difficulty of
implementation. VII. Port binding Efficient operations: Inform application services of the URI and port only

VI. Processes Cloud compatibility: Hand over status management to backend services

VIII. Concurrency Auto-scaling: Turn to cloud platforms and leverage PaaS

12
 Architecture: Applying the 12 Factors in Microservices Management

One-stop microservices management platform

3 12 Factors
Design and
5 development
Load Service 4 Security 1. Codebase
2 Dark launch 9
12 balancing registration

Microservices governance
2. Dependencies
Quick feedback and CD (DevOps)

Build Distributed configuration

1 Rate limiting
Call chain Service
and service 3. Config
tracing discovery Distributed messaging
Deployment degradation
4. Backing services

Cloud services/middleware
Distributed Distributed task
Metrics Communication scheduling 5. Build, release, run
Testing transaction

Distributed caches 6. Processes

Rollout 7. Port binding
Distributed databases
Java Go JS PHP Python .Net
Monitoring
8. Concurrency
and O&M Application performance
management 9. Disposability

Image
Application operations 10. Dev/Prod parity
Development Test 8 Auto-scaling
Container

management
services

environment environment management

10 6 11. Logs
Staging Production Resource Container
Distributed log
environment environment scheduling orchestration 7 11 12. Admin processes

13
 Organization — Cloud Native Microservices Architecture Embraces Agile
and DevOps
Moving towards cross-functional agile/DevOps teams based on cloud service/microservice-
oriented architecture
A team is responsible for the full lifecycle from planning, determination of requirement, design, development, and testing, to
independently deploy, deliver, and operate a specific feature, component, or service.

Before Transformation After

Agile transformation and horizontal Organization: Cross-functional service or microservices
integration of business, R&D, and teams led by the product manager
Team Process
operations teams and processes Process: E2E integration, automation, and fast releases

Business Go-to-
Business Plan
Requirements Cases Features Planning market Service 1 Product Technical Development Operations

Marketing and sales

Manager Manager engineers

Customer support

development
Product Technical Development

Business
Marketing, Service 2 Operations
Manager Manager engineers
requirement mgmt., Agile
and project mgmt... Service 3 Product Technical Development
Operations
Manager Manager engineers
... Product
Unit Technical Development
R&D (Dev) Design Developm ent Reconstruction
testing
Bug fix Deployment DevOps Service N Manager Manager engineers
Operations

Managers, R&D
engineers, test
engineers... Business
Requirements Planning Development Integration Testing Deployment Supply Monitoring
Plan

Operations (Ops) Supply Configuration Orchestration Deploym ent Report Monitoring

Platform and
infrastructure O&M

14
 Organization — Transforming into Autonomous Cross-Functional Teams

Insights Planning and Innovative Monitoring O&M Customer

design development Feedback

Architecture Evolution

Centralized decision-making Heuristic decision-making "Wild" growth

Product
PM/PO service team
PM Core AM/SEG
Product managers
team O&M PD SDE/SL SRE UX
AM SE Product
I&V service 1
Product Product
service 2 service 3
SRE
PL TL Microservices Microservices Microservices Security &
team 1 team 2 team 3 reliability
PD UX
Atomic
service 1 Atomic
SDE SDE SDE TE TE TE 2015– 2017– Atomic
PD FSD/SL service 3
SDE/SL
2016 2018 SDE/SL service 2

PM is responsible for results, reports to the The core team reports to its supervisor based on A microservices team makes its own decisions and
supervisor, and makes decisions directly based on business operations results for decision-making determines the future of the organization based on
business insights. support. Service teams run autonomously. digital operations results.

15
 Organization — Redefining Roles for Quick, Autonomous Decision-Making

From a large group to a squad: feature- or microservice-based,

SE
cross-functional, "two-pizza" teams, each consisting of fewer than PO
10 members

Sponsor: Project, business, or service stakeholders

PL Developer
Operations
PO: Product owner, responsible for product planning,
designing, and analysis
Operations: Responsible for product operation
UE: UCD engineer, responsible for user research, UI design, Tester
art design, and visual design
PL: Microservices/Feature leader, also responsible for agile
and +scrum UE
SE: System engineers, responsible for architecture and
system design
O&M
Developer: Responsible for code implementation
Tester: Responsible for testing and verification
O&M: Responsible for deployment, release, O&M,
and monitoring

16
 Organization — Enabling DevOps with Layered Software and Professional
Platforms
Leverage IaaS and PaaS services to dynamically allocate resources.

Business Business ...

aPaaS
1. Rely on dynamic resource
orchestration and scheduling
Resource orchestration and Distributed
Distributed MQ Database services
scheduling cache
2. Use existing services by
invoking APIs rather than
Image Security
Elastic computing Elastic network Elastic storage internally developing new
ECS, AS, and ELB VPC EVS, OBS...
Monitoring ... services

Professional cloud services ought to be provided by professional teams.

Complex system architecture is built on autonomous cloud services.

17
 Tools — DevOps Toolchain

E2E software R&D support covering delivery of requirement, code submission and compilation, testing, deployment, and O&M

Scrum/Kanban
Software
Image repository Configuration
repository service
Requirement status pushing Images or war

Environments
Application Performance
Requirement delivery
Create slave ECS RDS CCE Management (APM)
Deploy
Development

Application Testing
ECS RDS CCE
code Build Pipeline Deployment

Test code Verification (A/B/G)

Commit Application Operations
Deployment ECS RDS CCE Management (AOM)
script Configuration library service
IDE Verification
Intelligent Stress testing
operations
Environment
data analysis Data collection and reporting ECS RDS CCE

DIS CDM Kafka

Visualized Production
operations
data Logs, monitoring, and alarms
Data lake

18
 Tools — DevOps Toolchain and Environments Supporting Services and
Microservices

Concurrent development and testing of different microservices; verification in the Gamma environment;
dark launch and fast rollout; continuous feedback and evolution

Requirem
Process Design, development, and testing Release Production environment
ents

Microservices pipeline 1 Automated

Gamma monitoring
Source Build Alpha Beta Dark
Automated
Requirement launch Feedback
Key Microservices pipeline 2 release
analysis analysis
activities review
Source Build Alpha Beta Gamma Self-operated
cloud service
...
...

...
...
...

Return quality and usage data to R&D in real time

19
 Tools — Automated O&M for Data-Based Comprehensive Fault Monitoring

Automation of system deployment, upgrades, scaling, monitoring, alarms, fault demarcation and location, and self-healing.

Smaller granularity of services

requires stronger monitoring and
evaluation of the software and
hardware environments, along with
the services themselves. This can be
achieved by establishing data
collection and analysis platforms for
logs, KPI data, and alarm events.

20
CD — Essential Practices

Fast all-level delivery Short and frequent Automatic and visualized

iterations pipeline

Weekly iterations deliver small- Automatic pipeline–based fast

scale feature packages. delivery on all the four levels
Fast delivery on all four levels Delivery quality is close to TR5 Eliminates inefficient manual
Individual delivery: 30 min; quality in each iteration and operations and automates script
component-level delivery: 2 h; reaches TR5 quality after the last execution.
version-level delivery: 4 h integration and verification (I&V) Automates measurements at all
Daily E2E test version testing. pipeline checkpoints and
Versions can be used for solution visualizes delivery quality,
verification after each iteration. efficiency, and cycle time.

Automatic continuous Efficient and standardized

Reduced delivery time
deployment environments

Automatic deployment with Code coverage > 70%; Automation > Standardized environments on all the
installation and upgrade packages 90% four levels
within 15 min
Hierarchical case decoupling Automatic networking for service
Script-based process; automatic
(component-level verification: 2 h; environments; automatic
environment configuration and
version-level verification: 4 h) environment set-up
seamless reusability
SDV, SIT, and customer environment Component layer automated test Efficient support for R&D, testing,
verification supported by automatic cases ready within a day; Automated and verification; environments
deployment I&V cases ready within the iteration available at anytime

21
 CD — Full-Lifecycle Testing, as Early as Possible
Software quality management (SQM) aims to ensure that the software meets the
customer quality standards and any necessary regulatory requirements.
Require Test basis
-ments GUI
• Test requirements • Unit • Planning
• Tasks and progress • API • Preparations Testing Service
• Performance Fulfill
• Test cases • Execution
• Web automation Test object
• Defects • Report
• Continuous automatic R&D Unit
• Measurement testing… results

Testing pyramid
Test Automated
management testing Test procedure Relationship between requirement
determination, development, and testing

Requirement explanation Test preparations Test execution Test report • Self-testing by developers; special tests by
1–2
iterations testers
Microservices
Fulfill Fault • Manual, API, web automation, and online testing
developers Self-testing
requirements rectification
Organize by developers
release
Microservices review • Performance testing, reliability testing, security
testers Align developers Design test
cases Regression
testing, and online inspection by dedicated
and testers with Case testing
requirements Implement test testing testers
Dedicated cases
Prepare test
• Alpha: API testing; Beta: API testing, web
testers
environments automation testing, and security testing; Gamma:
Review test cases Confirm API testing, and web automation testing;
test
conclusion Production: online testing, performance testing,
reliability testing, and security testing 22
CD — Key Points for E2E CD Pipelines

A/B testing
No service
interruption
Static code
check

Microservices
testing and gate

Keys:
1. Smoke test: Verify new and modified code in the pipeline.
2. Andon Cord: Pipeline is stopped in case of a failure at any stage,
Test gate
such as a compilation alarm, coding style error, test lag, test failure, Proceed to the next
or violation of architecture principles. stage only if quality
is acceptable
3. Consistency: Build only once with consistent deployment mode
and environment.
4. Layering: Clarify requirements at all checkpoints.
5. Quick feedback: Automate measurement of efficiency, quality, and
cycle time, and visualize the pipeline.
6. R&D efficiency: The heartbeat of R&D, which reveals the actual
progress and quality.

23
 CD — Daily DevCloud CD Pipeline

Key actions:
1. 11:00 pm: Obtain code from each service's release branch for static checking, code download, compilation, build, archive, and release.
2. 12:00 am: Trigger automatic deployment of version packages.
3. 3:00 am: Trigger automated RF interface testing.

11:00 pm 12:00 am 3:00 am

Pipeline quality control

• Source code version control Code repository branches
Test Master
Release
• Branch strategy
• Static scanning Source code download, Component package Automated Automated RF
Static check
• > 80% code coverage compilation, and build archive and release deployment interface testing

• Vulnerability scanning
Daily online compilation and build using the release branch Automated deployment Automated testing using
• Open-source scanning Security scanning of version packages Robot Framework APIs with
results notified by email
• Version control
• Automated environment RF automated
cases
preparation Scanning result After compilation and build, archive
and release components using the
• Immutable servers automated release service.
Emailed check result
• Integration testing Code check notifications to ensure that
• Performance testing all issues are resolved
Email notification
• Automatic build, deployment, and CI01 environment

testing triggered by each commit

• Automatic change requests Check result
Archive components to
the software repository
• Low-risk releases
• Feature switch
24
Continuous Feedback — Automated Deployment and Rollback Driven by
Dark Launching
Level-1 dark launch users Level-2 dark launch users Level-3 dark launch users

1 Lv1 offline
3 Lv1 online 1. Fast rollback
SLB
Dark launch 4 Lv2 offline
6 Lv2 online
2. Online acceptance testing
strategy 7 Lv3-1 offline 3. A/B testing
9 Lv3-1 online
4. Key features first tested through
Lv1 Lv2 Lv3 private preview
dark launch dark launch dark launch

2 Lv1 deploy

5 Lv2 deploy
Deployment
8 Lv3-1 deploy

Cluster A 9% 45% 45%

1%
resource resource resource resource
pool pool pool pool
25
 Continuous Feedback — Integrated Efforts Supports Various Testing Stages

Service development Preview and testing by Open beta test or free trial Officially available for users
and verification select users for end tenants at market price
3–6 months 3–6 months

Cloud Code Complete Private Preview Public Preview General Availability

service (GA)

Invisible Limited number of All customers Unlimited

Custo customers
mer
Warm-up Special promotions Activities, traffic diversion,
Oper and promotions
ations
O&M Large-scale
Resource preparation Region-specific deployment
deployment/expansion

26
Continuous Feedback — Shifting from Manual to Automatic

Job Job
Configuration CMDB platform management

ZEUS
Automated
O&M
Platform
Assisted
Monitoring Monitoring
locating Analysis

O&M teams try everything possible to avoid manual operations because over 60% of incidents are
caused by maloperations.

27
 Continuous Feedback — Data Analysis, Dynamic Adjustment, and
Rectification Driven by VoC

Data

...
API calling PV
Instance type UV
collection
Instance
quantity
Product Website Heat map 1. User profile system: precise user research
...
...

Total
2. User behavior analysis system (high-frequency user
resources
Resource
Data O&M
SLA
Fault
Regional
Utilization operations and scenarios): PV/UV, intelligent routing, and
usage
...
Growth rate
...
Customer Consulting
growth hacking
Paid users User
service Fault reporting Data analysis
Conversion
Churn rate
Marketing Complaints
...
3. aPaaS: core service data and a North Star Metric system
...
Events, discounts, and traffic diversion 4. VoC system: user requirement feedback and analysis

Plan 5. Easy A/B testing with feature switches

adjustment

VoC: Voice of Customer

28
 Continuous Feedback — Layered Capability Maturity

Fast all-level delivery

Lead time
Short and frequent iterations
Individual
Team Delivery quality Automatic and visualized pipeline
Capability
Product maturity R&D efficiency
levels Automatic continuous deployment
Enterprise
Delivery stability Reduced time consumption

Efficient and standardized

environments

Category Agile Management CD O&M

Cross- Visualization Loosely

Require- Quality Continuous Automated Operation-
Key capabilities functional Planning Process CI Data Environment and coupled
ment assurance deployment O&M driven dev
team traceability architecture

Cloud infrastructure

29
 Summary — Cloud Native DevOps Practices
1. Agile Management
1. Joint agile delivery or crowd innovation with customers
2. Cross-functional, two-pizza teams
3. Service- or microservices-based, autonomous teams
Agile 4. Product management: product definition, competitor analysis, and
prioritization of requirements
2. CD 5. Epic-Feature-User Story: from strategic initiatives to implementation
6. Realizing customer value: independent stories available for delivery
1. Service/Microservices architecture
with traceable requirements
and decoupling
7. Scrum: stand-ups, review, Kanban, showcase acceptance
2. Reserve architecture optimization
and technology improvement CD 8. Dogfooding

channels
3. Code branching policy, fewer 3. Continuous Feedback
conflicts, and fast merging 1. Monitoring, O&M, logging, and application performance analysis
4. CI and automation pipelines
Continuous 2. VoC management and response
5. Chaos Monkey (resilience testing)
Feedback 3. Key customer engagement
6. Built-in security
4. Dark launching, private preview, public preview, and GA
7. Alpha/Beta/Staging environments
8. Automated deployment 5. Operations: warm-up, homepage promotions, and traffic diversion
6. Quick data-driven corrections and dynamic planning adjustment
7. Continuous learning and improvement: proactive suggestions and
encouragement for improvement...

30
 Summary — CD Implementation Framework
Improve delivery granularity, speed, and quality by focusing on seven dimensions
1 release every 100 days 1 release every 10 days 1 release per day 10 releases per day 100 releases per day

Waterfall and function-based Iteration and cross-functional

Team model
Iteration and function-based

Function-based branching Develop and release in trunk

Branching model
Develop in trunk, release in branch Develop features in branch and release in trunk

Function testing and automation testing by dedicated testing departments Automated pipeline unit, function, and performance testing
Test model
Function and automation testing by a team of both developers and testers

Technical Completely monolithic applications Independent products deployed separately or incrementally

architecture Comply with SOA principles, with backward-compatible microservices architecture

Strictly controlled release times and deployment windows Code allowed to go live directly in dark launch, A/B testing, and feature switches
Deployment
model Push code to the production environment based on demands

Strict infrastructure control by dedicated O&M Auto-scaling infrastructure as code service backed by PaaS or IaaS
Infrastructure
Automatic creation of production environment–like infrastructure

Manual data structure migration Forward/Backward compatibility of data structures

Database
Data migration with incremental scripts
31
 Summary — Cloud Native DevOps Framework with DevCloud

Regular review From decision making at stage gates to

Continuous planning OBP
Business 1 2 3 4 OBP-based periodic review
and regular review
decision-making
(Fixed frequency) Release Release Release Release

Service-oriented Responsible for customer experience operations E2E cross-functional, fully authorized
organization Cross-functional team of a product manager, developers, testers, O&M, and operations personnel team with integrated Dev and Ops

Architecture Service-oriented architecture: service decoupling and

decoupling Dark launching and rollback in seconds
independence
Service-based decoupling for independent
Approved planning Code delivery and easy change management
Portfolio and planning Service development
Dev&Ops Deployment
workflow Requirement identification Notification and
improvement O&M handover and release
Continuous feedback O&M management
Dev + Ops: shorter cycles and faster
feedback
Dev Release Ops
IT tools
and
environments
Production-like Data feedback
CD pipeline Fast release
environment platform
Lightweight DevOps toolchain consisting
of existing tools and advanced open-
Cloud service project initiation and service planning
source or commercial software
Continuous Public
PI execution Private preview GA Cloud fault handling
requirement planning preview
Service
process DOD Change
Fault handling SLA Independent delivery and automatic
Scrum management
environment deployment
Online and offline collaboration
32
Thank you. 把数字世界带入每个人、每个家庭、
每个组织，构建万物互联的智能世界。
Bring digital to every person, home and
organization for a fully connected,
intelligent world.

Copyright © 2019 Huawei Technologies Co., Ltd.

All Rights Reserved.

The information in this document may contain predictive

statements including, without limitation, statements regarding
the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially
from those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.