Scribd
Upload
415 views37 pages

Building Python Web Services with Ray

If you are building web services in Python that need to scale, this talk is for you! We show how the Python web serving ecosystem integrates with Ray and how Ray makes it easy to scale up web services and to manage their state. We address practical challenges such as how to use Ray tasks and actors effectively, how to write asynchronous code, and how to do type checking and testing. We also cover deployment challenges like monitoring and tracing in a distributed web service.

Uploaded by

Philipp Moritz
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
415 views37 pages

Building Python Web Services with Ray

If you are building web services in Python that need to scale, this talk is for you! We show how the Python web serving ecosystem integrates with Ray and how Ray makes it easy to scale up web services and to manage their state. We address practical challenges such as how to use Ray tasks and actors effectively, how to write asynchronous code, and how to do type checking and testing. We also cover deployment challenges like monitoring and tracing in a distributed web service.

Uploaded by

Philipp Moritz
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
  • Introduction: Introduction to the presentation topic, covering the development of Python web services with Ray.
  • What This Talk Is About: An overview of what will be covered, including Python web services using Ray and anyscale applications.
  • Requirements for a Web Service: Details key requirements such as availability, scalability, and library integration for web services.
  • Traditional Python Web Service Architecture: Illustrates traditional web service architecture and transitions to explaining Ray's architecture benefits.
  • Ray Web Service Architecture: Explains the architecture of web services using Ray, including advantages like a unified model and autoscaling.
  • The Anyscale Platform: Describes the features and capabilities of the Anyscale platform, emphasizing real-time collaboration and serverless experiences.
  • Scaling and Managing State with Ray: Demonstrates how to scale tasks and manage state using Ray actors and tasks.
  • Writing an API Server with fastAPI: Covers how to define a REST API using fastAPI and Ray with schema validation.
  • Ray AsyncIO Support: Provides insights about supporting async functions with Ray and handling object references.
  • Typing in Ray: Explains the use of typing in different parts of a Ray system for clarity and debugging.
  • Testing: Discusses different testing stages employed in Ray-based services, including integration and stress testing.
  • Metrics and Monitoring: Details how Ray's metrics API is used to monitor web service performance.
  • Tracing: Describes how tracing is implemented using OpenTelemetry for detailed analysis.
  • Deployment: Explains deployment strategies using Terraform, Docker, and Kubernetes.
  • Summary: Summarizes key points covered in the presentation about integrating Ray with Python web services.
  • Acknowledgments: Acknowledge contributions from the Anyscale team.

Building a Python

web service with Ray

Philipp Moritz
September 30, 2020
What this talk is about
What this talk is about

Show design patterns for building Python web services with


Ray using Ray tasks and actors
What this talk is about

Show design patterns for building Python web services with


Ray using Ray tasks and actors
Show how we are building Anyscale as a Ray application
What this talk is about

Show design patterns for building Python web services with


Ray using Ray tasks and actors
Show how we are building Anyscale as a Ray application
Show how to address practical challenges like type checking,
testing, tracing, monitoring and deployment
Requirements for a web service

Needs to be available 24/7


Requirements for a web service

Needs to be available 24/7

Needs to be scalable according to user demand


Requirements for a web service

Needs to be available 24/7

Needs to be scalable according to user demand

Needs to integrate external Python libraries and frameworks,


e.g. web serving frameworks or machine learning libraries
Traditional Python web service architecture

Flask server

aiohttp server

fastAPI server

Web logic
Traditional Python web service architecture

Redis

Flask server
Celery
Redis Queue
aiohttp server Multiprocessing

Service 1 Service 2 Service 3


fastAPI server

Web logic Business logic


Traditional Python web service architecture
Database
Redis

Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store

Service 1 Service 2 Service 3


fastAPI server

Web logic Business logic Data


Traditional Python web service architecture
Database
Redis

Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store

Service 1 Service 2 Service 3


fastAPI server
Challenges: Programming, scaling,
Web logic Business logic Data
monitoring, tracing, fault tolerance
Ray web service architecture
Database
Redis

Flask server
Celery
Redis Queue
aiohttp server Multiprocessing Blob store

Service 1 Service 2 Service 3


fastAPI server

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Flask server
Task Actor Task

Task Actor Task


aiohttp server Blob store
Actor Actor Task

Actor Actor
fastAPI server Task

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
aiohttp server Blob store
Actor Actor Task

Actor Actor
fastAPI server Task

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
Actor Actor Task

Actor Actor
fastAPI server Task

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task

Actor Actor
fastAPI server Task

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task
● Built-in facilities for monitoring
Actor Actor
fastAPI server Task

Web logic Business logic Data


Ray web service architecture
Database
Service 1 Service 2 Service 3
Advantages of Ray:
Flask server
Task Actor Task
● Unified programming model
Task Actor Task
● Automatic scheduling,
aiohttp server resource management Blob store
● Autoscaling Actor Actor Task
● Built-in facilities for monitoring
● Great
fastAPIsupport
server for ML
Actor Actor Task

Web logic Business logic Data


Reminder: The Anyscale Platform

1. Laptop experience with the power of a cluster


2. Serverless experience without serverless limitations
3. Real-time collaboration
Architecture of Anyscale

fastAPI server Service 1 Service 2 Database


Task Task
Task Task
Actor Actor

Web logic Business logic Data


Scaling up with Ray tasks

fastAPI server Sessions Websockets Database


Task Actor Session 1
Session 2

Web logic Business logic Data

/api/v2/session/start

Session 1

Session 2
Scaling up with Ray tasks

fastAPI server Sessions Websockets Database


Task Actor Session 1
Task Session 2
Task

Web logic Business logic Data

/api/v2/session/1/execute

/api/v2/session/1/execute

/api/v2/session/1/execute Session 1

Session 2
Scaling up with Ray tasks

fastAPI server Sessions Websockets Database


Task Actor Session 1
Task Session 2
Task
Task
Web logic Business logic Data
Web logic Business logic
/api/v2/session/1/execute

/api/v2/session/1/execute

/api/v2/session/1/execute Session 1

/api/v2/session/2/execute
Session 2
Managing state with Ray actors
Update
fastAPI server Sessions Notifications
Actor Actor

Task
Update
Web logic Business logic

/api/v2/session/start

Session
Writing an API server with fastAPI

● Makes it easy to define a REST API


● Schema validation
● Typing

@[Link](“/{command_id}/execution_logs”)

async def get_execution_logs(

command_id: int, ...) ->

Response[LogOutput]:
Ray asyncio support
Object Reference
Ray object references are awaitable!
async def get_execution_logs(session_record, session_command_id, log_params):

log = await session_tasks_service.get_execution_log.remote(

session_record["id"],

session_command_id,

logs_params

)
Ray asyncio support
Object Reference
Ray object references are awaitable!
async def get_execution_logs(session_record, session_command_id, log_params):

log = await session_tasks_service.get_execution_log.remote(


@[Link]
session_record["id"],
class WebSocketActor:

session_command_id, def __init__(self) -> None:


[Link] = [Link]()
logs_params
async def emit(self, message_name: str,
) data: Dict[str, Any]) -> None:
await [Link](message_name, data)
Ray actors can also be
Typing
fastAPI server Service 1 Service 2
Task Task
Task Task
Actor Actor

Web logic Business logic


Frontend
executeCommand({ async def execute_command( @[Link]
sessionId, session_id: int, options: Options): def execute_command(
options: { command_record = db.create_command( command: CommandRecord):
command: input session_id, options) runner = AnyscaleSessionRunner()
} execute_command.remote( runner.execute_command(command)
}) command_record)

TypeScript Python Python


Testing
Unit testing: Use the Ray local mode for unit testing:
[Link](local_mode=True)
Everything runs in a single process -> can mock out interfaces
Integration testing: Use a Ray instance running on the laptop/CI server,
testing web logic, business logic and database
End-to-end testing: Test full functionality in staging environment
Stress testing: Test scalability limits of the system
Metrics and Monitoring

Use Ray’s built in metrics API:


from [Link] import metrics

self.create_cluster_stats = [Link](

"Anyscale_create_cluster", "Num of seconds took to create cluster",

"second",

[float(i) for i in range(10, 300, 10)],

["step"],
Metrics and Monitoring

Use Ray’s built in metrics API:


from [Link] import metrics

self.create_cluster_stats = [Link](

"Anyscale_create_cluster", "Num of seconds took to create cluster",

"second",

[float(i) for i in range(10, 300, 10)],

["step"],
Tracing
We use OpenTelemetry for tracing

Can generate detailed traces for a number of Python libraries, including database
clients, web frameworks. [Link]:
opentelemetry-api
opentelemetry-sdk
opentelemetry-ext-asgi
Automatic tracing for Ray tasks and actors
opentelemetry-ext-asyncpg
opentelemetry-ext-botocore
opentelemetry-instrumentatio
Can also add custom traces: n-starlette
Tracing
We use OpenTelemetry for tracing

It can generate detailed traces automatically for a number of Python libraries,


including database clients, web frameworks.
[Link]
opentelemetry-api
opentelemetry-sdk

Full automatic tracing for Ray tasks and actors opentelemetry-ext-asgi


opentelemetry-ext-asyncpg
opentelemetry-ext-botocore
opentelemetry-instrumentation-starlette

Can also add custom traces:


Deployment
The cloud environment for our web service is set up with
Terraform, to make the setup easily reproducible for

● Development
● Staging,
● Production

The web service is deployed on Docker and Kubernetes, which


integrate well with Ray
Summary

We showed how the Python web serving ecosystem


integrates with Ray

We showed how Ray makes it easy to scale up your web


services and manage their state

We showed how to type, test, monitor and deploy your web


service with Ray
Thanks to the Team @ Anyscale

Building a Python web service with Ray Philipp Moritz September 30, 2020
What this talk is about
What this talk is about Show design patterns for building Python web services with Ray using Ray tasks and actors
What this talk is about Show design patterns for building Python web services with Ray using Ray tasks and actors Show how w
What this talk is about Show design patterns for building Python web services with Ray using Ray tasks and actors Show how w
Requirements for a web service Needs to be available 24/7
Requirements for a web service Needs to be available 24/7 Needs to be scalable according to user demand
Requirements for a web service Needs to be available 24/7 Needs to be scalable according to user demand Needs to integrate ex
Traditional Python web service architecture Web logic Flask server aiohttp server fastAPI server
Traditional Python web service architecture Celery Redis Queue Multiprocessing Service 1 Service 3 Business logic Service 2 R

You might also like