Scribd
Upload
657 views

Capacity Planning For MongoDB

Deploying MongoDB can be a challenge if you don't understand how resources are used nor how to plan for the capacity of your systems. If you need to deploy, or grow, a MongoDB single instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment. This talk will cover what resources MongoDB uses, and how to plan for their use in your deployment. Topics covered will include understanding how to model and plan capacity needs from the perspective of a new deployment, growing an existing one, and defining where the steps along scalability on your path to the top. The goal of this presentation will be to provide you with the tools needed to be successful in managing your MongoDB capacity planning tasks.

Uploaded by

Alvin John Richards
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
657 views

Capacity Planning For MongoDB

Deploying MongoDB can be a challenge if you don't understand how resources are used nor how to plan for the capacity of your systems. If you need to deploy, or grow, a MongoDB single instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment. This talk will cover what resources MongoDB uses, and how to plan for their use in your deployment. Topics covered will include understanding how to model and plan capacity needs from the perspective of a new deployment, growing an existing one, and defining where the steps along scalability on your path to the top. The goal of this presentation will be to provide you with the tools needed to be successful in managing your MongoDB capacity planning tasks.

Uploaded by

Alvin John Richards
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 36

#MongoDBdays

Capacity Planning: Choosing What to Deploy and When


Alvin Richards
@jonnyeight Technical Director, 10gen [email protected] alvinonmongodb.com

But rst a story

http://hci.stanford.edu/jheer/les/zoo/ex/maps/napoleon.html

Capacity Planning: Why, What, When

Why?
What are the consequences of not planning?

Capacity Planning: Why, What, When

Why?
What are the consequences of not planning?

Capacity Planning: Why, What, When

What?
Availability Throughput Latency

Capacity Planning: Why, What, When

When?
Before it's too late!

Start

Launch

Version 2

Capacity Planning: Why, What, When

When?
Before it's too late!

Start

Launch

Version 2

Problems
Capacity
Under Over Just right?

Prediction Models
User/Load System(s) Behavior

Change Velocity (reaction time)


Data/Resource-Allocation/Provisioning

Resource Usage
Storage
IOPS Size Data & Loading Patterns

CPU
Speed Cores

Memory
Working Set

Network
Latency Throughput

Storage Capability Example IOPs


7,200 rpm SATA 15,000 rpm SAS Amazon EBS/Provisioned Amazon SSD ~ 75-100 IOPS ~ 175-210 IOPS ~ 100 IOPS "up to" 2,000 IOPS 9,000 120,000 IOPS

Storage Capability Example IOPs


7,200 rpm SATA 15,000 rpm SAS Amazon EBS/Provisioned Amazon SSD Intel X25-E (SLC) Fusion IO Violin Memory 6000 ~ 75-100 IOPS ~ 175-210 IOPS ~ 100 IOPS "up to" 2,000 IOPS 9,000 120,000 IOPS ~ 5,000 ~ 135,000 IOPS IOPS

~ 1,000,000 IOPS

Storage Measuring and Monitoring

Storage Measuring and Monitoring

Storage Measuring and Monitoring

Storage
Active Archival Loading Patterns Integration (BI/DW)

stats()
db.blogs.stats() { "ns" : "test.blogs", "count" : 1338330, "size" : 46915928, "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "name_1" : 43982848 }, "ok" : 1 }

Size of data

Average document size Size on Disk Size of all indexes Size of each index

Memory
Sorting Aggregation Connections Working Set
Active Data in Memory Measured Over Periods

Ratio to Storage

Memory Measuring and Monitoring


New in 2.4 db.serverStatus( { workingSet: 1 } )

Memory and Storage MOPS: MongoDB Ops/Sec


MOPs

PFs

Memory and Storage

% Disk Util

MOPS

Estimating Sample Use Case


User log on

Read "user" collection based on supplied credentials Update "last seen" and "ip" on "user" collection Insert into "events" collection Update "activity" collection for time series metrics

Estimating Sample Use Case Worst Case


Index Read(s) Read "user" collection based on supplied credentials Update "last seen" and "ip" on "user" collection Insert into "events" collection Upsert "activity" collection for time series metrics 4k 4k Index Write(s) Journal Write Data Data Read(s) Write(s) 4k OpLog Write(s)

Maybe*

Maybe** sizeof(bson) 4k

4k

4k

4k per Index 4k per index

sizeof(bson) sizeof(bson) 4k

4k 4k

4k 4k

* Only if page is faulted out by the O/S before this operation executes ** if indexes exists on attributes updated, index update will also occur

Estimating Sample Use Case


User log on

Read "user" collection based on supplied credentials


8k Read 4k Read, 4k Write (single index), 4k Write OpLog 8k Write (single index), 4k Write OpLog 8k Read, 8k Write (single index), 4k Write OpLog

Update "last seen" and "ip" on "user" collection Insert into "events" collection Update "activity" collection for time series metrics

System load

20k Read, 36k Write

CPU
Non-indexed Data Sorting Aggregation
Map/Reduce Framework

Data
Fields Nesting Arrays/Embedded-Docs

CPU

MOPs

CPU

MOPs

CPU %

CPU
Non-indexed Data Sorting Aggregation
Map/Reduce Framework

Data
Fields Nesting Arrays/Embedded-Docs

Network
Latency
WriteConcern ReadPreference Batching Documents (and Collections)

Throughput
Update / Write Patterns Reads / Queries SAN / NAS Virtualization

Deployment Types
All of these use the same resources:
Single Instance Multiple Instances (Replica Set) Cluster (Sharding) Data Centers

Monitoring
Storage Memory CPU Network Application Metrics

Tools
MMS (MongoDB Monitoring Service) MongoDB: mongotop, mongostat Linux: iostat, vmstat, sar, etc Windows: Perfmon Load testing

Models
Load/Users
Response Time/TTFB

System Performance
Peak Usage Min Usage

Velocity of Change
Limitations
Data Movement Allocation/Provisioning (servers/mem/disk)

Improvement
Limit Size of Change (if you can) Increase Frequency Practice

Repeat (continuously)
Repeat Testing Repeat Evaluations Repeat Deployment

Starter Questions
What is the working set?
How does that equate to memory How much disk access will that require

How efcient are the queries? What is the rate of data change? How big are the highs and lows?

#MongoDBdays

Thank You
Alvin Richards
@jonnyeight Technical Director, 10gen [email protected] alvinonmongodb.com

You might also like