<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aaron Kalair on Medium]]></title>
        <description><![CDATA[Stories by Aaron Kalair on Medium]]></description>
        <link>https://medium.com/@AaronKalair?source=rss-834a463074f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VkYRP2sauIkXcCYI_9HXOg.jpeg</url>
            <title>Stories by Aaron Kalair on Medium</title>
            <link>https://medium.com/@AaronKalair?source=rss-834a463074f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:27:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AaronKalair/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[AWS ReInvent 2023 — Day 5— Friday]]></title>
            <link>https://medium.com/@AaronKalair/aws-reinvent-2023-day-5-friday-f9cf24273523?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9cf24273523</guid>
            <category><![CDATA[tech-conference]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 21:10:14 GMT</pubDate>
            <atom:updated>2023-12-09T21:10:14.711Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS ReInvent 2023 — Day 5— Friday</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86n92gUkNjzld_b4qptm6w.jpeg" /></figure><p>Watch today as a Vlog with a recap of the whole event — <a href="https://youtu.be/z9z5sn1p08M">https://youtu.be/z9z5sn1p08M</a></p><h3>Advanced VPC Designs and new Capabilities</h3><p>Youtube — <a href="https://www.youtube.com/watch?v=cRdDCkbE4es">https://www.youtube.com/watch?v=cRdDCkbE4es</a></p><p>Foundations</p><ul><li>EC2 classic global public network</li><li>VPC — Private network in the cloud, assumed everyone would just have one</li></ul><p>IP address management</p><p>Why is it important?</p><ul><li>v4 exhaustion</li><li>Cost optimisation</li><li>Global expansion</li></ul><p>IP address manager IPAM</p><ul><li>Address management at scale</li><li>Automate IP assignment</li><li>Monitor usage</li><li>Has a new free tier</li><li>Public IP insights in the free tier</li><li>IPv4 makes it easy to summarise IP space</li><li>IPv6 is harder, IPAM will give you contigious IPv6 space</li></ul><p>Expand VPCs or make the existing ones bigger?</p><ul><li>Can use a mix of both</li><li>Large VPCs can hit quota limits</li><li>Smaller VPCs needed to be connected</li><li>Blast Radius size differences</li></ul><p>Multi ENI Attachments</p><ul><li>1 EC2 instance can be in multiple VPCs</li></ul><p>NAT Gateways</p><ul><li>Can have up to 8 IP addresses associated with a a NAT Gateway to enable more connections</li></ul><p>Private link</p><ul><li>Consume services from other VPCs</li><li>Or private connections to AWS services</li></ul><p>CloudWAN</p><ul><li>The latest AWS product for building global networks</li><li>Define a JSON document that connects VPCs over multiple regions and also allows connections from VPNs etc</li><li>Attachments can be done automatically based on tags</li></ul><p>Application Networking</p><ul><li>ALBs support Mutual TLS</li><li>NLBs can have security groups, have a lot of limitations, can only be attached at creation time</li></ul><p>VPC Lattice</p><ul><li>0 trust service to service communication without having to worry about networking</li><li>Services can run on all AWS compute platforms</li><li>VPC lattice abstracts network addressing so services in VPCs with overlapping CIDR ranges can communicate</li></ul><p>EC2 Instance Connect</p><ul><li>Public IPv4 IPs no longer needed on EC2 instances</li></ul><p>Network Firewall</p><ul><li>Cloudnative traffic inspection</li><li>TLS inspection support</li></ul><p>VPN-Less Application Access</p><ul><li>Zero Trust access to applications inside a VPC</li></ul><p>Gen AI</p><ul><li>Can ask an AI assistant why your network doesn’t work</li></ul><h3>Ask the experts: Edge Compute with Amazon Cloudfront</h3><ul><li>After CDNs were provided people came up requirements to run compute on the edges</li><li>Altered cache keys on images to improve hit ratio ratio</li><li>Call LaunchDarkly when a request comes in to apply feature flags</li><li>Redirect multitentant SASS users to a specific EC2 instance for that customer</li><li>Originless less URL shortners</li></ul><p>Lambda@Edge</p><ul><li>Long running workloads</li><li>Complex use cases</li><li>Cloudfront functions</li><li>Lightweight</li><li>Low latency</li></ul><p>State Management</p><ul><li>Key Value store has seconds of replication time to all edge locations</li><li>Can store up to 5MB of data there</li><li>Microseconds read time</li><li>Ideal use case is write infrequently read frequently</li></ul><p>Redirection at the edge</p><ul><li>CF injects a header with the viewer country</li></ul><p>CF function runs at the edge locations</p><p>Lambda @ Edge runs on the regional edge cache</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9cf24273523" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ReInvent 2023 — Day 4— Thursday]]></title>
            <link>https://medium.com/@AaronKalair/aws-reinvent-2023-day-4-thursday-095aafec0deb?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/095aafec0deb</guid>
            <category><![CDATA[tech-conference]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 20:58:28 GMT</pubDate>
            <atom:updated>2023-12-09T20:58:28.150Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS ReInvent 2023 — Day 4— Thursday</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86n92gUkNjzld_b4qptm6w.jpeg" /></figure><p>View the Vlog of this day here — <a href="https://www.youtube.com/watch?v=qD9H7eb0494">https://www.youtube.com/watch?v=qD9H7eb0494</a></p><h3>Amazon EKS: Infrastructure as Code, GitOps or CI/CD?</h3><p>Developers</p><ul><li>Want full ownership of the application and deployment</li></ul><p>Platform Team</p><ul><li>Want to standardise deployment process and enforce standards</li></ul><p>Traditional CI/CD</p><ul><li>Build a Docker image for your application</li><li>The Infra dependencies like databases need to be provided by the platform team</li><li>This can take a while</li><li>Developers just focus on code, don’t need to learn IAC</li></ul><p>GitOps</p><ul><li>When you merge changes to a Git repo and produce a new image, the tool automatically notices and updates the cluster</li><li>If manual changes are made to the cluster its undone and reverted back to what’s in Git</li></ul><p>X-Plane Operator for K8s</p><ul><li>Can create AWS resources based on K8s manifest definitions</li><li>Can restrict what values in the AWS resource definitions can be modified</li><li>If manual changes are made x-plane reverts them</li><li>Initial setup can be difficult, you need to go install ArgoCD etc</li><li>Still need Terraform to create your EKS Cluster / VPC / Subnets etc.</li></ul><p>GitOps Bridge</p><ul><li>Terraform creates EKS Cluster and installs ArgoCD on the cluster</li><li>ArgoCD is then triggered to go install any other addons</li><li>Can then use X-Plane to create additional EKS clusters</li><li>The initial EKS cluster is a management cluster and its used to create additional ones</li><li>Need to really understand K8s if you go this deep into it</li></ul><h3>DocDB for Expert Practitioners</h3><ul><li>&lt;5 indexes per collection is recommended</li><li>MongoBetween connection pool for MongoDB</li></ul><p>Storage Engine</p><ul><li>Stores and retrieves data that a language lets users describe</li><li>Most storage engines are key values</li><li>Insert / Update / Delete</li><li>Range Lookup</li><li>Point lookup</li><li>Most basic is just a flat file you keep appending too</li><li>DocDB keeps an exact count of the number of docs stored</li></ul><p>B-Trees</p><ul><li>Provides n Log(n) lookup time for stored items</li><li>B-Tree leaf nodes point to an item in a heap</li></ul><p>Pages + Storage</p><ul><li>Page Size = 8k</li><li>When you store a node of an index or a data item, you store it in 8k blocks</li><li>Can have up to 16mb docs in DocDB</li><li>If doc &lt;8k, goes into an 8k page</li><li>If doc size is a factor of 8k it fits into the pages with less wastage</li><li>Big field names eat into the doc size, if you have a small amount of data and large field name your storage can be dominated by field names rather than real data</li><li>If a doc has an empty field its still stored but mapped with a null value</li><li>If one doc is 6k and another is 2k, there combined in a page to use the 8k limit</li><li>If doc &gt;8k it gets chopped into 2k chunks and stored in pages</li><li>Chunk references are external to this and don’t take up space</li><li>If your doc is 8k it takes one IO op to retrieve</li><li>If its 8.1k it takes two IO ops to retrieve</li><li>WireTiger has 32k pages</li></ul><p>MVCC</p><ul><li>Each doc inserted has a hidden timestamp start / timestamp end field</li><li>So you know at what point in time the document is visible, so queries that started before the time dont see it</li><li>Updates to the doc, create a new version with a different start and end timestmap</li><li>Any queries than come in before the end timestamp see the old one, any queries after see the new one</li><li>This means that indexes need pointers to multiple versions of the doc</li><li>Need to clear up old version’s of the documents and index pointers at some point</li></ul><p>Garbage Collection</p><ul><li>Once enough time is passed index is garbage collected and old pointers removed</li><li>The primary knows what timestamp all of the connections are at</li><li>If you have long running queries or long open cursors DocDB can’t garbage collect efficiently</li><li>Then once indexes are done heap is scanned and old documents are gone</li><li>This means that deletes are super fast</li><li>Metadata is kept inline on DocDB rather than off to the side which makes looking it up faster at a trade off for storage size</li><li>10k ops + 0.6% of the size of the collection is when GC is triggered</li></ul><p>IO Per Op</p><ul><li>If doc is &lt;8k 1 IO op to insert the Doc</li><li>Then IO operations to insert into the indexes</li><li>If you update a doc and dont touch any of the indexed fields it still triggers an update of all the indexes due to how MVCC is implemented</li></ul><p>IO Optimised</p><ul><li>Fixed IO pricing but you pay 3x for storage and 10% more for compute</li><li>Probably better for DBs with moderate size storage</li><li>Instances have a thread pool that talks to the storage layer</li><li>Bigger servers have a bigger thread pool</li><li>Each thread between server and IO can do 4K Writes</li><li>Thread waits for 4K of data or 10ms before doing the write</li><li>IO optimised waits for 1ms or 4K of data</li><li>Performance benchmarks suggest bigger docs improve write throughput by 2x</li><li>No known downside currently, reads might suffer if the writes only wait 1ms, writes could starve reads</li></ul><p>Compression</p><ul><li>Different to WireTiger</li><li>WireTiger compresses pages</li><li>DocDB compresses documents not pages</li><li>Smaller DocDB docs less likely to be compressed than in MongoDB as MongoDB compresses pages not docs, pages contain more data than docs</li><li>Docs need to be &gt;2k for DocDB to compress it</li><li>If you have a less than 2K doc add a dummy field with the same value, this gets heavily compressed and triggers doc compression</li><li>Compression can massively affect read performance</li><li>DocDB can compress the entire doc at once so good for bigger docs, MongoDB compresses pages so the biggest chunk it can ever compress is 32k, DocDB is potentially better at compression for bigger docs</li><li>DocDB calculates average doc size by sampling 20k docs</li><li>If your doc size changes dramatically over time, the average might be off</li><li>Compression not available in &lt;5.0</li><li>Compression when upgrading in place only available if you create a duplicate collection after upgrading to 5.0</li><li>Data is compressed on disk, and in the cache</li><li>Can fit a lot more documents in the cache</li><li>Every document access involves a decompression</li><li>If you heavily hit the cache, you might see more CPU usage as you decompress the doc from the cache before sending it</li><li>Have seen up to 20% CPU usage increase when compression is enabled</li><li>Can increase IO bound read workloads by 3x though</li></ul><p>Parallel Indexing</p><ul><li>Only applies on 4XL or greater</li><li>To avoid starving CPUs on smaller instances</li><li>Reduced index creation time from 14 days to 8 hours for one use case</li><li>2 threads = twice as fast</li><li>Can use up to 50% of the vCPUs for parallel index creation</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=095aafec0deb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Reinvent — Day 3 — Wednesday]]></title>
            <link>https://medium.com/@AaronKalair/aws-reinvent-day-3-wednesday-e8d2a144e0f1?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8d2a144e0f1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tech-conference]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 20:57:34 GMT</pubDate>
            <atom:updated>2023-12-09T21:20:39.370Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Reinvent 2023— Day 3 — Wednesday</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86n92gUkNjzld_b4qptm6w.jpeg" /></figure><p>Watch today as a Vlog here — <a href="https://www.youtube.com/watch?v=w1UGdEHQnPY">https://www.youtube.com/watch?v=w1UGdEHQnPY</a></p><h3>Safely Migrate Databases that serve millions of requests per second</h3><p>Video — <a href="https://www.youtube.com/watch?v=3bjnm1SXLlo">https://www.youtube.com/watch?v=3bjnm1SXLlo</a></p><ul><li>These serve millions of request per second</li></ul><p>Netflix has global databases in 4 AWS regions</p><ul><li>Have they globally distributed for access speed and redundancy</li></ul><p>More than one tool can satisfy requirements</p><ul><li>Requirements can change over time and you need a new type of data store</li><li>AWS can release new features that change what data stores can do</li><li>As you scale cost trade offs change</li><li>As you get larger, self managed solutions are most cost effective than AWS managed ones</li></ul><p>When you must migrate</p><ul><li>Understand how to do one</li><li>Automate safety measure</li><li>Execute migrations in parallel</li></ul><p>Techniques</p><ul><li>Abstract underlying storage engine APIs</li><li>Shadow idempotently across reads and writes</li><li>Merge in flight writes with backfilled data</li></ul><p>They implement a middleware proxy layer that clients connect to which dual writes to both data stores</p><p>Setup</p><ul><li>They provide libraries that provide methods like put and get, the middleware service that implements those APIs translates them to Redis / Cassandra API calls etc and return to the client</li><li>10% of the APIs aren’t generic and are specific to a data store type</li><li>Mutation operations require an idempotency token</li></ul><p>Shadow Traffic</p><ul><li>Middleware layer duplicates call to both data stores</li><li>Sometimes you need to change the client library and you cant use the middleware layer</li><li>This gives you bespoke interfaces but is harder to deploy and risks introducing new bugs when you update the library</li></ul><p>Backfill</p><ul><li>Read from source database, do any transformations needed, write to target</li><li>This is basically DMS</li></ul><p>Data Verification</p><ul><li>Can’t scan the entire datasets on a live database</li><li>Use a backup and compare offline</li><li>Combine with some live sampling</li></ul><p>Performance Verification</p><ul><li>Automated a process to compare performance metrics between the 2 databases</li></ul><h3>Why is it always DNS? Ask us Anything</h3><p>Registar Service</p><ul><li>Buying a domain name is paying someone to add an entry to a zone file on a root server</li></ul><p>Public Hosted Zone</p><ul><li>Let’s you put entries in the DNS zone you bought with some extra features like health checks etc</li></ul><p>VPC Resolver</p><ul><li>Sits on x.x.x.2 in the VPC</li></ul><p>Private Hosted Zone</p><ul><li>Don’t need to register the zone, its only available in your VPC</li></ul><p>Inbound Resolver Endpoint</p><ul><li>For exposing the VPC resolver to an on premises network</li><li>You should only have one of these</li></ul><p>Outbound Resolver Endpoint</p><ul><li>For exposing your VPC to your on premises resolver</li><li>You should only have one of these</li></ul><p>DNS Firewall</p><ul><li>Allow / Block lists for DNS resolution</li><li>AWS have blocklists for known malware domains</li><li>With alerting</li></ul><p>DNSSEC</p><ul><li>Resolver can authenticate answer was signed by a trusted entity</li><li>Does not encrypt your queries and hide them</li></ul><p>DoT/DoH</p><ul><li>Encrypts connection between end user and DNS resolver</li><li>Coming in the next year for AWS resolver</li></ul><p>Query Logging</p><ul><li>Logs when the resolver answers a query</li><li>Won’t log every query as other resolvers will cache the response and not forward it to AWS</li></ul><p>What if you hit DNS rate limits?</p><ul><li>The .2 address is special its not actually running in your VPC, its provided by the hypervisor</li><li>Container workloads can hit limits here as lots of containers run on a single instance</li><li>Network endpoints work around this and provide higher limits</li></ul><h3>Digging into AWS Network Performance</h3><p>Bandwidth</p><ul><li>Max amount of data you could transfer</li><li>AWS limits are both ways e.g you get 10 up AND down</li></ul><p>Throughput</p><ul><li>Actual transfer rate</li></ul><p>Assorted Q&amp;A Notes</p><ul><li>Placement groups increase bandwidth for EC2</li><li>ENA express offers faster network performance</li><li>Interface endpoints can do up to 100Gbps when they fully scale</li><li>Packets per second limit commonly hit with apps that send lots of small packets</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8d2a144e0f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Reinvent 2023 — Day 2 — Tuesday]]></title>
            <link>https://medium.com/@AaronKalair/aws-reinvent-2023-day-2-tuesday-28d8ca739396?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/28d8ca739396</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tech-conference]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 20:56:52 GMT</pubDate>
            <atom:updated>2023-12-09T20:56:52.274Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Reinvent 2023 — Day 2 — Tuesday</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86n92gUkNjzld_b4qptm6w.jpeg" /></figure><p>Watch the Vlog of this day here — <a href="https://www.youtube.com/watch?v=pEB9IoP7-8s">https://www.youtube.com/watch?v=pEB9IoP7-8s</a></p><h3>EKS Debugging</h3><p>YAML best practices</p><ul><li>Dry Run flag in Kubectl</li><li>Client and Server mode</li><li>OSS tools like kubeconform</li></ul><p>Networking Things to Remember</p><ul><li>CNIs are important</li><li>Networking issues are normally because of the CNI</li><li>Policy Management</li><li>Pod security / Network Policies</li><li>Security Groups</li><li>IP address exhaustion</li></ul><p><strong>Observing K8s</strong></p><p>Stern</p><ul><li>Nice replacement for kubectl logs</li></ul><h3>Performing Live Migrations to Amazon DocumentDB</h3><p>Why Move?</p><ul><li>Self Managed -&gt; AWS Managed</li><li>Remove the need to do work yourself</li><li>Get away from Legacy Databases</li><li>Use purpose built modern databases</li></ul><p><strong>Migration Methods</strong></p><p>Legacy</p><ul><li>Stop the database</li><li>Back it up</li><li>Restore on new Database</li></ul><p>Dual Writes</p><ul><li>Middleware layer sits in between app and database</li><li>Middleware proxies database connections and dual writes to both databases</li><li>Backfill new database</li><li>Turn off old database</li></ul><p>Stream Changes</p><ul><li>Build new database</li><li>Backfill data</li><li>Stream changes</li><li>Flip app to use new Database</li></ul><p>Moving</p><ul><li>Export and Import data</li><li>Use change streams to replicate data</li></ul><p>DMS can do this</p><ul><li>Uses DB connections</li><li>Can overload source, especially in multithreaded mode</li><li>Point DMS to the secondary to be safer</li><li>Can use hidden replicas to ensure the app isn’t using it</li><li>Use a big single instance node for the initial sync not multiple nodes</li></ul><p>DocDB uses write ahead logs</p><ul><li>All DocDB nodes use a shared data volume</li><li>Primary writes data, secondary only cares if the data that changes is in its cache</li></ul><p>Multithreaded DMS</p><ul><li>DMS runs single threaded by default</li></ul><p>258 million docs from 2018–2022</p><ul><li>Single thread -&gt; 9 mins</li><li>Multithreaded</li><li>Auto or Range mode</li><li>Auto — DMS figures out how to do it — 6 mins</li><li>Range — You tell DMS how your data is split — &lt;5 mins</li></ul><p>No reason not to multi thread</p><ul><li>Unless your source or target are too small</li></ul><p>Indexes</p><ul><li>Not ported by DMS</li></ul><p>Create indexes before or after data load?</p><p>Before</p><ul><li>Less data to go over to create the initial index</li></ul><p>After</p><ul><li>Less write penalty for each write during the load</li><li>For DocDB each document update requires the index to be updated even if the document doesn’t contain the indexed field</li><li>4XL instance or larger means you can use parallel index creation, linear increase in performance as you go up each instance size</li></ul><p>Always create index before loading data</p><ul><li>Performance hit to update the index is negligible</li><li>Doing it afterwards also uses a lot of IO to go fetch all the data and create the index afterwards</li><li>You pay for IO in DocDB</li></ul><p>Sync’ing Data</p><ul><li>DMS caches changes in RAM and if the RAM fills up, spills over to disk</li><li>You can’t modify a source in DMS you have to create a new one</li><li>This destroys all the cached data</li><li>If your secondary your syncing from fails and you replace it with a new one DMS will destroy the cached data UNLESS you used a DNS CNAME</li><li>Point the CNAME at the replacement secondary</li><li>DMS will automagically pickup where it left off</li></ul><p>Oplog in MongoDB is a capped collection</p><ul><li>Once the collection size limit is reached older data is deleted</li><li>Oplog stores every operation that happened in the cluster</li><li>If the DMS replication instance fails AWS gives you a new one with the cache in tact</li><li>Your OpLog needs to be big enough for the replacement instance to be able to pickup where it left off</li></ul><p>Can run multiple DMS tasks in parallel</p><ul><li>Each task moves a different table</li><li>Can use this if your collections have different access patterns</li><li>Use different tasks for high transaction workloads vs low transaction workloads</li><li>Can’t run multiple DMS tasks that point at the same collection use multi threading instead of this</li><li>If your using a single instance DocDB as the target your DMS instance needs to be able to cache 10 mins of changes whilst AWS build you a new DocDB target if it fails</li></ul><p>Need to test against DocDB before you flip your app</p><ul><li>You can point reads to DocDB, but you can’t point writes until your happy to not use Mongo anymore</li></ul><p>Volume cloning creates a copy on write version of your database using the same data as the original that you can test against</p><ul><li>Only pay for the data that you change in the new cluster</li></ul><p>Cutover</p><ul><li>Put your credentials in one location makes it easier to make sure everything knows about the new database</li></ul><p>Connection Pools need to be updated to know about the new database endpoint</p><ul><li>Need to restart the connection pools or have them error out</li><li>If you kill the source database it will force exceptions in the application and the connection pool will reset</li></ul><p>Once the first write happens on the DocDB side your committed to DocDB</p><ul><li>DMS can replicate data from DocDB to MongoDB even though its not officially supported</li></ul><h3>How to connect multiple VPCs across locations</h3><p>VPC Sharing</p><ul><li>Share a VPC amongst multiple AWS accounts in an org</li><li>Using RAM you can share subnets into other AWS accounts</li><li>Shared AWS account can create resources in the VPC</li><li>Can cross account reference SGs</li></ul><p>VPC Peering</p><ul><li>Create a link between VPCs in the same or different accounts / regions</li><li>Transitive routing not supported</li><li>Number of peering connections and route table entries needed quickly becomes unmanageable</li></ul><p>Transit Gateway</p><ul><li>Plug multiple VPCs into the Transit Gateway</li><li>All VPCs plugged into the TGW can potentially communicate</li><li>Can also attach VPNs into the TGW</li><li>TGW Peering allows you to connect multiple TGWs</li></ul><p>CloudWAN</p><ul><li>Let’s you build and manage a global unified network</li><li>VPCs and Physical networks all over the world</li><li>Let’s customers leverage the AWS backbone network</li><li>Core Network edges where you attach your VPCs</li></ul><p>Segments — Dedicated Routing Domains</p><ul><li>Dev / Prod etc. segments</li></ul><p>Attachments</p><ul><li>Where you add your other networks</li></ul><p>Segments are basically TGWs that AWS manage</p><ul><li>Can peer existing TGW to CloudWAN</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28d8ca739396" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ReInvent 2023 —  Day 1 — Monday]]></title>
            <link>https://medium.com/@AaronKalair/aws-reinvent-2023-day-1-monday-6a6c1c938b2e?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a6c1c938b2e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 20:55:04 GMT</pubDate>
            <atom:updated>2023-12-09T20:55:04.211Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS ReInvent 2023 — Day 1 — Monday</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86n92gUkNjzld_b4qptm6w.jpeg" /></figure><p>Checkout this day as a vlog — <a href="https://www.youtube.com/watch?v=Lw3f79M30WA">https://www.youtube.com/watch?v=Lw3f79M30WA</a></p><h3>Data Modelling Core Concepts for Amazon DynamoDB</h3><p>Youtube Link — <a href="https://www.youtube.com/watch?v=l-Urbf4BaWg">https://www.youtube.com/watch?v=l-Urbf4BaWg</a></p><p>You need to care about tables that have items</p><ul><li>Items need a partition key</li><li>This is where data goes behind the scenes in AWS’es servers</li></ul><p>Sort key is optional</p><ul><li>Sorts data within the partition</li><li>Partition Key + Sort Key = Primary Key</li><li>Needs to be unique</li><li>With no sort key you have a key value store</li></ul><p>Indexes</p><ul><li>Global Secondary Index</li><li>Table created on your behalf and data replicated there</li><li>Local Index</li></ul><p>Need to store data is intended to be read, no joins or calculations</p><p>Want to map out entities being stored, there sizes and cardinalities</p><p>How many tables do you need?</p><ul><li>Some entities will be stored in same table</li><li>Some will have entity per table</li><li>Workload priorities?</li><li>Do you access multiple entities together</li><li>Do you need specific access control</li><li>Do you have downstream services that rely on the data</li></ul><p>DynamoDB can’t do joins need to do it on the client if you use multiple tables</p><p>Multi Table Advantages</p><ul><li>Entities isolated</li><li>Indexes dont include unwanted data</li><li>Each table can have custom backup retention and storage class</li><li>Maps to traditional database concepts well</li></ul><p>Single Table</p><ul><li>Related data together mimicking joins</li><li>Throughput shared across all entities</li><li>Less monitoring and alerting as everything’s in the same table</li><li>Reduced RCU and latency compared to multiple queries</li></ul><p><strong>Patterns for scale</strong></p><p>Need to pick the right partition key</p><ul><li>This maps to a partition where DynamoDB stores your data behind the scenes</li><li>Sharded by default</li><li>High cardinality better</li><li>Ensure key is used in most access patterns</li><li>Don’t want partition key value to change as you’d have to go update every item</li><li>UUID / GUID etc is best</li></ul><p>Sort Keys</p><ul><li>Group related items under same partition key</li><li>Improve query performance by reading range of partition</li><li>Access split across multiple small items instead of one larger one</li></ul><p>Hierarchal Relationships</p><ul><li>Composite Sort Keys</li><li>Hash separates items</li><li>Query with begins_with function so do things like CART_ACTIVE or CART_SAVED to find appropriate items</li></ul><p>Cost optimisation</p><ul><li>1024 bytes = one write capacity unit</li><li>400kb max item size</li><li>Higher read write latency</li><li>Higher chance of getting throttled</li><li>If you have a large item and you update only part of it, you pay to write the entire item again including unchanged data</li></ul><p>Filtering Data using a Secondary Index</p><ul><li>A secondary table AWS builds for you</li><li>When you write to the main table AWS replicates the writes</li><li>Need high cardinality keys for the index, or it can cause throttling</li><li>Can chose what attributes go into the secondary index</li><li>These are eventually consistent</li><li>Have to pay for the additional storage costs</li><li>Act as sparse indexes so you can only include data your access patterns care about</li></ul><h3>Architecting for Data Residency and Data Protection at the Edge</h3><p>Residency</p><ul><li>AWS where you need it</li></ul><p>Hybrid Edge Services</p><ul><li>Cloudfront deployed in different POPs in metro areas in AWS operated facilities</li></ul><p>Nitro</p><ul><li>Local Zones and Outpost built from it</li><li>Allows AWS to segment where they have access and dont</li><li>All AWS can access is the Nitro card that runs alongside the server, not the customer data</li></ul><p>Outpost</p><ul><li>AWS gives you a server to run in your own data centre</li><li>Launch instances and databases etc onto it with the same control plane</li><li>Not all services supported, lots are</li><li>You can create subnets on the Outpost Server</li><li>Outpost is tied to a single AZ, extension of whatever AZ you choose</li></ul><p>Local Gateway</p><ul><li>Jump from Outpost to OnPrem Network</li><li>Can go out the IGW to the internet</li><li>Share routes with BGP</li><li>If the service link back to AWS dies your resources still run but you can’t make control plane changes etc.</li><li>Service linked is an encrypted VPN</li><li>IAM / KMS etc don’t run on the outpost they run in the AWS region</li><li>So if the service link dies you can do IAM actions etc.</li><li>30 ish versions of the Outpost Server available each server is 24xl worth of capacity</li><li>Power is generally the limiting factor for what people can run</li><li>Local Route53 resolver can run on the Outpost</li><li>Takes up compute budget on the server</li></ul><p>S3 on OutPost</p><ul><li>ARN namespace is different</li><li>Need to create an endpoint on the outpost</li><li>Access the data via an access point</li><li>Can replicate data across Outpost servers</li><li>If the service link goes down S3 breaks as it can talk to KMS</li><li>Service Link has Cloudwatch Metrics</li><li>Where do the IPs come from?</li><li>/26 needed for the Service Link</li></ul><p>AWS Dedicated Local Zones</p><ul><li>Extension of AZs</li><li>AWS Controlled environment that provides AWS services running in your facility</li><li>Fully AWS managed, think small cloud in your data centre</li><li>No Local Gateway Network link to your network</li><li>Network link from a dedicated local zone back to AWS fully managed by AWS</li><li>Each individual server has key material for encryption for data at rest on that server</li><li>You can physical unscrew this key material and destroy it</li></ul><p>Snow</p><ul><li>Has a local control plane</li><li>AWS services at the edge, can run without a connection to AWS</li><li>Same API calls you use to talk to “normal” AWS services</li></ul><h3>Netflix’s IPv6 journey: Building Network Configurations on AWS</h3><p>Why not IPv4 ?</p><ul><li>Need to choose a CIDR range for VPCs and make sure things dont overlap</li><li>Need to split up VPC CIDR range for subnets</li><li>Then create an IGW for Public Subnets</li><li>Then a NAT Gateway for Private Subnets</li><li>Need to ensure NAT Gateways can support enough connections</li></ul><p>How to connect to OnPrem</p><ul><li>VGW</li><li>TGW</li><li>If you can summarise your routes neatly you need to keep adding routes</li></ul><p>Need to do all of this because of IPv4</p><ul><li>If you have multiple VPCs you need Peering</li><li>Need to make sure peering is symmetrical or weird things happen</li></ul><p>What if you use IPv6 Instead</p><ul><li>Can assign variable CIDR sizes to subnets now</li><li>Your never going to run out of IP space so planning is easier</li><li>No NAT Gateways needed for V6</li><li>No need for VPN tunnels / VGW, IPv6 is all public by default</li><li>No need for public private subnet split</li><li>Multiple VPCs just talk over the internet no need for peering</li></ul><p>How do you get to using IPv6?</p><ul><li>AWS will provide IPv6 CIDR blocks — Might not get a sumariseable block for multiple allocations</li><li>Bring your Own IP</li><li>Need to provision per AWS account</li><li>Need to add IPv6 SG Rules</li><li>Need to add IPv6 Route Table entries</li><li>Need a AAAA record for DNS</li><li>IPv6 enabled != IPv6 used, need to ensure your app binds to an IPv6 Address</li><li>Some SDKs wont resolve the IPv6 address</li><li>You can use something like CoreDNS to change the response rather than upgrading all SDKs</li></ul><p>IPv4 External dependencies</p><ul><li>DNS64 / NAT64</li><li>DNS response includes IPV6 address in response</li><li>When the NAT Gateway sees this packet it translates it automatically</li><li>This introduces overhead per packet</li><li>Netflix tool uses Secomp to dynamically rewrite connect()calls</li></ul><p>How Netflix practically did this</p><ul><li>Kept scaling there IPv4 network until they hit the AWS limits</li><li>Assign a prefix list to an ENI rather than individual IPs</li><li>Tried to migrate the biggest network usage to IPv6 first</li></ul><p>Communication</p><ul><li>Got org wide buy in to migrate apps they owned and support them when things broke</li><li>Things broke along the way and they had to go back to IPv4 whilst they fixed it</li></ul><p>Cost Savings</p><ul><li>Different AWS services cost different amounts to move data around AWS</li><li>Spent millions of dollars talking to S3 across regions</li><li>Egressed out NAT Gateway and into another region</li><li>Could send traffic over VPC peer and then into S3 Endpoint</li><li>If you use a public IPv4 address you can just go over the internet</li><li>Have to pay for these IPs soon</li><li>Have to pay for inter region transfer</li><li>IPv6 is the cheapest possible, just pay for data transfer</li></ul><p>Multi Region Access Point?</p><ul><li>Didn’t know about it, would be another service though</li></ul><p>Lambda Not supporting IPv6 fully?</p><ul><li>Have to pick and choose battles, provided AWS with a list of services they want IPv6 support, don’t use Lambda much</li></ul><p>Security downsides to everything being public?</p><ul><li>NAT provides another layer of network access control</li><li>With V6 you have egress only IGW</li><li>You shouldn’t be relying on security through obscurity also</li></ul><p>What broke</p><ul><li>Different IPC clients behave differently — With GRPC</li><li>V6 Neighbour table limits hit</li><li>Triggered garbage collection</li><li>DNS spec chooses closest address in IPv6 causes hot spots in loadbalancing</li><li>Larger DNS responses trigger TCP connections</li><li>Canaries with IPv6 are chosen over the “normal” IPv4 clients and get all the traffic</li></ul><p>Positive surprise?</p><ul><li>Anyone who wanted IPC no longer needs a VPC peer, just use the IPV6 address</li><li>GRPC latency went down in one use case</li><li>Was able to transparently migrate people to IPv6</li><li>Automation made it possible to migrate 1000s of applications</li></ul><h3>The pragmatic serverless Python Developer</h3><p>YouTube Link — <a href="https://www.youtube.com/watch?v=52W3Qyg242Y">https://www.youtube.com/watch?v=52W3Qyg242Y</a></p><ul><li>Pragmatism, make decisions based on the context</li><li>Mkdocs-material</li><li>Pytest</li></ul><p>Pydantic</p><ul><li>Rewritten in Rust so its fast</li><li>Used to validate data structure</li></ul><p>Power tools for AWS Lambda</p><p>Leading CI Practices</p><ul><li>Project Setup</li><li>Pre Commit Checks</li><li>Pre Pull Request</li><li>Pull Request</li><li>Putting everything in one Lambda is fine to begin with but doesn’t scale</li><li>Can split code up into layers</li></ul><p>Handler</p><ul><li>Config</li><li>Validation</li><li>Calls domain layer</li><li>Passes response to caller</li></ul><p>Domain</p><ul><li>Stores data in DB</li><li>Passes response back to Handler</li></ul><p>Integration Layer</p><ul><li>Code that accesses an external API</li><li>Can you put this in the domain?</li><li>It depends</li><li>If integration layer code is mixed with Domain, it makes it harder to replace the external API</li><li>Can implement the adapter pattern</li></ul><p>Testing</p><ul><li>Serverless testing isn’t easy</li><li>Aim to never leave the IDE, work against real cloud resources to increase confidence it actually works</li><li>No mocks</li><li>AWS Powertools has mock events for many AWS services</li><li>Stubber from BotoCore can simulate AWS SDK errors</li><li>Lambda Event handler function needs to take the 2 AWS provided parameters but you can also pass in any other valid python</li><li>Can use this to inject dependencies</li><li>Lambda will always initialise additional kwargs as null, you can pass in an alternative when testing</li></ul><p>Event Driven Architecture</p><ul><li>Define the events in a Pydantic model to document the structure and validate events</li><li>Can include a version field to allow breaking changes</li><li>Add meta data so you know where events came from when it explodes</li></ul><p>Pytest Socket</p><ul><li>Library that kills all networking connections, useful for testing</li><li>Test double using Fakes rather than Mocks</li></ul><p>Tuna</p><ul><li>Visualises import time to debug cold start</li></ul><p>PySpy</p><ul><li>Create flame graph of function calls</li></ul><p>PyInstrument</p><ul><li>Visualisation for how long code takes to run</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a6c1c938b2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Summit London 2023 Notes]]></title>
            <link>https://medium.com/@AaronKalair/aws-summit-london-2023-notes-fd86f1edc782?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd86f1edc782</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 19:10:03 GMT</pubDate>
            <atom:updated>2023-06-07T19:10:03.592Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Power Of Cloud Network Innovation</h3><ul><li>Originally thought everyone only needed 1 VPC, had to call AWS to get more</li><li>VPC Peering used for 10–30 VPCs</li><li>If you have 100s, VPC peering doesn’t scale</li><li>AWS Transit Gateway is a virtual router that scales to thousands of VPCs</li><li>Firewalls, NAT Gateways, LBs make it all more complicated so you had to be a networking wizard to understand it</li></ul><p><a href="https://docs.aws.amazon.com/network-manager/latest/cloudwan/what-is-cloudwan.html"><strong>AWS CloudWAN</strong></a></p><ul><li>A way to unify multiple cloud and on premise networks</li><li>Define segments e.g. Prod Segment, Shared Services Segment</li><li>Defined in a JSON Blob</li></ul><p><strong>Service Mesh</strong></p><ul><li>Enables service to service connectivity</li><li>Have to deploy and manage the proxies</li><li>Only works with containers</li><li>Requires network expertise</li></ul><p><a href="https://aws.amazon.com/vpc/lattice/"><strong>VPC Lattice</strong></a></p><ul><li>No sidecars</li><li>Across all AWS services</li><li>No networking expertise</li><li>Traffic and Access Controls</li></ul><p><strong>Network Admins</strong></p><ul><li>Create an AWS Account for the networking resources, create a service network</li><li>Decide what AWS accounts have access to the networks created</li><li>Associate VPCs with the service network</li><li>Configure SGs</li></ul><p><strong>Developers</strong></p><ul><li>Build the services</li><li>Define routing policies</li><li>Associate services with the service network</li></ul><p><strong>Network Admins and Developers</strong></p><ul><li>Shared responsibility for security</li></ul><p><strong>Zero Trust</strong></p><ul><li><a href="https://aws.amazon.com/verified-access/">AWS Verified Access</a></li><li><strong>Secure access without a VPN</strong></li><li>Integrates with identity providers, sits in front of your applications associated with an ALB</li><li>New policy language to manage access</li><li>Policies can check if the device has appropriate security patches etc</li></ul><p><strong>SGs require IP addresses across regions, use Service Networks instead</strong></p><ul><li>One service network per region</li><li>Define policies for your service network at high level using IAM, e.g. all networks connecting need to be in my AWS org</li><li>Associate service network with VPC</li><li>Associate policies with services developers write</li><li>These policies replace SGs and control what services can talk to each other</li></ul><h3>Speeding up development of containerised applications</h3><ul><li>Inner loops — what developers do on local machines</li><li>Outer Loops — Outside of their machines e.g. CI / Deploy etc.</li></ul><p><strong>Containers</strong></p><ul><li>Require access to a Linux system</li><li>Is a grouping of other technologies</li></ul><p><strong>How to get access to Linux containers on Mac / Windows</strong></p><ul><li>Run a Linux VM</li><li>Connect to a remote Linux server</li></ul><p><strong>Linux VM Challenges</strong></p><ul><li>Security, there’s a whole new OS running, how do you keep it up to date</li><li>How do you access local files?</li><li>How do you access network services in the VM?</li></ul><p><a href="https://github.com/lima-vm/lima"><strong>Lima</strong></a></p><ul><li>Open Source tool for MacOS by ContainerD Maintainer</li><li>Handles files and networking out of the box</li><li>Part of a new generation of tools alongside WSL 2</li><li>The next step after things like Docker Machine</li></ul><p><a href="https://containerd.io/"><strong>ContainerD</strong></a></p><ul><li>Industry standard container runtime</li><li>Broken out of the Docker Project</li><li>Responsible for starting and stopping containers, and nothing else by design, is extendable for additional functionality</li><li>AWS is building on top of this, Fargate uses it, EKS uses it by default in the latest versions</li></ul><p><strong>ContainerD for Local Dev?</strong></p><p>Need a CLI that can talk to it</p><ul><li><a href="https://github.com/containerd/nerdctl">nerdctl</a></li></ul><p>Need a networking stack</p><ul><li><a href="https://github.com/containernetworking/cni">CNI</a></li></ul><p>Need to build and package images</p><ul><li><a href="https://docs.docker.com/build/buildkit/">BuildKit</a></li><li>Handles multi architecture well</li><li>This became an issue with M chips from Apple and Graviton instances</li><li>Can build an image for lots of different architectures through emulation</li></ul><p><a href="https://aws.amazon.com/blogs/opensource/introducing-finch-an-open-source-client-for-container-development/"><strong>Finch</strong></a></p><ul><li>Tool from AWS that packages all the above tools</li><li>Has the same API as Docker commands so is easy to switch to</li></ul><h3>Adopting AWS Serverless in Your Organisation</h3><p><strong>Serverless Compute</strong></p><ul><li>Serverless is more than compute, SQS / SNS, Step Functions, etc.</li><li>Lambda enables you to hand operational complexity to AWS</li></ul><p><strong>Cold Starts</strong></p><ul><li>Lambda takes a longer period of time to run your code if the environment isn’t ready</li><li>X-RAY lets you find expensive lines of code to help decrease cold start duration</li><li>Provisioned Concurrency lets you pay to eliminate the cold start if you know when a spike in traffic is coming</li></ul><p><strong>Resource Allocation</strong></p><ul><li>Graviton available for Lambda, cheaper and faster</li><li>CPU allocated is proportional to RAM</li><li><a href="https://github.com/alexcasalboni/aws-lambda-power-tuning">Lambda Power Tuning</a> finds optimal point of RAM / CPU to get best performance per $</li><li>Lambda Insights shows performance bottlenecks</li><li><a href="https://awslabs.github.io/aws-lambda-powertools-python/2.16.2/#local-development">Lambda Power Tools</a> library of Lambda Best Practices from AWS</li></ul><p><strong>Value of Abstractions</strong></p><ul><li>Embrace managed services, Serverless DBs mean no patching</li><li>The line item cost might be higher but the TCO is lower</li><li>Use orchestration to remove things like retries, branches etc from your code and let AWS provide that functionality</li><li>Let AWS write the Glue Code</li><li>AWS Step Functions</li></ul><p><strong>Integration Deep Dive</strong></p><ul><li>The appropriate level of coupling depends on the level of control you have over the endpoints</li></ul><p><strong>Embrace Async Communication</strong></p><p>Queues</p><ul><li>Sender pushes a message, Consumer picks them up when its ready</li></ul><p><strong>EventBus lets you do this is a centralised place</strong></p><ul><li>Event sources have 3rd party integrations, you can configure custom sources</li><li>Consumers can filter events</li><li>Rules route events to targets</li><li>Event Bridge can send events to SQS before you consume them to control the rate messages are consumed</li><li>API Gateways can send HTTP requests to a queue</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd86f1edc782" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Summit London 2022 Notes]]></title>
            <link>https://medium.com/@AaronKalair/aws-summit-london-2022-notes-a6ff695f8e6b?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6ff695f8e6b</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Thu, 28 Apr 2022 20:52:48 GMT</pubDate>
            <atom:updated>2022-04-28T20:52:48.845Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gareth Oakley on Twitter: &quot;#AWSSUMMITLONDON And this is all paid for by your Managed NAT Gateways pic.twitter.com/62dWethIDg / Twitter&quot;</h3><p>AWSSUMMITLONDON And this is all paid for by your Managed NAT Gateways pic.twitter.com/62dWethIDg</p><h3>Seven design patterns to increase the availability of your online applications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2M34w5rMt01HsXjInQ7H3g.jpeg" /></figure><p><strong>Food Tasting</strong></p><ul><li>Kings used to have squires taste food before eating it to make sure it was safe</li><li>Cloudfronts infrastructure is split into stripes, when configuration is updated each stripe pulls it at different times to make sure if there’s issues all of Cloudfront doesn’t break at the same time</li></ul><p><strong>Homogenous Platforms</strong></p><ul><li>If all your servers are the same then bugs in software updates break them all in the same way</li></ul><p><strong>Shuffle Sharding</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pn8_FotfOUqnZMhlZBU3w.jpeg" /></figure><ul><li>If all your workers can handle all your requests then things like DDOS attacks, or problematic messages on queues break the application for everyone</li><li>If you shard your workers so they only handle requests for certain combinations of customers then issues only affect certain customers and the others can continue to use your service without issue</li></ul><p><strong>Load Balancing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bP3gii2E544a2bQkRtBb8Q.jpeg" /></figure><ul><li>Virtual IPs (VIP) are associated with POPs</li><li>ECMP sends traffic to all the POPs</li><li>The Monitor process notices if servers go down and updates the router to send traffic to the other POPs</li><li>Each POP has a LoadBalancer in front of it, if the POP goes down the LoadBalancer sends traffic to other POPs instead</li></ul><p><strong>Origin Failure</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuEoBEDZB1jpc98qd9Vcfw.jpeg" /></figure><ul><li>Route53 resolves DNS to an origin</li><li>The origin is down, the DNS can resolve to a different origin</li><li>If you use Cloudfront, before the user gets the error, Cloudfront notices, and failovers to a different origin to get the object and respond to the user</li></ul><p><strong>Static Stability</strong></p><ul><li>If the dependencies fail the application can still function</li><li>In Cloudfront the Control Plane and Data Plane are seperate, so if the Control Plane goes down, the Data Plane still serves assets to the user, you just can’t update your deployment</li></ul><p><strong>Serverless at the Edge / Waiting Room Pattern</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ZUSePex6dISzC1roU0OGA.jpeg" /></figure><ul><li>Place a Lambda at Edge between the Origin and the User</li><li>The Lambda only lets x% of people to the origin, the others are redirect to a waiting room</li><li>The Lambda checks if the Origin is busy, or can accept new traffic and rewrites the URL appropriately</li></ul><p><strong>Reduce the Attack Surface</strong></p><ul><li>Can prevent your origin from being accessed by anyone except Cloudfront</li><li>All traffic then has to flow through a WAF and Cloudfront</li></ul><p><strong>Defend Against DDOS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oq6qBQnfyyWLuFqeMJtiSQ.jpeg" /></figure><ul><li>Everyone on AWS gets Shield for free</li><li>Shield Advanced is paid for and offers other features</li><li>It can automatically mitigate L7 attacks based on known patterns</li><li>WAFs can rate limit requests from certain IPs</li></ul><p><strong>Reduce False Positives</strong></p><ul><li>Apply changes to your WAF in batches not all at once</li></ul><h3>Scale Permissions Management in AWS w/ Attribute-Based Access Control</h3><p><strong>Permissions define who can access what</strong></p><ul><li>Who — People / Applications</li><li>What — S3, EC2 DynamoDB etc</li><li>IAM handles 0.5 billion checks per second</li></ul><p><strong>RBAC — Role Based Access Control</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ASwXATmqyIg2BmIbSlVsHQ.jpeg" /></figure><ul><li>Defines what resources a job role needs access to</li><li>As you get new resources you have to go and update the roles</li></ul><p><strong>ABAC — Attribute Based Access Control</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4DaSCTqCnDSYWwoCK6qIw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGBXiobIT98p6k0EhZy-5g.jpeg" /></figure><ul><li>Attribute — Key Value pairs</li><li>These can be defined by your identity provider</li><li>These can also be added as tags manually to resources</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BY36Fp-dQo3W53gs4_lDhA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzBXgbbOwjT1gskJNWxGnA.jpeg" /></figure><p>Admins and Developers work together to make ABAC work</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0jX1urEao9IqLjEEmv8cA.jpeg" /></figure><ul><li>Admins define the Roles and allowed attributes</li><li>Developers tag there resources with the right attributes</li></ul><p><strong>Demo</strong></p><ul><li>Create a role for cat and dog</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJxuZbQrkGbFIP76rTJqdg.jpeg" /></figure><ul><li>Developers are only allowed to create Secrets if they have the right tags</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGDjVHQn-IkUDfX0ecU-Ow.jpeg" /></figure><ul><li>Developers can then only manage existing Secrets if they have tags that match there access permissions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjunuQP9Pysov7Z952ZPYQ.jpeg" /></figure><ul><li>You need to ensure Developers can’t modify the Tag values to gain access to more resources</li><li>RequestTag in IAM is the Tags on the User creating the resource</li><li>ResourceTag in IAM the Tags on the existing resource</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fn9rMhMyP35-fxa9RBS3Q.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6ff695f8e6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Operation Jumbo Drop: How sending large packets broke our AWS network]]></title>
            <link>https://medium.com/in-the-hudl/operation-jumbo-drop-how-sending-large-packets-broke-our-aws-network-ff5041fc7a09?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff5041fc7a09</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Wed, 23 Mar 2022 21:21:33 GMT</pubDate>
            <atom:updated>2022-03-23T21:21:33.971Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Hudl recently ran into some really strange networking failures in our testing environment — we tracked them down to a pretty obscure network setting in AWS EC2.</p><p>If you’re interested in Chef or Linux debugging, AWS networking troubleshooting, or just love a good rabbit hole, read on.</p><h3>Background</h3><p>Every weekend at Hudl we run a process to restore a sanitised version of our production databases to our staging environment, to give our developers realistic data to test with before going to production.</p><p>This Monday we woke up to a Slack room full of errors from the restoration process, telling us that the Chef run which configures the databases had failed on every single EC2 instance.</p><p>The error we were presented with wasn’t super helpful, “EOFError, end of file reached.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*fFKxf3vNH0LNFxya" /></figure><p>It’s unclear what file it’s talking about. The recipe referenced does write multiple configuration files out, but they haven’t changed for years and it’s unclear why this would suddenly break.</p><p>When we logged on to the server ourselves and ran chef-client, we were able to replicate the exact error. So at least we had a starting point.</p><h3>Is it an access key error?</h3><p>Our first suspicions were that the access keys the server was using to talk to the Chef Server were invalid. We’d recently rotated the Chef keys associated with this process and could have broken them.</p><p>It seemed unlikely as chef-client had clearly been able to pull cookbooks down and start running, but it was the only thing we’d changed recently.</p><p>Knife is a CLI that administrators can use to interact with the Chef server. In theory it uses the same API calls as chef-client, so we attempted to use knife to communicate with the Chef server to see if the key was valid.</p><pre>knife node list -c /etc/chef/client.rb</pre><p>We were able to retrieve a list of all our nodes registered with Chef and the keys seemed fine.</p><h3>What do the network hangs mean?</h3><p>Running Chef again, we noticed that it hung for around two minutes before printing the EOFError, and if you killed the chef-client run with ctrl-c, the stack trace indicated it was stuck trying to write a node attribute back to the Chef Server.</p><pre>5: node.normal[:aws][:ebs_volume] = Hash[node[:volumes].map { |v| [<br>6: v.fetch(:name, v[:device][:path].split(‘/’)[-1]),<br>7: {:volume_id =&gt; v[:device].fetch(:volume_id, nil)}]}].reject { |k, v| v[:volume_id].nil? }<br>8&gt;&gt; node.save</pre><p>We repeated this a few times and it was always consistently stuck here.</p><p>Was this some sort of networking issue talking to the Chef Server?</p><p>It seemed weird we’d have a networking issue talking to the Chef Server. We’d successfully pulled down all the cookbooks from it, and had been able to talk to the Chef Server with knife.</p><p>We decided to use knife to try and save a node attribute; this was, after all, what the chef-client seemed to get stuck on.</p><p>We ran:</p><pre>EDITOR=vim knife node edit &lt;NODE_NAME&gt; -c /etc/chef/client.rb</pre><p>And edited a random attribute, which worked fine.</p><h3>What does the Chef Server See?</h3><p>We were slightly confused as to why chef-client was failing to save a node attribute to the server, as we could do this using knife. We decided to run chef-client with debug logs enabled, hoping that it would give us some context around the error.</p><pre>chef-client -l debug</pre><pre><strong>[2022–02–28T11:18:53+00:00] DEBUG: Initiating POST to </strong><a href="https://chef12-server.app.hudl.com/organizations/hudl/reports/nodes/"><strong>https://chef12-server.app.hudl.com/organizations/hudl/reports/nodes/</strong></a><strong>&lt;node name&gt;/runs/&lt;run_id&gt;</strong></pre><pre>[2022–02–28T11:18:53+00:00] DEBUG: Content-Length: 301<br>[2022–02–28T11:18:53+00:00] DEBUG: — — End HTTP Request Header Data — — <br>[2022–02–28T11:18:53+00:00] DEBUG: — — HTTP Status and Header Data: — — <br>[2022–02–28T11:18:53+00:00] DEBUG: HTTP 1.1 200 OK<br>[2022–02–28T11:18:53+00:00] DEBUG: server: openresty/1.7.10.1<br>[2022–02–28T11:18:53+00:00] DEBUG: date: Mon, 28 Feb 2022 11:18:53 GMT<br>[2022–02–28T11:18:53+00:00] DEBUG: content-type: application/json<br>[2022–02–28T11:18:53+00:00] DEBUG: content-length: 2<br>[2022–02–28T11:18:53+00:00] DEBUG: connection: close<br>[2022–02–28T11:18:53+00:00] DEBUG: — — End HTTP Status/Header Data — — <br><strong>[2022–02–28T11:18:53+00:00] DEBUG: EOFError: end of file reached</strong></pre><p>The logs seem consistent with the stack trace we got before. The HTTP POST is presumably the update to the node attribute.</p><p>We’d come as far as we could on this server, and decided to see if the logs on the other side of the HTTP connection had any more information.</p><p>The Nginx log on the Chef Server had more information on the request. It had returned a HTTP 408, which is a <a href="https://http.cat/408">timeout</a>.</p><pre>172.27.75.108 — — [28/Feb/2022:11:57:20 +0000] “PUT /organizations/hudl/nodes/&lt;node name&gt; HTTP/1.1” 408 “60.058” 0 “-” “Chef Client/12.14.89 (ruby-2.3.1-p112; ohai-8.20.0; x86_64-linux; +https://chef.io)&quot; “-” “-” “-” “12.14.89” “algorithm=sha1;version=1.1;” “&lt;node name&gt;” “2022–02–28T11:55:20Z” “&lt;random id&gt;” 1164</pre><p>We now seemed to be getting somewhere — but what on the Chef Server was causing the requests to timeout?</p><h3>Is the Chef Server healthy?</h3><p>The Chef Server runs a lot of different processes, from Rabbit, Solr and Redis, to the actual Chef Server processes itself. This makes it hard to understand the “health” of the Chef Server, so we had to check each process was running and read the logs from them one by one.</p><p>There were a few red herrings in the log files, errors that are apparently normal, but overall nothing seemed obviously broken.</p><h3>Is chef-client broken everywhere?</h3><p>Confused about what was happening, we started to look at if it was just our newly restored databases or a bigger problem.</p><p>All our production servers could still run Chef fine, but we could replicate the same error on all our servers inside our staging VPC.</p><h3>Has anyone else had this issue?</h3><p>Even more confused, we turned to Google to see if others had this oddly specific error where some servers couldn’t write to the Chef Server.</p><p>It turns out others had experienced almost the exact same error: <a href="https://github.com/chef/chef/issues/1937">https://github.com/chef/chef/issues/1937</a>.</p><p>Worryingly, the issue seemed to be related to low level networking settings: <a href="https://github.com/chef/chef/issues/1937#issuecomment-53661631">https://github.com/chef/chef/issues/1937#issuecomment-53661631</a>.</p><p>It seemed like a large MTU setting could cause servers to be unable to send data to the Chef Server if the requests were over a certain size in some environments.</p><h3>What is an MTU?</h3><p>The <a href="https://www.cloudflare.com/en-gb/learning/network-layer/what-is-mtu/">MTU</a> setting on a network interface specifies the maximum size of the packets the interface sends.</p><p>Due to the overhead of things like headers on the packets, it’s more efficient to send fewer but larger packets.</p><p>But the internet is littered with an assortment of networking devices, some of which can only handle smaller packets.</p><p>This means you may be forced to send smaller packets to ensure all the devices they pass through can handle them.</p><p>A generally accepted MTU is <a href="https://serverfault.com/a/325987">1500</a>. Some networks such as inside AWS VPC support MTUs of 9001 (referred to as Jumbo Frames), and it appears the AMIs AWS provides launch EC2 instances with the MTU set to 9001, which is what our EC2 instances have.</p><h3>Can we replicate the issue reported on Github?</h3><p>This issue seemed plausible for explaining why our tests worked with knife but the chef-client run still failed. We were still slightly sceptical and wanted to try and confirm this was the issue.</p><p>Both our Chef Server and Database had MTUs of 9001.</p><pre>[ec2-user@<strong>mongo</strong> ~]$ ifconfig<br>eth0 Link encap:Ethernet HWaddr 0E:36:F3:9C:96:6D<br>inet addr:172.27.75.108 Bcast:172.27.79.255 Mask:255.255.248.0<br>inet6 addr: fe80::c36:f3ff:fe9c:966d/64 Scope:Link<br>UP BROADCAST RUNNING MULTICAST <strong>MTU:9001</strong> Metric:1<br></pre><pre>[root@<strong>chef-server</strong> ~]# ifconfig<br>eth0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; <strong>mtu 9001<br></strong>inet 10.110.78.16 netmask 255.255.224.0 broadcast</pre><p>On the Chef Server we ran netcat in listening mode, printing out the number of bytes it received.</p><pre>[root@chef-server ~]# nc -l 9234 | wc — bytes</pre><p>On the MongoServer we used dd to generate fixed size amounts of data and send it to our netcat process.</p><pre>ec2-user@t-augmentation-mongo-rs1-use1d-520nmi ~]$ dd if=/dev/urandom count=1 bs=9002 | nc chef12-server.app.hudl.com 9234</pre><p>We couldn’t replicate the exact findings in the Github issue thread — packets of size 9002, 1500 and all sizes in between worked fine.</p><p>We randomly tried some larger sizes, and 20,000 byte requests seemed to replicate the issue as they were never received by the Chef Server.</p><p>Some binary searching led us to discover that 10,497 bytes was the tipping point. Any requests equal to or larger than this failed, but smaller than this worked.</p><p>It still isn’t clear why 10,497 is the tipping point, and not 9002, but presumably it’s related to headers on the packets or something like that.</p><h3>What’s actually happening to the packets?</h3><p>Curious about what was actually happening, we used tcpdump to capture the traffic on the database instance (we tried on the Chef Server, but installing tcpdump required updating a bunch of other packages that we weren’t sure were safe to upgrade), and the results were interesting.</p><p>Working requests were unexciting. The TCP connection was established, the packets sent, and the connection torn down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p4jxu7buXRJhILuA" /></figure><p>On the failed requests, the TCP connection was established and then after sending the first packet of data, no ACK is ever received and it just goes into a constant TCP retry loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y-R9rota3vWCCZKQ" /></figure><p>So the size of the packet does seem to affect if it reaches the destination.</p><h3>Does a lower MTU fix the issue?</h3><p>Now we lowered the MTU on the database server from 9001 to 1500 and repeated the tests to see if it made any difference.</p><p>It fixed the issue! Our chef-client run succeeded and by repeating the test with netcat, we confirmed data of all sizes made it to the server.</p><p>We’d narrowed down the issue but more questions still remained:</p><ol><li>What caused this to break? It worked fine last week.</li><li>How were we going to fix this? Do we really have to go and change our MTU everywhere in staging? Why does it still work in Prod?</li></ol><h3>Can tracepath help?</h3><p>Some Googling suggested that tracepath might be able to show us what the MTU was on the path between our database and Chef Server.</p><pre>[root@t-augmentation-mongo-rs1-use1d-520nmi ~]# tracepath 10.110.78.16</pre><pre>1?: [LOCALHOST] pmtu 1500<br>1: no reply<br>2: ip-10–110–78–16.ec2.internal 1.262ms reached</pre><pre>Resume: pmtu 1500 hops 2 back 1</pre><p>As expected it confirms 1500.</p><p>To experiment we set the database servers MTU back to 9100 and ran the tool again.</p><pre>[root@t-augmentation-mongo-rs1-use1d-520nmi ~]# tracepath 10.110.78.16</pre><pre>1?: [LOCALHOST] pmtu 9100<br>1: no reply<br>2: no reply<br>&lt;snip&gt;<br>31: no reply</pre><pre>Too many hops: pmtu 9100<br>Resume: pmtu 9100</pre><p>This time it didn’t seem to work. We were under the impression tracepath would try different MTUs until it found one that worked, but it just seems broken.</p><p>We played around with tracepath some more and were never able to get it work, unless we manually lowered the servers MTU to 1500.</p><h3>What about Path MTU Discovery?</h3><p>This wasn’t getting us any closer to a fix though, so next we turned our attention to Path MTU Discovery.</p><p>Path MTU Discovery is supposed to be a way for servers with differing MTUs to negotiate a mutually compatible MTU setting for their packets.</p><p>If a device between the servers, or the server itself, receives a message that is too large for it to process, it drops the packet and sends back an ICMP Fragmentation Needed packet. This signals to the client that it must reduce its MTU and send the packet again.</p><p>Why wasn’t this working for us, and allowing our servers to negotiate a compatible MTU?</p><p>We suspected that maybe the security groups were blocking the ICMP messages. As an experiment we altered the security groups for both the Chef Server and Database to allow ICMP packets in from anywhere, but it made no difference.</p><h3>How do the Prod and Staging networks differ?</h3><p>Running out of ideas, we turned our attention to try and work out why the Prod servers could reach the Chef Server but the staging servers could not. What were the differences in their networks?</p><p>Our Chef Server lives in its own VPC for “internal tools,” whilst Prod and Staging are separated into their own VPCs.</p><p>During this investigation, we discovered that just before the weekend, the staging VPC had been altered by another team to send its packets to the internal tools VPC via a Transit Gateway. However, the return packets still went down a VPC Peering connection.</p><p>The Prod VPC was untouched, and used a separate proxy for communicating with the Chef Server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdLkVjEMxljUpVOPSkcUrQ.jpeg" /></figure><p>This seemed like it must be the issue, but it was still unclear. Why had we configured our Transit Gateway incorrectly?</p><h3>How did the Transit Gateway cause the issue?</h3><p>Reading the AWS Documentation on MTUs, there’s a small note about MTUs and TransitGateways</p><blockquote>“For more information about supported MTU sizes for transit gateways, see MTU in Amazon VPC Transit Gateways.”</blockquote><p>Buried in the “Quotas” section of the MTU docs are these useful points:</p><blockquote>A transit gateway supports an MTU of 8500 bytes for traffic between VPCs, Direct Connect gateway, and peering attachments. Traffic over VPN connections can have an MTU of 1500 bytes.</blockquote><blockquote>When migrating from VPC peering to use a transit gateway, an MTU size mismatch between VPC peering and the transit gateway might result in some asymmetric traffic packets dropping. Update both VPCs at the same time to avoid jumbo packets dropping due to a size mismatch.</blockquote><blockquote>Packets with a size larger than 8500 bytes that arrive at the transit gateway are dropped.</blockquote><blockquote>The transit gateway does not generate the FRAG_NEEDED for ICMPv4 packet, or the Packet Too Big (PTB) for ICMPv6 packet. Therefore, the Path MTU Discovery (PMTUD) is not supported.</blockquote><p>We had essentially just reverse engineered all of this information.</p><p>Transit Gateways don’t support MTUs of 9001, they don’t support Path MTU Discovery, and asymmetric paths can cause issues, all of which we’d been experiencing.</p><p>Some more Googling revealed this fantastic blog post, where they’d experienced the exact same issue we had: <a href="https://medium.com/seek-blog/beware-of-transit-gateways-bearing-large-packets-77702c4c1b20">https://medium.com/seek-blog/beware-of-transit-gateways-bearing-large-packets-77702c4c1b20</a>.</p><p>We won’t repeat all of what they’ve explained excellently. Go and read that post for the full technical details on what went wrong here and then come back.</p><p>The short version is when using TCP connections:</p><ul><li>Networking devices such as TransitGateways can rewrite the value of a field in the TCP Header during connection establishment, called the Maximum Segment Size</li><li>This field tells the receiver the maximum size of the packets it should send in response</li><li>Packets sent from the database to the Chef Server would have had the MSS changed to 8500 as they passed through the TransitGateway</li><li>Packets sent from the Chef Server to the database would NOT have the MSS changed to 8500 as they don’t pass through the TransitGateway, so the database server doesn’t understand it shouldn’t send packets greater than 8500 bytes</li><li>The initial connection was able to be established via these asymmetric paths as the SYN / SYN ACK packets are naturally smaller than 8500 bytes</li></ul><h3>How did we verify Asymmetric Network Paths were indeed the issue?</h3><p>We altered the Route Tables to send the traffic in both directions through the TransitGateway and everything started working again with the MTU still set to 9001.</p><p>We took packet captures again, whilst sending 10,497 byte packets from the database to the Chef Server to verify the MSS is being altered with the fix in place.</p><p>Let’s look at what we have from the database client <strong>before</strong> the fix.</p><p>On the TCP SYN packet the Database sent to the Chef Server the MSS is 8961.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*NCMQHdC3DqGEberzqWggjw.png" /></figure><p>On the TPC SYN ACK, the Chef server sent back to the Database the MSS is 8961.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*MSa0bUxXQ5fQwJkRn2Br_g.png" /></figure><p>And now compare that to the databases perspective <strong>after</strong> the issue is fixed.</p><p>The SYN packet the database sends to the Chef Server still has a MSS of 8961.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*NCMQHdC3DqGEberzqWggjw.png" /></figure><p>The SYN ACK we get back from the Chef Server to the database has a smaller MSS of 8460.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*yCsJaKC83qwvKxXKSy4mog.png" /></figure><p>There we have it — verification that the MSS was not getting set to the smaller maximum supported value when we had the issue.</p><p>After the fix, when traffic in both directions was going through the TransitGateway, the TransitGateway alters the MSS to the correct lower value so the clients can send appropriately sized packets through it.</p><h3>Conclusion</h3><p>This investigation took around a day and a half to complete, and took us deep into the intricacies of how networking inside AWS works.</p><p>Without the Github issue and blog post from others experiencing similar issues, it may have taken significantly longer to fix this.</p><p>We hope this post might similarly prove useful as we highlighted some networking debugging tools and techniques you can use to troubleshoot issues.</p><p>It was an interesting dive into an area we don’t normally have to touch. Most importantly, we got everything working again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff5041fc7a09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-hudl/operation-jumbo-drop-how-sending-large-packets-broke-our-aws-network-ff5041fc7a09">Operation Jumbo Drop: How sending large packets broke our AWS network</a> was originally published in <a href="https://medium.com/in-the-hudl">In The Hudl</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Code Layout and Using Terragrunt]]></title>
            <link>https://medium.com/@AaronKalair/terraform-code-layout-and-using-terragrunt-db6864967916?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/db6864967916</guid>
            <category><![CDATA[terragrunt]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sun, 10 Oct 2021 12:54:43 GMT</pubDate>
            <atom:updated>2021-10-10T12:54:43.425Z</atom:updated>
            <content:encoded><![CDATA[<p>Terraform lets you organise your code however you want.</p><p>This gives you lots of flexibility and makes it easy to get started by just throwing a few resources into a file and running terraform apply</p><p>But as your environment grows, you’ll want to be more disciplined about how you structure your code.</p><p>This post talks about how we started with a simple freeform layout of files in a single folder, and then started moving to Terragrunt to address some of the issues that brought up.</p><p>The first thing you’ll need when working with Terraform is to manage the state file.</p><h3>State Management</h3><p>Terraform state is where Terraform stores information about the resources you’ve created with Terraform.</p><p>If you’re a single developer you could just keep the terraform.tfstate file locally on your machine (not checked into Git as it can contain potentially sensitive information) but when working on a team you need two things:</p><ol><li>A way to share this state file securely</li><li>A way to prevent multiple developers modifying the state file at the same time</li></ol><p>Thankfully the s3 backend type solves all of these problems — <a href="https://www.terraform.io/docs/language/settings/backends/s3.html">https://www.terraform.io/docs/language/settings/backends/s3.html</a></p><p>This stores the terraform.tfstate file in an S3 bucket, and uses DynamoDB to obtain an exclusive lock before performing actions that need the state file.</p><p>This does raise some new problems though, how do you initially create the S3 bucket and DynamoDB table?</p><p>There’s numerous solutions, you could do it with Cloudformation, or manually.</p><p>But we wanted to keep it all in Terraform, so we have a module that creates an S3 bucket and a DynamoDB table.</p><p>This is instantiated once in each environment in a bootstrap folder to give us environment specific S3 buckets and DynamoDB tables for the “main” Terraform State and the Terraform state for those 2 bootstrap resources is stored in Git.</p><p>These 2 initial resources are never modified after the initial creation so we’re not worried about having a lock to protect this state from concurrent modification and it doesn’t contain any sensitive date so we’re happy to commit it to Git.</p><h3>How to Layout the Terraform Code</h3><p>With the state file management sorted, you can move onto defining some resources you need to create.</p><p>One place to start is here:</p><pre>terraform/<br>  dev/<br>    bootstrap/<br>      bootstrap.tf<br>      terraform.tfstate</pre><pre>  staging/<br>    bootstrap/<br>      bootstrap.tf<br>      terraform.tfstate</pre><pre>  prod/<br>    bootstrap/<br>      bootstrap.tf<br>      terraform.tfstate</pre><p>Each of these folders is a seperate Terraform state and so if you make a mistake terraform applying in one environment it won&#39;t affect the others.</p><p>Inside each folder you’ll then need to decide how to split files containing module and resource definitions.</p><p>You could have just have one giant terraform.tf file and put every single definition in that.</p><p>But you’re going to get a lot of merge conflicts and find it nearly impossible to locate where specific resources are defined.</p><p>So a sensible layout is something like:</p><pre>/terraform<br>   dev/<br>     application_a.tf<br>        -&gt; resources / modules for app A<br>     application_b.tf<br>        -&gt; resources / module for app B<br>     vpc.tf<br>        -&gt; VPC / Subnets / NAT Gateways etc.<br>     eks.tf<br>        -&gt; K8s Cluster</pre><p>Tweak for your exact set of technologies but you get the idea.</p><h4>Variables</h4><p>The next thing you’ll want is a way to define variables, the modules for app A probably need to take the name of App A so they can tag resources / add identifiers so you’ll put a locals block in application_a.tf</p><pre>locals {<br>    app_a_name = &quot;A&quot;<br>}</pre><p>And then in the module definition reference it</p><pre>module app_a_resources {<br>    app_name = local.app_a_name<br>}</pre><p>Note that every file in the dev/ directory will have access to this locals block so you can’t just call the variable app_name</p><p>You’ll probably also have some variables that need to be the same across all your apps, for example maybe your company has agreements on when maintenance windows are for RDS instances.</p><p>So you either copy paste the window into the module definition for every database across all the apps, or define it as a local somewhere and reference the local in every module that needs it.</p><h4>Issues with this Approach</h4><p>This will probably work well for some time, but you’ll notice some issues start to creep in.</p><ol><li>As your entire environment is a single Terraform state terraform operations such as plan and apply will start taking longer and longer as they process every resource in the environment.</li><li>As your terraform repo grows the layout is going to become less consistent.</li></ol><p>People are going to name files e_resources.tf rather than application_e.tf</p><p>There going to forget / not be able to find the local definition for maintainence_window = &quot;mon12:13&quot; and create new local definitions such as maint_window or just hardcode a string in as the variable.</p><p>People are going to create Terraform modules with subtle variations on variable names, app_name will be app in some places, subnet_id will be subnet in others.</p><p>This isn’t a massive pain you can just remap the local name to match in the module definition.</p><pre>module a_resources {<br>   app = local.app_name<br>}</pre><p>But as people try to read / modify and extend the code its going to cause mistakes as people don’t catch the subtle variable name changes.</p><p>Eventually you’ll have another AWS region, and now you need the maintainence_window to be different to account for time zones and your customers being active at different times.</p><p>No worries, you create eu_maintainence_window and in all the eu module definitions write</p><pre>module eu_a_resources {<br>   maintainence_window = local.eu_maintainence_window<br>}</pre><p>One day someone will copy paste the maintainence_window definition when creating a EU module and forget to change it, so the database reboots during peak time for your EU customers.</p><p>Ideally you’d rename maintainence_window to us_maintainence_window everywhere, but in reality time pressures normally mean these refactors aren’t fully completed.</p><p>3. Your list of .tf files in a single directory is now getting quite large, you have to grep to find what you’re looking for quite often.</p><p>4. One day someone quickly defines an extra resource for app_c in devoutside of the app_c_resources module and directly in the dev/app_c.tf file, so they can quickly test something.</p><p>They forget to ever move the resource into the module and so when app_c goes live it’s missing a resource or changes haven’t been copy pasted across from dev/app_c.tf to prod/app_c.tf</p><p>Most of these problems can be addressed, rigorous code reviews and commitments to refactoring when variable names drift you could keep your Terraform repo neat and tidy.</p><p>The ever increasing size of your Terraform state is going to slow things down though.</p><p>Also remember only one person can obtain the lock to modify the state file so you may end up blocked waiting for a turn to run terraform apply</p><p>Finally if you accidentally corrupt the state, that’s a lot of resources potentially affected (make sure you have S3 versioning enabled!)</p><p>To address these issues we started using <a href="https://terragrunt.gruntwork.io/">Terragrunt</a> which has worked out really nicely.</p><h3>Terragrunt</h3><p>For a long time I struggled to understand what the point of Terragrunt was or what exactly it did.</p><p>The best way I can explain it is that Terragrunt places restraints on how you can organise your Terraform code, and forces you to use directory structure hierarchies and shared variable definition files to organise your code.</p><p>These restraints force your code to be more consistent and makes it harder to make mistakes, whilst reducing the amount of flexibility you have.</p><p>I don’t think I could have ever truly appreciated Terragrunt without first going through the above and seeing the issues with just being able to go freestyle.</p><h4>Logically Organising Your Infrastructure</h4><p>To use Terragrunt you must decide how your infrastructure can be logically broken down into smaller and smaller groups.</p><p>E.g. at the top level you could have an environment (e.g. dev), then a region (e.g. us-east-1), then an application (e.g. app-a), and then finally pieces of infrastructure that application needs (e.g. database, cache, s3 bucket).</p><p>And so you’ll have a folder structure something like</p><pre>dev/<br>  -&gt; us-east-1/<br>    -&gt; applications/<br>      -&gt; app-a<br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl</pre><p>Your directory structure represents how your infrastructure is organised.</p><p>The terragrunt.hcl files are what Terragrunt reads to understand what Terraform module to apply, more on those later.</p><p>Now then what if we add another app that just needs an IAM Role?</p><pre>dev/<br>  -&gt; us-east-1/<br>    -&gt; applications/<br>      -&gt; app-a<br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl<br><strong>      -&gt; app-b/<br>        -&gt; iam-role/<br>          -&gt; terragrunt.hcl</strong></pre><p>This is great, but what about resources shared across a region like an EKS Cluster?</p><p>You can put folders where ever you like, so lets make some at the environment level.</p><pre>dev/<br>  -&gt; us-east-1/<br><strong>    -&gt; eks-cluster/<br>      -&gt; terragrunt.hcl</strong></pre><pre>    -&gt; applications/<br>      -&gt; app-a<br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl<br>      -&gt; app-b/<br>        -&gt; iam-role<br>          -&gt; terragrunt.hcl</pre><p>Now why did we place app-a and app-b in a folder called applications ?</p><p>Couldn’t they have just been under us-east-1 ?</p><p>They could have, but if you have a lot of applications in your company, being able to group all of them in one sub folder might be nicer to reason about, and it gives you the option to have a shared values file exclusive to all your applications.</p><h4>Values Files</h4><p>Let&#39;s talk about shared values files, another big feature of Terragrunt.</p><p>Every Terraform module will take input variables that control the exact details of the resources it creates.</p><p>Some of these input variables are likely to be the same across all resources deployed in an environment, e.g. environment_name for things like tagging.</p><p>Rather than writing environment_name=dev in every single terragrunt.hcl file, lets define all those environment level variables in a file called environment.yaml</p><pre>dev/<br>  <strong>-&gt; environment.yaml</strong></pre><pre>  -&gt; us-east-1/<br>    -&gt; eks-cluster/<br>      -&gt; terragrunt.hcl</pre><pre>  -&gt; applications/<br>      -&gt; app-a/<br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl<br>      -&gt; app-b/<br>        -&gt; iam-role/<br>          -&gt; terragrunt.hcl</pre><p>And environment.yaml will look like:</p><pre>environment_name: dev</pre><p>Next we have some region level settings, for example in us-east-1 we might want our maintainence_window for RDS instances to be during mon05:00-07:00 a quiet time for our US customers.</p><pre>dev/<br>  -&gt; environment.yaml</pre><pre>  -&gt; us-east-1/<br>    <strong>-&gt; region.yaml</strong></pre><pre><strong>  </strong>-&gt; eks-cluster/<br>      -&gt; terragrunt.hcl</pre><pre>  -&gt; applications/<br>      -&gt; app-a/<br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl<br>      -&gt; app-b/<br>        -&gt; iam-role/<br>          -&gt; terragrunt.hcl</pre><p>And region.yaml will look like</p><pre>maintainence_window: mon05:00-07:00</pre><p>Now if we expanded into the EU and the quiet time for our EU customers was tue21:00-22:00 we could do something like:</p><pre>dev/<br>  -&gt; environment.yaml</pre><pre>  -&gt; us-east-1/<br>    <strong>-&gt; </strong>region.yaml</pre><pre>&lt;SNIP&gt;<br>  <br><strong>  -&gt; eu-west-1/<br>    -&gt; region.yaml<br>  -&gt; applications/<br>      -&gt; app-c/<br>        -&gt; database/<br>          -&gt; terragrunt.hcl</strong></pre><p>And eu-west-1/region.yaml would look like</p><pre>maintainence_window: tue21:00-22:00</pre><p>I’m sure you’re starting to get the idea now, you can easily propagate shared variable down the sub trees to easily be able to customise your infrastructure without having to copy paste values everywhere.</p><p>The final shared variable we’ll create will be for each application, the name of the app is likely to appear somewhere in the resources for it.</p><pre>dev/<br>  -&gt; environment.yaml</pre><pre>  -&gt; us-east-1/<br>    -&gt; region.yaml<br>    -&gt; eks-cluster/<br>      -&gt; terragrunt.hcl</pre><pre>  -&gt; applications/<br>      -&gt; app-a/<br>        <strong>-&gt; app.yaml</strong><br>        -&gt; database/<br>          -&gt; terragrunt.hcl<br>        -&gt; cache/<br>          -&gt; terragrunt.hcl<br>        -&gt; s3-bucket/<br>          -&gt; terragrunt.hcl<br>      -&gt; app-b/<br>        <strong>-&gt; app.yaml<br></strong>        -&gt; iam-role/<br>          -&gt; terragrunt.hcl</pre><p>And each app.yaml will look something like:</p><pre>app_name: &lt;NAME&gt;</pre><p>Now then, not every input variable will be generic, there’s going to be some specific to individual modules.</p><p>For those you can supply them directly to the module.</p><p>Lets look at what terragrunt.hcl actually is:</p><pre>terraform {<br>  source = &quot;Link to Terraform Module on Github&quot;<br>}</pre><pre>include {<br>  path = find_in_parent_folders()<br>}</pre><pre>inputs = {<br>  module_specific_variable = &quot;amazing&quot;<br>}</pre><p>It’s a fairly simple file.</p><p>You’ll need a link to the module you want to apply, it only works with Github links so no Terraform Module Registry references here.</p><p>Ignore include for now, it’s related to finding those shared variable files, we’ll come back to that</p><p>inputs is where you can pass any inputs to modules.</p><p>If an input variable has the same name as a variable defined in one of our shared variable .yaml files then it will be automatically picked up.</p><p>If the variable doesn’t come from a shared definition file you can enter it manually here.</p><p>There’s a few more things to mention about input variables whilst we’re here.</p><p>We’ll configure Terragrunt to use the first definition of a variable that it finds so that we can override generic values supplied by the .yaml files with more specific ones further down the tree.</p><p>You’ll probably also find situations where you’ve called the generic variable something like environment_name but some modules expect it to be called environment or env_name .</p><p>This is a nice feature of Terragrunt because it will quickly become clear where your modules aren’t consistent and you can work to bring things inline over time.</p><p>In the short term you can remap the generic variable names to the more specific module ones.</p><pre>locals {<br>  env_vars = yamldecode(<br>    file(&quot;${find_in_parent_folders(&quot;environment.yaml&quot;)}&quot;),<br>  )<br>}</pre><pre>inputs {<br>  env_name = local.env_vars[&#39;environment_name&#39;]<br>}</pre><h4>Dependencies</h4><p>Finally, it’s likely that the input to one module is going to be an output from another.</p><p>For example let&#39;s say our eks-cluster module outputs worker-sg-id the ID of the Security Group the K8s workers use.</p><p>Then our database module takes an input parameter of sg_to_allow_access_from the ID of a Security Group it will create an ingress rule for.</p><p>You can use the output of one module as an input to another like this …</p><pre>dependency &quot;k8s-worker-sg&quot; {<br>  config_path = &quot;../../eks-cluster&quot;<br>}</pre><pre>inputs = {<br>  sg_to_allow_access_from = dependency.k8s-worker-sg.outputs.worker-sg-id<br>}</pre><p>That covers pretty much everything you’ll need to get started with laying out the Terragrunt files and not copy pasting your variables everywhere.</p><h4>Terragrunt Configuration File</h4><p>But we still can’t run terragrunt yet there’s some configuration needed to tie it all together.</p><p>Previously we’d seen that the Terraform state file encompassed every resource in our environment.</p><p>This made Terraform commands take ever increasing amounts of time to run as our environment grew, and we risked affecting every resource if we corrupted the state somehow.</p><p>In this Terragrunt setup we can create one state file per “leaf node” of the directory tree, so essentially where ever there’s a terragrunt.hcl file defining a module to be applied we create a new state file.</p><p>This makes Terraform operations super fast, and reduces the consequences of corrupting a state file.</p><p>We need to create a terragrunt.hcl file under dev/us-east-1/terragrunt.hcl and rather than defining a Terraform module to apply, it defines all our Terragrunt configuration that every other terragrunt.hcl file imports with the include statement we saw earlier</p><pre>include {<br>  path = find_in_parent_folders()<br>}</pre><p>find_in_parent_folders is a built in Terragrunt function that returns the first terragrunt.hcl file it finds in the parent folders.</p><p>So let&#39;s start our dev/us-east-1/terragrunt.hcl file and define our state configuration</p><pre>remote_state {<br>  backend = &quot;s3&quot;<br>  config = {<br>    bucket         = &quot;S3 BUCKET NAME&quot;<br>    key            = &quot;${path_relative_to_include()}/terraform.tfstate&quot;<br>    region         = &quot;AWS REGION&quot;<br>    encrypt        = true<br>    dynamodb_table = &quot;DYNAMO DB TABLE&quot;<br>  }<br>}</pre><p>This uses S3 and DynamoDB to store state / obtain exclusive locks on the state file like we saw before without Terragrunt.</p><p>But the key = ... line means that inside the S3 Bucket they’ll be a directory structure mimicking your Terragrunt folder structure with state files.</p><p>Also, remember how in pure Terraform we had to do slightly hacky things to bootstrap the S3 Bucket and DynamoDB table?</p><p>Terragrunt will automatically create them if they don’t exist, solving that entire problem.</p><p>There is one downside to this though, all the Terraform modules you apply need to define</p><pre>terraform {<br>  backend &quot;s3&quot; {}<br>}</pre><p>In the module so that Terragrunt can fill in the details when it’s ran.</p><p>There’s a Github Issue thread from back in 2017 discussing this — <a href="https://github.com/gruntwork-io/terragrunt/issues/230">https://github.com/gruntwork-io/terragrunt/issues/230</a></p><p>We now need to tell Terragrunt where to find all those shared variable files we defined.</p><pre>inputs = merge(<br>  yamldecode(<br>    file(&quot;${find_in_parent_folders(&quot;environment.yaml&quot;, find_in_parent_folders(&quot;environment.yaml&quot;))}&quot;),<br>  ),<br>  yamldecode(<br>    file(&quot;${find_in_parent_folders(&quot;region.yaml&quot;, find_in_parent_folders(&quot;environment.yaml&quot;))}&quot;),<br>  ),<br>  yamldecode(<br>    file(&quot;${find_in_parent_folders(&quot;app.yaml&quot;, find_in_parent_folders(&quot;environment.yaml&quot;))}&quot;),<br>  ),<br>)</pre><p>As before find_in_parent_folders causes Terragrunt to search from where the modules defined up the tree to find the first occurence of the file.</p><p>The second parameter to file is a default to use if it can’t find the file, here we fallback to environment.yaml which we will make sure always exist, and it means that if we have a situation where a module isn’t nested deeply enough to have all the app.yaml , region.yaml and environment.yaml files in it’s parent directories it doesn’t explode.</p><p>merge is the standard Terraform function — <a href="https://www.terraform.io/docs/language/functions/merge.html">https://www.terraform.io/docs/language/functions/merge.html</a></p><p>It means that the any variables defined further down the tree override those further up.</p><p>One final configuration option and we’re done.</p><p>The AWS provider we use needs configuring, we can use Terragrunts ability to generate files to place an identical configuration in the working directory before Terraform is ran, rather than copy paste it 100’s of times.</p><pre>generate &quot;aws_provider&quot; {<br>  path      = &quot;aws_provider.tf&quot;<br>  if_exists = &quot;overwrite_terragrunt&quot;<br>  contents  = &lt;&lt;EOF<br>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}<br>EOF<br>}</pre><p>This will generate a file called aws_provider.tf in the working directory every time a command is ran that passes in our required options.</p><p>We can now run Terragrunt commands and build our infrastructure.</p><p>There are two options for us to run Terragrunt commands:</p><ol><li>Across multiple folders, allowing us to build multiple modules at the same time</li><li>In a single module for faster / more targeted applies.</li></ol><h4>run-all commands</h4><p>If you want to run commands across multiple modules at once the run-all commands can do that.</p><p>For example, if you run terragrunt run-all plan in the dev directory it will run terraform plan in every subdirectory and present you with the plan.</p><p>The first time you run commands in a folder it will check if the Terraform state files and lock tables exist and prompt you to create them if not.</p><h4>Single Module Applies</h4><p>If you just want to apply changes to a single Terraform module you can cd into that directory and run terragrunt &lt;COMMAND&gt; and it will only apply to the current working directory.</p><h4>Module Caching</h4><p>One annoying quirk of Terragrunt is that once a module has been downloaded it won’t refetch it if the source changes.</p><p>For example if you have ref = github.com/module?ref=my-branch and you push new changes to my-branch after having ran terragrunt apply it won’t notice the source has changed and fetch the new changes.</p><p>You’ll need to clear the Terragrunt cache with find . -type d -name &quot;.terragrunt-cache&quot; -prune -exec rm -rf {} \;</p><h3>Summary</h3><p>We’ve now got a fully working Terragrunt setup, to recap what we’ve gained:</p><ul><li>Small per module state files that make it faster to run terraform commands and reduce the impact of losing / corrupting a single state file</li><li>A well defined structure for laying out our infrastructure based on directory structures that mirror how the infrastructure is logically reasoned about</li><li>The ability to use shared variable files that allows variables to be defined at multiple levels and propagated down to the Terraform modules automatically to easily keep settings consistent at multiple levels, with the ability to override more generic values if required</li><li>The ability to declare that one module depends on another, creating the modules in the appropriate order and passing output variables between them</li><li>The inability to create random adhoc resources outside of a versioned Terraform module so we reduce the chances of resources / changes not being propagated to other environments</li><li>The ability to automatically retry Terraform commands if certain errors occur, reducing the impact of flaky / eventually consistent APIs</li></ul><p>It’s not perfect though some limitations / issues:</p><ul><li>Every Terraform module you use needs to define a blank backend block, meaning you may have to go modify every module you have, and you can’t use community modules</li><li>You can’t use Terraform module registry references, losing the ability to specify loosely locked versions and potentially meaning you need to provide Git authentication credentials so Terragrunt can pull modules from Github</li><li>You and everyone else on the team need to learn about Terragrunt</li><li>You now have another tool to keep up to date along side Terraform</li><li>If you haven’t been particularly disciplined about keeping your module input variables named consistently you’ll not get a lot of the benefits of the shared variable files without some refactoring</li></ul><h3>Example Repo</h3><p>There’s a lot of code blocks in this post, you can see them all together in an example repo here — <a href="https://github.com/AaronKalair/example-terragrunt-repo">https://github.com/AaronKalair/example-terragrunt-repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db6864967916" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Module Patterns]]></title>
            <link>https://medium.com/@AaronKalair/terraform-module-patterns-4ba2996f0b96?source=rss-834a463074f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ba2996f0b96</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aaron Kalair]]></dc:creator>
            <pubDate>Sun, 10 Oct 2021 12:13:54 GMT</pubDate>
            <atom:updated>2021-10-10T12:13:54.157Z</atom:updated>
            <content:encoded><![CDATA[<p>Terraform gives you a myriad of ways to organise your code, supporting almost any pattern you’d like to adapt.</p><p>Whilst flexibility can be nice, it can be really hard when you first start out to know if you’re organising your code sensibly or setting yourself up for difficulties in the future.</p><p>Here are some patterns for creating Terraform Modules that I’ve seen work well.</p><h3>Configuring AWS Resources with Company Specific Defaults</h3><p>The default configuration that Amazon believes to be acceptable, might not be the best for you.</p><p>Because of this, we create company-specific modules that wrap AWS resources and set defaults that make sense for us.</p><p>For example, we always want RDS storage_encrypted_enabled and S3 Public Access disabled.</p><p>So we create modules like company_name_s3_bucket that configure an S3 bucket with the company’s mandatory arguments and then allow the other arguments to be configured via input variables.</p><p>Then anyone who wants an S3 bucket uses the company_name_s3_bucket module instead of the Terraform s3_bucket resource from the AWS Provider and we never have to worry about people forgetting to set sensible defaults.</p><h3>Automatically Creating Lower Level AWS Resources</h3><p>Some AWS resources depend on others to be functional.</p><p>For example you can’t create an RDS instance without aws_db_subnets, so we have our company_name_mysql module create aws_db_subnets that are then passed to the aws_database_instance.</p><p>Now the user of the module doesn’t need to worry about creating these supporting resources and can just pass in the appropriate subnets they want the database created in.</p><h3>Grouping AWS Resources Needed to Make Functional Services</h3><p>We have Terraform modules that combine multiple AWS resources, and other Terraform modules to create functional services our applications need, one example of this is a MongoDB cluster.</p><p>Our company_name_mongodb_cluster module is organised as follows:</p><pre>company_name_mongodb_cluster<br>  -&gt; mongodb_node_a -&gt; company_name_mongo_node_module<br>  -&gt; mongodb_node_b -&gt; company_name_mongo_node_module<br>  -&gt; mongodb_node_c -&gt; company_name_mongo_node_module<br>  -&gt; cluster_security_group -&gt; aws_security_group_resource<br>  -&gt; cluster_backup_schedule -&gt; aws_dlm_lifecycle_policy_resouce<br>  -&gt; cluster_iam_policy -&gt; aws_iam_policy_resource<br>  -&gt; cluster_iam_role -&gt; aws_iam_role</pre><p>With company_name_mongo_node_module creating:</p><pre>company_name_mongo_node_module<br>  -&gt; cloudinit_data -&gt; template_cloudinit_config_resource<br>  -&gt; ebs_volume -&gt; aws_ebs_volume_resource<br>  -&gt; database_instance -&gt; aws_instance_resource<br>  -&gt; dns_entry -&gt; aws_route53_record_resource</pre><p>Here we nest two modules we’ve written ourselves.</p><p>The company_name_mongo_node_module allows us to consistently create EC2 instances than run mongod without lots of copy and pasting</p><p>The company_name_mongodb_cluster_module allows us to create resources that are shared across the nodes like a security_group and iam_role that can then be passed into the child module which creates the actual MongoDB servers.</p><p>The user of company_name_mongodb_cluster simply passes in a few required parameters and gets a MongoDB cluster configured following our requirements with regards to availability (3 nodes), security (appropriately scoped IAM role, and Security Group isolating the cluster) and backup with the DataLifeCycleManager.</p><p>The module can also output the cluster endpoint that can be passed into other modules or given to an application.</p><h3>How deep to nest modules</h3><p>The above example uses two custom modules we’ve written and brings up another question.</p><p>How small and specific should your modules be and should you create long chains of modules with reusable components?</p><p>For example imagine:</p><pre>networking_module -&gt;<br>  vpc_module -&gt;<br>    subnet_module -&gt;<br>      route_table_module</pre><p>This at first seems quite nice.</p><p>Every environment you create will need a VPC, some Subnets and Route Tables, but what we’ve seen is that practically working with this can be painful.</p><p>If you want to update the route_table_module you then need to version bump every parent module (3 of them) to release that change.</p><p>If you want to pass route_table specific parameters down to the route_table_module you have to pass it through multiple other modules, even if they don’t care about them.</p><p>Similarly if you need to pass data back up as an output from the route_table_module you have to pass it through multiple layers.</p><p>Further this assumes a perfect world where every environment is identical and needs all of the resources.</p><p>In practice we find this not to be true and so the module ends up taking variables like enable_subnet_b that is passed into a count statement like this for the specific resource we only want to create in certain environments:</p><p>count = var.enable_subnet_b ? 1 : 0</p><p>This leads to the number of input variables the module takes growing quickly, as it will need to take inputs for the large number of resources it creates, and inputs to decide if it should even create the resource in the first place.</p><p>We’ve found this style of module quite hard to work with and no longer design them this way.</p><h3>Where to store the modules</h3><p>Once you’ve got some modules you’ll need a place to store them so you can reference them in your Terraform runs.</p><p>Initially we had a monorepo that stored all our modules under the modules/ directory and our actual instatiations of the modules were under environment specific directories in the same repo e.g.</p><pre>environments/<br>  dev/<br>    vpc.tf<br>  prod/<br>  modules/<br>    vpc/<br>    mongo/</pre><p>And reference the modules with …</p><p>source = &quot;../modules/vpc&quot;</p><p>This works well when you first start out.</p><p>You can super easily iterate on modules and make changes but it quickly causes issues.</p><p>As all of the modules are unversioned and live in the same Git repo, it’s impossible for modules in prod/ to use a different version of a module to dev/.</p><p>Of course you can use -ignore or tell people not to run apply in prod/ as you’re testing changes, but this is an accident waiting to happen.</p><p>So you’ll want to move your modules out into their own Git repos</p><p>Once they’ve been split out you have two ways to release them.</p><h4>Git Refs</h4><p>You can use Git refs in the source field to reference a module e.g.</p><p>source = &quot;github.com/hashicorp/example?ref=&lt;BRANCH OR TAG&gt;&quot;</p><p>If you tag each merge to master with a version number you can then easily control what version of the module is deployed in each environment.</p><p>This even works if your modules are in a private git repo, the system running Terraform just needs access to the private repository either with an SSH key or username and password.</p><p>This is by far the best way I’ve seen to store and use Terraform modules, mostly due to the downsides of the Terraform Module Registry discussed next.</p><h4>Terraform Module Registry</h4><p>The Terraform module registry can import modules from Github and allows you to reference them like …</p><pre>source = &quot;app.terraform.io/&lt;YOUR COMPANY&gt;/&lt;MODULE&gt;/&lt;NAMESPACE&gt;&quot;<br>version = &quot;~&gt; 1.1.0&quot;</pre><p>It’s free for 5 users and they have paid plans for additional users.</p><p>The biggest advantage of the Terraform Module Registry is fuzzy version locking e.g. ~&gt; 1.1.0 means the latest minor version from the 1.1 tree.</p><p>So without having to change any code you can run terraform init -upgrade and pull any bug fix releases from the module registry.</p><p>We initially switched from Git references to the module registry for this fuzzy version locking, but ultimately it turned out not to be used that often and we prefered to manually control any version bumps.</p><p>Our initial need for the fuzzy version locking was for our long chain of nested Terraform modules described above. In reality that pattern was a bad idea, and as we migrated away from it, our need for loose versioning lessened.</p><p>It’s important to remember that with the version loosley locked like that everyone needs to have run terraform init -upgrade at the same times to ensure you all use the same versions of the modules, and someone else running terraform apply doesn’t undo changes.</p><p>We use the free plan for the Terraform Module registry and our daily job which builds a reference version of our Terraform modules frequently fails because either the Terraform module registry is down or we hit API limits.</p><p>Also as you don’t want to put untested module versions into the registry you’ll find yourself using the Git Refs still to test changes to a module on a branch you push to Github to verify it works before merging.</p><p>Finally, it’s worth noting that if you plan to use Terragrunt it doesn’t support the Terraform module registry.</p><h3>Bash scripts as modules</h3><p>Terraform supports data resources which can be a variety of different things such as AMIs or EC2 Instance UserData.</p><p>You can do neat things with this like creating modules for UserData that you need to run as part of the Cloudinit process when an EC2 instance comes up.</p><p>For example, if you need to kick off a chef-client run when your EC2 Instance comes up, you can create a module that defines a templated bash script like …</p><pre>… &lt;assorted setup work&gt;<br>chef-client -E ${environment} -N $node_name …</pre><p>And output it from a module like ..</p><pre>output &quot;file&quot; {<br>  value = templatefile( <br>    &quot;${path.module}/chef.sh.tmpl&quot;, {<br>      environment = var.environment<br>    }<br>  )<br>}</pre><p>Then every EC2 instance that needs to run it can use it to build up their template_cloudinit_config</p><pre>module &quot;chef_userdata&quot; {<br>  source      = &quot;&lt;MODULE_SOURCE&gt;&quot;<br>  environment = &quot;var.environment&quot;<br>}</pre><pre>data &quot;template_cloudinit_config&quot; &quot;service&quot; {<br>  part {<br>    filename     = &quot;chef.cfg&quot;<br>    content_type = &quot;text/cloud-config&quot;<br>    content      = module.chef_userdata.file<br>  }<br>}</pre><p>Now you have a fully versioned set of scripts you can use to bootstrap your instances, and you don’t need to copy paste them around everywhere!</p><h3>Assorted Ad-hoc resources for applications</h3><p>Aside from the larger resources, like databases and caches, your applications may also need an assortment of “smaller” things like IAM Roles, SSL certificates or random strings for secrets.</p><p>You could just define these as individual resources in app.tf files inside your Terraform environment and terraform apply them e.g.</p><pre>resource &quot;aws_acm_certificate&quot; &quot;app&quot; {<br>  ...<br>}</pre><pre>resource &quot;random_password&quot; &quot;password&quot; {<br>  length = 64<br>}</pre><pre>resource &quot;aws_secretsmanager_secret&quot; &quot;secret&quot; {<br>  name = &quot;password&quot;<br>}</pre><pre>resource &quot;aws_secretsmanager_secret_version&quot; &quot;secret&quot; {<br>  secret_id     = aws_secretsmanager_secret.secret.id<br>  secret_string = random_password.password.result<br>}</pre><pre>resource &quot;aws_iam_policy&quot; &quot;&quot;{<br>  ...<br>}</pre><p>But now when you want to create those same resources for your application in say prod you have to copy paste them all over.</p><p>If you change any of them in dev you need to remember to exactly reproduce the changes in prod .</p><p>Finally, if you use Terragrunt you can’t just define random resources at the top level like this.</p><p>So we decided it was better to adopt a pattern of app-resource modules.</p><p>If required, each app gets a module called terraform-&lt;BUSINESS_NAME&gt;-app-resources-&lt;APP_NAME&gt;. In here you place all of these assorted resources and then you instantiate the module in your terraform environment instead of defining all the separate resources.</p><h3>Linking Terraform created resources with your applications</h3><p>Once you’ve created your various AWS resources in Terraform, it’s likely your applications will need to know something about them, like S3 Bucket Names, Database DNS addresses etc.</p><p>You could manually copy and paste these into your application settings file, or take advantage of Terraforms ability to create AWS Parameter Store and Secrets Manager entries to make this a less manual process.</p><p>E.g. Our Database module creates Parameter Store entries for the DNS address of the database…</p><pre>resource “aws_ssm_parameter” “dns_address” {<br>  name  = /dev/app-a/database_address<br>  type  = “String”<br>  value = database_module.outputs.dns_address<br>}</pre><p>And then you can either use something like<a href="https://github.com/external-secrets/kubernetes-external-secrets"> ExternalSecrets</a> to fetch that data and inject it into a K8s pod, or have your app talk to the AWS APIs itself to retrieve the values.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ba2996f0b96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>