Zach Wilson

I’ll admit, that imposter syndrome is real, Even after over a decade in the tech industry and founding my own company, there are still days when I wonder, “Who am I ,to be doing this?” "Am I doing it right?" Many questions, But you know what? I’ve realized that a little healthy amount of self-doubt isn’t always a bad thing. It alerts me to constantly push myself to learn and improve, And reminds me that I don’t have all the answers (and that’s okay!). One area I’ve really been focused on improving is my marketing skills. When I first started out, I’ll be honest – I was clueless. How do you create an awesome social media strategy? What even is SEO? And don’t get me started on building an email list... But bit by bit, I’ve been leveling up. Taking online courses, reading marketing blogs, and connecting with experienced entrepreneurs who’ve been there, done that. And you know what? It’s been SO rewarding to see the needle move, even in small ways. So if you’re feeling like a total imposter, just know – you’re not alone. We’re all on this entrepreneurial roller coaster together. 🎢 The key is to embrace the journey, celebrate the small wins, and know that you’ve got this. Ps:Let me know what areas of your business you’re working to improve! I’d love to hear your story. Enjoy this? ♻️ Follow Santosh Patil for more. #impostersyndrome #socialmedia #SEO #marketing #britwise

17

next post on my series about #AngularQuery Below is a first simple example on how to use injectQuery to fetch data. The best? You get respective states like loading for free! No more cumbersome #RxJs tap()ing needed #Angular

61

Apache XTable had its 1st release today 🎉 It's been only 4 months since we donated Apache XTable (Incubating) to the Apache Software Foundation. The project has had some great momentum since it was introduced last year. The community engagement, diverse set of contributors, mentors & users are something that drives all of this. It has also made some real impact to enterprise solutions and open source projects to help table formats be 'interoperable'. So, customers can choose their desired compute for that specific format. ✅ Microsoft Fabric + Snowflake -> uses XTable to support Iceberg along with Delta Lake, thereby allowing to interoperate between these 2 platforms. ✅ Apache Hudi support in Delta Lake UniFormat is facilitated by XTable. Having worked with multiple users/customers to implement lakehouses in the past couple years, I know some of the pressing problems that interoperability can solve. It is important for table formats to be 'interoperable' in an open data architecture. Today, 0.1.0 is out. Although this was focused mostly on improvements & getting ASF complaint (codebase, infra), we are getting the foundations robust. Check out the release notes (link in comments)! 🌟 Interested in applying XTable or contributing to it? Reach out or add a PR. #dataengineering #softwareengineering

148

5 Comments

Understanding distributed SQL is the key to cracking the performance gains of MPP (Massive Parallel Processing)

7

When working with an unfamiliar JSON string and needing to identify all keys, using a JSON_KEYS extractor can be extremely useful. #bigquery https://lnkd.in/eY3nKTE6

6

You probably don't need Scratch Data to stream information into Redshift and BigQuery. AWS has a 950-word document on how to do this. GCP's guide is 2,500 words. Our docs show how to do this 138 characters - less than an (OG) tweet. As I said, you don't need Scratch Data. But your developers probably have better things to do.

15

1 Comment

How to build a wildly successful app in 2024? You only need one framework. Problem → Solution → Maintenance 1/ Find a “real” problem There are millions of apps already. You need to address a real problem, And find people who want the solution to it. 2/ Create the solution Now, your solution (app) must be really effective. Otherwise, what’s the point? 3/ Iterate and improve Fix issues and incorporate more features that users want, Double down on what works, and make your app a necessity. Now this maybe cliche, But it’s the only framework that works. Do you agree? #appdeveloper #founder #b2b

14

12 Comments

My first piece of the new year! 'Database Trends: A 2024 Review and a Look Ahead', a round-up of the big database influences in 2024 — like vector store, GraphQL, and open table formats — and what they portend for 2025. https://buff.ly/40mmS15 @thenewstack.io #Database #Analytics #AI #DataManagement #VectorDatabases #ApacheIceberg #DeltaLake

14

I made lots of new connections on this app recently. For those, who don't know, I write about Cloud, Serverless, at The Cloud Handbook ☁ 🚀 ✅ Subscribe. It's FREE: 👉 cloudhandbook.substack.com Also, feel free to share your newsletter if you have any in the comments, I would be happy to recommend them. #aws #awscommunity #cloud #softwaredevelopment

30

1 Comment

I almost made the same mistake Notion did in 2021. While demoing 𝐕𝐚𝐝𝐞 𝐒𝐭𝐮𝐝𝐢𝐨 to prospects, I realized the interactions were slower than I hoped it would be. So I started benchmarking my schema design. While auditing Postgres database, I discovered our 𝐉𝐒𝐎𝐍 𝐛𝐥𝐨𝐛 storage approach was heading down a dangerous path. Here's what I learned from Notion's near-death experience with Postgres: 𝐓𝐡𝐞 𝐏𝐫𝐨𝐛𝐥𝐞𝐦: • Notion's database was ready to explode • Postgres was at terabytes of volume • Service became unbearably slow • VACUUM process started failing • Transaction IDs were about to wrap around 𝐓𝐡𝐞 𝐑𝐨𝐨𝐭 𝐂𝐚𝐮𝐬𝐞: • Everything in Notion is a "block" • Each block = one database row • Simple document = 1000s of rows • Multiply that by millions of users • Single monolithic database couldn't handle it 𝐅𝐢𝐫𝐬𝐭 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧 (32 𝐒𝐡𝐚𝐫𝐝𝐬): • Sharded by workspace ID • 32 physical databases • 15 logical shards per database • Total: 480 shards • Used PgBouncer for connection pooling 𝐏𝐥𝐨𝐭 𝐓𝐰𝐢𝐬𝐭: 𝐋𝐚𝐭𝐞 2022, 𝐩𝐫𝐨𝐛𝐥𝐞𝐦𝐬 𝐞𝐦𝐞𝐫𝐠𝐞𝐝: • 90% CPU utilization • IOPS reaching limits • Connection pools maxing out 𝐅𝐢𝐧𝐚𝐥 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧 (96 𝐒𝐡𝐚𝐫𝐝𝐬): • Tripled physical databases • Reduced schemas per instance • Optimized index creation • Dark reads for testing • Nearly 100% data consistency 𝐓𝐡𝐞 𝐫𝐞𝐬𝐮𝐥𝐭𝐬? CPU usage dropped from 90% to 20% during peak traffic. 𝐓𝐡𝐞 𝐛𝐢𝐠𝐠𝐞𝐬𝐭 𝐥𝐞𝐬𝐬𝐨𝐧? Sometimes the simplest storage solution (like our JSON blob approach) can become your biggest nightmare at scale. P.S. How do you store hierarchical data in your applications? JSON blobs or relational tables? #Database #notion #Developer #founder

4

3 Comments

I built my own RAG ingestion and indexing repository about 8 months ago before Amazon Bedrock Knowledge Base (KB) was a thing. I'm very happy that my design is about 95% similar, using Opensearch Serverless and S3, metadata files etc. I cannot wait to replace 4 Lambdas and a distributed Map Step Function with Amazon Bedrock KB when we have time. But, until then, what has been super helpful is to look into the Bedrock invocations when doing a retrieve_and_generate to see how they are doing the system prompting for the RAG and include inline citations in the response. Well worth it if you want to improve your own system prompting for RAG 👍

28

4 Comments

I appreciate there are only a few Organizations with Data Infrastructure spanning across 2100 Clusters having 16k nodes and three regions. But still, if that’s your thing, and especially on MySQL – Uber recently shared their in-place MySQL 5.7 to version 8.0 upgrade (https://lnkd.in/gav6QFAm). It’s a great resource and calls out pre-process-post steps along with learnings and outcomes. Also, there are some learnings which could vary or be re-purposed for any such initiatives: 1. Cost to Value: Overall upgrade spanned a year. With 16000 nodes and 2100 clusters, that’s about ~1334 nodes or 175 clusters upgraded every month. Even with automation, that’s a considerable cost with testing and ongoing maintenance. Though with this, Uber realized 29% and 33% improvement in write and read ops. Considering consumption of 3mil queries per second – that’s a substantial saving. So, whilst multiplier effect lands a greater value in Uber’s case – lack of scale could outweigh the advantage and leave with higher costs. 2. Delayed version upgrade: For Prod, I have generally seen Org’s staying 2 steps behind latest release. Considering that, and even with MySQL debate with version 6 and 7, I would have preferred to upgrade earlier than v5.7 reaching EOL. 3. Client Library Incompatibility: For a rollout of this scale – its imperative to plan across the ecosystem, including client end consumption (or connect libraries). A vendor would almost always test only the core operation and functionality, interoperability or connect libraries often leading to interesting challenges. Plan for the core change, but then also look at the edges and end-to-end flow. #engineering #upgrade #mysql #uber #learnings

7

🚀 Building the Architecture for APIOps: Speed, Consistency, and Reliability Simplified 🌐 In the world of APIs, delivering faster, more reliable, and consistent solutions is no longer optional - it's essential. That’s where APIOps comes in, blending automation, best practices, and collaboration to revolutionize API development and operations. 🛠️💡 Here’s a high-level blueprint for creating an effective APIOps architecture: 🌟 1. Foundation: A Solid #API Platform API Gateway for routing and traffic control (e.g., #Kong, #AWS API Gateway). API Management for versioning, lifecycle management, and monitoring. Developer Portals for seamless API documentation and onboarding. 🤖 2. Automation is Key Automated Testing: Functional, performance, and security testing with tools like #Postman and #OWASP ZAP. CI/CD Pipelines: Automate deployment using Jenkins or GitHub Actions. Code Reviews: Ensure quality through tools like SonarQube. 📋 3. Reusable Templates & Best Practices Pre-built API contracts, deployment configurations, and security guidelines ensure every team stays aligned while moving fast. 🔄 🔍 4. Observability & Monitoring Gain real-time insights with #Prometheus or #DataDog. Trace and debug with tools like #Jaeger or #Elastic Stack (ELK). 🛠️ 5. Self-Service Tools for Developers Empower teams with pre-configured tools for deployment, testing, and monitoring. 🚀 💬 6. Feedback Loops for Continuous Improvement Listen to developers and end-users, and use analytics to refine your APIs continuously. 📈 By leveraging APIOps, organizations can deliver APIs that are fast, reliable, and scalable, all while adhering to best practices. 🏆 📢 Let’s embrace the future of API development! How are you automating your API workflows? Share your thoughts below 👇 #APIOps #APIManagement #DevOps #Automation #APIDevelopment #DataManagement #AI #Mindset #DataManagement Picture Credits: Kong Inc

6

If you're intrigued by dbt (data build tool) but unsure whether to choose dbt Cloud or dbt Core, you're in the right place. Maybe you're already using one and considering a switch. No matter your current stance, understanding the differences between these options is essential for making an informed choice. In this article, we'll explore the key distinctions between dbt Cloud and dbt Core to help you decide which is the best fit for your needs. https://lnkd.in/gS2xZ9Cx

19

1 Comment

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

How I Made $35k Selling My First Online Course - A Retrospect 💰 A one-year retrospect after creating and selling my first online course: my strategy, the actual journey, mistakes, and recommendations 💡 https://lnkd.in/dwyB2VaP

11

2 Comments

Have you ever noticed the number of services you deploy alongside your Rails app? What if you could reduce your service sprawl? In under 5 minutes integrate your database and caching service providers into one with Couchbase. You might've not known that Couchbase can serve as a caching layer for your app, and that the Couchbase Ruby SDK provides a convenient extension for ActiveSupport to set Couchbase as your caching store, but now you do! Follow my quick tutorial and within just minutes you'll reduce your service sprawl by at least one less service deployed alongside your Rails app. https://lnkd.in/e5GdA9fU

13