Andrew Fong’s Post

Things Data Engineers Often Overlook 1. Cost of Scalability Data Engineers often address performance issues by scaling infrastructure. While this can provide a quick fix, it's not always the most efficient solution and can lead to unnecessary expenses. Understanding the root cause of slowness is critical before scaling. 2. Avoiding Notebooks Many Data Engineers shy away from using notebooks, preferring to engineer everything through scripts or pipelines. However, for certain exploratory tasks, notebooks can save time and effort. Ignoring them might lead to overengineering. 3. Sticking to Familiar Tools Data Engineers often default to familiar data technologies, even when they might not be the best fit for a specific use case. This comfort zone can limit innovation and efficiency. Evaluate the tools based on the problem, not familiarity.

Data teams have become more rigorous by adopting version control, testing, and contracts for their data models. But they've lost something along the way... SPEED. Is it possible to have it all? Let’s look to software engineering for inspiration. In software, a just-in-time (JIT) compiler only interprets and processes code as it needs it, rather than doing it all up front before a program is run. Applying the same idea to data modeling: Just-in-time data modeling means building your data model as you do analysis. Need a new dimension or measure? Just add it to your analysis, and get the answers you need. But this only works if you close the loop by allowing the ad-hoc definitions you’ve made to become part of your organization’s data model. This is the workflow data organizations need to balance speed and quality: build analysis quickly, distill out the useful parts into the organization’s data model, then repeat by building more analysis on top of that data model. Link to my full blog post introducing just-in-time data modeling below.

Garbage in, garbage out. It’s the oldest rule in software engineering, but it holds true for every business today. Poor data quality can lead to costly mistakes, from inventory mishaps to financial losses. Here's how Six Sigma can identify errors and fix poor data quality: https://lnkd.in/g2XxuUiU #dataquality #sixsigma #businessstrategy #datamanagement

CI/CD explained very simple. So it’s easy to understand the benefits of it. But implementation of such process can be a tricky and again fall under manual work. This happens when edge cases are still left to be managed in spreadsheets or send via email by someone in your org and most importantly it follows random formatting each time.

Six (6) Reasons why to use to augment your Enterprise Development Teams and Data Engineering Teams with Engineer Up! www.engineerup.com Expertise - you don't know how to do x and don't want to blow a ton of money with a failed strategy/implementation. Better investment to pay those who know x than risk failure. Lack of time - current team is too busy with other things to work only on how to do x for 5 months. Industry benchmarks - sometimes consultants help bring you things other people are doing that you didn't know. Politics1 - it can be an easier sell if doing x is what the experts recommend vs one guy who is in a struggle with another guy (aka objectivity). Politics2 - if you botch the implementation later you can blame the consultants' ideas. Budgets - you don't really need the consultants right now but you've got some money you'd rather do something productive with.

How would you get started with unifying enterprise applications, to have a consistent language and be a coherent suite? How would you go about prioritizing which applications to attack first? A big bang approach is not ideal, certainly its too much to chew in one bite. A top-down approach, would be to pick applications based on architecture factors of coupling, complexity, overlap etc. This would be my instinct if I were a purist, I think. Another approach can be to go bottom up, based on data. This could be data from production operations and development process, with more bias to operations. Basically saying, attend to those babies first that cry the most in production. Theoretically speaking, a meet-in-the-middle approach that uses both facets from above, can also be a method to prioritize. But they may or may not converge. What do you think?

In our 3rd article we move from going deeper into problem analysis into the concept of systems thinking, a holistic approach that empowers us to perceive the world through the lens of interconnectivity. By embracing this paradigm, we transcend the limitations of traditional problem-solving methods and gain the ability to unravel the intricate web of relationships and feedback loops that govern complex systems. As a distributed systems engineer and solutions architect, learning systems thinking has been an invaluable mindset to understanding the often unseen (not obvious) relationships between domains, especially when they bridge boundaries like people, process, and technology. Designing a solution to fit only one may be just fixing the symptom of a much larger complex problem. Akhil Bhaskar Arthi Jaganathan Tod Golding #SolutionsArchitecture #SystemsThinking #Interconnectivity #FeedbackLoop #Complex #Systems #ProblemSolving https://lnkd.in/gutJnfXg

Building an effective RAG pipeline requires careful attention. Starting with document processing, it's crucial to implement proper chunking strategies, thorough text cleaning, and metadata extraction while preserving document structure integrity. The embedding generation phase demands thoughtful model selection, efficient batch processing, and careful consideration of dimensionality, along with consistent quality evaluation of the generated embeddings. For vector storage, selecting the appropriate database, implementing optimal indexing strategies, and planning for scale and performance are essential, including a robust update strategy. The retrieval process hinges on choosing the right search algorithms, fine-tuning k-nearest neighbors parameters, and implementing effective filtering and re-ranking mechanisms. In the response generation stage, focus should be placed on sophisticated prompt engineering, optimizing the context window, and implementing thorough response verification procedures. Finally, the evaluation and monitoring phase requires clear metrics definition, continuous performance monitoring, and systematic feedback collection mechanisms. All these components work together in a continuous flow, where each stage's performance directly impacts the overall system effectiveness. Success lies in balancing these considerations while maintaining system efficiency and result quality. Here is my complete article that helps you understand some advanced RAG techniques: https://lnkd.in/gdyVqgaY But, no matter what type of RAG approach you choose, it is always a must to pick a database that helps you build robust RAG applications. SingleStore is an all-in-one data platform to help you build solid RAG applications. Try SingleStore for Free: https://lnkd.in/gcvWifQX

🛠️ Building an Effective RAG Pipeline: A Systematic Approach 🤖 Constructing a robust Retrieval-Augmented Generation (RAG) pipeline is as much an art as it is a science. Every stage, from document preprocessing to response generation, plays a pivotal role in the system's overall performance and accuracy. 📄 In the document processing stage, thoughtful chunking strategies, effective text cleaning, and structured metadata extraction set the foundation. Preserving document integrity ensures that the context remains intact throughout downstream tasks. 🧠 Moving to embedding generation, selecting the right embedding model is key. Efficient batch processing and dimensionality optimization help maintain performance, while embedding quality evaluations ensure semantic consistency across the dataset. 🗂️ Vector storage introduces architectural considerations, from choosing the optimal database (like SingleStore) to fine-tuning indexing strategies for fast and reliable retrieval. Scalability and update mechanisms must be planned upfront to prevent bottlenecks as data grows. 🔍 The retrieval phase relies on fine-tuned search algorithms, parameter optimization for k-nearest neighbors (k-NN), and intelligent re-ranking strategies. These steps ensure that only the most contextually relevant information is passed to the generation stage. 📝 Finally, response generation and evaluation close the loop. Sophisticated prompt engineering, optimized context windows, and clear evaluation metrics guarantee quality outputs, while systematic feedback mechanisms enable continuous improvement. This article is an excellent resource for anyone diving deeper into advanced RAG techniques. And tools like SingleStore, with their all-in-one platform, simplify many of the complexities in storage and retrieval workflows. Thanks for sharing these insights! A well-structured RAG pipeline is essential for building AI systems that deliver consistent and accurate results. Looking forward to exploring these techniques further. 💡 #RAG #GenerativeAI #AIEngineering #MachineLearning #DataInfrastructure #SingleStore #AI #DataScience