Architecting Big Data & Analytics Solutions - Integrated with IoT & Cloud

Introduction

Purpose of this book

IoT, Big Data, and Cloud Computing are three distinct technology domains with overlapping use cases. Each technology has its own merits; however, the combination of three creates a synergy and the golden opportunity for businesses to reap the exponential benefits. This combination can create technological magic for innovation when adequately architected, designed, implemented, and operated.

Integrating Big Data with IoT and Cloud architectures provide substantial business benefits. It is like a perfect match. IoT collects real-time data using smart objects. Big Data optimises data management solutions. Cloud collects, hosts, computes, stores, and disseminates data rapidly.

Based on these compelling functions and business propositions, the primary purpose of this book is to provide practical guidance on creating Big Data solutions integrated with IoT and Cloud service models. To this end, the book offers an architectural overview, solution practice, governance, and underlying technical approach for creating integrated Big Data, Cloud, and IoT solutions. 

The book offers an introduction to the process of solution architecture, three distinct chapters comprising Big Data, Cloud, and the IoT with the final chapter, including conclusive remarks to consider for Big Data solutions. These chapters include essential architectural points, solution practice, methodical rigour, techniques, technologies, and relevant tools.

Creating Big Data solutions are complex and complicated from multiple angles. However, with the awareness and guidance provided in this book, the Big Data solutions architects can be empowered to provide useful and productive solutions with growing confidence.

Audience

This book has a specific focus on Big Data solution architects; however, its content can be an interest in all types of architects in Big Data, IoT, and Cloud Computing areas.

Apart from architects, this book can provide useful insights to senior technical leaders such as CTOs (Chief Technology Officers), CDO (Chief Digital Officers), Chief Data Officers, and CIOs (Chief Information Officers) to understand the architectural considerations for these critical technologies ubiquitously taking over the Information Technology Services globally.

This book can also be useful for the advanced tertiary students planning a career in these growing technology areas, understand the best practices, and see the big picture for their desired professions reflected from practice al work settings.

Summary of Chapters

Chapter 1:  Architectural Solution Process

This chapter provides an overview of the architectural solution process by introducing the critical solution work-products.

Chapter 2: IoT, Big Data and Cloud Relationships

This chapter provides a high-level view of these technologies (IoT, Big Data, Cloud) setting the architectural framework by defining them from architectural perspectives.  It provides an overview of relationships for these three distinct technologies in terms of creating the synergies and potential benefits for business with an agile approach.

Chapter 3: Big Data Architecture Process

This core chapter provides Big Data architectural solutions, providing an overview of the process, technologies, methods, and tools.

Chapter 4: Cloud for Big Data

This chapter provides an overview of Cloud Computing architecture and solutions related to Big Data solutions. It provides an overview of the process, technologies, methods, and tools.

Chapter 5: IoT for Big Data

This chapter is dedicated to IoT architecture and solutions, providing an overview of the process, technologies, methods, and tools within the context of Big Data.

Chapter 6: Conclusions

This chapter, to bring all together, provides conclusive remarks and critical considerations for the use of IoT and Cloud Computing services for Big Data solutions.

Chapter 1: Architectural Solution Process

Purpose of this Chapter

Before starting the significant themes of the book, in this section, we have an overview and a high-level walkthrough of the primary architectural tasks for creating Big Data solutions. Experienced solution architects may skip this section. However, it can be useful to have a quick scan of the common work-products covered in this chapter.

Understanding the context is vital for solution architects. Therefore, this section sets the context for the solution architecture process. By understanding the context, we can have a common understanding of what an architectural solution means and what it covers.

In this section, we touch on general architectural practice and knowledge. This practice can be transferred or applied to any architectural solutions. They are not just unique to Big Data solutions.  

The view I provide here depicts only fundamental aspects and covers very condensed content to start the solution journey. Those who are starting solution architecture practice, I’d recommend undertaking an architectural method course which can provide a comprehensive and an end to end view. Learning the purpose and content of the solution work-products are essential for any architects. Otherwise, it can be difficult, messy, and time-consuming to produce desired results.

Without further ado, let’s start by presenting an overview of the fundamental architectural approach for creating solutions. Architectural tasks are covered under the name of ‘solution work-products’. Architectural methods also call these tasks ‘work-products’.

Solution Work-Products

Awork-product is an essential term used in architectural vernacular. Work-products are standard terms in the industry. Solution architects create and deal with multiple architectural work-products.

These work-products are template documents that prompt the solution architects to enter the required information with guiding notes. Work-product templates are created based on experience and lessons learned from previous successful projects. There can be many work-products for a specified solution.

The standard view of work-products in an organisation can save time to the architects, reviewers, specialists, and other content contributors from various domains. Many organisations have their mandatory work-products used for solution architectures in their method. Let’s understand what an architectural method is.

Architectural Solution Method

An architectural method is a step by step guide to complete a solution providing templates, solution building blocks, techniques, annotations, and references. To create solution architectures, rather than proverbial re-inventing the wheel, we can follow an established method to save time. Following an established method allows solution architects to achieve predictable and repeatable results. These methods have evolved through their multiple uses. In turn, the techniques covered in these methods have also evolved through the reuse of successful implementations, as well as the lessons learnt from failed projects.

There can be hundreds of work-products in sophisticated solution methods. To keep it concise and see the big picture, we only focus on the primary work-products, that matter,  related to our solution scope. Please keep in mind that your organisation may mandate more work-products to comply with the solution delivery requirements. If your organisation is a service provider, your customers may also have their proprietary method which you need to use to create your solutions. To further complicate the situation, your organisation may mandate some work-products in your method, and your customer organisation may mandate additional work-products in their method. As solution architects, we need to be flexible to meet methodical requirements for both stakeholders.

Strategy and Architectural Thinking

An architectural solution starts with strategic thinking. The architectural strategy takes place at the beginning of a solution. Setting the strategy upfront help the solution team to understand where they are now and where they are going. With this strategic approach in mind, an established method can help us reach our destination most effectively. This means that setting the strategy and following the method are two essential tasks in the earlier phase of solution development.

The solution strategy process provides an objective assessment of the current situation.