Search Results for "distributed storage"
Sort By:
Showing 64 open source projects for "distributed storage"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Zipkin
Distributed tracing system to gather timing data
Zipkin is a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data. If you have a trace ID in a log file, you can jump directly to it. Otherwise, you can query based on attributes such as service, operation name, tags and duration. Some interesting data will be summarized for you, such as the percentage of time spent in a service, and whether or not operations... -
2
Go Micro
A framework for distributed systems development
Go Micro is a framework for distributed systems development. Go Micro provides the core requirements for distributed systems development including RPC and Event driven communication. The Micro philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly but everything can be easily swapped out. Auth is built in as a first class citizen. Authentication and authorization enable secure zero trust networking by providing every service an identity and... -
3
Jaeger
Monitor and troubleshoot transactions in complex distributed systems
...It is used for monitoring and troubleshooting microservices-based distributed systems. OpenTracing compatible data model and instrumentation libraries include Go, Java, Node, Python, C++ and C#. Jaeger uses consistent upfront sampling with individual per service/endpoint probabilities and it has multiple storage backends: Cassandra, Elasticsearch, memory. -
4
Foundatio
Pluggable foundation blocks for building distributed apps
Pluggable foundation blocks for building loosely coupled distributed apps. Includes implementations in Redis, Azure, AWS, RabbitMQ and in memory (for development). When building several big cloud applications we found a lack of great solutions (that's not to say there aren't solutions out there) for many key pieces to building scalable distributed applications while keeping the development experience simple. Wanted to build against abstract interfaces so that we could easily change... -
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
5
Harvester
Open source hyperconverged infrastructure (HCI) software
...It is an open-source alternative designed for operators seeking a cloud-native HCI solution. Harvester runs on bare metal servers and provides integrated virtualization and distributed storage capabilities. In addition to traditional virtual machines (VMs), Harvester supports containerized environments automatically through integration with Rancher. It offers a solution that unifies legacy virtualized infrastructure while enabling the adoption of containers from core to edge locations. Harvester is an enterprise-ready, easy-to-use infrastructure platform that leverages local, direct attached storage instead of complex external SANs. ... -
6
MinIO Client Quickstart Guide
Fast tool to manage MinIO clusters
MinIO Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports filesystems and Amazon S3-compatible cloud storage service (AWS Signature v2 and v4). MinIO is distributed under GNU AGPLv3. See the source tree for complete information on all components of the work and their licenses. MinIO is a high-performance object storage solution that provides an Amazon Web Services S3-compatible API and supports all core S3 features. MinIO is built to deploy anywhere - public or private cloud, bare metal infrastructure, orchestrated environments, and edge infrastructure. ... -
7
Bacalhau
Community-driven, simple, yet powerful framework
Bacalhau is a decentralized compute platform for running jobs on data stored across distributed networks, like IPFS or Filecoin, without moving the data to centralized cloud environments. It allows developers to run containerized workloads close to where the data lives, reducing latency, cost, and privacy risks. Bacalhau supports various runtime environments and is designed to make decentralized data processing as accessible as traditional cloud computing. It’s especially useful for... -
8
Apache Drill
Apache Drill is a distributed MPP query layer for self describing data
Apache Drill is a distributed MPP query layer that supports SQL and alternative query languages against NoSQL and Hadoop data storage systems. It was inspired in part by Google's Dremel. Get faster insights without the overhead (data loading, schema creation and maintenance, transformations, etc.) Analyze the multi-structured and nested data in non-relational datastores directly without transforming or restricting the data. -
9
Datahike
A durable Datalog implementation adaptable for distribution
Datahike is a durable Datalog database powered by an efficient Datalog query engine. This project started as a port of DataScript to the hitchhiker-tree. All DataScript tests are passing, but we are still working on the internals. Having said this we consider Datahike usable for medium sized projects, since DataScript is very mature and deployed in many applications and the hitchhiker-tree implementation is heavily tested through generative testing. We are building on the two projects and... -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
Smallpond
A lightweight data processing framework built on DuckDB and 3FS
smallpond is a lightweight distributed data processing framework built by DeepSeek, designed to scale DuckDB workloads over clusters using their 3FS (Fire-Flyer File System) backend. The idea is to preserve DuckDB’s fast analytics engine but lift it from single-node to multi-node settings, giving you the ability to operate on large datasets (e.g. petabyte scale) without moving to a heavyweight system like Spark. -
11
CubeFS
cloud-native file store
...Support replicas and erasure coding engines, users can choose flexibly according to business scenarios. Easy to build a PB or EB-scale distributed storage service, and each module can be expanded horizontally. Supports multi-tenant management and provides fine-grained tenant isolation policies. Supports multi-level caching, and multiple high-performance replication protocols, and optimizes specific performance for small files. -
12
Encore
The infrastructure platform for the intelligence era
Encore is an infrastructure platform designed to help developers and AI coding agents build, test, and deploy production applications with less manual infrastructure work. It provides an open source infrastructure SDK for TypeScript and Go that lets developers declare databases, Pub/Sub, object storage, caches, cron jobs, secrets, and other resources directly in application code. Encore can run these resources locally with production-like infrastructure and automatically provision corresponding services in AWS or Google Cloud when applications are deployed. The platform supports local development, per-pull-request preview environments, distributed tracing, and production deployments using a consistent infrastructure model. ... -
13
Google Cloud Platform Go Samples
Sample apps and code written for Google Cloud
...It also showcases integrations with services such as Pub/Sub, BigQuery, Cloud Functions, and Vertex AI, helping developers understand how to orchestrate distributed systems in Go. -
14
Apache Spark
A unified analytics engine for large-scale data processing
...The built-in MLlib library provides scalable machine learning algorithms, while GraphX enables graph computations integrated with data pipelines. Spark supports multiple languages—Scala, Java, Python, R—and connects with many storage systems like HDFS, S3, Cassandra, and streaming platforms like Kafka, making it a versatile choice for big data workloads in analytics, ETL, and data science. -
15
M3
Distributed TSDB, Aggregator and Query Engine, Prometheus Sidecar
M3 is a Prometheus-compatible, easy-to-adopt metrics engine that provides visibility for some of the world’s largest brands. M3 is the obvious choice for Cloud Native companies looking to scale up their Prometheus-based monitoring systems. M3 can be used as Prometheus Remote Storage and has 100% PromQL compatibility. M3 was originally developed at Uber in order to provide visibility into Uber’s business operations, microservices, and infrastructure. With its ability to horizontally scale... -
16
Zeebe
Distributed Workflow Engine for Microservices Orchestration
Automate processes at scale with unprecedented performance and resilience. Zeebe is the workflow and decision engine that powers Camunda Platform 8. Zeebe’s cloud-native design provides the performance, resilience, and security enterprises need to future-proof their process orchestration efforts. Zeebe distributes data across all brokers in a cluster with storage directly on the server filesystem. If one broker goes down, another can replace it with no data loss. This pre-configured... -
17
Micro Cloud
A distributed OS built for the Cloud
Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provides a set of services which act as the building blocks of a platform. Micro deals with the complexity of distributed systems and provides simpler programmable abstractions to build on. Micro is the all encompassing end to end platform experience from source to running and beyond built with a developer first focus. Micro’s goal is to abstract away the... -
18
System Design 101
Explain complex systems using visuals and simple terms
System Design 101 is an educational repository that explains software architecture and infrastructure concepts through visual diagrams and concise explanations. It is designed for engineers preparing for system design interviews as well as developers who want to understand how large systems operate. Topics cover APIs, networking, databases, caching, distributed systems, software architecture, payments, security, and development practices. Real-world case studies examine architectures used by... -
19
JuiceFS
JuiceFS is a distributed POSIX file system built on top of Redis
...Purposely built to serve big data scenarios such as self-driving model training, recommendation engine, and Next-generation Gene Sequencing, JuiceFS specializes in high performance and easier management of tens of billion of files management. We bring JuiceFS to developers with the hope that it will be easy to use, reliable, high-performance, and solve all your file storage problems in a cloud environment. -
20
OpenNext for Cloudflare
Open Next.js adapter for Cloudflare
This sibling project targets Cloudflare’s edge platform, adapting Next.js apps to run on Workers and related services. It converts server components, API routes, and middleware into Worker scripts, while placing static assets in globally distributed storage for low-latency delivery. The adapter accounts for edge constraints—limited runtime APIs, streaming semantics, and per-request cold starts—so Next.js features remain functional at the perimeter. It integrates with platform primitives like key-value stores and durable/relational options where needed for sessions, caches, or data hydration. ... -
21
Redpanda
Redpanda is a streaming data platform for developers
...We are building a real-time streaming engine for modern applications - from the enterprise to the solo dev prototyping a react application on her laptop. We go beyond the Kafka protocol, into the future of streaming with inline WASM transforms and geo-replicated hierarchical storage. A new platform that scales with you from the smallest projects to petabytes of data distributed across the globe. We provide a very simple build system that uses your system libraries. We recommend users leverage our pre-built stable releases which are vetted, tested, and reproducible with exact versions of the entire transitive dependency graph, including exact compilers all built from source. ... -
22
KubeSphere
The container platform tailored for Kubernetes multi-cloud, datacenter
KubeSphere is a distributed operating system for cloud-native application management, using Kubernetes as its kernel. It provides a plug-and-play architecture, allowing third-party applications to be seamlessly integrated into its ecosystem. KubeSphere is also a multi-tenant container platform with full-stack automated IT operation and streamlined DevOps workflows. It provides developer-friendly wizard web UI, helping enterprises to build out a more robust and feature-rich platform, which... -
23
learn_python3_spider
Python crawler tutorial series, learning Python crawlers from 0 to 1
learn_python3_spider is a tutorial and example-code collection for learning web scraping with Python 3 from beginner to advanced topics. It introduces request capture, browser and mobile traffic inspection, and core scraping libraries such as Requests, BeautifulSoup, Selenium, Appium, and Scrapy. Lessons also cover JSON parsing, regular expressions, concurrency, proxy usage, login handling, and CAPTCHA-related techniques. Data can be stored with CSV, MySQL, or MongoDB examples. More advanced... -
24
Substrate Node Template
A rust implementation of the Quip Protocol forked from Substrate
...The node supports development and testing environments with configurable genesis states and pre-funded accounts, making it suitable for experimentation and rapid prototyping. It also includes tools for running single-node or multi-node networks, enabling developers to simulate distributed consensus systems locally. The architecture is designed for extensibility, allowing custom consensus algorithms, storage models, and transaction logic to be integrated. -
25
System Design
Learn how to design systems and prepare for system design interviews
This project is an open, course-style repository designed to help you learn system design from fundamentals through advanced, interview-ready thinking. It organizes core networking and distributed-systems concepts into a structured path, so you can build intuition before jumping into “design X” exercises. It covers the building blocks that show up in real architectures, such as DNS, load balancing, caching, CDNs, proxies, scalability and availability tradeoffs, and storage patterns, then connects those pieces to larger system goals. ...
