Search Results for "cluster computing"
Sort By:
Showing 154 open source projects for "cluster computing"
View related business solutions-
Veeam Data Platform v13.1Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
1
Kraken
P2P Docker registry capable of distributing TBs of data in seconds
Kraken is a P2P-powered Docker registry that focuses on scalability and availability. It is designed for Docker image management, replication, and distribution in a hybrid cloud environment. With pluggable backend support, Kraken can easily integrate into existing Docker registry setups as the distribution layer. Kraken has been in production at Uber since early 2018. In our busiest cluster, Kraken distributes more than 1 million blobs per day, including 100k 1G+ blobs. At its peak... -
2
Apache Sedona
Cluster computing framework for processing large-scale geospatial data
Apache Sedona™ is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines. According to our benchmark and third-party research papers, Sedona runs 2X - 10X faster than other Spark-based geospatial data systems on computation-intensive query workloads. ... -
3
Dagger.jl
A framework for out-of-core and parallel execution
Dagger.jl is a framework for out-of-core and parallel computing in Julia that allows users to construct and execute dynamic task graphs. It is designed for large-scale, distributed, and memory-efficient computations. Dagger supports lazy evaluation and scheduling across multiple threads or machines, enabling high-performance workflows for data processing, scientific computing, and machine learning. -
4
kubeinvaders
Gamified Chaos Engineering Tool for Kubernetes
This project is part of the landscape of Cloud Native Computing Foundation in the Observability and Analysis - Chaos Engineering section. Backed by the teams at platform engineering.it and devopstribe.it, which provides enterprise-grade features and certified resilience services for your Kubernetes infrastructure. -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
LINKERD
Ultralight, security-first service mesh for Kubernetes
Enterprise power without enterprise complexity. Linkerd adds security, observability, and reliability to any Kubernetes cluster. 100% open source, CNCF graduated, and written in Rust. Instantly add latency-aware load balancing, request retries, timeouts, and blue-green deploys to keep your applications resilient. Incredibly small and blazing fast Linkerd2-proxy micro-proxy written in Rust for security and performance. Self-contained control plane, incrementally deployable data plane, and... -
6
MicroK8s
Single-package Kubernetes for developers, IoT and edge
Low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. MicroK8s automatically chooses the best nodes for the Kubernetes datastore. When you lose a cluster database node, another node is promoted. No admin needed for your bulletproof edge. MicroK8s is small, with sensible defaults that ‘just work’. A quick install, easy upgrades and great security make it perfect for micro clouds and edge computing. As the publishers of MicroK8s, we deliver the world’s most efficient multi-cloud, multi-arch Kubernetes. ... -
7
sparklyr
R interface for Apache Spark
sparklyr is an R package that provides seamless interfacing with Apache Spark clusters—either local or remote—while letting users write code in familiar R paradigms. It supplies a dplyr-compatible backend, Spark machine learning pipelines, SQL integration, and I/O utilities to manipulate and analyze large datasets distributed across cluster environments. -
8
Redis
An in-memory database that persists on disk
...Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. You can run atomic operations on these types, like appending to a string; incrementing the value in a hash; pushing an element to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set. To achieve top performance, Redis works with an in-memory dataset. Depending on your use case, you can persist your data either by periodically dumping the dataset to disk or by appending each command to a disk-based log. -
9
Kubernetes Handbook
Cloud native application architecture practice handbook
...Kubernetes is a container orchestration and scheduling engine developed by Google in June 2014 based on its internal Borg system. Google contributed it as an initial and core project to the CNCF (Cloud Native Computing Foundation). In recent years, it has gradually developed a cloud native Ecology. The goal of Kubernetes is to provide a specification to describe the architecture of the cluster. -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
10
PelicanHPC is an iso-hybrid (CD or USB) image that let's you set up a high performance computing cluster in a few minutes. A Pelican cluster allows you to do parallel computing using MPI. You can run Pelican on a single multiple core machine to use all cores to solve a problem, or you can network multiple computers together to make a cluster. The frontend node (either a real computer or a virtual machine) boots from the image.
-
11
Smile
Statistical machine intelligence and learning engine
Smile is a fast and comprehensive machine learning engine. With advanced data structures and algorithms, Smile delivers the state-of-art performance. Compared to this third-party benchmark, Smile outperforms R, Python, Spark, H2O, xgboost significantly. Smile is a couple of times faster than the closest competitor. The memory usage is also very efficient. If we can train advanced machine learning models on a PC, why buy a cluster? Write applications quickly in Java, Scala, or any JVM... -
12
AlphaZero.jl
A generic, simple and fast implementation of Deepmind's AlphaZero
Beyond its much publicized success in attaining superhuman level at games such as Chess and Go, DeepMind's AlphaZero algorithm illustrates a more general methodology of combining learning and search to explore large combinatorial spaces effectively. We believe that this methodology can have exciting applications in many different research areas. Because AlphaZero is resource-hungry, successful open-source implementations (such as Leela Zero) are written in low-level languages (such as C++)... -
13
tsuru
Open source and extensible Platform as a Service (PaaS)
Tsuru is an open source Platform as a Service software, focused in Developer productivity. Tsuru goes beyond 12 factor apps. Run any application written in any language or framework. The deploy process is really simple with just one command. Grow your application dynamically allocating resources with ease. Let your developers code and understand the business instead of solving infrastructure problems or handling large configuration files. We built the platform as a service atop of... -
14
Covalent workflow
Pythonic tool for running machine-learning/high performance workflows
Covalent is a Pythonic workflow tool for computational scientists, AI/ML software engineers, and anyone who needs to run experiments on limited or expensive computing resources including quantum computers, HPC clusters, GPU arrays, and cloud services. Covalent enables a researcher to run computation tasks on an advanced hardware platform – such as a quantum computer or serverless HPC cluster – using a single line of code. Covalent overcomes computational and operational challenges inherent in AI/ML experimentation. -
15
AWS ParallelCluster Cookbook
The Chef cookbook used to build and bootstrap AWS ParallelCluster
AWS ParallelCluster is an AWS supported Open Source cluster management tool that makes it easy for you to deploy and manage High Performance Computing (HPC) clusters in the AWS cloud. Built on the Open Source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and a shared filesystem and offers a variety of batch schedulers such as AWS Batch and Slurm. -
16
Kubeflow Trainer
Distributed AI Model Training and LLM Fine-Tuning on Kubernetes
...The platform supports a wide range of machine learning frameworks, including PyTorch, JAX, Hugging Face, DeepSpeed, and XGBoost, making it highly flexible for different AI use cases. One of its key innovations is the integration of MPI-based distributed computing within Kubernetes, allowing efficient communication between nodes for high-performance training. It also includes advanced scheduling capabilities through integrations with tools like Kueue and Volcano, enabling topology-aware resource allocation and multi-cluster job orchestration. -
17
AWS ParallelCluster Node
Python package installed on the Amazon EC2 instances
aws-parallelcluster-node is the python package installed on the Amazon EC2 instances launched as part of AWS ParallelCluster. AWS ParallelCluster is an AWS-supported Open Source cluster management tool that makes it easy for you to deploy and manage High-Performance Computing (HPC) clusters in the AWS cloud. Built on the Open Source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and a shared filesystem and offers a variety of batch schedulers such as AWS Batch and Slurm. ... -
18
Apache Seata
High-performance, open source distributed transaction solution
Seata is a distributed transaction solution for microservices that provides consistent, cross-service commits without forcing every team to adopt the same persistence model. Its architecture separates responsibilities into a global coordinator and per-service participants, so business services remain decoupled while transactions are orchestrated centrally. Multiple modes are supported—AT (automatic, SQL-based with undo logs), TCC (try-confirm-cancel), Saga (long-running compensation), and... -
19
Matrix
Multi-Agent daTa geneRation Infra and eXperimentation framework
Matrix is a distributed, large-scale engine for multi-agent synthetic data generation and experiments: it provides the infrastructure to run thousands of “agentic” workflows concurrently (e.g. multiple LLMs interacting, reasoning, generating content, data-processing pipelines) by leveraging distributed computing (like Ray + cluster management). The idea is to treat data generation as a “data-to-data” transformation: each input item defines a task, and the runtime orchestrates asynchronous, peer-to-peer agent workflows, avoiding global synchronization bottlenecks. That design makes Matrix particularly well-suited for large-batch inference, model benchmarking, data curation, augmentation, or generation — whether for language, code, dialogue, or multimodal tasks. ... -
20
Java Multiprocessing
Java explicit multiprocessing, SSI and cluster management tool
▪ JMP library allows to do multiprocessing in Java with a simple API that resembles Java multithreading. JMP allows running a piece of Java code on a separate process either locally or on remote machine. ▪ If the code in question is not present on the target machine, it can be dynamically uploaded. ▪ JMP allows setting affinity for a new process explicitly instead of relying on the OS. ▪ JMP also contains a simple cluster management / load balancing tool called JMP node. ▪ JMP = SSI +... -
21
Bufferstack.IO Super Container OS
Live OS running Docker Container Engine, Kubernetes, Xen Hypervisor
Super Container OS makes use of os level virtualization to image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings, all this by making use of Docker Container Engine and systemd-nspawn Also included in is Xen Hypervisor and Kubernetes to create full fledged container cluster After installing this OS on HDD, please use Portainer to manage your Docker images. Portainer 2.0.0 is now preinstalled along with Helm (tool for managing Charts. Charts are packages of pre-configured Kubernetes resources) Run portainer as root and use your browser to log in portainer running on port number 9000 Refer details of Helm here - https://github.com/helm/helm Super Container OS is based on Bufferstack,IO Computing Platform, for more details, please check https://bufferstack.io -
22
Amplicon_Sequencing_Worfklow
Analyzing amplicon data from sequences to stats
...All you need is to have the latest version of R installed, some experience with the command line & shell, and enough memory to run all of the programs. There are also instructions provided in case you are running these analyses via a computing cluster/Slurm workload manager. You can choose to go through the workflow using either an Rmd script, an html file, or a PDF, or via the homepage link provided. If you have questions or concerns, please don't hesitate to reach out. Thanks! -
23
KubeOperator
An open source, lightweight Kubernetes distribution
...Compared with heavyweight PaaS platforms such as OpenShift, KubeOperator only focuses on solving one problem, which is to help enterprises plan (Day 0), deploy (Day 1), and operate (Day 2) production-level K8S clusters, and achieve the ultimate. Supports multiple computing, storage and networking scenarios. Integrate Ansible and Terraform. Support online environment and offline environment deployment. Provides a visual web UI. Supports cluster planning, deployment and operations. Easily run workloads like machine learning, high-performance computing, and more. Quickly deploy and manage applications in K8S. ... -
24
Mars Framework
Mars is a tensor-based unified framework for large-scale data
Mars is a distributed computing framework designed to scale scientific computing and data science workloads across large clusters while preserving the familiar programming interfaces of common Python libraries. The project provides a tensor-based execution model that extends the capabilities of tools such as NumPy, pandas, and scikit-learn so that large datasets can be processed in parallel without rewriting code for distributed environments. -
25
SuperEdge
An edge-native container management system for edge computing
SuperEdge is an open-source container management system for edge computing to manage compute resources and container applications in multiple edge regions. These resources and applications, in the current approach, are managed as one single Kubernetes cluster. A native Kubernetes cluster can be easily converted to a SuperEdge cluster. SuperEdge extends the powerful container orchestration and scheduling capabilities of Kubernetes to the edge.
