Open Source Big Data Tools - Page 2
Sort By:
Big Data Tools
View 283 business solutions-
PRTG Catches Network Issues Before They Cause DowntimeReactive troubleshooting usually means hearing about a problem from frustrated users, not your monitoring tool. PRTG sets threshold-based alerts across devices, servers and applications, notifying your team by email, SMS or push the moment a metric crosses a set limit. That means catching a failing disk or overloaded server before it becomes an outage and getting time back from firefighting. Start a free trial and set your first alerts today.
-
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
1
Augustus
PMML-compliant scoring engine and analytic toolkit
Augustus development has moved to google code. The new project page is augustus.googlecode.com. New releases of the project are not currently being released to sourceforge. Augustus is designed for statistical and data mining models and produces and consumes models with 10,000s of segments. Versions of Augustus support PMML 3, 4.0.1, and 4.1. -
2
Parkiet
Parquet format file GUI editor
Parquet file viewer and editor written in Java and SWT. It uses Apache Avro library for reading and writing edited parquet files. Only Parquet files with simple data type columns are supported. -
3
Faum
Fast Autonomous Unsupervised Multidimiensional Classification
This is the proof-of-concept implementation of the FAUM Clustering method. This implementation was used to perform the published results and is now released in the hope that it will be useful. -
4
NCHC-Storm
NCHC's Storm Team
Sharing the applications of storm which developed by NCHC's Storm Team. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
5
.NET for Apache Spark
A free, open-source, and cross-platform big data analytics framework
.NET for Apache Spark provides high-performance APIs for using Apache Spark from C# and F#. With these .NET APIs, you can access the most popular Dataframe and SparkSQL aspects of Apache Spark, for working with structured data, and Spark Structured Streaming, for working with streaming data. .NET for Apache Spark is compliant with .NET Standard - a formal specification of .NET APIs that are common across .NET implementations. This means you can use .NET for Apache Spark anywhere you write .NET code allowing you to reuse all the knowledge, skills, code, and libraries you already have as a .NET developer. .NET for Apache Spark runs on Windows, Linux, and macOS using .NET Core, or Windows using .NET Framework. It also runs on all major cloud providers including Azure HDInsight Spark, Amazon EMR Spark, AWS & Azure Databricks. -
6
AXYZ
Newsfeed aggregator
The article explains the design and the characteristics of a new open source content aggregation system. Among the features of the program, stands out a new processing engine of syndication channels, monitoring capability of information recovery in real time, the possibilities of the configuration of aggregator behavior, automatic classification of contents and new models for representation of information from relational interactive maps. On the other hand, the aggregation program, which named AXYZ, is designed to manage thousands of syndication channels of RSS format. Furthermore it also provides statistics that can be used to study the production of any producer subject and the impact of the information that published in other sources. The result that has been obtained in the research, allow to create modules capable of compare the relationship between different news or information from different sources, their degree of influence and their detection by the patterns. -
7
Apache Doris
MPP-based interactive SQL data warehousing for reporting and analysis
Apache Doris is a modern MPP analytical database product. It can provide sub-second queries and efficient real-time data analysis. With it's distributed architecture, up to 10PB level datasets will be well supported and easy to operate. Apache Doris can meet various data analysis demands, including history data reports, real-time data analysis, interactive data analysis, and exploratory data analysis. Make your data analysis easier! Support standard SQL language, compatible with MySQL protocol. The main advantages of Doris are the simplicity (of developing, deploying and using) and meeting many data serving requirements in a single system. Doris mainly integrates the technology of Google Mesa and Apache Impala, and it is based on a column-oriented storage engine and can communicate by MySQL client. -
8
Apache Hudi
Upserts, Deletes And Incremental Processing on Big Data
Apache Hudi (pronounced Hoodie) stands for Hadoop Upserts Deletes and Incrementals. Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage). Apache Hudi is a transactional data lake platform that brings database and data warehouse capabilities to the data lake. Hudi reimagines slow old-school batch data processing with a powerful new incremental processing framework for low latency minute-level analytics. Hudi provides efficient upserts, by mapping a given hoodie key (record key + partition path) consistently to a file id, via an indexing mechanism. This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the mapped file group contains all versions of a group of records. -
9
Apache Polaris
Apache Polaris, the interoperable, open source catalog
Apache Polaris is an open-source metadata catalog and data management service designed to manage Apache Iceberg tables in modern data lakehouse environments. It provides a centralized catalog that allows multiple compute engines and analytics systems to interact with the same datasets through a standardized interface. By implementing the Iceberg REST catalog API, Polaris enables distributed data platforms to access shared table metadata without tightly coupling storage systems and query engines. This design allows organizations to run queries on the same Iceberg tables using tools such as Apache Spark, Flink, Trino, and other analytics engines while maintaining consistency across platforms. Polaris also focuses on data governance, security, and interoperability within large-scale cloud data architectures. Because Iceberg tables often exist across many services in a distributed ecosystem, the catalog helps coordinate metadata, schemas, and access policies in a unified system. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
Apache RocketMQ
Distributed messaging and streaming platform with low latency
Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. Messaging patterns including publish/subscribe, request/reply and streaming. Financial grade transactional message. Built-in fault tolerance and high availability configuration options base on DLedger. A variety of cross language clients, such as Java, C/C++, Python, Go. Pluggable transport protocols, such as TCP, SSL, AIO. Built-in message tracing capability, also support opentracing. Versatile big-data and streaming ecosytem integration. Message retroactivity by time or offset. Reliable FIFO and strict ordered messaging in the same queue. Efficient pull and push consumption model. Million-level message accumulation capacity in a single queue. Multiple messaging protocols like JMS and OpenMessaging. Flexible distributed scale-out deployment architecture. Lightning-fast batch message exchange system. -
11
Arroyo
Distributed stream processing engine in Rust
Arroyo is a distributed stream processing engine written in Rust, designed to efficiently perform stateful computations on streams of data. Unlike traditional batch processing, streaming engines can operate on both bounded and unbounded sources, emitting results as soon as they are available. -
12
BEAR
CBR Meets Big Data
Case-based regression learner for big data. The package contains source and binary files for running BEAR's method. BEAR utilizes EAR4 and locality sensitive hashing in its implementation. -
13
BIRT iHub F-Type
Free report server for creating data-driven apps
Open Source BIRT (Business Intelligence and Reporting Tools) is a visual-based development tool used to create data visualizations and reports that can be embedded into rich client and web applications. BIRT is a top-level designer tool within the Eclipse Foundation, an independent not-for-profit open source community and consortium of software vendors. For the first time ever, Open Source developers using BIRT can now leverage technology previously reserved for commercial applications with BIRT iHub F-Type, a free enterprise-grade server for managing and delivering BIRT content to any number of users, while instantly enhancing Open Source reporting functionality and reducing deployment time down to minutes. With out-of-the-box Big Data support, BIRT iHub F-Type connects to all relational databases and allows unlimited data-in for creating data visualizations and dynamic report views. -
14
Big Sack
Big Sack: A lightweight Java Key/Value store with undo and disk cache.
Big Sack is a Java persistence mechanism that allows storage of key value pairs following the popular Big Data paradigms. Its a very simple and straightforward way to bridge the gap between in-memory data structures and long-term storage. It has the convenience of Java SDK TreeMap and TreeSet classes and is used the same easy way, but it includes rollback through undo logging to checkpoint data so it does not wind up in an unknown state regardless of failures. Data storage in the exabyte range is possible using filesystem and/or memory-mapped IO. Three levels of configurable write-through caching at different granularities ensure performance. -
15
Blue Whale Configuration Platform
Blue Whale smart cloud configuration platform
Has accumulated experience in supporting hundreds of Tencent businesses, compatible with various complex system architectures, born in operation and maintenance, and proficient in operation and maintenance. From configuration management to job execution, task scheduling and monitoring self-healing, and then through operation and maintenance big data analysis to assist operational decision-making, it covers the full-cycle assurance management of business operations in a comprehensive manner. The open PaaS has a powerful development framework and scheduling engine, as well as a complete operation and maintenance development training system, which helps the rapid transformation and upgrading of operation and maintenance. Through the Blue Whale intelligent cloud system, it can help enterprises quickly realize the automation of basic operation and maintenance services, thereby accelerating the transformation of DevOps, realizing a tool culture, and maximizing operational efficiency. -
16
Chinese I Ching R Algorithms
Chinese I Ching Algorithms implemented with R programming
I Ching offers an idea to summary the world by constructing functional mappings in finite groups. Much of ancient Chinese natural and social science grew from its concepts as root. The derived theories like "Eight Diagrams", "Five Elements" became the foundation of nearly all academic fields in acient Chinese. As a result, the theories are also the high concentration of many practical and statistical experience in China during her thousands-of-year history. This project tries to realize these algorithms with statistical programming. Overcoming the long span has important meanings. First, we can verify the validity of algorithm with big data. Second, we can put them into unprecedented wide application in the age of data. Third, the unique inspection of I Ching that linked "all the universe" open new eyes for us in thought and methodology about interdisciplinary use of data. This program is open-source for human. If you are interested in it, welcome to discuss and work together. -
17
Chinese I Ching R Algorithms
Chinese I Ching Algorithms implemented with R programming
I Ching offers an idea to summary the world by constructing functional mappings in finite groups. Much of ancient Chinese natural and social science grew from its concepts as root. The derived theories like "Eight Diagrams", "Five Elements" became the foundation of nearly all academic fields in acient Chinese. As a result, the theories are also the high concentration of many practical and statistical experience in China during her thousands-of-year history. This project tries to realize these algorithms with statistical programming. Overcoming the long span has important meanings. First, we can verify the validity of algorithm with big data. Second, we can put them into unprecedented wide application in the age of data. Third, the unique inspection of I Ching that linked "all the universe" open new eyes for us in thought and methodology about interdisciplinary use of data. This program is open-source for human. If you are interested in it, welcome to discuss and work together. -
18
Chordalysis
Log-linear analysis (data modelling) for high-dimensional data
===== Project moved to https://github.com/fpetitjean/Chordalysis ===== Log-linear analysis is the statistical method used to capture multi-way relationships between variables. However, due to its exponential nature, previous approaches did not allow scale-up to more than a dozen variables. We present here Chordalysis, a log-linear analysis method for big data. Chordalysis exploits recent discoveries in graph theory by representing complex models as compositions of triangular structures, also known as chordal graphs. Chordalysis makes it possible to discover the structure of datasets with thousands of variables on a standard desktop computer. Associated papers at ICDM 2013, ICDM 2014 and SDM 2015 can be found at http://www.francois-petitjean.com/Research/ YourKit is supporting Chordalysis open source project with its full-featured Java Profiler. YourKit is the creator of innovative and intelligent tools for profiling Java and .NET applications. http://www.yourkit.com -
19
CrispI
A hybrid graph database with analysis
A graph database layer in Java which allows mixed-mode database handling. For example, using an underlying oodb, a OneToMany and ManyToOne relationships can be implemented allowing quick and robust hierarchies to be built (a subset of a graph). The system also includes a Big Data implementation, other analytics and visualization. The current invokation is based on a Versant Db. Examples for use can be found in CrispI-Examples. -
20
Cube Platform is a decentralized grid computing system that uses P2P Pastry protocol for communication between nodes. It's a big data storage written in Java.
-
21
Custom Apache Big data Distribution
A Custom Apache Distribution including Spark and Hadoop, for Windows.
This Distribution has been customized to work out of the box. So, just download it, and unzip it. Set the Path variables for bin folders, HADOOP_HOME, SPARK_HOME, and JAVA_HOME. That's it..! use Hadoop and Spark natively on Windows. -
22
DSTK - DataScience ToolKit
DSTK - DataScience ToolKit for All of Us
DSTK - DataScience ToolKit is an opensource free software for statistical analysis, data visualization, text analysis, and predictive analytics. Newer version and smaller file size can be found at: https://sourceforge.net/projects/dstk3/ It is designed to be straight forward and easy to use, and familar to SPSS user. While JASP offers more statistical features, DSTK tends to be a broad solution workbench, including text analysis and predictive analytics features. Of course you may specify JASP for advanced data editing and RapidMiner for advanced prediction modeling. DSTK is written in C#, Java and Python to interface with R, NLTK, and Weka. It can be expanded with plugins using R Scripts. We have also created plugins for more statistical functions, and Big Data Analytics with Microsoft Azure HDInsights (Spark Server) with Livy. License: R, RStudio, NLTK, SciPy, SKLearn, MatPlotLib, Weka, ... each has their own licenses. -
23
FinMind
Open Data, more than 50 financial data
In the era of big data, data is the foundation of everything. We collect more than 50 kinds of Taiwan stock related information and provide download, online analysis, and backtesting. Regardless of the program, you can download data through the api provided by FinMind, or you can download data directly from the website. After data is available, statistical analysis, regression analysis, time series analysis, machine learning, and deep learning can be performed. For individual stocks, provide visual analysis of technical, fundamental, and chip levels. According to different strategies, back-test analysis is performed to provide performance, profit and loss, and stock selection targets of different strategy investment portfolios. -
24
Fluid
Fluid, elastic data abstraction and acceleration for BigData/AI apps
Fluid, elastic data abstraction and acceleration for BigData/AI applications in the cloud. Provide DataSet abstraction for underlying heterogeneous data sources with multidimensional management in a cloud environment. Enable dataset warmup and acceleration for data-intensive applications by using a distributed cache in Kubernetes with observability, portability, and scalability. Taking characteristics of application and data into consideration for cloud application/dataset scheduling to improve the performance. -
25
FrincBackup
Incremtal backup tool supporting removable storage devices
FrincBackup means free incremental backup. It is developed for backing up a x TB NAS with storage devices in a logical volume to multiple removable storage devices, such as 500 GB USB hard drives. Files are backuped as files (not as an archive) and are readable without the need of a tool and without the need of FrincBackup itself (allthough there is a restore mode for better handling).
