Open Source Big Data Tools - Page 3
Sort By:
Big Data Tools
View 283 business solutions-
Earn up to 16% annual interest with Nexo.Generate interest, borrow against your crypto, and trade a range of cryptocurrencies — all in one platform. Geographic restrictions, eligibility, and terms apply.
-
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
1
GOBIG is a toolbox that can be used for detecting genetic variations. The project is intended to handle big data. What's more important is that it be used to detect clusters of SNP variants. It is the intention to use the toolbox with common and rare variants. To use it, for example, to find the genetic map of genes causing complex diseases.
-
2
Genie
Distributed Big Data Orchestration Service
Genie is a completely open source distributed job orchestration engine developed by Netflix. Genie provides REST-ful APIs to run a variety of big data jobs like Hadoop, Pig, Hive, Spark, Presto, Sqoop and more. It also provides APIs for managing the metadata of many distributed processing clusters and the commands and applications which run on them. -
3
GnuCopy is an Open-Source tool to copy and archive all your important data. It supports all important archive typs like Zip and Tar to guaranty an easy and secure exchange between all types of operating systems. Additionally, you can create profiles to blacklist or whitelist specific file types or folders to seperate your big data stores for backups.
-
4
HPCC Systems
End-to-end big data in a massively scalable supercomputing platform.
Important: As of April 20, 2026, this project can now be found at https://github.com/hpcc-systems/HPCC-Platform/releases. HPCC Systems® (www.hpccsystems.com) from LexisNexis® Risk Solutions is a proven, open source solution for Big Data insights that can be implemented by businesses of all sizes. With HPCC Systems, developers can design applications with Big Data at their core, enabling businesses to better analyze and understand data at scale, improving business time to results and decisions. HPCC Systems offers a consistent data-centric programming language, two processing platforms and a single, complete end-to-end architecture for efficient processing. -
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
5
HSRA
Hadoop spliced read aligner for RNA-seq data
HSRA is a MapReduce-based parallel tool for mapping reads from RNA sequencing (RNA-seq) experiments. RNA-seq analyses typically begin by mapping reads to a reference genome in order to determine the location from which the reads were originated, which is a very time-consuming step. This tool allows bioinformatics researchers to efficiently distribute their mapping tasks over the nodes of a cluster by combining a fast multithreaded spliced aligner (HISAT2) with Apache Hadoop, which is a distributed computing framework for scalable Big Data processing. HSRA currently supports single-end and paired-end read alignments from FASTQ/FASTA datasets. Moreover, our tool uses the Hadoop Sequence Parser (HSP) library (link above) to efficiently read the input datasets stored on the Hadoop Distributed File System (HDFS), being able to process datasets compressed with Gzip and BZip2 codecs. -
6
The idea about this project is use it to create and provide a big data structures library, like trees, list and other.
-
7
JuiceFS
JuiceFS is a distributed POSIX file system built on top of Redis
A POSIX, HDFS and S3 compatible distributed file system for cloud. JuiceFS is designed to bring back the gold-old memories and experience of file systems in local disks to the cloud. JuiceFS is POSIX compliant and is fully compatible with HDFS and S3. Cloud app building or migrating, file sharing cross-geo and cross-cloud has become easier than ever before. Whether it's a public cloud, private cloud, or hybrid cloud, JuiceFS is available on any cloud of your choice and delivers flexibility, availability, scalability and strong consistency for your data-intensive applications. Purposely built to serve big data scenarios such as self-driving model training, recommendation engine, and Next-generation Gene Sequencing, JuiceFS specializes in high performance and easier management of tens of billion of files management. We bring JuiceFS to developers with the hope that it will be easy to use, reliable, high-performance, and solve all your file storage problems in a cloud environment. -
8
Logan
Logan is a lightweight case logging system based on mobile platform
Logan is a log platform with the ability to collect, store, upload and analyze front-end logs. We provide five components, including iOS SDK, Android SDK, Web SDK, analysis services Server SDK and LoganSite. In addition, we also provide a Flutter plugin Flutter Plugin. LoganSite provides a visualized way for developers to scan and search logs uploaded from App and Web. To put it simply, the traditional idea is to piece together the problems that appear in the logs of each system, but the new idea is to aggregate and analyze all the logs generated by the user to find the scenes with problems. In the future, we will provide a data platform based on Logan big data, including advanced functions such as machine learning, troubleshooting log solution, and big data feature analysis. -
9
LogicalSets
Integrated Comprehensive Data Architecture & Methodology
This is an advanced data architecture and methodology. A comprehensive Enterprise Resource Management System. A re-usable database with rules for customization, While being a data driven transaction processing engine, this system has very advanced reporting capabilities. This design eliminates up to 90% of business logic due to the way the data is structured. Uses a concept called Table Sets. Has a compound key that tells the programmer what tableset, which record which applet will view/edit the data. Developed in SAP PowerDesigner, for (Sybase) SQL Anywhere. Don't let the date fool you, this system is ahead of its time. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
MapReduce Brazil
Aggregates MapReduce projects
Nowadays the production and storage of Big Data is common, both in the academy and in the enterprises. To process this huge amount of data it is essential the use of high performance platforms and programming models like MapReduce -
11
MarDRe
MapReduce-based tool to remove duplicate DNA reads
MarDRe is a de novo MapReduce-based parallel tool to remove duplicate and near-duplicate DNA reads through the clustering of single-end and paired-end sequences from FASTQ/FASTA datasets. This tool allows bioinformatics to avoid the analysis of not necessary reads, reducing the time of subsequent procedures with the dataset. MarDRe is the Big Data counterpart of ParDRe (link above), which employs HPC technologies (i.e., hybrid MPI/multithreading) to reduce runtime on multicore systems. Instead, MarDRe takes advantage of the MapReduce programming model to significantly improve ParDRe performance on distributed systems, especially on cloud-based infrastructures. Written in pure Java to maximize cross-platform compatibility, MarDRe is built upon the open-source Apache Hadoop project, the most popular distributed computing framework for Big Data processing. -
12
Modin
Scale your Pandas workflows by changing a single line of code
Scale your pandas workflow by changing a single line of code. Modin uses Ray, Dask or Unidist to provide an effortless way to speed up your pandas notebooks, scripts, and libraries. Unlike other distributed DataFrame libraries, Modin provides seamless integration and compatibility with existing pandas code. Even using the DataFrame constructor is identical. It is not necessary to know in advance the available hardware resources in order to use Modin. Additionally, it is not necessary to specify how to distribute or place data. Modin acts as a drop-in replacement for pandas, which means that you can continue using your previous pandas notebooks, unchanged, while experiencing a considerable speedup thanks to Modin, even on a single machine. Once you’ve changed your import statement, you’re ready to use Modin just like you would pandas. -
13
Nebula Graph
A distributed, fast open-source graph database
The graph database built for super large-scale graphs with milliseconds of latency. Optimized SUBGRAPH and FIND PATH for better performance. Optimized query paths to reduce redundant paths and time complexity. Optimized the method to get properties for better performance of MATCH statements. Nebula Graph adopts the Apache 2.0 license, one of the most permissive free software licenses in the world. Free as in freedom, because, under the Apache 2.0 license, you can use, copy, modify and redistribute Nebula Graph, even for commercial purposes, all without asking for permission. We believe that great open source projects are not built in isolation, but rather by a community of contributors. We welcome contributions to Nebula Graph from anyone regardless of skill level or background in software development. If you have an idea for a feature you would like to see added, or you have identified a bug that needs fixing, please don't hesitate to submit an issue to our Github repository. -
14
Neuro
The Neuro crypto currency
The Neuro NRO cryptocurrency is designed to support solutions of machine learning tasks, big data and neural networks. Neuro is a scientific-technical project uniting scientists, engineers and programmers inspired by the idea to build something big, kind and bright. From the first stages of work, we will be engaged in the development of new architectures and algorithms of neural networks. Someday we will undoubtedly enter the annual ImageNet Challenge contest to compete with such giants as GoogLeNet Inception and Microsoft ResNet. At further stages of the work, we adapt the neural networks to calculate molecular interactions in protein environments. Our system will help to look for new types of drugs for cancer, Alzheimer's and other serious problems of modern medicine. We plan to make a serious contribution to the increase of human life expectancy. -
15
OCW Test - Out of Commerce Works
Program for out of commerce works detection
The OCW Test program has been designed to provide assistance in the detection of works outside trade, taking as reference a list of works from a specific bibliographic catalog. In this first version, the program operates on the identifiers of the books of the library of the Complutense University of Madrid. However, the program can be reedited, to work on any bibliographic catalog. -
16
ODD Platform
First open-source data discovery and observability platform
Unlock the power of big data with OpenDataDiscovery Platform. Experience seamless end-to-end insights, powered by unprecedented observability and trust - from ingestion to production - while building your ideal tech stack! Democratize data and accelerate insights. Find data that fits your use case and discover hints left by your peers to leverage existing knowledge. Explore tags, ownership details, links to other sources and other information to shorten and simplify data discovery phase. Forget unnerved stakeholders and wasting too much time on digging the root cause of data issues when it fails. With ODD’s automatic company-wide ingestion-to-product lineage you’ll have answers in just seconds and stakeholders won’t need to wait. Sleep well, knowing all your data is in check. Forget manual testing, days of debugging, and weeks of worrying. Know the impact of each code change with automatic testing. Enjoy lineage and alerts powered with data quality information. -
17
Oasis Development Tool
OASIS Development Tool
The OASIS Development Tool is an innovative IDE for Code Generation-, Code Debugging- and Visual Coding- using the OASIS Programming Language. The OASIS Programming Language is a 4GL Concurrency- and Database Language running round a distributed OASIS Runtime Machine Environment (RME) as interpreted OASIS Scripts sequenced into OASIS Polyglot Runtime Components (PRC) with just in time patterns. The IDE is designed specifically for the OASIS Programming Language. The IDE is focused around the concept of Visual-, Online-, Data-Centric-, Concurrent-, and Runtime- Code, whilst remaining an IDE to handle OASIS Programming. The IDE has a number of visual code drag and drop features. The Tool is by no means a representative of the Cyclical UML Model- and Code concept, but rather a replacement. The IDE Tool is focused around (Team Based) System Engineering, Meta Programming, Visual Coding, Concurrent Processing and, Databases and Big Data. -
18
Oblivious Bloom Intersection
Oblivious Bloom Intersection
This page is about the PSI implementation described in the paper: When Private Set Intersection Meets Big Data: An Efficient and Scalable Protocol (CCS 2013). -
19
Occursions
Fast customizable time series web database for big data like log files
Our goal is to create the world's fastest extendable, non-transactional time series database for big data (you know, for kids)! Log file indexing is our initial focus. For example append only ASCII files produced by libraries like Log4J, or containing FIX messages or JSON objects. Occursions was built by a small team sick of creating hacks to remotely copy and/or grep through tons of large log files. We use it to index around a terabyte of new log data per day. You can use it too. Who doesn't have `just too many' log files? Occursions asynchronously tails log files and indexes the individual lines in each log file as each line is written to disk so you don't even have to wait for a second after an event happens to search for it. Occursions uses custom disk backed data structures to create and search its indexes so it is very efficient at using CPU, memory and disk. You can extend Occursions with shared libraries to support your own file formats, even binary file formats! -
20
Open Source Data Quality and Profiling
World's first open source data quality & data preparation project
This project is dedicated to open source data quality and data preparation solutions. Data Quality includes profiling, filtering, governance, similarity check, data enrichment alteration, real time alerting, basket analysis, bubble chart Warehouse validation, single customer view etc. defined by Strategy. This tool is developing high performance integrated data management platform which will seamlessly do Data Integration, Data Profiling, Data Quality, Data Preparation, Dummy Data Creation, Meta Data Discovery, Anomaly Discovery, Data Cleansing, Reporting and Analytic. It also had Hadoop ( Big data ) support to move files to/from Hadoop Grid, Create, Load and Profile Hive Tables. This project is also known as "Aggregate Profiler" Resful API for this project is getting built as (Beta Version) https://sourceforge.net/projects/restful-api-for-osdq/ apache spark based data quality is getting built at https://sourceforge.net/projects/apache-spark-osdq/ -
21
PROPER is a package for visual evaluation of ranking classifiers for biological big data mining studies in the mathematical language MATLAB. It is an efficient tool for optimization and comparison of the state-of-the-art ranking classifiers by generating over 20 different high quality two- and three-dimensional performance curves.
-
22
PROPER is a package for visual evaluation of ranking classifiers for biological big data mining studies in the mathematical language MATLAB. It is an efficient tool for optimization and comparison of the state-of-the-art ranking classifiers by generating over 20 different high quality two- and three-dimensional performance curves.
-
23
PanoramaServer
Open Source Panorama Server for free virtual tour of 360 degrees views
Ideal for creating virtual tours of panoramic views for all sorts including property exhibition for brokers at real estate agencies/property agents, tour guide for indoor/outdoor venues, information to public/private facilities for curators, travel journal for tourist as log book, backdrop setting for storytelling, treasure hunt like games, big data mining for pattern through computer vision in artificial intelligence, etc. It is like creating your own Google Map Street View. All is required by the user is to have photos of equirectangular format (panorama) taken from 3D cameras common for on-site premises. These images can be referenced by the PanoramaServer to create virtual travels with 360 degrees view where viewers can navigate to different locations, view information, etc. If made available online to general public over the internet, can even share the link of your virtual trips. PanoramaServer is free as it is open source licensed. -
24
PartialLoader
A class library for issuing big data in parts.
-
25
Querybook
Big Data Querying UI, combining collocated table metadata
Querybook is Pinterest’s open-source big data IDE via a notebook interface. Querybook’s core focus is to make composing queries, creating analyses, and collaborating with others as simple as possible. Organize rich text, queries, and charts into a notebook to easily document your analyses. Work collaboratively with others in a DataDoc and get real-time updates. The Query Editor is aware of your tables and their columns, as such it provides autocompletion, syntax highlighting, and the ability to hover or click on a table to view its information. No need to leave Querybook to create charts to quickly visualize your results. With a familiar interface easily create line, bar, stacked area, pie, horizontal bar, donut, scatter, and table charts. Add them then to your DataDoc to complete your data narrative.
