CSCI E-63 Big Data Analytics (24038) 2016 Spring term (4 credits)

Zoran B. Djordjevi, PhD, Senior Enterprise Architect, NTT Data, Inc.

The recent explosion of social media and the computerization of every aspect of economic
activity resulted in creation of large volumes of mostly unstructured data: web logs, videos,
speech recordings, photographs, e-mails, Tweets, and similar. In a parallel development,
computers keep getting ever more powerful and storage ever cheaper. Today, we have the
ability to reliably and cheaply store huge volumes of data, efficiently analyze them, and extract
business and socially relevant information. This course introduces you to several key IT
technologies that you will be able to use to manipulate, store, and analyze big data. We will
look at the basic tools for statistical analysis, R and Python, and a few key methods used in
Machine Learning. We will review MapReduce techniques for parallel processing and Hadoop,
an open source framework that allow us to cheaply and efficiently implement MapReduce on
internet scale problems. We will spend considerable time mastering Spark, a memory based
evolution of Hadoop. We will touch on related tools that provide SQL-like access to
unstructured data like Hive. We will analyze so-called NoSQL storage solutions exemplified by
Cassandra for their critical features: speed of reads and writes, data consistency, and ability to
scale to extreme volumes. We will examine memory resident databases (VoltDB) and streaming
technologies which allow analysis of data in flight, i.e. near real time. Students will gain the
ability to design highly scalable systems that can accept, store, and analyze large volumes of
unstructured data in batch mode and/or real time.
Prerequisites: Familiarity with Intermediate Java is advised. Most assignments could easily be
done in Python, Scala, C# or Perl too. We will assume no familiarity with Linux and will
introduce you to all essential Linux commands. Students need access to a computer with a 64
bit operating system and at least 4 GB of RAM. Note: 8 GB or more of RAM is strongly advised.
Lectures: Lectures will be delivered live and simultaneously made available after lectures for
online viewing through BlackBoard Collaborate Web Conferencing tool. Streaming recording
might also be available. Links to BlackBoard Collaborate recorded lectures will be accessible on
the course Web site within a few hours after the end of the lecture. If streaming video is
provided, recorded lectures will become available with a delay of up to two days.
References: Detailed handouts with references to material on the Web will be handed out
every week. There is no required text book.
Grading: Practically every class will be followed by a homework assignment. Grades on the
solutions for class assignments constitute approximately 85% of the final grade. 15% of the
grade will be earned through the final project. Final projects will be assigned a few weeks
before the end of the class. You will produce a paper (10+ pages of MS Word text, 10+
PowerPoint Slides, a working demo, 15 minute YouTube video of your presentation and a brief
2 minute YouTube video that might be presented to the class on the day of final presentations.
Several students will be invited to present their final projects live to the entire class. Grades:
95% or higher cumulative grade on all assignments and the final project gives you an A as the
final grade in the course, 90-94.9% gives you an A-, 85-89.9% a B+, 80-84.9% a B, etc.

Communications: [email protected], also Canvas message box once the class starts.

1
Tentative List of Class Topics:
Date Topic
1 01/29/16 R a language developed by statisticians for statisticians. Whatever you do with Big
Data has something to do with statistics. Learning R might be a good idea.
2 02/05/16 MapReduce Framework and Hadoop. Embarrassingly parallel processes and other
design patterns for big data processing. Cloudera virtual machine. HDFS -
Hadoop Distributed Filesystem, YARN - Yet Another Resource Negotiator.
3 02/12/16 MapReduce 2 API . We will examine some of advanced details of Hadoop
MapReduce Java API
4 02/19/16 Spark. A memory based evolution of MapReduce framework with considerable
improvement in execution speed. Spark RDD-s
5 02/26/16 Hive is a data warehouse built atop of HDFS and Hadoop. It allows SQL queries
over data stored in HDFS.
6 03/04/16 Spark Data Frames and Spark SQL are tools (APIs) within Spark ecosystem allowing
you to manipulate data (RDD-s) in a most efficient manner.
7 03/11/16 NoSQL Databases and Cassandra are non-traditional database engines build with
some relaxed features like consistency but providing very high performance of
reads or writes.
03/18/16 No Class Spring Break
8 03/25/16 Spark Streaming, Kafka and Cassandra is becoming a standard stack for processing
of fast data
9 04/01/16 Visualizing Large Data Sets with D3. We will introduce a Java Script API and
techniques that enable more insightful use of graphs and charts to present the
content and features of large data set
10 04/08/16 Neo4J, a Graph Database. A storage and retrieval system based on hierarchical
structures which have proven themselves very efficient for fast queries among
highly correlated data.
11 04/15/16 Natural Language Processing. Basic mechanisms for processing and analysis of
written text.
12 04/22/16 Spark MLLib, Machine Learning with Spark. We will review a few algorithms that
can learn from and make predictions on data
13 04/29/16 Compute Unified Device Architecture (CUDA) is a parallel computing platform
and API model created by NVIDIA. It allows software developers to use graphics
processing units (GPU) for general purpose processing. Basic Programming
techniques.
14 05/06/16 Data Flow Computing is a new revolutionary way of performing computations,
completely different to computing with conventional CPUs or GPUs. Dataflow
computers focus on optimizing the movement of data in an application and utilize
massive parallelism between thousands of tiny dataflow cores to provide order of
magnitude benefits in performance, space and power consumption. We will learn
to perform Data Flow computations using Maxelers technology.
15 05/13/16 Presentations of selected student projects.

Tentative List of Lab Topics

2
 Date Topic
1 01/30/16 R Zoran
2 02/06/16 MapReduce Framework and Hadoop. Zoran
3 02/13/16 MapReduce 2 API Marina
4 02/20/16 Spark. Spark RDD-s Diane
5 02/27/16 Hive Rahul
6 03/05/16 Spark Data Frames and Spark SQL Olena
7 03/12/16 NoSQL Databases and Cassandra Blagoje (AWS)
03/19/16 No Class Spring Break
8 03/26/16 Spark Streaming, Kafka and Cassandra Marina
9 04/02/16 Visualizing Large Data Sets with D3. Diane
10 04/09/16 Neo4J, a Graph Database. Zoran
11 04/16/16 Natural Language Processing Blagoje
12 04/23/16 Spark MLLib Rahul
13 04/30/16 CUDA Blagoje/Zoran
14 05/07/16 Data Flow Computing Guest
15 05/13/16 Presentations