ITE 3 – DATA STRUCTURES AND ALGORITHMS

BIG O NOTATION

 Big O is the most commonly used mathematical tool to compare the efficiency of algorithms.
 It tells us about the asymptotic behavior of a function that is how fast a function f of n grows as n becomes
large.
 This Big O examines the rate of growth of a function by comparing it with some standard functions whose
rate of growth is known.
 Big O Notation is the language we use to describe the complexity of an algorithm.
 In other words, Big O Notation is the language we use for talking about how long an algorithm takes to
run.
 It is how we compare the efficiency of different approaches to a problem.
 With Big O Notation we express the runtime in terms of — how quickly it grows relative to the input, as
the input gets larger.

COMMON ASYMPTOTIC FUNCTIONS

A. Constant – O(1)

Constant time algorithms are (asymptotically) the quickest.

Key to understanding Big O is understanding the rates at which things can grow. The rate in question here is time
taken per input size.

Consider this simple piece of code:

Clearly, it doesn't matter what n is, above. This piece of code takes a constant amount of time to run. It's not
dependent on the size of n.

Similarly:

The above example is also constant time. Even if it takes 3 times as long to run, it doesn't depend on the size of
the input, n. We denote constant time algorithms as follows: O(1). Note that O(2), O(3) or even O(1000) would
mean the same thing.

We don't care about exactly how long it takes to run, only that it takes constant time.

B. Logarithmic – O(log n)

Logarithmic time is the next quickest. Unfortunately, they're a bit trickier to imagine.

One common example of a logarithmic time algorithm is the binary search algorithm.

What is important here is that the running time grows in proportion to the logarithm of the input (in this case, log
to the base 2):

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

C. Linear – O(n)

If we say something grows linearly, we mean that it grows directly proportional to the size of its inputs.

Think of a simple for loop:

How many times does this for loop run? n times, of course! We don't know exactly how long it will take for this
to run – and we don't worry about that.

What we do know is that the simple algorithm presented above will grow linearly with the size of its input.

We'd prefer a run time of 0.1n than (1000n + 1000), but both are still linear algorithms; they both grow directly in
proportion to the size of their inputs.

Again, if the algorithm was changed to the following:

The runtime would still be linear in the size of its input, n. We denote linear algorithms as follows: O(n).

As with the constant time algorithms, we don't care about the specifics of the runtime. O(2n+1) is the same
as O(n), as Big O Notation concerns itself with growth for input sizes.

D. n Log n – O(n log n)

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

E. Polynomial – O(np)

F. Exponential – O(kn)

G. Factorial – O(n!)

This class of algorithms has a run time proportional to the factorial of the input size.

Let’s look at a simple O(n!) algorithm:

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

In this table, we have values of some of these functions for different values of n. We can see that some functions
grow faster than the others. For example growth rate of this function n3 is much higher than growth rate of n or
n log n. So if we have 2 functions f1 and f2 where f1 is O(n) and f2 is O(n3), then we can say that rate of growth
of f2 is more than that of f1, because growth rate of this function n3 is more than that of n.

FINDING BIG O

Suppose we have this function f and we have to find order of this function that is we have to find a function g
such that f of n is big O of g of n. The rules for doing this are simple, but before using those rules we'll try to
understand the logic behind them. When value of n is 10, the value of this function f is this: f(n) = 300 + 40 + 15.
Now in this value, the contribution of this term 3n2 is 84.5%, contribution of this term 4n is 11% and contribution
of this constant 15 is 4%. In the table, we can see the contribution of these 3 different terms at different values
of n. So we can see that as the value of n grows, this term 3n2 dominates the value of the function and this
smaller term and this constant become insignificant. This is because n2 grows more rapidly than n and the
constant doesn't grow at all.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

Now in Big O analysis, we are concerned with the value of the function as n becomes very large and here we
have seen that when n becomes large, the fastest growing term in the function becomes dominant and the lower
terms and the constants become insignificant, so we can just ignore them.

So for finding big O for a function, we just have to keep the fastest growing term of the function and we can
discard all the lower terms and constants.

RULE FOR FINDING BIG O

 Keep the fastest growing term and discard the lower terms and constants.
 Ignore coefficients

For example if f is equal to 5n2, then we can ignore this coefficient 5 and say that f is order of n square, similarly
here we can ignore this 67 and say that f is order of n.

Now in the example function f = 3n2 + 4n + 15, according to the first rule we ignore the 2 terms: 4n and 15. So
we are left with this term: 3n2, in this term also we can ignore this coefficient 3 and we can say that function f(n)
is O(n2).

 If f(n) is constant, then we say that f(n) is O(1)

Now the next rule is simple, if we have a function f that is constant, then we can say that f is order of 1.

For example all the above functions are order of 1. The proof of this rule is very simple.

 Base of logarithm is not important

Now the next rule states that while computing the O notation, base of the logarithm is not important.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

For example for both the above functions, we'll just say that f is order of log n without specifying any base. Now
let us see the reason for this. We know that logarithms with different base are related by a constant factor. This
is the constant factor. Now Big O ignores constants, so that is why there is no need to associate any base with
the logarithms while specifying the order of a function. In algorithm analysis we'll mostly encounter logarithms
with base 2.

EXAMPLES:

 Example 1 is a constant function, so it is order of 1.

 Example 2, n3 is the fastest growing term so this is order of n3.
 Example 3, in this function n is the fastest growing term so this is order of n.
 Example 4, in this function n2 is the fastest growing term so this is order of n2.
 Example 5, in this function n log n is the fastest growing term so this is order of n log n. There is no need
to specify the base.
 Example 6, in this function 2 raised to the power n is the fastest growing term so this is order of 2 raised
to the power n.
 Example 7, in this function this 6 raised to the power n is the fastest growing term so this is order of 6
raised to the power n.
 Example 8, in this function this factorial n is the fastest growing term so this is order of factorial n.

BIG O ANALYSIS OF ALGORITHMS

We have seen what big O notation is, and how to find the order of a function. Now we’ll see how we can use big
O to analyze the running time of our algorithms. For this, first we have to express the running time of an algorithm
as function of input size. Suppose this function T represents the running time of our algorithm in terms of n, which
is the input size. This is also called the time complexity of the algorithm.

Now the next thing that we have to do is find big O for this function T.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

Now suppose we have 4 algorithms for solving a problem and we want to compare them. So first we will find out
this function T for all the 4 algorithms. These functions represent the running time of these algorithms in terms
of input size n. In the previous lectures we have seen that if the running time of an algorithm increases very
rapidly with the input size, then that algorithm is not efficient as it would take lot of time for large inputs. So to
compare these algorithms we actually need to compare the rates at which their running times grow. So basically
we have to compare the rate of growth of these functions. For that we have to determine the big O for these
functions. We have already seen how to do that
 In function A, this n3 is the fastest growing term so this is of order n3
 In function B, this n2 is the fastest growing term so this is order of n2
 In function C, order of n, and
 In function D, order of n2.

So based on this analysis we can say that Algorithm C is the best, because its rate of growth is the least. These
algorithms B and D are both of order of n2, so they have same asymptotic behavior.

FINDING TIME COMPLEXITY

Running time of an algorithm will be proportional to the number of primitive operations that are executed during
run time. Some examples of these primitive operations are comparisons, arithmetic operations, assignments or
input and output of primary data types. To find function T, we have to express the number of primitive operations
executed during run time, in terms of input size n.

Suppose we have this algorithm that performs some simple operations. Each of these operations will
take a certain amount of time, depending upon the computer that you are using. Suppose this input takes time
c1, this assignment takes time c2, this comparison takes time c3 and this arithmetic operation takes time c4.
These constants can be in any unit. Now to calculate the total running time of this algorithm, we have to add the
time taken by all these operations and for that we have to find the number of times each operation is executed.
This input, assignment and arithmetic operations are executed only one time and this comparison is executed n

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

times, so the total running time is this c1 + c2 + c3*n + c4. Now we have to find this function T that represents
the running time in terms of n. So the next step is to find big O for this function. Now while finding big O we can
drop the constants and ignore the coefficients, so we drop these constants c1, c2, c4 and ignore this coefficient
c3 and we can say that T is order of n. So we have seen that in big O analysis, we can ignore all the system
dependent constants, and therefore this method allows us to analyze algorithms in a way that is independent of
the computer.

So in asymptotic analysis, we are not interested in how much time an operation takes to execute on
a computer, we are interested in how many times that operation is executed. So the exact computation of
the function T is not necessary. While writing this function T we'll be concerned only about those primitive
operations whose number depends on input size n, because these are the operations whose number can
increase dramatically with increase in input size. And in asymptotic analysis we are interested in seeing what
happens when input size becomes large. So basically we have to look for operations that are performed in those
loops whose number of iterations depend on n. For example, here in this algorithm we are interested only in this
comparison operation, because for large values of n, this will account for most of the running time.

BIG O ANALYSIS OF ALGORITHMS: EXAMPLES

A.

In this algorithm, we'll ignore these assignment and output operations as they don’t depend on n. We can ignore
this loop also because it will always execute 6 times no matter what the size of input is. Even when n becomes
1 million it will be executed 6 times only. This loop will execute n - 2 times and this loop will execute n times, so
time complexity is (n-2) + n, and so this algorithm is order of n.

B.

Now let’s take another algorithm. Here this loop will execute almost n by 4 times, so this algorithm is O(n).

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

C.

Now here this loop will execute almost n by 5 times, so this algorithm is also O(n).

So we can say that the time complexity of loops in which the control variable is incremented or decremented by
a constant amount is order of n.

Now we will consider loops in which the control variable is divided or multiplied by a constant, in each iteration.

D.

Here in this loop, the control variable is divided by 2 in each iteration. If value of n is 51 then this loop will have
6 iterations. So the number of iterations of this loop is given by log base 2 n. We have seen that there is no need
to specify the base of the logarithm in big O, so we can just say that this algorithm is of order log n.

E.

Now in this loop, control variable is multiplied by 2 in each iteration, if value of n is 51 then this loop will also have
6 iterations. So number of iterations for this loop is also given by log base 2n, and so this algorithm is also of
order log n.

Now instead of this 2, if we divide or multiply this control variable by any other constant then also the complexity
will be of order log n. So if in a loop, the control variable is divided or multiplied by a constant amount, then
the time complexity of that loop is order of log n.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

Now we'll consider nested loops. When we have nested loops, we will be interested in knowing how many times
the innermost loop executes, because the iterations of innermost loop will always be more than those of outer
loops.

F.

Now here in this algorithm this outer loop executes n times, and for each iteration of outer loop the inner loop
will execute n times. So the total number of times the inner loop executes is equal to n*n. Now the time complexity
is equal to n + n2 because this assignment executes n times and this comparison executes n2 times. This n2 is
the faster growing term so we can say that this algorithm is of order n2.

G.

Now in this algorithm, this outer loop iterates n times, and for each iteration of outer loop this first inner loop
iterates log n times, and this inner loop iterates n times. So the total number of iterations of this loop is n log n
and the total number of iterations of this loop is n2. So this arithmetic operation executes n log n times and this
comparison executes n2 times, and so the time complexity is n log n + n2 which is order of n2.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

H.

Now in this algorithm the total number of iterations of this innermost loop is n3 and the total number of iterations
of this inner loop is 10 * n so this comparison executes n3 times and this assignment executes 10*n times, so the
time complexity is n3 + 10*n which is order of n3.

Now we'll see some nested loops in which iterations of inner loop depend on the index of outer loop. So in these
cases, we cannot find the total iterations of inner loop by just multiplying the iterations of inner and outer loops.

I.

Now in this algorithm there is a function call in this loop, and this function is of order log n so the time complexity
is given by n log n.

J.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

Now here in this algorithm we have an if else statement. When this function is executed, either this block will
execute or this block will execute. While calculating time complexity we generally consider the block whose order
is larger. Now here this block is of order n2 and this block is of order n, so we'll consider this if else statement as
order of n2. So time complexity is n log n + n2 which is order of n square.

Some algorithms perform the same number of operations every time they are executed, these algorithms are
said to take constant time and they are all order of 1.

K.

For example this algorithm is order of 1 because it will always execute constant number of steps, whether the
input size is one hundred or 1 million. The number of steps does not depend on input size n.

Some examples of constant time functions are a function that prints first element of an array of size n, a function
that swaps first and last elements of an array of size n, or a function that returns the size of a stack.

WORST, AVERAGE AND BEST CASE ANALYSIS

We have seen that running time of an algorithm depends on the input. But some algorithms might take different
running time for different inputs of same size. In these algorithms the running time depends on the type of input
data also.

Suppose we have an algorithm and these are the different input data sets which are all of same size and now
the running time in all these cases is different. So we can see the behavior of this algorithm depends upon the
size of input as well as type of input. When analyzing these type of algorithms we generally consider three cases:
worst case, best case, and average case.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

So we divide the time complexity of the time in these three cases: Worst case time complexity gives us a
measure of maximum running time that an algorithm will take for input size n. Best time complexity gives us a
measure of minimum running time that an algorithm will take for input size n. An average time complexity gives
us a major average running time that an algorithm will take for input size n.

 In worst case, we consider an input for which an algorithm takes the longest time. This input causes
the maximum number of operations to be executed. The example when we search for an element in an
array using the linear search. The worst case would be the element to be searched as at the last place
or is not present in the array. In that case there would be no comparisons. So worst case time complexity
of linear search is order of n.
 In best case, we consider an input for which algorithm takes shortest time. So this input causes the
minimum number of operations to be executed. In linear search, the best case will be the element to be
searched is present at the first location of the array. In this case there will be only one comparison. So
the best case time complexity of linear search is order of one.
 In an average case, because we consider all possible inputs and computed average. In this case we
have to use the probabilistic distribution for the input data. The most common assumption is that all
possible permutations of an input are equally likely.

In practice the worst case analysis is usually done to analyze algorithms because in this case we get to know
the maximum time that the algorithm can take. The actual running time can never be worse than the best case.
Best case analysis is not very useful and the best cases don't go very often. This is not very informative as
it just gives us the minimum time the algorithm can take. And the real running time can be much more than this
minimum time. For example an algorithm that takes some seconds in best case might take days to execute in
worst case or average case. This average case analysis is very rarely done because calculating time
complexity in average case is generally very difficult. So the worst case analysis is most useful and
informative and can be easily done. That's why by analyzing algorithms we generally consider the worst case.

Suppose we have an algorithm which behaves in order of n2 in worst case or order n log n in an average case
and order of one in best case. So we do see that this algorithm is order n square. To be more specific you can
explicitly state the order in these three different cases.

COMMON COMPLEXITIES

Usually, the complexity of an algorithm is a function relating the input length/size to the number of fundamental
steps (time complexity) or fundamental storage locations (space complexity).

If time complexity is O (1), then we say that algorithm's running time is constant. In this case all operations are
executed once or only a fixed number of times. Such an algorithm will take same time for any input size. For
example, the algorithm for inserting a new element in the beginning of a linked list will take constant time.

If time complexity is O (log n), the algorithm is said to be logarithmic. Algorithms that solve problems by dividing
into smaller problems have this type of complexity. Such algorithms generally reduce the number of elements to
be considered in each step. An example of this type of time complexity is binary search algorithm which is used
to search an element in a sorted array.

If the time complexity is O (n), the algorithm is said to be linear. This is the case when each element of the input
is processed in some way. Here the running time is linearly proportional to n. That is, if the size of input is doubled
then the running time will also be doubled. For example the algorithm that prints all the elements of an array, or
the algorithm that inserts a new element at the end of a linked list will both take linear time.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

If time complexity is O (n log n), the algorithm is said to be linear logarithmic. An example of this type of time
complexity is merge sort algorithm.

If time complexity is O (nsqaure), then algorithm is said to be quadratic. This occurs when an algorithm
processes all pairs of input data items. In this case if n doubles, the run time will quadruple. This complexity
generally occurs in algorithms that involve nested iterations over the input. Bubble sort and selection sort
algorithms have quadratic complexity.

If time complexity is O (n cube), then algorithm is said to be cubic. This complexity occurs when algorithm
processes all triplets of input data items. An example of this complexity is matrix multiplication.

If time complexity is O (n raised to power k) for some constant k, then algorithm is said to be polynomial, An
algorithm is called exponential if it has running time of O (a raised to the power n) where a is some number more
than 1. The runtime for these algorithms increases very rapidly, for example if we have an algorithm with
complexity 2 raised to the power n, then the running time for that algorithm will double with each additional
element in the input data. An example of exponential algorithm is tower of Hanoi problem.

Now since running times for these algorithm increases very rapidly, they can be used only for small input size
and therefore are not of much practical use.

Abstract Data Type

 Abstract data type is a mathematical model or a concept that defines a data type logically.
 The definition of an Abstract data type specifies a set of data and a set of operations that can be
performed on that data.
 The definition only specifies what operations are to be performed on the data, it does not mention
anything about how these operations will be implemented. So basically it does not specify how the data
will be organized in memory and what algorithms will be used for implementing the operations.
 The dictionary meaning of abstract is "Existing as an idea but not having a physical existence" so this
abstract data type is only a concept or a theoretical model that gives us an implementation independent
view of a data type.
 An abstract data type (ADT) is the way we look at a data structure, focusing on what it does and ignoring
how it does its job.
 It refers to a set of data values and associated operations that are specified accurately, independent of
any implementation.
 It consists of a set of definitions that allow us to use the functions while hiding the implementation.
 For example, when we use a stack or a queue, the user is concerned only with the type of data and the
operations that can be performed on it. Therefore, the fundamentals of how the data is stored should be
invisible to the user. They should not be concerned with how the methods work or what structures are
being used to store the data. They should just know that to work with stacks, they have push() and pop()
functions available to them. Using these functions, they can manipulate the data (insertion or deletion)
stored in the stack.

Now let's see specifications of some abstract data types. Figure 11 shows the definition of List abstract
data type, stack abstract data type and queue abstract data type. We can see that these definitions only
specify the operations that have to be performed, they don't mention how these abstract data types will be
represented and how the operations will be carried out. An abstract data type is just a theoretical concept
and there can be different ways to implement it. For example the list abstract data type can be implemented
using arrays, or single linked list or double linked list. Similarly this stack abstract data type or queue abstract
data type can also be implemented using either arrays or linked lists. The different implementations of an
abstract data type are compared based on time and space and the implementation that best suits the
requirement of the user is used. For example if a user wants to use a list in a program which involves lots of
insertions and deletions from the list, then it is better to use the linked list implementation of list because
insertion and deletion in arrays require lot of movement of data.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES
ITE 3 – DATA STRUCTURES AND ALGORITHMS

Figure 1. Definition of List Abstract Data Type, Stack Abstract Data Type, and
Queue Abstract Data Type

Abstract Data Types and Data Structure

Data Structure is the physical implementation of Abstract data type. Abstract Data Type is the logical
view of data and the operations to manipulate the data while Data structure is the actual representation of data
and the algorithms. Abstract Data Type is a logical description while Data Structure is concrete so an abstract
data type tells us what is to be done and data structure tells us how to do it. We can say that Abstract Data Type
is a specification language for data structures. A program that uses a data structure is generally called a client
and the specification of Abstract Data Type is called interface and this interface is the only thing that is visible
to the clients who use data structures. The clients have no knowledge of how the data is represented and how
operations are implemented. For example when we use int data type we only need to know the values that it can
take and operations that can be performed on it, we don't need to know how it is implemented. Similarly if
someone wants to use a stack, he can simply use push and pop operations without any knowledge of how they
are implemented. The representation or algorithms that implement the stack can be changed but the client code
will not be affected as long as the interface remains the same. For example if the implementation of stack is
changed from array to linked list, the client program will work in the same way and it need not be changed. So
this abstraction helps the user of a data structure focus on his program rather than going into the details of the
data structure.

Three Stages of ADT

1. Specification
 This is an identification of the data’s components, properties, relationships among its components,
and the operations performed on the data in a problem.
 It only identifies what are involved in the problem and not how to solve the problem.
2. Representation
 This is a design on how to store instances of the ADT.
 This design aims to show relationships among the components of the data in storage. The result
is called a data structure.
3. Implementation
This is a procedural description of how each operation is carried out using the data structure. Each procedural description
is an algorithm.

CRISTY S. CARPON, MSIT

COLLEGE OF COMPUTING AND INFORMATION SCIENCES