Linear algebra explained in four pages Excerpt from the No BULLSHIT GUIDE TO LINEAR ALGEBRA by Ivan Savov Abtract—This document wil review the fundamental ideas of linear algebra. ‘We will learn about matrices, matrix operations, near transformations and discuss both the theoretieal and computational aspects of linear algebra. The tools of linear algebra open the gateway to the study of more advanced mathematics A lo of knowledge Bice awaits you if you choot to follow the path of undertanding, instead of trying to memorize bunch of formulas 1. EyrRopuction Linear algebra is the math of vectors and matrices. Let n be a positive integer and let B denote the set of real numbers, then isthe set of all rituples of veal numbers. A vector #€ IR” is an ntuple of real numbers. The notation “eS” is read “element of $." For example, considera vector that has three components R F= (m0) € (RRR) A matrix A © R™ is a rectangular array of real numbers with m rows and n columns. For example, a 3 x 2 matrix looks like this: [2S] [EB The purpose of this dacument is to introduce you to the mathematical ‘operations that we can perform on vectors and matrices and to give you @ feel of the power of linear algebra. Many problems in science, business, and technology can be described in terms of vectors and matrices so itis Important that you understand how fo work with thes. Prerequisites The ony prereqitite fr this oil ea basic understanding of high school rath coneeps! like numbers, variables, equations, and the fundamental aithmetic operations on real numbers: adtion (denoted +), subtraction (denoted —), mukipication (denoted impliily), and division (Gracin). You should also be falar with functions that take real numbers 28 inputs and ive real numbers as ouputs, f : R —+ R, Recall that, by definition, the inverse function f-' undoes the effect of f. If you are given f(z) and you want to find x, you can use the inverse function as follows: J (f(2)) 2. For example, the function (2) ~ In(e) has the inverse f~!(2) — e%, and the inverse of g(x) — JZ is g(x) — 2? WL, Drnnrrios A. Vector operations ‘We now define the math operations for vectors, The operations we ean perform on vectors = (1s,¥a,s) and = (0, 2,25) ate: addition, subtraction, scaling, norm (length), dot product, and cross product: TH o= (ut sya teats +9) (41 v2 — tay — 29) (cus, aa, ens) ‘The dot product and the cross prauet of two vectors ean also be deseribed fn terms of the angle @ between the two vectors, The formula for the dot product of the vectors isi. = |isj] cos. We say two vectors if and 1 axe orthogonal ifthe angle between them is 90°. The dot product of orthogonal vectors is zex0: - = |] 20a(90°) = 0 ‘The norm of the cross product is given by |i x i] = | | sind. Tae cross product is not commutative: 2x 0-7 # x, in fat DF = FAL "A good textbook to (lean high school math is B. Matrix operations We denote by A the matix as a whole and sefer to its enties as ay The mathematical operations defined for mattices axe the following + dition (enoted +) CHAtB ey aay thy + suburaction (the inverse of addition) ‘+ matrix product. The product of matrices Ac R™*" and Bc is another matix C € R”™** given by the formula ae ABe Sant, Laas asa] P29) Laasbis + asabar aabia + asabaa) 1+ matrix inverse (denoted A~!) ‘+ mattix wanspose (denoted ") beal-B al + determinant (denoted det(A) or Al) Note that the matrix product is not a commutative operation: AB Y BA. ©. Matriv-vector product ‘The matrix-vector product is an important special case of the matrix ratsix product. The product of a 3 x 2 matrix A and the 2 x 1 column vector # results in a 3 x 1 vector given by: BRE Spay sss en fet san :] © [lar aia): 2] re J ® L(asa, a2) - 2 There are two? fundamentally different yet equivalent ways to interpret the matrix-vector product. In the column picture, (C), the multiplication of the rmateix A by the vector # produces linear combination of the columns of the matrix: y= AZ = 21Ap.1) + #2Aj,aj, where Ajai and Aja] are the first and second columns of the matrix In the row picture, (R), multiplication of the matix A by the vector 2 produces a column vector with coelficients equal to the dot products of rows of the matrix with the vector #, D. Linear transformations ‘The matrix-vector product is used to define the notion of a linear transformation. which is one of the Key notions in the study of linear algebra. Multiplication by a matrix A ¢ R*" can be thought of as computing a linear transformation T's that takes n-vectors a8 inputs and produces m-vectors as outputs: Ta sR" +R or mote info see he video of Prof. Suang's MIT lett: bit .1y/10vnKteLInstead of writing J = T'a(Z) for the linear transformation T's applied to the vector Z, we simply wnte = AZ. Applying the linear transformation Ta to the vector Z corresponds to the product of the mattix A and the column vector # We say To is represented by the matiix A. ‘You can think of linear tansformations as “vector funetions” and deseribe their properties in anslogy with the regular functions you are familiar with: function f:R OR © 7B input 2 R > ourpat f(z) & gofaaista)) & function inverse fo! €> linear transformation Tx :” input #¢ R” output T(z) To(Ta(a)) = BAZ rmatex inverse A“! tA) (A) = column space of A zeros of f null space of A range of f ange of Ts Note thatthe combined effect of applying the transformation T' followed by Tz on the input vector # i equivalent to the matix product Az’ E, Fundamental vector spaces A vector space consists of a set of weetors and all linear combinations of these vectors. For example the vector space S = span{,%} consists of all vectors of the form # = a, + 9, where av and 3 ate real numbers. ‘We now define three fundamental vector spaces associated with a mats A. ‘The columm space ofa mats Ais the set of veetors that ean be produced as linear combinations of the columns of the matrix A. (A) = (FE R™ | F= AF for some FER" ‘The cofumn space is the range of the linear transformation Ty (he set of possible outputs). You ean convince yourself ofthis fat by reviewing the definition of the matrix-vector product in the column picture (C). The vector A contains #1 times the 1" column of A, x2 times the 2* column of A, et. Varying over all posible inputs Z, we obtain all possible linear combinations of the columns of A, henee the name “eolumn space” ‘The mull space N'A) of a matrix Ae 2"*" consists ofall the vectors that the matrix sends to the zero vector: (A) = (@ER" | AZ= 5) The vectors in the null space ate orthogonal to all the rows of the matrix. We can see tis ftom the row picture (R): the ouput vectors is 6 if and only f the input vector 2 is orthogonal to all the rows of A. ‘The row space of & matrix A, denoted R(A), is the set of linear combinations of the rows of A. The row space R(A) isthe orthogonal complement of the null space (A). This means that forall vectors © RCA) and all vectors w & NA}, we have 7 a3 = 0. Together, the sll space and the row space form the domain of the ansformation T's, RY =N(A) 0 R(A), where © stands for orthogonal direct sum E Matrix inverse By definition, the inverse mateix A~* undoes the effect ofthe mateix A ‘The cumulative effect of applying AW? after A isthe identity matix | ° tas ok ‘The identity matrix (ones on the diagonal and zeros everywhere else) comesponds tothe identity wansformation: T:(Z) = 13 =, forall 7 “The matrix inverse is useful for solving matrix equations. Whenever we want to get rid of the matrix A in some matrix equation, we can “hit” A with its inverse A to make it disappear. For example, to solve for the rmateix X in the equation XA = B, multiply both sides of the equation by A” from the right: X = BA, To solve for X in ABCXD multiply both side of the equation by D~' on the right and by A~ and C™? fin that order) from the left: X= "1B! AED IML, CoMPUTATIONAL. LINEAR ALGEBRA (Okay, hear what you ae saying “Due, enough with the theory tll, let's see some calculations” In tis section we'll look at one ofthe fundamental algorithms of linear algebra called GaussJordan elimination. A. Solving systems of equations Suppose were asked to solve the following system of equations: tay +28 =5, Se +922 =2 ° Without s knowledge of linear algebra, we could use substitution, elimina tio, subwaction to find the values ofthe two unknowns #3 and = CGasse-lordan elimination isa systematic procedure for solving systems of equations based the following row operations: 19) Adding a multiple of one row to another row 8) Swapping two rows 7) Moltplying a row by a constant ‘These row operations allow us to simplify the system of equations without changing thei solution “oillustate the Gause-Jordan elimination procedure, we'll now show the sequence of row operations required to solve the system of linea equations described above, We sat by conseuting a augmented marisa follows: 12/5 [3 5 [a] ‘The fist columa inthe augmented matex coresponds tothe coelcials of the variable, the second column corresponds to the coeticients of 2, and the third column contains the constants from the right-hand side ‘The Gauss-Jordan elimination procedure consis of wo pases, During the first phase, we proced lef-o-ight by choosing a row with a leading one inthe leftmost calm (called pvr) and systematically subirating that row from allows below it o gt rezoe below in the entre column. Ia the second phase, we start with the rightmost pivot and use it to eliminate all te numbers above iin the samme column, Let's te this in ation 1) ‘The frst sep is o wie the pivot in the fist column to eliminate the viable 1 in the second row. We do this by subtracting thee times the fis tow fom the second tow, denoted Ra — Ra 3%, 12/5] 036 2) Next, we create pivot in the second row using Re Re 12]s oil2 3) We now start the backward phase and eliminate the second variable from the first row. We do this by subtracting two times the second row from the frst row Ri: © Ry 2Re aot oie ‘The matrix is now in reduced row echelon form (RREE), which is its “simplest” form it could be ia, The solutions ate: 21 = 1, 22 = 2. B. Systems of equations as matrix equations We will now discuss another approach for solving the system of equations. Using the definition of the matrix-vector product, we can express this system of equations (1) as a matrix equation: 2) fn] _ [5] 39 [22 ~ [2 “This matrix equation had the form AZ is the vector of unknowns, and is a vector of constants. We ean solve for 2 by muliplying both sides of the equation by the matsix inverse A warnie=[E]-aw- [8 fF where A is 82% 2 matrix, 2 But how did we know what the inverse matrixIV, ComPUTING THE INVERSE OF A MATRIX. In thie section we'll look at several differen approaches for computing the inverse of a matsix, The matiix inverse is unigue so no matter which method we use to find the inverse, we'll always obtain the same answer, A. Using row operations ‘One approach for computing the inverse is to use the Gauss—Jordan climination proceduse, Slart by creating an array comlaining the enties of the matix A on the left side and the identity matix on the right sie: 12}io sofoa [Now we perform the Gauss-lordan elimination procedure on thie array. 1) The first row operation isto subtract thee times the frst row from the second row: Ry = Ro ~ $R. We obtain: r2]io o3faa 2) The second row operation is divide the second row by 8: Ra — 12]1 0 on 4 3) The third row operation is Ry — Ry — 2Rs [ 10/3 3 | o1f-a ‘The array is now in seduced row echelon form (RREF), The inverse mattix appears on the right side ofthe ary. ‘Observe that the sequence of row operations we used to solve the specific system of equations in A = Bin the previous section are the same as the row operations we used inthis scetion to find the inverse matrix. Indeed, sn both eases the combined effect of the thee row operations isto “undo” the effects of A. The right side of the 2 x 4 array is simply a convenient ‘way to record this sequence of operations and thus obtain AW? B Using elementary matrices Every row operation we perform on a matrix is equivalent to a left multiplication by an elementary matrix. There are three types of elementary rmateies in correspondence withthe three types of row operations Behe mimt o B[") remem 6 eff 3] RyRy mR Let's revisit the row operations we used to find AW? jn the above section representing each row operation as an elementary matsix multiplication 1) The first row operation Rs — Ry —3Ry corresponds to a miltipi- cation by the elementary matsix Ey [3 Jb 3-b 2) The second row operation Ry $y corresponds to am EMG a-b 4 3) The final step, Ry + Ry ~2Re, corresponds to the mattix Ey ab YE so the product By. must be equal @ A! oe eee TM AL = L can be decomposed asthe product of elementary matrices. Since we know A= BBE then A= (A ES ESE," BA FlELA) Ex(EaBiA) Note that EE: EA ©. Using a computer ‘The last (and most practical) approach for finding the inverse of a max is to use 2 computer algebra system like the one at 1ive.sympy-Org. poo A= Matedet (1123030311 } # ezine A a, 21 Be 8 bo> Ain) 4 calle the Sav mechod on A (3, 2/3) [261 ‘You can use sypy (0 “check” your answers on homework problems, V. Orne Topics Well now discuss a number of other important topics of linear algebra A. Basis Inuitively, a basis is any set of vectors that can be used as a coordinate system for a vector space. You ae certainly famuliar with the standard basis for the y-plane that is made up of two orthogonal axes: the x-axis and the yeanis. A vector «can be described as a coordinate pair (0,1) With respect to these axes, or equivalently as = wyi+,j, where t= (1,0) (0,2) ace unit vectors that point along the z-axis and y-axis respectively. However, other coordinate systems aze also possible Definition (Bass). A basis for a n-dimensional vector space S is any set of m linearly independent vectors that are part of S. ‘Agy set of wo linearly independent vectors (1, a) ean seve as a basis for R?. We can wite any vector 7 € RE asa linear combination ofthese basi vetors = 116s + ¥26a Note the same vector eatesponds to diferent coordinate pais depend ing on the basis used: (vz, u) inte standard basis B, = {3,3}, and ust) inthe basis By = {é), 2} Therefore, itis important to keep in mind the basis with respect to which the cocticints are taken, and if accesaryspeily the bass as asubscpt ©, (es), ef (01,02), Converting a coordinate vector from the basis By to the bass By is performed as 4 mulipiation by a change of Basie max: rele] Ce ee EL coordinate system. The change of asi ftom the B,-basis to the Be-bass is accomplished using the inverse max: .||n, ~ (ps [L)me) 2 A, Matrix representations of linear transformations ‘Bases play an important role inthe representation of linear transforma tions T: R” —+ R". To fully describe the matrix tat corresponds to some linear tansformation 7, itis suficiens to know the effects of T to the n veetots ofthe standard basis forthe input space, For a linear transformation TT: R? +R, the matrix representation corresponds to 1 se [ro | ews rot ‘Asa first example, consider the transformation Tl, which projec veto, conto the a-axis, For any vector ~ (0,1), We have IIs (1) — (0) “The mateixsepresealaion of Ie is ve-[m(E7]) (GD ]-[5 3] ‘As a second example, le’s find the matrix representation of Ry, the counterclockwise rotation by the angle 0: weet 8) (ED) et a vector 129 = (cos, sin)", The second column shows that Ry maps the vector j= 125 to the vector LZ(E +0) = (—sind, cos)" eral. Dimension and bases for vector spaces ‘The dimension of a vector space is defined as the number of vectors in a basis for that veetor space. Consider the following veetor space 'S = span{(1,0,0), (0, 1,0), (1, 1,0)}. Seeing that the space is desribed by three vectors, we might think thet S is Stimensional. This is not the case, however, since the three vectors are not linearly independent so they don't form a bass for S. Two vectors are sufiient to describe any vector in S; we ean write S = span{(,0,0),(0,1,0)}, and we see these «wo vectors are linearly independent so they form a basis and dima(S) There is a general procedure for finding a basis for a vector space Suppose you ate given a description of a vector space in terms of m vectors Y= span{i,%,---, thy) and you are asked to find a basis for V and the dimension of V. To find a basis for V. you must find a sot of linearly independent vectors that span V. We ean use the Gauss-Tordan elimination procedure to accomplish this task, Write the vectors 7 a6 the rows of @ ‘matrix M. The veetor space V corresponds tothe row space of the matrix [M. Next, use row operations to find the reduced row echelon form (RRI of the matrix M. Since row operations do not change the row space of the smattx, the row space of reduced row echelon form of the matrix M is the same as the row space of the original set of vectors, The nonzero rows in the RREF of the matrix form a basis for vector space Vand the numbers fof nonzero rows is the dimension of V. D. Row space, columns space, and rank of « mari Recall she fundamental vector spaces for mattices that we defined in Section ILE: the column space (A), the null space AV(A), and the row space R(A). A standard linear algebra exam question 1s to give you a certain matrix A and ask you to find the dimension and a basis for each ofits fundamental spaces In the previous section we described a procedure based on Gauss-Jordan climination which canbe used “dixill” «set of lineazy independent vectors whieh form a basis forthe row space 72(A), We will now illstate this procedure with an example, and also show how to wse the RREF of the satrix A to find bases for C(A) and (4). ‘Consider the following mattix and its reduced row echelon form 1333 130 0 A=[2 6 7 6| meay=fo 0 1 a 39 9 10 e001 ‘The reduced row echelon form of the matix A contains thre pivots. The locations of the pivots will play an important role in the following steps ‘The vectors {(1,3,0,0), (0,0, 1,0), (0,0, 0,1)} form a basis for (A). ‘To find a basis for the column space C(A) of the matrix A we need w0 find which of the columns of A ae linearly independent, We can do this by identifying the columns which contain the leading ones in reef(A). The comesponding columns in the original matrix form a basis for the column space of A, Looking at mef(A) we see the first third and fourth columns of the matix are linearly independent 0 the vectors {(2,2.3)", (8,7, 9)", (,6,10)") form a bass for (4) Now let's find a basis forthe null space, V(A) = (2 € R! | Az = 0} The second column does not contain a pivot, therefore it corresponds to a ‘ree variable, which we will denote», Weare logking fora vector with three unknowns and one fice variable (21, 3,5, 2)" that obeys the conditions oo 1 of[sf= lol > ln = 0 ooo af} [ol in = 0 Let's express the unknowns 21, 7s, and 24 in terms of the fre variable s ‘We immediately see that 25 = 0 and #4 = 0, and we can wiite x1 = —8s. ‘Therefore, any vector of the form (—3s, 0,0), for any # € R, isin the null space of A, We waite (A) — span{ (3, 1,0,0)"). ‘Observe that the diim(C(A)) = dira(R(A)) = 5, this is known as the rank of the matrix A. Also, dima(R(A)) + dim('(A)) = 3-41 = 4, ‘hich isthe dimension of the input space ofthe linear transformation Ty E, Invertible matrix theorem ‘There isan important distinction between matrices that ae invertible and those that are not as formalized by the following theorem. Theorem. For an n xn matrix A, the fellowing statements are equivalent: 1) A ts imertite 2) The RREF of A isthe 1 identity matrix 3) The rank of the matrix ism 4) The row space of Ais 53) The column space of A is 8" ©) A doesn't havea nll space (only the zero vector N(A) = (8) 1) The determinant of Ais nonzero det(A) #0 For a given matrix A, the above statements are either all true ofall false, ‘An invertible mattix A corresponds toa linear tansformation T', whieh ruaps the n-dimensional input vector space tothe n-dimensional outpst vector space B” such that there exists an inverse transformation T’,* that can faithfully undo the effects of Ts ‘On the other hand, an n x n matrix B that is not invertible maps the input vector space BR" to a subspace C(B) © R and has a nonempty null space. Once Ty sends a vector wi € A(B) to the zero vector, there i no Ty" that ean undo this operation, E. Determinants ‘The determinant of a matrix, denoted det(A) of |A|, isa special way to combine the entries ofa matrix that serves to check if @ matrix is invertible for not, The determinant formulas for 2 x 2 and 3 x 3 matrices are If the [Al = 0 then A is not invertible. IF [Al # 0 then A is invertible G. Eigenvalues and eigenvectors ‘The set of eigenvectors of a matrix isa special set of input vectors for Which the action of the mattix is described as a simple sealing, When matrix is multiplied by one of its eigenvectors the output is the same eigenvector multiplied by a constant Az, = Ac. The constant is ealled an eigenvalue of A “To find the eigenvalues of a matrix we star fom the eigenvalue equation Aes = és, inset the identity £, and rewnite it as a mull-space problem: Aby=ME = (AALS This equation will havea solution whenever |A—X{| = 0. The eigenvalues of AC RY", denoted (3, Aa, polynomial p{3) = |A~ All. The eigenvectors astociated with the eigenvalue 2 are the vectors inthe null space of the matiix (A — 244) Certain matrices can be written entirely in terms of their eigenvectors and their eigenvalues. Consider the matrix A that has the eigenvalues of the matrix A on the diagonal, and the matrix Q constructed from the eigenvectors of A as columas: Dra} ae the 10085 of ‘characteristic os o | o} @-fé, af ten A= eng 0 0 dn | Maitices that can be written this way are called diagonaliable. ‘The decomposition of a matrix into its eigenvalues and eigenvectors ives valuable insights into the properties of the matsix. Google's orginal Pagesank algorithm for ranking webpages by “importance” can be formalized as an eigenvector ealeulation on the matvix of web hyperlinks VI. Texraoox Pius Tr youre interested in Tearing more about linear algebra, you can check ‘out my new book, the No BULLSHIT GUIDE TO LINEAR ALGEBRA. A pre-release version ofthe book is available here: gum .co/noBSLA