10. Linear Systems of Equations#

10.1. Gaussian Elimination#

The technique studied here makes solving large systems of equations (e.g. 200 equations in 200 unknowns) practical, and it can be implemented on a computer in a basic programming language. You might be surprised to learn that it is necessary to solve vast systems of linear equations as a matter of routine in many practical scientific applications.

10.1.1. Motivation#

Gaussian elimination is a systematic technique for solving systems of linear equations, which are of the form

(10.1)#

\begin{aligned} a_{1, 1} x_{1} + a_{1, 2} x_{2} + a_{1, 3} x_{3} + \dots + a_{1, n} x_{n} & = b_{1} \\ a_{2, 1} x_{1} + a_{2, 2} x_{2} + a_{2, 3} x_{3} + \dots + a_{2, n} x_{n} & = b_{2} \\ ⋮ \\ a_{m, 1} x_{1} + a_{m, 2} x_{2} + a_{m, 3} x_{3} + \dots + a_{m, n} x_{n} & = b_{m} \end{aligned}

where $a_{i, j}$ are constants. Usually there are the same number of equations as unknowns, so $m = n$ . If $m < n$ then the system is undetermined, if $m > n$ , the system is over constrained.

Example

(10.2)#

\begin{aligned} 4 x_{1} - x_{2} & = 1 \\ - 2 x_{1} + 3 x_{2} & = 12 \end{aligned}

Each equation here defines a line, and we are looking for a point which satisfies both equations, which means that the lines intersect.

From the first line we obtain $x_{2} = 4 x_{1} - 1$ , and by substituting this into the second line, we obtain $x = \frac{3}{2}$ , $y = 5$ . Two equations with two unknowns will always give a unique solution, unless the lines are parallel (and so the equations are just a scaling of each other).

If they are parallel and distinct, there are no solutions because there are no points that lie on both lines.
If they are parallel and coincident (same line), there are an infinite number of solutions.

10.1.2. A systematic technique for solving systems of equations#

We will begin by finding a solution to the following system:

\begin{aligned} (r_{1}) & x_{1} & + & x_{2} & + & 2 x_{3} & = & 2 \\ (r_{2}) & 2 x_{1} & + & 3 x_{2} & - & 4 x_{3} & = & 1 \\ (r_{3}) & - x_{1} & + & x_{2} & + & x_{3} & = & 3 \end{aligned}

The equations have been labelled $r_{1}, r_{2}, r_{3}$ .

First, we will use $r_{1}$ to eliminate $x_{1}$ from $r_{2}$ and $r_{3}$ . This gives two equations in two unknowns. Then, we will use $r_{2}$ to eliminate $x_{2}$ from $r_{3}$ .

The steps are written out below:

\begin{aligned} (r_{1}) & x_{1} & + & x_{2} & + & 2 x_{3} & = & 2 \\ (r_{2}) & 2 x_{1} & + & 3 x_{2} & - & 4 x_{3} & = & 1 \\ (r_{3}) & - x_{1} & + & x_{2} & + & x_{3} & = & 3 \\ (r_{1}) & x_{1} & + & x_{2} & + & 2 x_{3} & = & 2 \\ r_{2} & - & 2 r_{1} & ⟶ & (r_{2}) & x_{2} & - & 8 x_{3} & = & - 3 \\ r_{3} & + & r_{1} & ⟶ & (r_{3}) & 2 x_{2} & + & 3 x_{3} & = & 5 \\ (r_{1}) & x_{1} & + & x_{2} & + & 2 x_{3} & = & 2 \\ (r_{2}) & x_{2} & - & 8 x_{3} & = & - 3 \\ r_{3} & - & 2 r_{2} & ⟶ & (r_{3}) & 19 x_{3} & = & 11 \end{aligned}

The solution for $x_{3}$ can now be read off from $r_{3}$ , $x_{2}$ can be obtained from $r_{2}$ using the result for $x_{3}$ and $x_{1}$ can be obtained from $r_{1}$ using the results for $x_{1}$ and $x_{2}$ . This is known as back-substitution.

These manipulations can be conveniently done by looking only at the coefficients, which we collect together in a form called the augmented matrix:

\begin{array}{r} \begin{array}{rc} \begin{matrix} (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{matrix} & (\begin{array}{cccc} 1 & 1 & 2 & 2 \\ 2 & 3 & - 4 & 1 \\ - 1 & 1 & 1 & 3 \end{array}) \\ \begin{array}{ccr} (r_{1}) \\ r_{2} - 2 r_{1} & ⟶ & (r_{2}) \\ r_{3} + r_{1} & ⟶ & (r_{3}) \end{array} & (\begin{array}{cccc} 1 & 1 & 2 & 2 \\ 0 & 1 & - 8 & - 3 \\ 0 & 2 & 3 & 5 \end{array}) \\ \begin{array}{ccr} (r_{1}) \\ (r_{2}) \\ r_{3} - 2 r_{2} & ⟶ & (r_{3}) \end{array} & (\begin{array}{cccc} 1 & 1 & 2 & 2 \\ 0 & 1 & - 8 & - 3 \\ 0 & 0 & 19 & 11 \end{array}) \end{array} \end{array}

We can see that the algorithm (described in the box below) works by eliminating the coefficients below the leading diagonal, which is highlighted.

Naive Gaussian elimination algorithm (obtaining upper triangular form)

Step 1 Choose initial pivot

We choose the first element from the leading diagonal as the pivot element.

Step 2 Row reduction step

Add multiples of the pivot row to the rows below, to obtain zeros in the pivot column below the leading diagonal.

Step 3 Choose new pivot

The pivot moves to the next element on the leading diagonal.

Repeat Repeat from Step 2 until the matrix is in upper triangular form (containing all zeros below the leading diagonal).

The solutions can then be obtained by back-substitution.

Exercise 10.1

Write the following system as an augmented matrix then solve it using the Naive Gaussian elimination algorithm.

\begin{array}{r} \begin{aligned} x & - & 2 y & + & z & = & 0 \\ 2 x & + & y & - & 3 z & = & 5 \\ 4 x & - & 7 y & + & z & = & - 1 \end{aligned} \end{array}

10.1.3. Generalisation#

The naive algorithm introduced here can be generalised to include additional row operations. In general, the acceptable row operations that we can perform are:

multiplication of any row by a constant (e.g. $\frac{1}{2} r_{1} \to r_{1}$ )
addition of (a multiple of) any row to any other (e.g. $r_{2} + 2 r_{1} \to r_{2}$ )
swapping any two rows (e.g. $r_{1} \leftrightarrow r_{2}$ )

It is often possible to apply these steps creatively to get a result with greater efficiency than using the naive algorithm described above.

It is also not necessary to stop at upper triangular form. Once the last row has been fully simplified, it can be used to obtain zeros above the main diagonal in the last column. Then, the second-last row is used to obtain zeros in the second-last column above the main diagonal, and so-on until the only non-zero elements remaining are on the main diagonal. Then the solutions can be simply read off from each row. For instance, continuing with the naive row reduction for the example shown in the previous section, we obtain:

\begin{array}{r} \begin{array}{rc} \begin{array}{ccr} (r_{1}) \\ (r_{2}) \\ \frac{1}{19} r_{3} & ⟶ & (r_{3}) \end{array} & (\begin{array}{cccc} 1 & 1 & 2 & 2 \\ 0 & 1 & - 8 & - 3 \\ 0 & 0 & 1 & \frac{11}{19} \end{array}) \\ \begin{array}{ccr} r_{1} - 2 r_{3} & ⟶ & (r_{1}) \\ r_{2} + 8 r_{3} & ⟶ & (r_{2}) \\ (r_{3}) \end{array} & (\begin{array}{cccc} 1 & 1 & 0 & \frac{16}{19} \\ 0 & 1 & 0 & \frac{31}{19} \\ 0 & 0 & 1 & \frac{11}{19} \end{array}) \\ \begin{array}{ccr} r_{1} - r_{2} & ⟶ & (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{array} & (\begin{array}{cccc} 1 & 0 & 0 & - \frac{15}{19} \\ 0 & 1 & 0 & \frac{31}{19} \\ 0 & 0 & 1 & \frac{11}{19} \end{array}) \end{array} \end{array}

We have obtained row-reduced form and the solutions for $x_{1}, x_{2}, x_{3}$ can now be read off from the final column.

10.2. Row Echelon Form#

In this section we present an algorithm for solving a linear system of equations. The system algorithm works for any linear system of equations, regardless of whether there are zero, one or infinitely many solutions. It works by converting system of equations to reduced row echelon form.

Definition

A matrix is in row echelon form if:

All zero rows are at the bottom.
The first nonzero entry of a row is to the right of the first nonzero entry of the row above.
Below the first nonzero entry of a row, all entries are zero.

(10.3)#

(\begin{matrix} ⋆ & ⋆ & ⋆ & ⋆ & ⋆ \\ 0 & ⋆ & ⋆ & ⋆ & ⋆ \\ 0 & 0 & 0 & ⋆ & ⋆ \\ 0 & 0 & 0 & 0 & 0 \end{matrix})

$⋆ =$ any number
$⋆ =$ any non-zero number.

A pivot is the first nonzero entry of a row of a matrix in row echelon form.

A matrix is in reduced row echelon form if it is in row echelon form, and:

Each pivot is equal to 1.
Each pivot is the only nonzero entry in its column.

(10.4)#

(\begin{matrix} 1 & 0 & ⋆ & 0 & ⋆ \\ 0 & 1 & ⋆ & 0 & ⋆ \\ 0 & 0 & 0 & 1 & ⋆ \\ 0 & 0 & 0 & 0 & 0 \end{matrix})

$⋆ =$ any number

Exercise 10.2

Which of the following matrices are in (a) row echelon form (b) reduced row echelon formed?

\begin{array}{r} \begin{aligned} 1. (\begin{array}{c} 1 & 0 & 2 \\ 0 & 1 & - 1 \end{array}) 2. (\begin{array}{c} 2 & 1 \\ 2 & 1 \end{array}) 3. (\begin{array}{cccc} 2 & 7 & 1 & 4 \\ 0 & 0 & 2 & 1 \\ 0 & 0 & 1 & 3 \end{array}) \\ 4. (\begin{array}{cccc} 2 & 7 & 1 & 4 \\ 0 & 0 & 2 & 1 \\ 0 & 0 & 0 & 3 \end{array}) 5. (\begin{array}{c} 0 & 1 & 8 & 0 \end{array}) 6. (\begin{array}{c} 2 & 1 & 3 \\ 0 & 0 & 0 \end{array}) \end{aligned} \end{array}

10.2.1. Gaussian Elimination for Reduction to Echelon Form#

Row Reduction Algorithm (Gaussian Elimination)

Step 1 If necessary, swap the 1st row with a lower one so a leftmost nonzero entry is in the 1st row.

Step 2 Multiply the 1st row by a nonzero number so that its first nonzero entry is equal to 1.

Step 3 Replace all lower rows with multiples of the first row so all entries below this 1 are 0.

Step 4 Repeat Steps 1-3 for row 2, row 3 and so on.

The matrix is now in row echelon form. To convert it to reduced row echelon form:

Step 5 Replace all rows above with multiples of the final pivot row to clear all entries above the pivot.

Step 6 Repeat step 5 for each of the other pivot rows, working from bottom to top.

Example: Gaussian Elimination

Use Gaussian elimination to reduce the following matrix to reduced row echelon form:

\begin{array}{r} (\begin{array}{c} - 2 & - 4 & - 2 & - 10 & 0 \\ 2 & 4 & 1 & 9 & - 2 \\ 3 & 6 & 1 & 13 & - 4 \end{array}) \end{array}

Solution

First get a 1 pivot in the first row and zeros below in the same column:

\begin{array}{r} \begin{array}{rc} \begin{array}{ccr} (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{array} & (\begin{matrix} - 2 & - 4 & - 2 & - 10 & 0 \\ 2 & 4 & 1 & 9 & - 2 \\ 3 & 6 & 1 & 13 & - 4 \end{matrix}) \\ \begin{array}{ccr} - \frac{1}{2} r_{1} & ⟶ & (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{array} & (\begin{matrix} 1 & 2 & 1 & 5 & 0 \\ 2 & 4 & 1 & 9 & - 2 \\ 3 & 6 & 1 & 13 & - 4 \end{matrix}) \\ \begin{array}{ccr} (r_{1}) \\ r_{2} - 2 r_{1} & ⟶ & (r_{2}) \\ r_{3} - 3 r_{1} & ⟶ & (r_{3}) \end{array} & (\begin{matrix} 1 & 2 & 1 & 5 & 0 \\ 0 & 0 & - 1 & - 1 & - 2 \\ 0 & 0 & - 2 & - 2 & - 4 \end{matrix}) \end{array} \end{array}

We have zeros in the second column so the next pivot will be in column 3:

\begin{array}{r} \begin{array}{rc} \begin{array}{ccr} (r_{1}) \\ - r_{2} & ⟶ & (r_{2}) \\ (r_{3}) \end{array} & (\begin{matrix} 1 & 2 & 1 & 5 & 0 \\ 0 & 0 & 1 & 1 & 2 \\ 0 & 0 & - 2 & - 2 & - 4 \end{matrix}) \\ \begin{array}{ccr} (r_{1}) \\ (r_{2}) \\ r_{3} + 2 r_{2} & ⟶ & (r_{3}) \end{array} & (\begin{matrix} 1 & 2 & 1 & 5 & 0 \\ 0 & 0 & 1 & 1 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix}) \end{array} \end{array}

The matrix is now in echelon form. To achieve reduced row echelon form, we eliminate the values above the pivots:

\begin{array}{r} \begin{array}{rc} \begin{array}{ccr} r_{1} - r_{2} & ⟶ & (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{array} & (\begin{matrix} 1 & 2 & 0 & 4 & - 2 \\ 0 & 0 & 1 & 1 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix}) \end{array} \end{array}

10.3. Parametric Solutions to a Linear System#

The reduced row echelon form of the augmented matrix allows us to determine the solutions to the system. There are three cases:

1. Every column except the last column is a pivot column. In this case, the system of equations is consistent and there is a single unique solution. For example,

\begin{array}{r} (\begin{array}{cccc} 1 & 0 & 0 & 5 \\ 0 & 1 & 0 & 2 \\ 0 & 0 & 1 & - 1 \end{array}) \end{array}

has the solution $(\begin{matrix} x_{1} \\ x_{2} \\ x_{3} \end{matrix}) = (\begin{matrix} 5 \\ 2 \\ - 1 \end{matrix})$ .

2. The last column is a pivot column. In this case the system is inconsistent and there are no solutions. For example,

\begin{array}{r} (\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array}) \end{array}

is equivalent to $(\begin{matrix} x_{1} \\ x_{2} \\ 0 \end{matrix}) = (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix})$ which is inconsistent. There are no values of $x_{1}, x_{2}$ and $x_{3}$ which solve this system.

3. The last column is not a pivot column, and some other column is also not a pivot column. In this case, there are infinitely many solutions. For example,

\begin{array}{r} (\begin{array}{ccccc} 1 & 2 & 0 & 3 & 1 \\ 0 & 0 & 1 & - 2 & - 2 \end{array}) \end{array}

has infinitely many solutions since columns 2 and 4 are non-pivot columns. The variables corresponding to a non-pivot column of the matrix are termed free variables. In the next section will explore how these solutions can be determined.

10.3.1. General Solution Set#

The following system of equations,

\begin{array}{r} \begin{aligned} x_{1} & + & 2 x_{2} & + & 9 x_{3} & = & - 1 \\ 2 x_{1} & + & x_{2} & + & 12 x_{3} & = & 1 \end{aligned} \end{array}

is represented by the following augmented matrix,

\begin{array}{r} (\begin{array}{cccc} 1 & 2 & 9 & - 1 \\ 2 & 1 & 12 & 1 \end{array}) \end{array}

which through Gaussian elimination may be reduced to the following echelon form,

\begin{array}{r} (\begin{array}{cccc} 1 & 0 & 5 & 1 \\ 0 & 1 & 2 & - 1 \end{array}) \end{array}

corresponding to the following pair of equations:

\begin{array}{r} \begin{aligned} x_{1} & + & 5 x_{3} & = & 1 \\ x_{2} & + & 2 x_{3} & = & - 1 & . \end{aligned} \end{array}

Rearranging slightly,

\begin{array}{r} \begin{aligned} x_{1} & = & 1 & - & 5 x_{3} \\ x_{2} & = & - 1 & - & 2 x_{3} & . \end{aligned} \end{array}

For any value of $x_{3}$ , there is exactly one value of $x_{1}$ and $x_{2}$ that satisfy the equations. But we are free to choose any value of $x_{3}$ , and so we have found all the solutions: the set of all values $x_{1}, x_{2}, x_{3}$ where

\begin{array}{r} \begin{aligned} x_{1} & = & 1 & - & 5 t \\ x_{2} & = & - 1 & - & 2 t \\ x_{3} & = & t \end{aligned} \end{array}

for any $t \in R$ .

This is the general solution in parametric form of the system of equations, and $x_{3}$ is a free variable.

In a later section we will see how to write this in vector form as follows:

\begin{array}{r} (\begin{array}{c} x_{1} \\ x_{2} \\ x_{3} \end{array}) = t (\begin{array}{c} - 5 \\ - 2 \\ 1 \end{array}) + (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) . \end{array}

The solution set is a straight line.

Parametric solution to system of linear equations

Suppose we have a linear system of $m$ equations in $n$ unknowns. Determine the parameteric form of the solution set as follows:

Write the system of as an augmented matrix.
Use Gaussian elimination to reduce to reduced row echelon form.
Write the corresponding system of linear equations.
Move all free variables to the right hand side.

Example

Write the solution to the system represented by the following augmented matrix in parametric form.

\begin{array}{r} (\begin{array}{ccccc} 1 & 2 & 0 & 3 & 1 \\ 0 & 0 & 1 & - 2 & - 2 \end{array}) \end{array}

Solution

The matrix is already in reduced row echelon form. The pivot variables are $x_{1}$ and $x_{3}$ ; the free variables are $x_{2}$ and $x_{4}$ . It corresponds to the following equations:

\begin{array}{r} \begin{aligned} x_{1} & + & 2 x_{2} & + & 3 x_{4} & = & 1 \\ x_{3} & - & 2 x_{4} & = & - 2. \end{aligned} \end{array}

Move the free variables to the right hand side to give the parametric solution:

\begin{array}{r} \begin{aligned} x_{1} & = & - 2 x_{2} & - & 3 x_{4} & + & 1 \\ x_{3} & = & - & 2 x_{4} & - & 2 \end{aligned} \end{array}

for any $x_{2}, x_{4} \in R$ .

This is the equation of a (2d) plane.

Exercise 10.4

The reduced row echelon form of the matrix for a linear system in four variables $x_{1}, x_{2}, x_{3}, x_{4}$ is

\begin{array}{r} (\begin{array}{ccccc} 1 & 0 & 0 & 3 & 2 \\ 0 & 0 & 1 & 4 & - 1 \end{array}) . \end{array}

Identify the pivot variables and free variables.
Write the solution to the system in parametric form.

10.4. Solutions#

Solution to Exercise 10.1

Gaussian elimination

\begin{array}{r} \begin{array}{rc} \begin{matrix} (r_{1}) \\ (r_{2}) \\ (r_{3}) \end{matrix} & (\begin{array}{cccc} 1 & - 2 & 1 & 0 \\ 2 & 1 & - 3 & 5 \\ 4 & - 7 & 1 & - 1 \end{array}) \\ \begin{array}{ccr} (r_{1}) \\ r_{2} - 2 r_{1} & ⟶ & (r_{2}) \\ r_{3} - 4 r_{1} & ⟶ & (r_{3}) \end{array} & (\begin{array}{cccc} 1 & - 2 & 1 & 0 \\ 0 & 5 & - 5 & 5 \\ 0 & 1 & - 3 & - 1 \end{array}) \\ \begin{array}{ccr} (r_{1}) \\ (r_{2}) \\ r_{3} - \frac{1}{5} r_{2} & ⟶ & (r_{3}) \end{array} & (\begin{array}{cccc} 1 & - 2 & 1 & 0 \\ 0 & 5 & - 5 & 5 \\ 0 & 0 & - 2 & - 2 \end{array}) \end{array} \end{array}

Back-substitution

\begin{aligned} - 2 z = - 2 & \Rightarrow z = 1. \\ 5 y - 5 z = 5 & \Rightarrow y = 1 + z = 2. \\ x - 2 y + z = 0 & \Rightarrow x = 2 y - z = 3. \end{aligned}

The solution is $(\begin{matrix} x \\ y \\ z \end{matrix}) = (\begin{matrix} 3 \\ 2 \\ 1 \end{matrix})$ .

Solution to Exercise 10.2

1, 4, 5, 6 are in row echelon form.

1, 5 are in reduced row echelon form.

Solution to Exercise 10.3

Row echelon form:

\begin{array}{r} (\begin{array}{c} 3 & - 9 & 12 & - 9 & 6 & 15 \\ 0 & 1 & - 2 & 2 & 1 & - 3 \\ 0 & 0 & 0 & 0 & 1 & 4 \end{array}) \end{array}

Reduced row echelon form:

\begin{array}{r} (\begin{array}{c} 1 & 0 & - 2 & 3 & 0 & - 24 \\ 0 & 1 & - 2 & 2 & 0 & - 7 \\ 0 & 0 & 0 & 0 & 1 & 4 \end{array}) \end{array}

Solution to Exercise 10.4

1. The pivots are $x_{1}, x_{3}$ ; the free variables are $x_{2}, x_{4}$ .

2. Write the system of equations:

\begin{array}{r} \begin{aligned} x_{1} & + & 3 x_{4} & = & 2 \\ x_{3} & + & 4 x_{4} & = & - 1 \end{aligned} . \end{array}

In parametric form:

\begin{array}{r} \begin{aligned} x_{1} & = & 2 & - & 3 x_{4} \\ x_{3} & = & - 1 & - & 4 x_{4} \end{aligned} . \end{array}

What happened to $x_{2}$ ? It is a free variable, but no other variable depends on it. The general solution is:

\begin{array}{r} (\begin{array}{c} x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{array}) = (\begin{array}{c} 2 - 3 x_{4} \\ x_{2} \\ - 1 - 4 x_{4} \\ x_{4} \end{array}) \end{array}

for any values of $x_{2}$ and $x_{4}$ .

Mathematics for Natural Sciences 2

Linear Systems of Equations

Contents