Visualizing Linear Transformation

Elan Ding

Modified: June 23, 2018

Twisting, turning of the plane

A 2x2 matrix looks very simple. For example,

$$ A=\begin{bmatrix} 2 & -1 \\ 1 & 2 \end{bmatrix} $$

And yet they can show us so much about the beauty of linear transformation.

Any linear transformation $T : \mathbb{R}^2 \to \mathbb{R}^2$ can be represented by a 2x2 matrix. For the matrix $A$ above, we can see a nice visualization using Python's plotly package.

In [1]:
import plotly.offline as pyo
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode
from IPython.display import display, HTML
import numpy as np
In [2]:
init_notebook_mode(connected=True)
In [3]:
def eigenplot(inputs):

    t=np.linspace(0, 2*np.pi, 50)
    x=np.cos(t)
    y=np.sin(t)

    mat = np.reshape(inputs, [2,2])

    new_x = mat[0,0]*x + mat[0,1]*y
    new_y = mat[1,0]*x + mat[1,1]*y

    lim = max(max(abs(x)), max(abs(new_x)), max(abs(y)), max(abs(new_y)))

    eig = np.linalg.eig(mat)

    # Grabbing the eigenvectors and normalize them
    eig1 = eig[1].transpose()[0]/np.linalg.norm(eig[1].transpose()[0])
    eig2 = eig[1].transpose()[1]/np.linalg.norm(eig[1].transpose()[1])

    new_eig1 = [mat[0,0]*eig1[0] + mat[0,1]*eig1[1],
                    mat[1,0]*eig1[0] + mat[1,1]*eig1[1]]

    new_eig2 = [mat[0,0]*eig2[0] + mat[0,1]*eig2[1],
                    mat[1,0]*eig2[0] + mat[1,1]*eig2[1]]

    data = [go.Scatter(x=x,
                       y=y,
                       mode='markers',
                       name='unit circle',
                       marker=dict(size=5,
                                   color='black')
                      ),

            go.Scatter(x=[0,eig1[0]],
                       y=[0,eig1[1]],
                       mode='lines',
                       name='first eigenvector',
                       line=dict(width=3,
                                 color='blue')
                      ),

            go.Scatter(x=[0,eig2[0]],
                       y=[0,eig2[1]],
                       name='second eigenvector',
                       mode='lines',
                       line=dict(width=3,
                                 color='red')
                      )
           ]

    frames=[dict(data=[go.Scatter(x=new_x,
                                  y=new_y,
                                  mode='markers',
                                  marker=dict(size=10,
                                              color='red')
                                 ),

                       go.Scatter(x=[0,new_eig1[0]],
                                  y=[0,new_eig1[1]],
                                  mode='lines',
                                  name='eigenvector 1',
                                  line=dict(width=3,
                                            color='blue')
                                 ),

                       go.Scatter(x=[0,new_eig2[0]],
                                  y=[0,new_eig2[1]],
                                  mode='lines',
                                  name='eigenvector 2',
                                  line=dict(width=3,
                                            color='red')
                                 )
                      ]
                ),

            dict(data=[go.Scatter(x=x,
                                  y=y,
                                  mode='markers',
                                  name='unit circle',
                                  marker=dict(size=5,
                                              color='black')
                                 ),

                       go.Scatter(x=[0,eig1[0]],
                                  y=[0,eig1[1]],
                                  mode='lines',
                                  name='eigenvector 1',
                                  line=dict(width=3,
                                            color='blue')
                                 ),

                       go.Scatter(x=[0,eig2[0]],
                                  y=[0,eig2[1]],
                                  name='eigenvector 2',
                                  mode='lines',
                                  line=dict(width=3,
                                            color='red')
                                 )
                      ]
                )
           ]

    layout = go.Layout(xaxis=dict(range=[-lim, lim],
                                  autorange=False,
                                  zeroline=False),

                       yaxis=dict(range=[-lim, lim],
                                  autorange=False,
                                  zeroline=False),

                       title='Linear transformation of plane', hovermode='closest',

                       updatemenus=[{'type': 'buttons',
                                     'buttons': [{'label': 'Transform',
                                                  'method': 'animate',
                                                  'args': [None]}
                                                ]
                                    }]
                      )

    figure = go.Figure(data=data, layout=layout, frames=frames)

    return figure
In [4]:
pyo.plot(eigenplot([2, -1, 1, -2]), filename='plot1.html')
Out[4]:
'file:///Users/yirending/Documents/Github/Jupyter-blog/plot1.html'

By clicking on the Transform button multiple times, we can clearly see the picture. The 2x2 matrix

$$ A=\begin{bmatrix} 2 & -1 \\ 1 & 2 \end{bmatrix} $$

has two linearly independent eigenvectors, which are still parallel to themselves after the transformation.

What about a projection matrix? Let's take a look at the matrix

$$ A = \begin{bmatrix} 0.5 & 0.5 \\ 0.5 & 0.5 \end{bmatrix} $$

In [5]:
pyo.plot(eigenplot([0.5, 0.5, 0.5,0.5]), filename='plot2.html')
Out[5]:
'file:///Users/yirending/Documents/Github/Jupyter-blog/plot2.html'

You see that the whole space is compressed onto a single line. Notice that even though the red vector is an eigenvector, it is not the only one! Any vector that is not parallel to the blue vector can be an eigenvector since the corresponding eigenvalue is 0. (For projection matrix, the eigenvalues are always 1 and 0.) Since we can find two independent eigenvectors, the matrix is still diagonalizable even though it is singular.

You can experiment with your own 2x2 matrix using my app here.

Diagonalization

First, we will focus on symmetric matrices, which can always be orthogonally diaganalized by the spectral theorem. Geometrically, the nice thing about an orthogonal matrix is that it preserves length. Hence all orthogonal matrices are simply rotations and reflections. Let's look at

$$ A = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} $$

You can go ahead and enter this matrix in the app to see the end result of the transformation. Next we are going to break it down into steps by using diagonalization.

First we solve the characteristic equation for the eigenvalues.

$$ \text{det}(A-\lambda I) = 0 $$

$$ (1-\lambda)^2 - 4 = 0 $$

$$ \lambda^2 -2\lambda -3 = 0 $$

$$ (\lambda-3)(\lambda+1) = 0 $$

Hence we have that the eigenvalues are $\lambda_1=-1$ and $\lambda_2 =3$.

Given a symmetric matrix $A = \begin{bmatrix} a & b \\ b & c \end{bmatrix}$ with eigenvalues $\lambda_1$ and $\lambda_2$, a shortcut for computing the corresponding eigenvectors are

$$ v_1 = \begin{bmatrix} b \\ \lambda_1-a \end{bmatrix} = \begin{bmatrix} 2 \\ -2 \end{bmatrix}, \quad v_2 = \begin{bmatrix} \lambda_2-c \\ b \end{bmatrix} = \begin{bmatrix} 2 \\ 2 \end{bmatrix} $$

Note that $v_1$ and $v_2$ are automatically orthogonal. After normalizing them, we get the eigenvectors

$$ q_1 = \begin{bmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \end{bmatrix}, \quad q_2 = \begin{bmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix} $$

The orthonormal vectors $q_1$ and $q_2$ can form the columns of $Q$, so that $A$ is diagonalized into:

$$ A = Q\Lambda Q^{\text{T}} = \begin{bmatrix} 1/\sqrt{2} & 1/\sqrt{2} \\ -1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix}\begin{bmatrix} -1 & \\ & 3 \end{bmatrix}\begin{bmatrix} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix} $$

We can verify this using numpy.linalg.svd function:

In [6]:
A = np.array([[1,2],[2,1]])
np.linalg.svd(A, full_matrices=True)
Out[6]:
(array([[-0.70710678, -0.70710678],
        [-0.70710678,  0.70710678]]),
 array([3., 1.]),
 array([[-0.70710678, -0.70710678],
        [ 0.70710678, -0.70710678]]))

Note that the singular value decomposition, which we will talk about later, is equivalent to diagonalization for a real symmetric matrix. The result above is different from our answer except for the sign and the order of the eigenvalues. This is completely justified since we can rewrite the diagonal matrix $\Lambda$ as

$$ \Lambda = \begin{bmatrix} -1 & \\ & 3 \end{bmatrix} = \begin{bmatrix} & -1\\ -1 & \end{bmatrix}\begin{bmatrix} 3 & \\ & 1 \end{bmatrix}\begin{bmatrix} & -1 \\ 1 & \end{bmatrix} $$

where, in the product, the first off diagonal matrix switches the rows and changes the signs of both rows, and the third off diagonal matrix switches the columns and only changes sign of the first column. Subsituting this into our original decomposition, we get the same as what numpy produces:

$$ A = \begin{bmatrix} -1/\sqrt{2} & -1\sqrt{2} \\ -1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix} \begin{bmatrix} 3 & \\ & 1 \end{bmatrix} \begin{bmatrix} -1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{bmatrix} $$

Note that this is far from the only choice of factorization. We can also do

$$ \Lambda = \begin{bmatrix} -1 & \\ & 3 \end{bmatrix} = \begin{bmatrix} & 1\\ 1 & \end{bmatrix}\begin{bmatrix} 3 & \\ & 1 \end{bmatrix}\begin{bmatrix} & 1 \\ -1 & \end{bmatrix} $$

From playing with 2x2 matrices, we can learn so much about eigenvalues and eigenvectors! We leanred that

  • If we change the order of the eigenvalues, the order of the columns of $Q$ changes accordingly
  • The signs of the eigenvalues and eigenvectors can be picked according to our needs (they don't have to match)

For our demonstration, I will create a visualization. You can obtain the source code on Github. The following is the visualization.

The plot breaks linear transformation of $A$ into three steps. First we see a rotation by the orthogonal matrix $Q^{\text{T}}$. Next, the matrix $\Lambda$ stretches and coordinate axis. Finally we multiply by the matrix $Q$ again for a rotation!

Singular Value Decomposition

For a symmetric matrix, we can always factor it as $Q\Lambda Q^{\text{T}}$, where $Q$ is an orthogonal matrix. The good news is that for any matrix $A$ of rank $r$, we can always find the singular value decomposition:

$$ A = U\Sigma V^{\text{T}} $$

The first $r$ columns of $U$ are eigenvectors of $AA^{\text{T}}$. They form the orthonormal basis for the row space of $A$. The first $r$ columns of $V$ are the eigenvectors of $A^{\text{T}}A$. They form the orthonormal basis for the column space of $A$. The matrix $\Sigma = \text{diag}(\sigma_1^2, \sigma_2^2,..., \sigma_r^2)$ consists of the nonzero eigenvalues of $AA^{\text{T}}$ and $A^{\text{T}}A$.

The leftover $m-r$ columns of $U$ form the orthonormal basis for the left null space of $A$, and the leftover $n-r$ columns of $V$ form the orthonormal basis for the nullspace of $A$.

Singular value decomposition involves all four subspaces, so it can deal with all situations! And the awesome thing is that $U$ and $V$ are both square matrices, so they can be viewed geometrically as rotations or reflections. The diagonal matrix $\Sigma$ is simply a stretching of the coordinate axis.

Let's find the singular value decomposition of the matrix $A$ below.

$$ A = \begin{bmatrix} 2 & 2 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} -1 & 0 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 2\sqrt{2} & \\ & \sqrt{2} \end{bmatrix}\begin{bmatrix} -1/\sqrt{2} & -1/\sqrt{2} \\ -1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix} = U\Sigma V^{\text{T}} $$

In [7]:
A = np.array([[2, 2], [-1, 1]])
np.linalg.svd(A)
Out[7]:
(array([[-1.00000000e+00,  1.23907179e-16],
        [ 8.64164897e-17,  1.00000000e+00]]),
 array([2.82842712, 1.41421356]),
 array([[-0.70710678, -0.70710678],
        [-0.70710678,  0.70710678]]))

Compared to orthogonal diagonalization, where only one matrix $Q$ is involved, here we have two orthogonal matrices $U$ and $V$. But this does not take away its geometric significance! The linear transformation can always be broken down into three steps:

  • rotation or reflection
  • stretching
  • a different rotation or reflection

Let's visualize it!

Linear algebra is beautiful!