View source for Applied linear operators and spectral methods/Lecture 1

Linear operators can be thought of as infinite dimensional matrices.  Hence we
can use well known results from matrix theory when dealing with linear operators.
However, we have to be careful.  A finite dimensional matrix has an inverse if
none of its eigenvalues are zero.  For an infinite dimensional matrix, even 
though all the eigenvectors may be nonzero, we might have a sequence of 
eigenvalues that tend to zero.  There are several other subtleties that we will
discuss in the course of this series of lectures.

Let us start off with the basics, i.e., linear vector spaces.

==Linear Vector Spaces (''S'')==
Let <math>\mathcal{S}</math> be a linear vector space.  

===Addition and scalar multiplication===
Let us first define addition and scalar
multiplication in this space.  The addition operation acts completely in <math>\mathcal{S}</math> 
while the scalar multiplication operation may involve multiplication either
by a real (in <math>\mathbb{R}</math>) or by a complex number (in <math>\mathbb{C}</math>).  These
operations must have the following closure properties:

#If <math>\mathbf{x}, \mathbf{y} \in \mathcal{S}</math> then <math>\mathbf{x} + \mathbf{y} \in \mathcal{S}</math>.
#If <math>\alpha \in \mathbb{R}</math> (or <math>\mathbb{C}</math>) and <math>\mathbf{x} \in \mathcal{S}</math> then <math>\alpha~\mathbf{x} \in \mathcal{S}</math>.

And the following laws must hold for addition

#<math>\mathbf{x} + \mathbf{y}</math> = <math>\mathbf{y} + \mathbf{x} \qquad</math> Commutative law.
#<math>\mathbf{x} + (\mathbf{y} + \mathbf{z})</math> = <math>(\mathbf{x} + \mathbf{y}) + \mathbf{z} \qquad</math> Associative law.
#<math>\exists \mathbf{0} \in \mathcal{S}</math> such that <math>\mathbf{0} + \mathbf{x} = \mathbf{x} \quad \forall \mathbf{x} \in \mathcal{S} \qquad </math> Additive identity.
#<math>\forall \mathbf{x} \in \mathcal{S} \quad \exists -\mathbf{x} \in \mathcal{S}</math> such that <math>-\mathbf{x} + \mathbf{x} = \mathbf{0} \qquad </math> Additive inverse.

For scalar multiplication we have the properties

#<math>\alpha~(\beta~\mathbf{x}) = (\alpha~\beta)~\mathbf{x}</math>.
#<math>(\alpha + \beta)~\mathbf{x} = \alpha~\mathbf{x} + \beta~\mathbf{x}</math>.
#<math>\alpha~(\mathbf{x}+\mathbf{y}) = \alpha~\mathbf{x} + \alpha~\mathbf{y}</math>.
#<math>\mathbf{1}~\mathbf{x} = \mathbf{x}</math>.
#<math>\mathbf{0}~\mathbf{x} = \mathbf{0}</math>.

====Example 1: ''n'' tuples====
The <math>n</math> tuples <math>(x_1, x_2, \dots, x_n)</math> with
:<math>
  \begin{align}
  (x_1, x_2, \dots, x_n) + (y_1, y_2, \dots, y_n)  & = 
  (x_1 + y_1, x_2 + y_2, \dots, x_n + y_n) \\
  \alpha~(x_1, x_2, \dots, x_n)  & = 
  (\alpha~x_1, \alpha~x_2, \dots, \alpha~x_n) 
  \end{align}
</math>
form a linear vector space.

====Example 2: Matrices====
Another example of a linear vector space is the set of <math>2 \times 2</math> matrices 
with addition as usual and scalar multiplication, or more generally <math>n\times m</math>
matrices.
:<math>
  \alpha\begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix} = 
  \begin{bmatrix} \alpha~x_{11} & \alpha~x_{12} \\ \alpha~x_{21} & \alpha~x_{22} 
  \end{bmatrix}
</math>

====Example 3: Polynomials====
The space of <math>n</math>-th order polynomials forms a linear vector space.
:<math>
  p_n = \sum_{j=1}^n \alpha_j~x^j
</math>

====Example 4: Continuous functions====
The space of continuous functions, say in <math>[0, 1]</math>, also forms a linear vector 
space with addition and scalar multiplication defined as usual.

===Linear Dependence===
A set of vectors <math>\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_n \in \mathcal{S}</math> are said to be linearly
dependent if <math>\exists~ \alpha_1, \alpha_2, \dots, \alpha_n</math> not all zero such that
:<math>
  \alpha~\mathbf{x}_1 + \alpha~\mathbf{x}_2 + \dots + \alpha~\mathbf{x}_n = \mathbf{0}
</math>
If  such a set of constants <math>\alpha_1, \alpha_2, \dots, \alpha_n</math> does not exist 
then the vectors are said to be linearly independent.

====Example====
Consider the matrices
:<math>
  \boldsymbol{M}_1 = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}, 
  \boldsymbol{M}_2 = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}, 
  \boldsymbol{M}_3 = \begin{bmatrix} 0 & 0 \\ 0 & -1 \end{bmatrix} 
</math>
These are linearly dependent since <math>\boldsymbol{M}_1 - \boldsymbol{M}_2 + 2~\boldsymbol{M}_3 = \mathbf{0}</math>.

===Span===
The span of a set of vectors <math>(\boldsymbol{T})</math> is the set of all vectors that are
linear combinations of the vectors <math>\mathbf{x}_i</math>.  Thus
:<math>
  \text{span}(\boldsymbol{T}) = \{\boldsymbol{T}_1, \boldsymbol{T}_2, \dots, \boldsymbol{T}_n\} 
</math>
where
:<math>
  \boldsymbol{T}_i = \alpha_1~\mathbf{x}_1 + \alpha_2~\mathbf{x}_2 + \dots + \alpha_n~\mathbf{x}_n
</math>
as <math>\alpha_1, \alpha_2, \dots, \alpha_n</math> vary.

====Spanning set====
If the span = <math>\mathcal{S}</math> then <math>\boldsymbol{T}</math> is said to be a spanning set.

===Basis===
If <math>\boldsymbol{T}</math> is a spanning set and its elements are linearly independent then we call 
it a basis for <math>\mathcal{S}</math>.  A vector in <math>\mathcal{S}</math> has a unique representation as a 
linear combination of the basis elements.

This is because adding two vectors is the same as adding the coefficients of the basis elements, and if any vector had two distinct representations, the basis would no longer be linearly independent, as <math display="block">\vec{v} = a_1 \vec e_1 + a_2 \vec e_2 + \dots = b_1 \vec e_1 + b_2 \vec{e}_2 + \dots</math>implies that<math display="block">\vec v - \vec v = (a_1-b_1)\vec e_1 + (a_2-b_2)\vec e_2 + \dots = \vec 0</math>which is only permissible (by linear independence) if the <math display="inline">a_i - b_i</math> are all zero.

====Dimension====
The dimension of a space <math>\mathcal{S}</math> is the number of elements in the basis.  This is 
independent of actual elements that form the basis and is a property of <math>\mathcal{S}</math>.

====Example 1: Vectors in ''R<sup>2</sup>''====
Any two non-collinear vectors <math>\mathbb{R}^2</math> is a basis for <math>\mathbb{R}^2</math> 
because any other vector in <math>\mathbb{R}^2</math> can be expressed as a linear 
combination of the two vectors.

====Example 2: Matrices====
A basis for the linear space of <math>2 \times 2</math> matrices is
:<math>
  \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}, 
  \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}, 
  \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}, 
  \begin{bmatrix} 1 & 3 \\ 1 & 1 \end{bmatrix} 
</math>
Note that there is a lot of nonuniqueness in the choice of bases.  One 
important skill that you should develop is to choose the right basis to solve
a particular problem.

====Example 3: Polynomials====
The set <math>\{1, x, x^2, \dots, x^n\}</math> is a basis for polynomials of degree <math>n</math>.
  
====Example 4: The natural basis====
A natural basis is the set <math>\{ \mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_n\}</math> where the <math>j</math>th 
entry of <math>\mathbf{e}_k</math> is
:<math>
  \delta_{jk} = \begin{cases} 1 & \mbox{for} ~j = k \\
      0 & \mbox{for}~ j \ne k \end{cases}
</math>
The quantity <math>\delta_{jk}</math> is also called the Kronecker delta.

===Inner Product Spaces===
To give more structure to the idea of a vector space we need concepts such as
magnitude and angle.  The inner product provides that structure.

The inner product generalizes the concept of an angle and is defined as a 
function
:<math>
  \langle\bullet,~\bullet\rangle : \mathcal{S}\times\mathcal{S} \rightarrow \mathbb{R}
  \quad (\text{or}~\mathbb{C}~\text{for a complex vector space})
</math>
with the properties

#<math>\langle\mathbf{x},~\mathbf{y}\rangle = \overline{\langle\mathbf{y},~\mathbf{x}\rangle} \qquad</math> overbar indicates complex conjugation.
#<math>\langle\alpha~\mathbf{x},~\mathbf{y}\rangle = \alpha~\langle\mathbf{x},~\mathbf{y}\rangle \quad</math> Linear with respect to scalar multiplication.
#<math>\langle\mathbf{x}+\mathbf{y},~\mathbf{z}\rangle = \langle\mathbf{x},~\mathbf{z}\rangle + \langle\mathbf{y},~\mathbf{z}\rangle \quad</math> Linearity with respect to addition.
#<math>\langle\mathbf{x},~\mathbf{x}\rangle > \mathbf{0}</math> if <math>\mathbf{x} \ne 0</math> and <math>\langle\mathbf{x},~\mathbf{x}\rangle = \mathbf{0}</math> if and only if <math>\mathbf{x} = \mathbf{0}</math>.

A vector space with an inner product is called an inner product space.

====Example 1:====
:<math>
  \langle\mathbf{x},~\beta~\mathbf{y}\rangle = \overline{\langle\beta~\mathbf{y},~\mathbf{x}\rangle} 
    = \overline{\beta}~\overline{\langle\mathbf{y},~\mathbf{x}\rangle} 
    = \overline{\beta}\langle\mathbf{x},~\mathbf{y}\rangle
</math>

====Example 2: Discrete vectors====
In <math>\mathbb{R}^n</math> with <math>\mathbf{x} = \{x_1, x_2, \dots, x_n \}</math> and 
<math>\mathbf{y} = \{y_1, y_2, \dots, y_n \}</math> the Eulidean norm is given by
:<math>
  \langle\mathbf{x},~\mathbf{y}\rangle = \sum_n x_n~y_n 
</math>
With <math>\mathbf{x}, \mathbf{y} \in \mathbb{C}^n</math> the standard norm is
:<math>
  \langle\mathbf{x},~\mathbf{y}\rangle = \sum_k x_k~\overline{y_k}
</math>

====Example 3: Continuous functions====
For two complex valued continuous functions <math>f(x)</math> and <math>g(x)</math> in <math>[0, 1]</math>
we could approximately represent them by their function values at 
'' equally spaced points.''

Approximate <math>f(x)</math> and <math>g(x)</math> by
:<math>
  \begin{align}
  F & = \{f(x_1), f(x_2), \dots, f(x_n)\} \qquad \text{with} ~x_k = \cfrac{k}{n}\\
  G & = \{g(x_1), g(x_2), \dots, g(x_n)\} \qquad \text{with} ~x_k = \cfrac{k}{n}
  \end{align}
</math>
With that approximation, a natural norm is
:<math>
  \langle F,~G\rangle = \cfrac{1}{n}~\sum_{k=1}^n f(x_k)~\overline{g(x_k)}
</math>
Taking the limit as <math>n \rightarrow \infty</math> (show this)
:<math>
  \langle f,~g\rangle = \int_0^1 f(x)~\overline{g(x)}~dx 
</math>

If we took '' non-equally spaced'' yet smoothly distributed points we would get
:<math>
  \langle f,~g\rangle = \int_0^1 f(x)~\overline{g(x)}~w(x)~dx 
</math>
where <math>w(x) > 0</math> is a smooth weighting function (show this).

There are many other inner products possible.  For functions that are not only
continuous but also differentiable, a useful norm is
:<math>
  \langle f,~g\rangle = \int_0^1 \left[f(x)~\overline{g(x)} + 
      f^{'}(x)~\overline{g^{'}(x)}\right]~dx
</math>

We will continue further explorations into linear vector spaces in the next 
lecture.

[[Category:Functional analysis]]
{{lecture}}