[Linear Algebra Series Part 14] Basis and Dimension: The Minimum Coordinates Needed to Describe a Space

What this post covers

This post introduces basis and dimension.

What a basis is
Why dimension means the minimum number of independent directions
How span and independence combine into one concept
Why coordinates, compression, and latent structure all connect here

Key terms

basis: a spanning set with no redundancy
dimension: the number of vectors in a basis
coordinate: the numeric description of a vector relative to a chosen basis
span and independence: the two ingredients of a basis

Core idea

In this post, “space” means a vector space, especially the finite-dimensional ones programmers see first. You can picture it as the collection of all vectors you can reach from some building blocks.

A basis is a set of vectors that satisfies two conditions at once:

it spans the whole space
it is linearly independent

So a basis is a spanning set with no redundancy.

If you remove too many vectors, you lose spanning power. If you add too many, you create dependence.

The dimension of the space is the number of vectors in any basis. A basis theorem guarantees that every basis of the same finite-dimensional space has the same size, which is why dimension is well defined.

Step-by-step examples

Example 1) The standard basis of the plane

(1, 0) and (0, 1) span all of R^2 and are independent, so they form a basis. Therefore the dimension of the plane is 2.

Example 2) A different basis for the same space

(1, 1) and (1, -1) also form a basis of R^2.

For example,

(3, 1) = 2(1, 1) + 1(1, -1)

So relative to this basis, the coordinate vector is (2, 1) instead of (3, 1).

The coordinates changed, but the space did not. More precisely, (3, 1) are the standard coordinates of the vector, while (2, 1) are the coordinates relative to the basis ((1, 1), (1, -1)).

Example 3) Effective dimension in data

A movie-preference vector may be stored in 100 coordinates, but the real variation may sit close to a much smaller subspace. In data work, people sometimes call this the effective dimension: the number of directions you need to keep the approximation error acceptably small. That is an approximate, data-driven notion, not the same as the ambient dimension of the full vector space.

Math notes

A basis must span the space and remain independent. Miss either one, and you lose unique representation.
Once a basis is fixed, every vector has a unique coordinate representation because linear independence prevents two different coefficient lists from describing the same vector.
Coordinates depend on the basis, but dimension does not.
That is why basis turns an abstract space into something we can compute with.

Common mistakes

Thinking only the standard basis counts

The standard basis is just the most familiar one. Many different bases can describe the same space.

Thinking dimension is just the number of columns in a dataset

Raw ambient dimension and effective dimension are not always the same.

Treating basis as only abstract theory

Basis is really about finding the smallest clean coordinate system that still explains the whole space.

Practice or extension

Are (1, 0), (0, 1), (1, 1) a basis of the plane?
Why are (1, 1) and (2, 2) not a basis?
Why can the same plane have many bases but only one dimension?

Wrap-up

This post introduced basis and dimension.

A basis spans a space without redundancy.
Dimension counts how many basis vectors are needed.
Coordinates depend on the chosen basis.
Next, we study important subspaces created by a matrix.