> For the complete documentation index, see [llms.txt](https://ayushs-organization-15.gitbook.io/kee_pingupwithml/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ayushs-organization-15.gitbook.io/kee_pingupwithml/foundations-of-math-for-machine-learning/learning-algebra-essentials.md). # Learning Algebra Essentials This chapter introduces the core concepts of vectors, matrices, and tensors, which are crucial for understanding data structures and transformations in machine learning. Key topics like eigenvalues, eigenvectors, and Singular Value Decomposition (SVD) are covered, as they form the basis for dimensionality reduction and feature extraction techniques used in ML algorithms. *Connection*: Essential for grasping the inner workings of algorithms like Principal Component Analysis (PCA), Singular Value Decomposition (SVD), Support Vector Machines (SVMs), and deep learning networks. ## Why This Chapter Is Required This chapter lays the foundational language of machine learning: **vectors and vector spaces**. In machine learning, data is not abstract—it *lives in high-dimensional space*, often represented as vectors in $$\Reals^n$$. Models interpret these vectors, apply transformations using matrices, and optimize functions over them to make predictions. Whether you're working with tabular data, images, or embeddings from text, you're manipulating points and directions in vector spaces. Understanding vectors isn’t just academic—it’s *practical geometry applied to data*. Feature vectors, weight parameters, gradients, projections, and distances—every core idea in ML depends on the structure and behavior of vectors. This chapter is essential because it introduces you to this *language of geometry* that machine learning speaks fluently. Without a firm grasp of vectors and vector spaces, you're flying blind through the mathematics that powers every model. ## Objectives of this Chapter By the end of this chapter, you will be able to: * Understand the *definition of vectors* as elements of $$\mathbb{R}^n$$, and distinguish between *row* and *column* vectors. * Develop a *geometric intuition* for vectors as points or arrows in space. * Perform fundamental operations like *vector addition* and *scalar multiplication*, and interpret their geometric significance. * Construct and interpret *linear combinations* of vectors. * Understand the formal definition of a *vector space*, along with its *axioms* and their practical meaning in ML contexts. * Explore concepts like *subspaces*, *span*, and *closure under operations*, which form the basis of understanding dimensionality and linear structure. * Relate mathematical concepts to real-world ML constructs such as *feature vectors*, *word embeddings*, and *latent spaces*. * Build a solid conceptual bridge between abstract linear algebra and its *applications in machine learning models and algorithms*. These objectives ensure that you not only learn the algebra but also develop the insight necessary to *engineer, interpret, and innovate* in machine learning workflows. ## Introduction: Why Linear Algebra for Machine Learning? Machine Learning is not just a statistical framework—*it is geometry applied to data*. Every data point in machine learning is encoded as a *vector*, which is either a point in space or a direction from the origin. Collections of data form matrices, and every operation—from feature transformations to model predictions—is a manipulation of these geometric objects. In this view, *linear algebra becomes the native language* of machine learning. Data, whether it’s an image, a document, or a biological sequence, is typically represented in the *vector space* $$\mathbb{R}^n$$, where each feature corresponds to a dimension. For instance, a grayscale image of size 28×28 pixels is represented as a vector in $$\mathbb{R}^{784}$$, and a one-hot encoded word or embedding might exist in $$\mathbb{R}^{300}$$ or $$\mathbb{R}^{1024}$$. Thus, *data lives in high-dimensional real spaces*, and understanding how these spaces behave is critical to how we train, interpret, and optimize models. Machine learning models operate by *optimizing functions over these vector spaces*. Parameters such as *weights* in neural networks or *coefficients* in linear models are also vectors, and *gradients*—used in optimization techniques like gradient descent—are directions in $$\mathbb{R}^n$$ that guide how the model learns. At every iteration, we’re computing derivatives, updating vectors, and transforming matrices. Linear algebra provides the tools for these operations. Several core ML techniques are built directly on linear algebra foundations. *Principal Component Analysis (PCA)* identifies the most informative directions in the data by computing *eigenvectors of the covariance matrix*, revealing the axes of maximum variance. *Singular Value Decomposition (SVD)*, a generalization of eigendecomposition, is at the heart of *data compression*, *latent semantic analysis*, and *noise filtering*. In *deep learning*, every layer of a neural network is a *matrix multiplication followed by a non-linear transformation*, and backpropagation involves computing and chaining gradients across these matrices. To truly understand, design, and debug machine learning systems, it is essential to grasp the linear algebra that governs them. This chapter sets the stage by equipping you with the geometric and algebraic intuition behind vectors and vector spaces—providing the mathematical lens through which machine learning becomes not only powerful but interpretable. ## Vectors and Vector Spaces This section introduces the core algebraic and geometric properties of vectors—the foundational building blocks of all ML data representations—and extends the concept to vector spaces. We aim to deeply understand the structure of $$\mathbb{R}^n$$, how data behaves inside it, and why machine learning thrives on this structure. A **vector** is an ordered list of real numbers that can be visualized as a point or a direction in space. Formally, an n-dimensional vector is an element of the vector space: $$\mathbf{v} \in \mathbb{R}^n = { (v\_1, v\_2, \dots, v\_n) \mid v\_i \in \mathbb{R} }$$ There are two main notations used: * **Column Vector**: $$\mathbf{v} = \begin{bmatrix} v\_1 \ v\_2 \ \vdots \ v\_n \end{bmatrix}$$ #### Row vector: $$\mathbf{v} = \[v\_1, v\_2, \dots, v\_n]$$ While mathematical formulations often prefer column vectors for consistency in linear transformations, many ML frameworks (like [Numpy](https://numpy.org/)) commonly use row vectors in dataframes or input arrays. The orientation is implementation-dependent, but mathematically, they represent the same object: an element in $$\mathbb{R}^n$$. **Example 1:** 2D Vector as a Point Let’s take a simple vector in $$\mathbb{R}^n$$: $$\mathbf{v} = \begin{bmatrix} 3 \ 4 \end{bmatrix}$$ This can be plotted as the point *(3, 4)* in a 2D Cartesian plane. Visually, it is a directed arrow from the origin *(0, 0)* to the point *(3, 4)*.
> *In machine learning, such vectors can represent simple data points — for example, a sample with two features like height and weight.* **Example 2:** 3D Vector in Numpy ```python import numpy as np v = np.array([1, 2, 3]) # Row vector by default in Numpy print("Shape:", v.shape) # Output: (3,) ``` > In NumPy, a 1-dimensional array representing a vector will have a shape of `(n,)`. To obtain a 2-dimensional row vector with shape `(1, n)` or a column vector with shape `(n, 1)`, you need to explicitly define it as a 2D array. To covert it to a column vector, use `.reshape(-1, 1)` : ```python v_col = v.reshape(-1, 1) print(v_col) ``` This yields: ```python [[1] [2] [3]] ``` > Most ML libraries use row vectors when handling datasets, where each row corresponds to a sample and each column corresponds to a feature. ### Types of Vectors In the context of *Euclidean space* ℝⁿ and its generalizations, vectors can be categorized based on their properties, orientation, or role within mathematical operations. Understanding these distinctions helps in interpreting data structures and transformations in machine learning models. **1. Zero Vector** A *zero vector* has all its components equal to zero:\ *v = \[0, 0, ..., 0]* ∈ ℝⁿ.\ It represents the origin in space and is the additive identity in any vector space, meaning *v + 0 = v*. > *In ML, the zero vector can represent missing features or a neutral initialization point.* **2. Unit Vector** A *unit vector* has a *magnitude (or norm)* of 1. It is typically used to represent *direction* without considering magnitude.\ For example, *u = \[1/√2, 1/√2]* ∈ ℝ² is a unit vector in 2D space. > ℓ₂-Norm: ‖**u**‖₂ = √(∑ᵢ uᵢ²) = 1 > *In ML, unit vectors are used in cosine similarity, where we care about direction (angle) more than length.* **3. Standard Basis Vectors** The *standard basis vectors* in ℝⁿ are vectors with a 1 in one coordinate and 0 elsewhere.\ For example, in ℝ³: * *e₁ = \[1, 0, 0]* * *e₂ = \[0, 1, 0]* * *e₃ = \[0, 0, 1]* > *These vectors form the basis for coordinate representation, crucial in constructing feature spaces in ML.* **4. Row vs. Column Vectors** * *Row vector*: 1 × n matrix: *v = \[v₁, v₂, ..., vₙ]* * *Column vector*: n × 1 matrix: ``` v = [v₁ v₂ ⋮ vₙ] ``` > *Linear transformations assume column vectors, while data libraries like NumPy/Pandas often store row vectors (samples as rows).* **5. Position Vector** A *position vector* points from the origin to a particular point in space. For point P with coordinates (x, y, z), the position vector is **OP** = \[x, y, z]. > *Useful in computer graphics and in ML embeddings where vector represents position in latent space.* **6. Displacement Vector** A *displacement vector* captures the change from point A to point B:\ $$\overrightarrow{AB}$$ = \[x₂ − x₁, y₂ − y₁, z₂ − z₁]\ It emphasizes *direction and relative change*. > *Used in optimization algorithms like gradient descent to describe update steps.* **7. Random Vectors** A *random vector* is a vector whose components are random variables. For example:\ $$\overrightarrow{X}$$ *= \[X₁, X₂, ..., Xₙ]*\ Each *Xᵢ* is a scalar random variable. > *Essential in probabilistic machine learning and Bayesian models where inputs/outputs are distributions.* ### Vector Addition and Scalar Multiplication This section introduces two fundamental operations on vectors: **vector addition** and **scalar multiplication**. These are not just algebraic rules; they are *geometrically meaningful transformations* and *building blocks of linear models and feature interactions*. Here’s what to cover step-by-step: #### **Vector Addition** **Definition:**\ Given two vectors **u** = \[u₁, u₂, ..., uₙ] and **v** = \[v₁, v₂, ..., vₙ], their sum is: $$\mathbf{u}+\mathbf{v}=\[u\_1+v\_1, \mathbf{u\_2}+v\_2, \dots, u\_n+v\_n]$$ **Geometric Interpretation:** * Think of vector addition as placing **v** starting from the head (tip) of **u**, forming a parallelogram. * The resulting vector points from the origin to the opposite corner of the parallelogram. **Python Visualization:** ```python import numpy as np import matplotlib.pyplot as plt # Define vectors u = np.array([2, 1]) v = np.array([1, 3]) sum_vec = u + v # Plot plt.figure(figsize=(6, 6)) origin = [0, 0] plt.quiver(*origin, *u, color='r', angles='xy', scale_units='xy', scale=1, label='u') plt.quiver(*u, *v, color='b', angles='xy', scale_units='xy', scale=1, label='v (from u)') plt.quiver(*origin, *sum_vec, color='g', angles='xy', scale_units='xy', scale=1, label='u + v') plt.xlim(0, 5) plt.ylim(0, 5) plt.grid() plt.legend() plt.title("Vector Addition: u + v") plt.show() ``` **ML Tie-In:** * **Feature augmentation:** Adding two feature vectors can combine different sources of input data. * **Gradient updates:** The next point in optimization is `w_new = w_old + Δw`. #### Scalar Multiplication **Definition:**\ Given a scalar $$\alpha \in \mathbb{R}$$ and a vector **v** = \[v₁, ..., vₙ]: $$\alpha \mathbf{v} = \[\alpha v\_1, \alpha v\_2, \dots, \alpha v\_n]$$ **Geometric Interpretation:** * *Scales* the magnitude of the vector. * *Preserves* direction if α > 0, *reverses* direction if α < 0. **Python Visualization:** ```python alpha = 2 scaled_v = alpha * v plt.figure(figsize=(6, 6)) plt.quiver(*origin, *v, color='b', angles='xy', scale_units='xy', scale=1, label='v') plt.quiver(*origin, *scaled_v, color='m', angles='xy', scale_units='xy', scale=1, label='2v') plt.grid() plt.xlim(0, 5) plt.ylim(0, 10) plt.legend() plt.title("Scalar Multiplication: 2 × v") plt.show() ``` **ML Tie-In:** * **Weighting features:** Adjusting feature importance via scaling. * **Regularization penalties** in loss functions scale weights to control model complexity. Before we move onto the next section here are important algebraic properties of the vectors that you should remeber: > Algebraic Properties of $$\mathbb{R}^n$$ > > For all vectors $$\mathbf{u}, \mathbf{v}, \mathbf{w}$$ in $$\mathbb{R}^n$$ and all scalars $$c$$ and $$d$$: > > (i) $$\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}$$ (Commutativity of addition)\ > (ii) $$(\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})$$ (Associativity of addition)\ > (iii) $$\mathbf{u} + \mathbf{0} = \mathbf{0} + \mathbf{u} = \mathbf{u}$$ (Existence of a zero vector)\ > (iv) $$\mathbf{u} + (-\mathbf{u}) = (-\mathbf{u}) + \mathbf{u} = \mathbf{0}$$, where $$-\mathbf{u}$$ denotes the additive inverse of $$\mathbf{u}$$ (Existence of additive inverses)\ > (v) $$c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}$$ (Distributivity of scalar multiplication over vector addition)\ > (vi) $$(c + d)\mathbf{u} = c\mathbf{u} + d\mathbf{u}$$ (Distributivity of scalar multiplication over scalar addition)\ > (vii) $$c(d\mathbf{u}) = (cd)\mathbf{u}$$ (Associativity of scalar multiplication)\ > (viii) $$1\mathbf{u} = \mathbf{u}$$ (Multiplicative identity) > > These properties are the axioms that define a vector space, ensuring consistency in vector arithmetic. ### **Linear Combination of Vectors** The concept of a *linear combination* lies at the heart of linear algebra and is fundamental in understanding how vectors form spaces. A **linear combination** involves creating new vectors by *scaling* existing vectors and *adding* them together. #### Definition Given vectors $$\vec{v}\_1, \vec{v}\_2, \dots, \vec{v}\_n \in \mathbb{R}^n$$ and scalars $$a\_1, a\_2, \dots, a\_n \in \mathbb{R}$$, a **linear combination** is any vector of the form: $$\vec{w} = a\_1 \vec{v}\_1 + a\_2 \vec{v}\_2 + \cdots + a\_n \vec{v}\_n$$ The coefficients $$a\_i$$ are scalars. The vectors $$\vec{v}\_i$$ form the basis or generating set. The result $$\vec{w}$$ is a new vector in the same vector space. **Intuition** Imagine you're navigating a 2D plane with two arrows $$\vec{v}\_1=\[1,0]$$ and $$\vec{v}\_2=\[0,1]$$. A linear combination like: $$2\vec{v}\_1 + 3\vec{v}\_2 = 2\[1,0] + 3\[0,1] = \[2,3]$$ means you move 2 units in the x-direction and 3 units in the y-direction. You can visualize this as moving along the two arrows scaled appropriately — reaching a new point/vector. > **Geometric Interpretation** > > * **2 Vectors in 2D**: If they’re not scalar multiples, their linear combinations span the entire 2D plane. > * **3 Vectors in 3D**: If all 3 are linearly independent, their combinations can describe any point in 3D space. > * **Collinear vectors**: Only span a line. > * **Coplanar but not collinear**: Span a plane. ```python import numpy as np import matplotlib.pyplot as plt v1 = np.array([1, 0]) v2 = np.array([0, 1]) a, b = 2, 3 w = a * v1 + b * v2 plt.quiver(0, 0, v1[0], v1[1], angles='xy', scale_units='xy', scale=1, color='r', label='v1') plt.quiver(0, 0, v2[0], v2[1], angles='xy', scale_units='xy', scale=1, color='g', label='v2') plt.quiver(0, 0, w[0], w[1], angles='xy', scale_units='xy', scale=1, color='b', label='w = 2v1 + 3v2') plt.xlim(-1, 4) plt.ylim(-1, 4) plt.grid() plt.axhline(0, color='black') plt.axvline(0, color='black') plt.legend() plt.title('Linear Combination of Vectors') plt.show() ``` This code visually shows how two vectors combine linearly to form a new vector, making the abstract idea of *linear combination* tangible and memorable. **Example 1: Is a vector a linear combination of others?** Let’s check if the vector $$\mathbf{v} = \begin{bmatrix} 3 \ 4 \end{bmatrix}$$ can be expressed as a linear combination of $$\mathbf{a} = \begin{bmatrix} 1 \ 0 \end{bmatrix}, \quad \mathbf{b} = \begin{bmatrix} 0 \ 2 \end{bmatrix}$$ **Solution:** We want to find scalars $\alpha, \beta$ such that: $$\alpha \mathbf{a} + \beta \mathbf{b} = \mathbf{v} \Rightarrow \alpha \begin{bmatrix} 1 \ 0 \end{bmatrix} + \beta \begin{bmatrix} 0 \ 2 \end{bmatrix} = \begin{bmatrix} 3 \ 4 \end{bmatrix}$$ Breaking it down: * First component: $\alpha = 3$ * Second component: $2\beta = 4 \Rightarrow \beta = 2$ So, $$\mathbf{v} = 3\mathbf{a} + 2\mathbf{b}$$ is a valid linear combination. **Python Visualization:** ```python import numpy as np import matplotlib.pyplot as plt # Define vectors a = np.array([1, 0]) b = np.array([0, 2]) # Calculate the linear combination v = 3 * a + 2 * b # Set up the plot plt.figure(figsize=(7, 7)) plt.axhline(0, color='grey', lw=0.5) plt.axvline(0, color='grey', lw=0.5) # Plot vectors as arrows from the origin plt.quiver(0, 0, a[0], a[1], angles='xy', scale_units='xy', scale=1, color='r', width=0.006, label='a') plt.quiver(0, 0, b[0], b[1], angles='xy', scale_units='xy', scale=1, color='g', width=0.006, label='b') # Plot the resulting linear combination vector plt.quiver(0, 0, v[0], v[1], angles='xy', scale_units='xy', scale=1, color='b', width=0.008, label='v = 3a + 2b') # Add intermediate vectors for visualization plt.quiver(0, 0, (3*a)[0], (3*a)[1], angles='xy', scale_units='xy', scale=1, color='r', linestyle=':', width=0.003) plt.quiver((3*a)[0], (3*a)[1], (2*b)[0], (2*b)[1], angles='xy', scale_units='xy', scale=1, color='g', linestyle=':', width=0.003) # Set plot limits and labels plt.xlim(-1, 5) plt.ylim(-1, 5) plt.grid(True, linestyle='--', alpha=0.6) plt.gca().set_aspect('equal') # Ensure equal scaling on both axes plt.legend() plt.title("Linear Combination of Vectors a and b to form v") plt.xlabel("X-axis") plt.ylabel("Y-axis") plt.show() ``` **Example 2: Linear combination in** $$\mathbb{R}^3$$ Given: $$\mathbf{u}\_1 = \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix}, \quad \mathbf{u}\_2 = \begin{bmatrix} -1 \ 2 \ 1 \end{bmatrix}$$ Can the vector $$\mathbf{w} = \begin{bmatrix} 2 \ 1 \ 1 \end{bmatrix}$$ be written as a linear combination of $$$ $$\mathbfu\_1$$ $$$ $$\mathbf{u\_1}$$ and $$\mathbf{u\_2}$$? Set up: We want: $$\alpha \mathbf{u}\_1 + \beta \mathbf{u}\_2 = \mathbf{w} \Rightarrow \alpha \begin{bmatrix} 1 \ 1 \ 0 \end{bmatrix} + \beta \begin{bmatrix} -1 \ 2 \ 1 \end{bmatrix} = \begin{bmatrix} 2 \ 1 \ 1 \end{bmatrix}$$ System of equations: 1. $$\alpha - \beta = 2$$ 2. $$\alpha + 2\beta = 1$$ 3. $$\beta = 1 \quad \text{(from 3rd row directly)}$$ Substitute $$\beta$$=1 into the first equation: $$\alpha - 1 = 2 \Rightarrow \alpha = 3$$ So, $$\mathbf{w} = 3\mathbf{u}\_1 + 1\mathbf{u}\_2$$ **Example 3: Machine Learning Tie-In: Word Embeddings** In NLP, word vectors such as those in Word2Vec can be combined linearly to explore semantic relationships. Suppose: $$\text{vec}(\text{"king"}) = \mathbf{k}$$ $$\text{vec}(\text{"man"}) = \mathbf{m}$$ $$\text{vec}(\text{"woman"}) = \mathbf{w}$$ We often explore: $$\text{vec}(\text{"queen"}) \approx \mathbf{k} - \mathbf{m} + \mathbf{w}$$ This is a linear combination capturing a semantic relationship via vector arithmetic in high-dimensional space (usually $$mathbbR^{300}$$). ### Vector Spaces: Axioms and ML Interpretations A **vector space** (or linear space) is a fundamental structure in linear algebra consisting of a set of vectors along with two operations—vector addition and scalar multiplication—that satisfy a collection of rules known as axioms. Formally, a vector space over a field $$\mathbb{R}$$ (or $$\mathbb{C}$$) is a set $$V$$ such that for all $$u, v, w \in V$$ and scalars $$\alpha, \beta \in \mathbb{R}$$, the following axioms hold: * **Closure under addition:** $$u + v \in V$$ * **Commutativity:** $$u + v = v + u$$ * **Associativity of addition:** $$(u + v) + w = u + (v + w)$$ * **Additive identity:** There exists a zero vector $$0 \in V$$ such that $$v + 0 = v$$ * **Additive inverse:** For every $$v \in V$$, there exists $$-v \in V$$ such that $$v + (-v) = 0$$ * **Closure under scalar multiplication:** $$\alpha v \in V$$ * **Distributivity over vector addition:** $$\alpha(u + v) = \alpha u + \alpha v$$ * **Distributivity over scalar addition:** $$(\alpha + \beta)v = \alpha v + \beta v$$ * **Associativity of scalar multiplication:** $$\alpha(\beta v) = (\alpha\beta)v$$ * **Multiplicative identity:** $$1v = v$$ **Example 1: Closure under Addition and Scalar Multiplication** Let $$\mathbf{v}\_1 = \begin{bmatrix} 1 \ 2 \end{bmatrix}, \quad \mathbf{v}\_2 = \begin{bmatrix} 3 \ 4 \end{bmatrix}, \quad \text{and scalar } \alpha = 2.$$ Check if their sum and scalar multiple are in $$\mathbb{R}^2$$. **Vector Addition:** $$\mathbf{v}\_1 + \mathbf{v}\_2 = \begin{bmatrix} 1 + 3 \ 2 + 4 \end{bmatrix} = \begin{bmatrix} 4 \ 6 \end{bmatrix} \in \mathbb{R}^2$$ **Scalar Multiplication:** $$\alpha \cdot \mathbf{v}\_1 = 2 \cdot \begin{bmatrix} 1 \ 2 \end{bmatrix} = \begin{bmatrix} 2 \ 4 \end{bmatrix} \in \mathbb{R}^2$$ Hence, $$\mathbb{R}^2$$ is closed under both operations. This property is critical in ML for operations like linear regression updates, where weights remain valid within the same space after every iteration. **Example 2: Additive Identity and Inverse** Let $$\mathbf{v} = \begin{bmatrix} -5 \ 3 \end{bmatrix}$$. **Zero Vector (Additive Identity):** $$\mathbf{v} + \mathbf{0} = \begin{bmatrix} -5 \ 3 \end{bmatrix} + \begin{bmatrix} 0 \ 0 \end{bmatrix} = \begin{bmatrix} -5 \ 3 \end{bmatrix}$$ **Additive Inverse:** $$\mathbf{v} + (-\mathbf{v}) = \begin{bmatrix} -5 \ 3 \end{bmatrix} + \begin{bmatrix} 5 \ -3 \end{bmatrix} = \begin{bmatrix} 0 \ 0 \end{bmatrix}$$ In Neural Networks, this shows that vector cancellation is meaningful—e.g., subtracting gradients or balancing forces in optimization flows. > These axioms define a stable structure where linear operations behave consistently. This is essential when designing ML models that rely on combinations, projections, or transformations of feature vectors. In the context of *Machine Learning*, vector spaces provide the theoretical foundation for understanding *features*, *embeddings*, and *parameter optimization*. For instance, when we represent a data point as a vector of real numbers, we're essentially placing it in the vector space ℝⁿ. Each transformation—whether it’s dimensionality reduction (like PCA), gradient updates in optimization, or hidden representations in neural networks—operates within or across vector spaces. The properties like *linearity* and *closure* ensure that learned representations remain valid elements of their feature space during transformations. Thus, understanding vector space axioms is not merely theoretical—it aligns with how models learn, generalize, and interpolate in high-dimensional environments. **Example 3: Feature Space** Suppose you're building a logistic regression model with 3 features: $$\mathbf{x}\_1 = \begin{bmatrix} 0.2 \ 0.7 \ 0.9 \end{bmatrix}, \quad \mathbf{x}\_2 = \begin{bmatrix} 0.5 \ 0.3 \ 0.4 \end{bmatrix}$$ The feature space is $\mathbb{R}^3$. Any linear combination: $$\mathbf{x}\_3 = a \cdot \mathbf{x}\_1 + b \cdot \mathbf{x}\_2$$ remains in $$\mathbb{R}^3$$, satisfying the vector space axioms. That means weights and feature vectors can be combined arbitrarily (as long as the model uses linear operations), and the result will still be a valid input vector—preserving consistency in training and inference. ```python import numpy as np # Define vectors v1 = np.array([1, 2]) v2 = np.array([3, 4]) alpha = 2 # Closure under addition print("v1 + v2 =", v1 + v2) # Closure under scalar multiplication print("alpha * v1 =", alpha * v1) # Additive identity and inverse zero_vector = np.zeros_like(v1) print("v1 + 0 =", v1 + zero_vector) print("v1 + (-v1) =", v1 + (-v1)) ``` Output: ```python v1 + v2 = [4 6] alpha * v1 = [2 4] v1 + 0 = [1 2] v1 + (-v1) = [0 0] ``` ### Subspaces, Span, and Closure Under Operations A *subspace* is a subset of a vector space that is also a vector space under the same operations. In machine learning, subspaces often arise naturally—for example, in the space spanned by learned feature representations or in dimensionality reduction techniques such as PCA. **Definition** A subset $$W \subseteq V$$, where $$V$$ is a vector space, is called a **subspace** if: * **Zero Vector Inclusion:** $$\vec{0} \in W$$ * **Closure under Addition:** If $$\vec{u}, \vec{v} \in W$$, then $$\vec{u} + \vec{v} \in W$$ * **Closure under Scalar Multiplication:** If $$\vec{u} \in W$$, $$c \in \mathbb{R}$$, then $$c\vec{u} \in W$$ This means every linear combination of vectors in $$W$$ is also in $$W$$, preserving the structure of the vector space. **Span** The **span** of a set of vectors $${\vec{v}\_1, \vec{v}\_2, \dots, \vec{v}\_k}$$ in $$\mathbb{R}^n$$ is the set of all linear combinations of those vectors: $$\text{Span}(\vec{v}\_1, \dots, \vec{v}\_k) = \left{ c\_1\vec{v}\_1 + \cdots + c\_k\vec{v}\_k \mid c\_i \in \mathbb{R} \right}$$ The span is itself a subspace. If the vectors are linearly independent, they form a basis for the subspace. Let’s define two vectors and show their span graphically in 2D using Python. ```python import numpy as np import matplotlib.pyplot as plt # Define two vectors v1 = np.array([1, 2]) v2 = np.array([2, 1]) # Generate linear combinations coeffs = np.linspace(-10, 10, 20) span_vectors = np.array([a * v1 + b * v2 for a in coeffs for b in coeffs]) # Plot plt.figure(figsize=(8, 8)) plt.quiver(0, 0, v1[0], v1[1], angles='xy', scale_units='xy', scale=1, color='r', label='v1') plt.quiver(0, 0, v2[0], v2[1], angles='xy', scale_units='xy', scale=1, color='b', label='v2') plt.scatter(span_vectors[:, 0], span_vectors[:, 1], alpha=0.4, color='gray', s=10) plt.xlim(-20, 20) plt.ylim(-20, 20) plt.grid() plt.legend() plt.title("Span of v1 and v2 in ℝ²") plt.gca().set_aspect('equal') plt.show() ``` This visualization shows how combining $$\vec{v}\_1$$ and $$\vec{v}\_2$$ in various proportions fills up the 2D plane—i.e., their span is $$\mathbb{R}^2$$. **Example 1: Subspace Test in** $$\mathbb{R}^3$$ Let’s consider the subset $$W = \left{ \begin{bmatrix} x \ y \ z \end{bmatrix} \in \mathbb{R}^3 \mid x + y + z = 0 \right}$$ We test if this is a subspace of $\mathbb{R}^3$. **Check Axioms:** **Zero Vector:** $$\mathbf{0} = \begin{bmatrix} 0 \ 0 \ 0 \end{bmatrix} \Rightarrow 0 + 0 + 0 = 0 \Rightarrow \mathbf{0} \in W$$ **Closure under Addition:** Let $$\mathbf{u} = \begin{bmatrix} 1 \ 2 \ -3 \end{bmatrix}, \quad \mathbf{v} = \begin{bmatrix} 2 \ -1 \ -1 \end{bmatrix}$$ Then $$\mathbf{u} + \mathbf{v} = \begin{bmatrix} 1+2 \ 2-1 \ -3-1 \end{bmatrix} = \begin{bmatrix} 3 \ 1 \ -4 \end{bmatrix}$$ Check sum for $$\mathbf{u} + \mathbf{v}$$: $$3 + 1 - 4 = 0 \Rightarrow \mathbf{u} + \mathbf{v} \in W$$. **Closure under Scalar Multiplication:** Let $$\alpha \in \mathbb{R}$$. $$\alpha \cdot \mathbf{u} = \alpha \cdot \begin{bmatrix} 1 \ 2 \ -3 \end{bmatrix} = \begin{bmatrix} \alpha \ 2\alpha \ -3\alpha \end{bmatrix}$$ Check the sum of components: $$\alpha + 2\alpha - 3\alpha = 0$$. Thus, $$\alpha \cdot \mathbf{u} \in W$$. Therefore, $$W$$ is a subspace of $$\mathbb{R}^3$$. **Example 2: Span and Subspace Construction** Let $$\mathbf{v}\_1 = \begin{bmatrix} 1 \ 0 \ 1 \end{bmatrix}, \quad \mathbf{v}\_2 = \begin{bmatrix} 0 \ 1 \ 1 \end{bmatrix}$$ Find the subspace spanned by $${ \mathbf{v}\_1, \mathbf{v}\_2 }$$, and test if $$\mathbf{u} = \begin{bmatrix} 2 \ 3 \ 5 \end{bmatrix}$$ belongs to the span. **Solution:** We want scalars $$a, b$$ such that: $$a \cdot \mathbf{v}\_1 + b \cdot \mathbf{v}\_2 = \mathbf{u} \Rightarrow a \cdot \begin{bmatrix} 1 \ 0 \ 1 \end{bmatrix} + b \cdot \begin{bmatrix} 0 \ 1 \ 1 \end{bmatrix} = \begin{bmatrix} 2 \ 3 \ 5 \end{bmatrix}$$ Break into equations: * $$a = 2$$ * $$b = 3$$ * $$a + b = 5 \Rightarrow 2 + 3 = 5$$ (This equation is consistent with the values found for $$a$$ and $$b$$) Hence, $$\mathbf{u} \in \text{span}({\mathbf{v}\_1, \mathbf{v}\_2})$$. **Python Visualization: Vectors in Span** (ℝ²) ```python import numpy as np import matplotlib.pyplot as plt v1 = np.array([2, 1]) v2 = np.array([1, 3]) # Grid of coefficients a_vals = np.linspace(-10, 10, 20) b_vals = np.linspace(-10, 10, 20) # Compute combinations points = np.array([a * v1 + b * v2 for a in a_vals for b in b_vals]) # Plot plt.figure(figsize=(8, 8)) plt.scatter(points[:, 0], points[:, 1], alpha=0.4, color='gray', label='Span(v1, v2)') plt.quiver(0, 0, v1[0], v1[1], angles='xy', scale_units='xy', scale=1, color='red', label='v1') plt.quiver(0, 0, v2[0], v2[1], angles='xy', scale_units='xy', scale=1, color='blue', label='v2') plt.axhline(0, color='black', lw=0.5) plt.axvline(0, color='black', lw=0.5) plt.grid() plt.legend() plt.title("Span of Two Vectors in ℝ²") plt.axis("equal") plt.show() ``` Understanding *subspaces*, their *span*, and *closure under addition and scalar multiplication* is critical for reasoning about learned representations, optimization paths, and transformations in machine learning. These concepts form the algebraic bedrock for PCA, projections, dimensionality reduction, and even neural network transformations. ### The Relevance Room Understanding *vectors* and *vector spaces* is not just a mathematical exercise—it’s the **bedrock of how modern machine learning models think and operate**. In machine learning, every data point is typically represented as a *feature vector* in $$\mathbb{R}^n$$, where each coordinate encodes a specific attribute or measurement. This transformation of raw data into numerical vector form is crucial because **most machine learning algorithms operate in vector spaces**: they perform linear transformations, compute distances, define decision boundaries, or optimize loss functions—all over these structured vector representations. A particularly profound application lies in *embeddings*—vector representations of complex entities like words, images, molecules, or users. For instance, *word embeddings* such as Word2Vec or GloVe map discrete linguistic tokens into continuous vector spaces, where **semantic relationships are preserved via vector geometry**: *"king" - "man" + "woman" ≈ "queen"* is one famous example of this property. Similarly, in computer vision, a deep network transforms an image into a high-dimensional embedding vector whose structure captures shape, texture, or semantic content. Moreover, *subspaces* play a key role in dimensionality reduction techniques like **PCA**, where data is projected onto the most significant subspace capturing the variance. Concepts such as the *span* and *closure* are foundational to understanding how models generalize from limited data—because every hypothesis or output space must be **closed under operations** defined by the model’s architecture. By mastering vector algebra and the structure of vector spaces, you’re essentially learning how machines see and manipulate the world. These geometric interpretations fuel innovations in representation learning, manifold learning, and neural network optimization. Hence, this foundational chapter serves as your **gateway to understanding the geometric backbone of machine learning algorithms**. ### Excercises 1. Given the following 3 points in $$\mathbb{R}^2$$:\ $$A=(2,3), B=(-1,5), C=(4,-2)$$, write their column vector representations. Then compute the vector $$\vec{AB}$$ and interpret its geometric meaning. 2. Let $$\vec{u} = \begin{bmatrix} 2 \ -3 \ 1 \end{bmatrix}$$ and $$\vec{v} = \begin{bmatrix} -1 \ 4 \ 2 \end{bmatrix}$$. Compute $$2\vec{u} + 3\vec{v}$$ and represent it geometrically (if possible). Discuss how this resembles weighted feature combination in ML. 3. Let $$\vec{a} = \begin{bmatrix} 1 \ 2 \end{bmatrix}, \vec{b} = \begin{bmatrix} 3 \ 5 \end{bmatrix}$$, and $$\vec{c} = \begin{bmatrix} 9 \ 16 \end{bmatrix}$$. Check if $$\vec{c}$$ can be expressed as a linear combination of $$\vec{a}$$ and $$\vec{b}$$. If yes, find the coefficients. 4. Given vectors $$\vec{u} = \begin{bmatrix} 1 \ 1 \end{bmatrix}$$ and $$\vec{v} = \begin{bmatrix} -1 \ 2 \end{bmatrix}$$, describe the span of $${ \vec{u}, \vec{v} }$$. Does the span form a subspace of $$\mathbb{R}^2$$? Justify using subspace axioms. 5. Consider the set of vectors in $$\mathbb{R}^3$$ where the last component is always zero, i.e.,\ $$V = { (x, y, 0) \mid x, y \in \mathbb{R} }$$. True or False: $$V$$ is a vector space. Prove or disprove using vector space axioms. 6. A one-hot encoded vector of size 4 is used to represent categorical variables in a dataset. Explain how these one-hot vectors lie in a subspace of $$\mathbb{R}^4$$. Can the space be spanned by fewer vectors? 7. For a neural network input layer, input is often represented as a row vector of shape $$(1 \times n)$$. Explain the reason for this convention in matrix multiplication. Convert an example feature vector $$\vec{x} = \[5, 1, 3]$$ into both row and column formats and perform a dot product with weight vector $$\vec{w} = \[0.1, -0.3, 0.5]$$. 8. Plot the vectors $$\vec{a} = \begin{bmatrix} 2 \ 0 \end{bmatrix}, \vec{b} = \begin{bmatrix} 1 \ 1 \end{bmatrix}$$, and their linear combination $$3\vec{a} - 2\vec{b}$$ on the 2D plane. Discuss how linear combinations relate to basis selection in feature transformation (e.g., PCA). 9. In NLP, word embeddings like Word2Vec or GloVe embed words into $$\mathbb{R}^{300}$$. Explain why this 300-dimensional space must satisfy vector space properties. Illustrate with an example of two word embeddings $$\vec{v}{\text{king}}, \vec{v}{\text{man}} \in \mathbb{R}^{300}$$, why $$\vec{v}{\text{king}} - \vec{v}{\text{man}} + \vec{v}\_{\text{woman}}$$ makes semantic sense. 10. Write a Python function that takes a list of 2D vectors and returns whether a given vector lies in their span. Use NumPy's `np.linalg.lstsq` or `np.linalg.solve` appropriately. Test with vectors $$\vec{v}\_1 = \[1, 0], \vec{v}\_2 = \[0, 1]$$ and target vector $$\[2, 3]$$.