Introduction to Diffusion Models (Part II: Math Intuitions)

Abstract.
This article delves into the mathematical and intuitive underpinnings of diffusion models, bridging the gap between traditional diffusion processes and their application in deep learning. It provides a comprehensive overview of the diffusion equation, its theoretical foundation, and its relevance in generative modeling. Through a blend of physical intuition and mathematical rigor, readers will gain a holistic understanding of how substances spread in space and time and how these principles can be adapted for data generation in the realm of machine learning.

Learning Outcomes

1. Understanding of Diffusion in Physical and Mathematical Contexts: Readers will comprehend the fundamental principles of diffusion, both from a physical standpoint of how substances spread and from a mathematical perspective of how the diffusion equation captures this process.
2. Applicability in Deep Learning: By the end of the article, readers will recognize how the principles of diffusion have been adapted in deep learning, particularly in generative modeling. They’ll appreciate the significance of iterative refinement, noise addition, and the role of neural networks in driving the diffusion process in data generation.
3. Insight into Theoretical Foundations: Readers will gain a deeper understanding of the Markov processes, random walks, and Brownian motion. They will discern how these foundational theories influence and shape the diffusion models in machine learning and their iterative nature.

Photo of a diffusion experiment setup, with a clear glass beaker containing a light blue liquid. A drop of dark blue dye is being added to the center, showing the initial stage of diffusion. Arrows around the drop indicate the direction of molecule movement outward.

Detailed Introduction to Diffusion Models

Intuition Behind the Diffusion Equation

In this section, we explain the intuition behind the diffusion equation.

Figure 2. sequence illustrates the concept of the diffusion model. Starting from the left, a glass of clear water with a freshly dropped ink droplet at the center. As the sequence progresses to the right, the ink gradually spreads throughout the water. Superimposed on the images, a mathematical equation predicts the rate and direction of the ink’s spread.

Figure2. Illustration of the diffusion model concept

The diffusion equation, often termed Fick’s second law, mathematically represents the process we observe when substances like ink spread out or diffuse over time. At its core, it’s defined by:

Figure 3. Diffusion Model Equation

Here’s how to decipher this:

• ϕ: is the concentration of the substance. It symbolizes the concentration of our substance, much like the density of ink in a particular water region.
• t: represents the time.
• ∂ϕ/∂t: This rate component tells us how this concentration changes as time passes. It’s like observing how quickly our ink spreads through the water.
• D: Termed the diffusion coefficient, this factor embodies the inherent “spread-speed” of our substance. It’s the reason why some substances disperse swiftly while others take their time.
• ∇²ϕ: Known as the Laplacian operator, which represents the second spatial derivative. This part of the equation gives us insights into the direction and extent of our substance’s spread. It’s akin to determining in which direction our ink wants to travel and how much it wants to disperse.

Summary.
In a nutshell, the diffusion equation serves as a bridge between abstract mathematics and the tangible world, providing us with a formulaic lens to predict and understand the spreading behavior of substances in various mediums over time.

Discretization

For computational purposes, the continuous diffusion equation is often discretized, allowing for its application in iterative deep-learning frameworks. This involves breaking down time and space into discrete steps and units.

How does Discretization work?

Discretizing the diffusion equation means transforming our continuous view of time and space into small, distinct chunks, akin to converting a smooth gradient into a set of individual color blocks. For computational models, like those in deep learning, this stepwise perspective is essential.

Now, for the diffusion equation:

When we discretize it, we can represent changes in concentration, time, and space using discrete steps:

Let’s use:

• Δt for the discrete time step.
• Δx for the discrete space step in the x-direction (and similarly for y, z, if it’s 2D or 3D).

Our discretized diffusion equation can look something like this:

Figure 4. Finite Difference Approximation for the Diffusion Equation

This equation essentially states that the change in concentration at any point over a small time step (Δt) depends on the difference in concentration between neighboring points (determined by Δx) at the current time.

The connection between Diffusion Equations and Generative Modeling:

1. Diffusion as a Generative Process: In deep learning, the diffusion equation can be adapted to describe the generative process. The idea is to consider the data distribution as a ‘substance’ that diffuses over time. The process begins from a simple distribution and, through iterative steps, approaches the target complex distribution.
2. Noise Addition and Iterative Refinement: By adding noise, we push the data towards a simpler distribution. The iterative refinement, facilitated by neural networks, tries to reverse this process. It’s akin to ‘denoising’ the data at each step, gradually moving it closer to the target distribution.
3. Probabilistic Interpretation: The diffusion process can be viewed probabilistically. At each step, given the current state, there’s a probability distribution over the next state. Neural networks in diffusion models are trained to capture this transition probability, aiding in the generation process.

Summary.
Diffusion models in the context of deep learning leverage the principles of diffusion — traditionally understood in terms of substances spreading through space and time — to describe the process of data generation. They use the power of neural networks to drive this diffusion process, starting from simple distributions and iteratively refining them to resemble complex real-world data distributions.

Theoretical Foundations

Markov Processes and Random Walks:

1. Markov Process: A Markov process is a mathematical model that describes a sequence of events where the probability of each event depends only on the state of the previous event. It’s “memoryless”, meaning the future state only depends on the current state and not the sequence of states preceding it.
2. Random Walk: A special case of a Markov process where an entity takes steps in random directions. In a one-dimensional space, this might be visualized as a particle moving left or right with equal probability at each step.

Brownian Motion In-depth

1. Mathematical Description: Brownian motion B(t) can be described as a continuous-time stochastic process that satisfies two properties:
   - B(0)=0: It starts at zero.
   - The increments are independent and normally distributed with
   B(t)−B(s)∼N(0,t−s) for 0≤s<t.
2. Properties

• It has stationary and independent increments.
• It’s continuous everywhere but differentiable nowhere.
• It’s a limit of scaled random walks as the step size goes to zero and the number of steps goes to infinity.

1. Connection to Diffusion: The Brownian motion is closely related to the diffusion equation. The equation can describe the evolution of the probability distribution of a particle undergoing Brownian motion.

Understanding the Diffusion Equation in Detail

1. Physical Intuition: The diffusion equation describes how substances spread through space and time. The rate of change of the substance’s concentration at a point in space is proportional to the curvature of the concentration around that point, which is the essence of the equation.
2. Mathematical Analysis: Solutions to the diffusion equation can be analyzed using various mathematical techniques, like separation of variables or Fourier transforms, to understand how different initial distributions evolve.
3. Boundary and Initial Conditions: Solutions to the diffusion equation often require specifying boundary conditions (how the substance behaves at the edges of the domain) and initial conditions (the initial distribution of the substance). These conditions heavily influence the evolution of the substance over time.

Summary.
By understanding these foundational concepts, one gains a deeper appreciation for how diffusion models in deep learning work. The iterative refinement seen in diffusion models is, in many ways, a discretized version of the continuous diffusion processes described by these theories.