Understanding the Efficacy of Over-Parameterization in Neural Networks

Understanding the Efficacy of Over-Parameterization in Neural Networks Understanding the Efficacy of Over-Parameterization in Neural Networks: Mechanisms, Theories, and Practical Implications Introduction Deep neural networks (DNNs) have become the cornerstone of modern artificial intelligence, driving advancements in computer vision, natural language processing, and myriad other domains. A key, albeit counter-intuitive, property of contemporary DNNs is their immense over-parameterization: these models often contain orders of magnitude more parameters than the number of training examples, yet they generalize remarkably well to unseen data. This phenomenon stands in stark contrast to classical statistical learning theory, which posits that models with excessive complexity relative to the available data are prone to overfitting and poor generalization. Intriguingly, empirical evidence shows that increasing the number of parameters in DNNs can lead ...

The Origin of Singularities: Unpacking the Laplacian of 1/r & the Birth of the Delta Function!

The Origin of Singularities: Unpacking the Laplacian of 1/r & the Birth of the Delta Function!

The Origin Story of a Singularity: Unpacking the Laplacian of 1/r (and the Birth of the Delta Function!)

Ever wondered what happens when smooth math slams into an infinite wall? What if a function tries to break free from its well-behaved nature... and creates a mathematical black hole at its core?
In our last journey, we brushed against the edges of functions that "break," hinting at something deeper: singular functionals. Today, we dive headfirst into the most iconic of them all — the mysterious behavior of \( 1/r\) under the Laplacian.
Get ready for a revelation: how this innocent-looking function hides the secret origin of the Dirac delta function, a tool so powerful it reshaped fields from quantum mechanics to engineering.

The Curious Case of 1/r: A Harmonic Hero with a Hidden Flaw

Recall the 3D radial distance:\[ \sqrt{(x_1^2+ x_2^2 + x_3^2} \]The function 1/r is everywhere in physics:

  • It's the gravitational potential around a star.
  • It's the electric potential of a point charge.
  • It's a fundamental solution of Laplace’s equation.

It’s harmonic (meaning \(\Delta \left(\frac{1}{r} \right) = 0\) everywhere except at the origin.
But what happens at the origin?
That’s where things get spicy. At \( \quad r = 0, \quad \frac{1}{r} \) explodes into a singularity. It’s like the function folds space into an infinite spike. How do we make sense of this in mathematics?

The Detective Work: Laplacian as a Generalized Function

To investigate, we enter the world of generalized functions (distributions), where we don’t evaluate functions directly — we test them using smooth “probe” functions \(\) The Laplacian of \(\frac{1}{r} \) becomes:\[(\Delta (1/r), \varphi) = \left( \frac{1}{r}, \Delta \varphi \right) = \iiint_{\mathbb{R}^3} \frac{1}{r} \Delta \varphi \, dV \]But wait — we can’t just integrate over all of space. That singularity at the origin is a no-go zone. So we cut out a tiny ball of radius ๐œ–, do the integration outside it, and then shrink ฯต→0.

Green's Theorem: The Mathematical Microscope

Using Green’s identity on a ball with a spherical hole at the center: \[ \iiint_{r \geq \epsilon} \frac{1}{r} \Delta \varphi \, dV = \iiint_{r \geq \epsilon} \varphi \Delta \left(\frac{1}{r} \right) \, dV - \iint_{S_{\epsilon}} \left( \varphi \frac{\partial}{\partial n} \left(\frac{1}{r} \right) - \frac{1}{r} \frac{\partial \varphi}{\partial n} \right) \, dS\] Breakdown:

  • Volume Integral: Vanishes because \(\Delta \left( \frac{1}{r} \right) = 0 \) for \( r \neq 0.\)
  • First Surface Term: \[\frac{\partial}{\partial n} \left( \frac{1}{r} \right) = -\frac{1}{r^2} \Rightarrow \iint_{S_{\epsilon}} \varphi \cdot \frac{1}{\epsilon^2} \, dS \to 4\pi \varphi(0)\]
  • Second Surface Term: Goes to 0 as ฯต→0
    Result: \[ (\Delta (1/r), \varphi) = 4\pi \varphi(0) = 4\pi (\delta(x), \varphi) \]

The Grand Reveal

Thus, \[ \Delta (1/r) = 4\pi \varphi(x) \] This means all the "action" is concentrated at the origin. The Laplacian detects the singularity and encodes it as a delta function.

SageMath Spotlight: Visualizing the Delta’s Ghost

You can’t graph an infinite spike — but you can approximate it! Try this interactive visualization with SageMath:

      
# Define symbolic variables
x = var('x')
epsilon = var('epsilon')

def approx_delta(x, epsilon):
    return (1/(epsilon * sqrt(pi))) * exp(-x^2 / epsilon^2)

# Plot for different epsilons
p1 = plot(approx_delta(x, 0.5), (x, -2, 2), color='blue', legend_label='ฮต = 0.5')
p2 = plot(approx_delta(x, 0.1), (x, -2, 2), color='red', legend_label='ฮต = 0.1')
p3 = plot(approx_delta(x, 0.05), (x, -2, 2), color='green', legend_label='ฮต = 0.05')

show(p1 + p2 + p3, title="Approximating the Dirac Delta Function")

# Integral check
test_epsilon = 0.01
integral_val = integrate(approx_delta(x, test_epsilon), x, -infinity, infinity)
print(f"Integral ≈ {integral_val.n()}")
	
    

๐Ÿ’ก Try It Yourself! Now You can copy and paste directly into here Run SageMath Code Here

Challenge: Set \(epsilon = 0.001\) and rerun it. What happens to the peak? Does the area stay close to 1?

Why It Matters: Singularities in the Wild

Singularities aren't just math tricks — they model real, jaw-dropping physics:

  • Electrostatics: A point charge’s potential is 1/๐‘Ÿ , and the charge distribution is ๐›ฟ(๐‘ฅ).
  • Fluid Dynamics: Shockwaves create discontinuities in pressure and flow — modeled as singularities.
  • Material Science: Cracks concentrate stress into tiny zones — mathematically, a singularity.
  • Black Holes: General relativity predicts infinite curvature — singularities of spacetime itself!
  • Quantum Mechanics: Wavefunction collapse = instant transition — modeled using delta-like behavior.
  • Phase Transitions: Supercooling or nucleation involves sudden changes — modeled with singular terms.

The Dirac delta is the mathematical embodiment of everything happening at a single point — whether it's a charge, a crack, a collapse, or a cosmic mystery.

Beyond 3D: A Unified Theory of Point Sources

This isn’t just about 3D. In \( n \geq 3 \)dimensions:\[ \quad \Delta \left( \frac{1}{r^{n-2}} \right) = - (n-2) \Omega_n \delta(x)\] For n = 3, we recover: \[ \quad \Delta \left( \frac{1}{r} \right) = -4\pi \delta(x) \] In 2D, we switch to: \[ \quad \Delta \left( \ln \left(\frac{1}{r} \right) \right) = -2\pi \delta(x) \]

These are the fundamental solutions — the “fingerprints” of localized sources in space.

Where We’re Headed: Differentiation Without Limits

Coming soon: How to treat singular objects like delta functions and even 1/๐‘Ÿ as if they were smooth, by upgrading our idea of differentiation itself. This will open doors to:

  • Solving PDEs with singular sources
  • Creating Green's functions
  • Making physics and engineering models more precise

Your Turn: Think Like a Physicist

Here’s your challenge:

If \( \Delta( 1/r)=4\pi\delta(x) \) and 1/r is the potential of a point charge, how does this deepen your understanding of the delta function as a source? Can you feel why the Laplacian acts like a “charge detector”? What happens when there are multiple sources — or a continuous distribution?

Share your thoughts, insights, or even code experiments in the comments. Let’s explore the singular frontier together.

Comments

Popular posts from this blog

๐ŸŒŸ Illuminating Light: Waves, Mathematics, and the Secrets of the Universe

Spirals in Nature: The Beautiful Geometry of Life