Contrastive divergence

Contrastive Divergence (CD) is an algorithm used for training generative models, particularly in the context of Boltzmann Machines (BMs) and Restricted Boltzmann Machines (RBMs).

CD is a practical approximation of the more computationally intensive Markov Chain Monte Carlo (MCMC) methods, making it feasible to train BMs effectively.

The primary objective of CD is to adjust the weights of the BM to maximize the likelihood of generating observed data samples. The training process involves two main steps: positive phase and negative phase.

In the positive phase, the BM receives a training sample and activates its neurons to reproduce the observed data pattern. The activations of the visible and hidden neurons are updated according to the current weights and biases. This step captures the joint distribution of the observed data and the hidden representations.

The negative phase, also known as the sampling phase, is where the model generates synthetic samples. Starting from the activations obtained in the positive phase, a Markov Chain process is initiated by repeatedly sampling the states of the visible and hidden neurons. As the Markov Chain evolves, the model generates new samples by repeatedly updating the neuron states based on the current weights.

The key idea behind CD is to approximate the difference in statistics between the positive and negative phases. Instead of running the Markov Chain until it reaches convergence, CD typically performs a few steps of sampling. This approximation reduces computational costs and makes the training process more efficient. By comparing the statistics of the observed data and the generated samples, CD computes the gradient of the model’s parameters and updates the weights accordingly.


Just in

Raspberry Pi is now a public company — TC

Raspberry Pi priced its IPO on the London Stock Exchange on Tuesday morning at £2.80 per share, valuing it at £542 million, or $690 million at today’s exchange rate, writes Romain Dillet. 

AlphaSense raises $650M

AlphaSense, a market intelligence and search platform, has raised $650 million in funding, co-led by Viking Global Investors and BDT & MSD Partners.

Elon Musk’s xAI raises $6B to take on OpenAI — VentureBeat

Confirming reports from April, the series B investment comes from the participation of multiple known venture capital firms and investors, including Valor Equity Partners, Vy Capital, Andreessen Horowitz (A16z), Sequoia Capital, Fidelity Management & Research Company, Prince Alwaleed Bin Talal and Kingdom Holding, writes Shubham Sharma. 

Capgemini partners with DARPA to explore quantum computing for carbon capture

Capgemini Government Solutions has launched a new initiative with the Defense Advanced Research Projects Agency (DARPA) to investigate quantum computing's potential in carbon capture.