Best blog post i've seen describing contrastive learning for machine learning researchers
https://lilianweng.github.io/lil-log/2021/05/31/contrastive-representation-learning.html
Contrastive Representation Learning
2021-05-31
Lilian Weng
<div><div>
 
 <p>
 <time>
 
 May 31, 2021
 </time>
 
 by Lilian Weng
 
 
 
 
 
 </p>
 </div>||||I||||">
 Lil'Log
 🙂 Emoji
 🦄 Contact
 🙋 FAQ
 ⌛ Archive

 Contrastive Representation Learning

 May 31, 2021 by Lilian Weng

 ← Reducing Toxicity In Language Models Diffusion Models →
2021 © by Lilian Weng. All Rights Reserved. Built by Jekyll. | View on Github | Tags | Contact | FAQ

">
 Lil'Log
 🙂 Emoji
 🦄 Contact
 🙋 FAQ
 ⌛ Archive
 Contrastive Representation Learning
 May 31, 2021 by Lilian Weng
 ← Reducing Toxicity In Language Models Diffusion Models →
2021 © by Lilian Weng. All Rights Reserved. Built by Jekyll. | View on Github | Tags | Contact | FAQ
Contrastive Representation Learning
https://lilianweng.github.io/lil-log/2021/05/31/contrastive-representation-learning.html
https://kyunghyuncho.me/restricted-boltzmann-machines-or-contrastive-learning/
Restricted Boltzmann machines or contrastive learning?
2021-07-10
Kyunghyuncho
<div><article><div><p>my inbox started to over-flow with emails that urgently require my attention, and my TODO list (which doesn’t exist outside my own brain) started to randomly remove entries to avoid overflowing. of course, this is perfect time for me to think of some random stuff.</p><p>This time, this random stuff is contrastive learning. my thought on this stuff was sparked by Lerrel Pinto’s message on #random in our group’s Slack responding to the question “<em>What is wrong with contrastive learning?</em>” thrown by Andrew Gordon Wilson. Lerrel said,</p><blockquote><p><em>My understanding is that getting negatives for contrastive learning is difficult. </em></p><cite>Lerrel Pinto (2021)</cite></blockquote><p>i haven’t worked on the (post-)modern version of contrastive learning, but every time i hear of “<em>negative samples</em>” i am reminded me of my phd years. during my phd years, i’ve mainly worked on a restricted Boltzmann machine which defines a distribution over the observation space as</p><p>$$p(x; W, b, c) \propto \exp(x^\top b) \prod_{j=1}^J (1+\exp(x^\top w_{\cdot, j} + c_j)),$$</p><p>where $W$, $b$ and $c$ are the weight matrix, visible bias and hidden bias. for simplicity, i’ll assume the visible bias is $0$, which is equivalent to saying that the input is on expectation an all-zero vector. This makes the definition above a bit simpler, and especially so when we look at the log-probability:</p><p>$$\log p(x; W, c) = \sum_{j=1}^J \log (1+\exp(x^\top w_{\cdot, j} + c_j)) – \log Z,$$</p><p>where $\log Z$ is the log-partition function or log-normalization constant.</p><p>the goal of learning with a restricted Boltzmann machine is then to maximize the log-probabilities of the observations (training examples):</p><p>$$\max_{W, c} \mathbb{E}_{x \sim D} [\log p(x; W, c)],$$</p><p>using stochastic gradient descent with the stochastic gradient derived to be</p><p>$$g_{\theta} = \sum_{j=1}^J \nabla_\theta \log (1+\exp(x^\top w_{\cdot,j} + c_j)) – \mathbb{E}_{x_- \sim p(x; W,c)} [\sum_{j=1}^J \nabla_\theta \log (1+\exp({x_-}^\top w_{\cdot,j} + c_j)].$$</p><p>the first term ensures that each hidden unit (or expert) $j$ is well aligned with the correct observation $x$ drawn from the data distribution (or training set.) not too surprising, since the alignment (dot product) between the expert weight $w_{\cdot, j}$ and a given observation gives rise to the probability of $x$.</p><p>the second term corresponds to computing the expected negative energy (ugh, i hate this discrepancy; we maximize the probability but we minimize the energy) over all possible observations according to the model distribution. what this term does is to look for all input configurations $x_-$ that are good under our current model and to make sure the hidden units (or experts) are not well aligned with them.</p><p>you can imagine this as playing whac-a-mole. we try to pull out our favourite moles, while we “whac” any mole that’s favoured by the whac-a-mole machine.</p><p>in training a restricted boltzmann machine, the major difficulty lies with how to efficiently and effectively draw negative samples from the model distribution. a lot of bright minds at the University of Toronto and University of Montreal back then (somewhere between 2006 and 2013) spent years on figuring this out. unfortunately, we (as the field) have never got it to work well, which is probably not surprising since we’re talking about sampling from an unnormalized (often discrete) distribution over hundreds if not thousands of dimensions. if it were easy, we would’ve solved most of problems in ML already.</p><h2>Data augmentation creates a restricted Boltzmann machine</h2><p>let’s consider a stochastic transformation $T: \mathcal{X} \to \mathcal{X}$, where $\mathcal{X}$ is the input space. given any input $x \in \mathcal{X}$, this transformation outputs $\tilde{x} \sim T$ that highly likely maintains the same semantics as the original $x$. this is often used for data augmentation which has been found to be a critical component of contrastive learning (or as a matter of fact any so-called self-supervised learning algorithms).</p><p>imagine a widely used set of input transformations in e.g. computer vision. $T$ would include (limited) translation, (limited) rotation, (limited) color distortion, (limited) elastic distortion, etc. we know these transformations often in advance, and these are often domain/problem-specific.</p><p>what we will now do is to create a very large set of hidden units (or experts) by drawing transformed inputs from the stochastic transformation $T$ for one particular input $x$. that is, we have $J$-many $\tilde{x}_j \sim T(x)$. in the case of computer vision, we’ll have $J$-many possible distortions of $x$ that largely maintain the semantics of $x$.</p><p>these hidden units then define a restricted Boltzmann machine and allow us to compute the probability of any input $x’$:</p><p>$$\log p(x’ | \tilde{x}_1, \ldots, \tilde{x}_J) = \sum_{j=1}^J \log (1+\exp(s(x’,\tilde{x}_j))) – \log Z,$$</p><p>where i’m now using a compatibility function $s: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ instead of the dot-product for more generality.</p><p>starting from here, we’ll make two changes (one relaxation and one restriction). first, we don’t want to only use $J$ many transformed copies of the input $x$. we want to in fact use all possible transformed versions of $x$ out of $T$. in other words, we want to relax our construction that this restricted Boltzmann machine has a finite number of hidden units. this turns the equation above to be:</p><p>$$\log p(x’ | x, T) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log Z.$$</p><p>second, we will assume that the input space $\mathcal{X}$ coincides with the training set $D$ which has a finite number of training examples, i.e., $D=\left\{ x_1, \ldots, x_N \right\}$. this second change only affects the second term (the log-partition function) only:</p><p>$$\log p(x’ | T(x)) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log \frac{1}{N} \sum_{n=1}^N \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x_n,\tilde{x})))\right].$$</p><p>to summarize what we’ve done so far: we build one restricted Boltzmann machine for a given input $x \in \mathcal{X}$ by drawing the hidden units (or experts) from the transformation distribution $\tilde{x} \sim T(x)$. the support of this restricted Boltzmann machine is restricted (pun intended) to be a training set.</p><h2>Contrastive learning trains <em>N</em> restricted Boltzmann machines</h2><p>what would be a good training criterion for one such restricted Boltzmann machine? the answer is almost always maximum likelihood! in this particular case, we want to ensure that the original example $x$ is most likely under the restricted Boltzmann machine induced by itself:</p><p>$$\max_{\theta} \log p(x | T(x)),$$</p><p>where $\theta$ is the parameters for defining the compatibility function $s$ from above.</p><p>we do so for all $N$ restricted Boltzmann machines induced from $N$ training examples:</p><p>$$\max_{\theta} \frac{1}{N} \sum_{n=1}^N \log p(x_n | T(x_n)).$$</p><p>since it’s decomposed over the training examples, let’s consider only one example $x \in D$. we then train the induced restricted Boltzmann machine with stochastic gradient descent, following</p><p>$$\frac{1}{M} \sum_{m=1}^M \nabla_{\theta} \log (1+\exp(s(x, \tilde{x}_m; \theta))) – \frac{1}{M} \sum_{m=1}^M \sum_{n=1}^N p(x_n|T(x)) \nabla_{\theta} \log (1+\exp(s(x_n, \tilde{x}_m; \theta))),$$</p><p>where we use $M$ transformed copies to approximate the two expectations over $T(x)$ but not $p(x_n|T(x))$. we probably should use another set of $M$ transformed copies to get the unbiased estimate.</p><p>this does look quite similar to more recently popular variants of contrastive learning. we start from a training example $x$, generate a transformed version $\tilde{x}$, maximize the compatibility between $x$ and $\tilde{x}$, and minimize the compatibility between $\tilde{x}$ and all the training examples (including $x$). there are minor differences, such as the choice of nonlinearity, but at the high level, it turned out we can derive contrastive learning from the restricted Boltzmann machine.</p><p>perhaps the only major difference is that this formulation gives us a clear guideline on how we should pick the negative examples. that is, according to this formula, we should either use all the training examples weighted according to how likely they are under this $x$-induced restricted Boltzmann machine or use a subset of training examples drawn according to the $x$-induced restricted Boltzmann machine without further weighting. of course, another alternative is to use uniformly-selected training examples as negative samples but weight them according to their probabilities under the $x$-induced restricted Boltzmann machine, <em>à la</em> importance sampling.</p><p>so, yes, contrastive learning can be derived from restricted Boltzmann machines, and this is advantageous, because this tells us how we should pick negative examples. in fact, as i was writing this blog post (and an earlier internal slack message,) i was reminded of a recent workshop i’ve attended together with Yoshua Bengio. there was a talk on how to choose <em>hard</em> negative samples for contrastive learning (or representation learning) on graphs, and after the talk was over, Yoshua raised his hand and made this remark</p><blockquote><p><em>That’s called Boltzmann machine learning!</em></p><cite>Yoshua Bengio (2019, paraphrased)</cite></blockquote><p>Indeed…</p><h2>Data augmentation is what matters</h2><p>Based on this exercise of deriving modern contrastive learning from restricted Boltzmann machines, we can now have a meta-framework for coming up with a contrastive learning recipe. Any recipe must consist of three major ingredients:</p><ol><li><strong>A per-example density estimator</strong>: i used the restricted Boltzmann machine, but you may very well use variational autoencoders, independent component analysis, principal component analysis, sparse coding, etc. these will give rise to different variants of self-supervised learning. the latter three are particularly interesting, because they are fully described by a set of basis vectors and don’t require any negative samples for learning. i’m almost 100% certain you can derive all these non-contrastive learning algorithms by choosing one of these three.</li><li><strong>A compatibility function</strong> $s$: this is the part where we design a network “architecture”, and how the output from this network is used to compute a scalar that indicates how similar a pair of examples is. it looks like the current practice is to use a deep neural net with a cosine similarity to implement this compatibility function.</li><li><strong>A stochastic transformation</strong> <strong>generator</strong>: this generator effectively generates a density estimator for each example. this is very important, since it defines the set of bases used by these density estimators. any aspect of data cannot be modelled if these generated bases do not cover them.</li></ol><p>we have a pretty good idea of what kind of density estimator is suitable for various purposes. we have a pretty good idea what’s the best way to measure the similarity between two highly-complex, high-dimensional inputs (thanks, deep learning!) but, we cannot know what the right stochastic transformation generator should be, because it is heavily dependent on the problem and domain. for instance, the optimal transformation generator for static, natural images won’t be optimal for e.g. natural language text.</p><p>so, my sense is that the success of using contrastive learning (or any self-supervised learning) for any given problem will ultimately boil down to the choice and design of stochastic transformation, since there’s a chance that we may find a near-optimal pair of the first two (density estimator and compatibility function) that works well across multiple problems and domains.</p></div></article>||||I|||| Skip to content

 Kyunghyun Cho

 July 10, 2021July 10, 2021 kyunghyuncho

 Restricted Boltzmann machines or contrastive learning?

 Research

 my inbox started to over-flow with emails that urgently require my attention, and my TODO list (which doesn’t exist outside my own brain) started to randomly remove entries to avoid overflowing. of course, this is perfect time for me to think of some random stuff.

 This time, this random stuff is contrastive learning. my thought on this stuff was sparked by Lerrel Pinto’s message on #random in our group’s Slack responding to the question “What is wrong with contrastive learning?” thrown by Andrew Gordon Wilson. Lerrel said,

 My understanding is that getting negatives for contrastive learning is difficult.

 Lerrel Pinto (2021)

 Restricted Boltzmann Machines

 i haven’t worked on the (post-)modern version of contrastive learning, but every time i hear of “negative samples” i am reminded me of my phd years. during my phd years, i’ve mainly worked on a restricted Boltzmann machine which defines a distribution over the observation space as

 $$p(x; W, b, c) \propto \exp(x^\top b) \prod_{j=1}^J (1+\exp(x^\top w_{\cdot, j} + c_j)),$$

 where $W$, $b$ and $c$ are the weight matrix, visible bias and hidden bias. for simplicity, i’ll assume the visible bias is $0$, which is equivalent to saying that the input is on expectation an all-zero vector. This makes the definition above a bit simpler, and especially so when we look at the log-probability:

 $$\log p(x; W, c) = \sum_{j=1}^J \log (1+\exp(x^\top w_{\cdot, j} + c_j)) – \log Z,$$

 where $\log Z$ is the log-partition function or log-normalization constant.

 the goal of learning with a restricted Boltzmann machine is then to maximize the log-probabilities of the observations (training examples):

 $$\max_{W, c} \mathbb{E}_{x \sim D} [\log p(x; W, c)],$$

 using stochastic gradient descent with the stochastic gradient derived to be

 $$g_{\theta} = \sum_{j=1}^J \nabla_\theta \log (1+\exp(x^\top w_{\cdot,j} + c_j)) – \mathbb{E}_{x_- \sim p(x; W,c)} [\sum_{j=1}^J \nabla_\theta \log (1+\exp({x_-}^\top w_{\cdot,j} + c_j)].$$

 the first term ensures that each hidden unit (or expert) $j$ is well aligned with the correct observation $x$ drawn from the data distribution (or training set.) not too surprising, since the alignment (dot product) between the expert weight $w_{\cdot, j}$ and a given observation gives rise to the probability of $x$.

 the second term corresponds to computing the expected negative energy (ugh, i hate this discrepancy; we maximize the probability but we minimize the energy) over all possible observations according to the model distribution. what this term does is to look for all input configurations $x_-$ that are good under our current model and to make sure the hidden units (or experts) are not well aligned with them.

 you can imagine this as playing whac-a-mole. we try to pull out our favourite moles, while we “whac” any mole that’s favoured by the whac-a-mole machine.

 in training a restricted boltzmann machine, the major difficulty lies with how to efficiently and effectively draw negative samples from the model distribution. a lot of bright minds at the University of Toronto and University of Montreal back then (somewhere between 2006 and 2013) spent years on figuring this out. unfortunately, we (as the field) have never got it to work well, which is probably not surprising since we’re talking about sampling from an unnormalized (often discrete) distribution over hundreds if not thousands of dimensions. if it were easy, we would’ve solved most of problems in ML already.

 Data augmentation creates a restricted Boltzmann machine

 let’s consider a stochastic transformation $T: \mathcal{X} \to \mathcal{X}$, where $\mathcal{X}$ is the input space. given any input $x \in \mathcal{X}$, this transformation outputs $\tilde{x} \sim T$ that highly likely maintains the same semantics as the original $x$. this is often used for data augmentation which has been found to be a critical component of contrastive learning (or as a matter of fact any so-called self-supervised learning algorithms).

 imagine a widely used set of input transformations in e.g. computer vision. $T$ would include (limited) translation, (limited) rotation, (limited) color distortion, (limited) elastic distortion, etc. we know these transformations often in advance, and these are often domain/problem-specific.

 what we will now do is to create a very large set of hidden units (or experts) by drawing transformed inputs from the stochastic transformation $T$ for one particular input $x$. that is, we have $J$-many $\tilde{x}_j \sim T(x)$. in the case of computer vision, we’ll have $J$-many possible distortions of $x$ that largely maintain the semantics of $x$.

 these hidden units then define a restricted Boltzmann machine and allow us to compute the probability of any input $x’$:

 $$\log p(x’ | \tilde{x}_1, \ldots, \tilde{x}_J) = \sum_{j=1}^J \log (1+\exp(s(x’,\tilde{x}_j))) – \log Z,$$

 where i’m now using a compatibility function $s: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ instead of the dot-product for more generality.

 starting from here, we’ll make two changes (one relaxation and one restriction). first, we don’t want to only use $J$ many transformed copies of the input $x$. we want to in fact use all possible transformed versions of $x$ out of $T$. in other words, we want to relax our construction that this restricted Boltzmann machine has a finite number of hidden units. this turns the equation above to be:

 $$\log p(x’ | x, T) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log Z.$$

 second, we will assume that the input space $\mathcal{X}$ coincides with the training set $D$ which has a finite number of training examples, i.e., $D=\left\{ x_1, \ldots, x_N \right\}$. this second change only affects the second term (the log-partition function) only:

 $$\log p(x’ | T(x)) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log \frac{1}{N} \sum_{n=1}^N \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x_n,\tilde{x})))\right].$$

 to summarize what we’ve done so far: we build one restricted Boltzmann machine for a given input $x \in \mathcal{X}$ by drawing the hidden units (or experts) from the transformation distribution $\tilde{x} \sim T(x)$. the support of this restricted Boltzmann machine is restricted (pun intended) to be a training set.

 Contrastive learning trains N restricted Boltzmann machines

 what would be a good training criterion for one such restricted Boltzmann machine? the answer is almost always maximum likelihood! in this particular case, we want to ensure that the original example $x$ is most likely under the restricted Boltzmann machine induced by itself:

 $$\max_{\theta} \log p(x | T(x)),$$

 where $\theta$ is the parameters for defining the compatibility function $s$ from above.

 we do so for all $N$ restricted Boltzmann machines induced from $N$ training examples:

 $$\max_{\theta} \frac{1}{N} \sum_{n=1}^N \log p(x_n | T(x_n)).$$

 since it’s decomposed over the training examples, let’s consider only one example $x \in D$. we then train the induced restricted Boltzmann machine with stochastic gradient descent, following

 $$\frac{1}{M} \sum_{m=1}^M \nabla_{\theta} \log (1+\exp(s(x, \tilde{x}_m; \theta))) – \frac{1}{M} \sum_{m=1}^M \sum_{n=1}^N p(x_n|T(x)) \nabla_{\theta} \log (1+\exp(s(x_n, \tilde{x}_m; \theta))),$$

 where we use $M$ transformed copies to approximate the two expectations over $T(x)$ but not $p(x_n|T(x))$. we probably should use another set of $M$ transformed copies to get the unbiased estimate.

 this does look quite similar to more recently popular variants of contrastive learning. we start from a training example $x$, generate a transformed version $\tilde{x}$, maximize the compatibility between $x$ and $\tilde{x}$, and minimize the compatibility between $\tilde{x}$ and all the training examples (including $x$). there are minor differences, such as the choice of nonlinearity, but at the high level, it turned out we can derive contrastive learning from the restricted Boltzmann machine.

 perhaps the only major difference is that this formulation gives us a clear guideline on how we should pick the negative examples. that is, according to this formula, we should either use all the training examples weighted according to how likely they are under this $x$-induced restricted Boltzmann machine or use a subset of training examples drawn according to the $x$-induced restricted Boltzmann machine without further weighting. of course, another alternative is to use uniformly-selected training examples as negative samples but weight them according to their probabilities under the $x$-induced restricted Boltzmann machine, à la importance sampling.

 so, yes, contrastive learning can be derived from restricted Boltzmann machines, and this is advantageous, because this tells us how we should pick negative examples. in fact, as i was writing this blog post (and an earlier internal slack message,) i was reminded of a recent workshop i’ve attended together with Yoshua Bengio. there was a talk on how to choose hard negative samples for contrastive learning (or representation learning) on graphs, and after the talk was over, Yoshua raised his hand and made this remark

 That’s called Boltzmann machine learning!

 Yoshua Bengio (2019, paraphrased)

 Indeed…

 Data augmentation is what matters

 Based on this exercise of deriving modern contrastive learning from restricted Boltzmann machines, we can now have a meta-framework for coming up with a contrastive learning recipe. Any recipe must consist of three major ingredients:

 1. A per-example density estimator: i used the restricted Boltzmann machine, but you may very well use variational autoencoders, independent component analysis, principal component analysis, sparse coding, etc. these will give rise to different variants of self-supervised learning. the latter three are particularly interesting, because they are fully described by a set of basis vectors and don’t require any negative samples for learning. i’m almost 100% certain you can derive all these non-contrastive learning algorithms by choosing one of these three.
 2. A compatibility function $s$: this is the part where we design a network “architecture”, and how the output from this network is used to compute a scalar that indicates how similar a pair of examples is. it looks like the current practice is to use a deep neural net with a cosine similarity to implement this compatibility function.
 3. A stochastic transformation generator: this generator effectively generates a density estimator for each example. this is very important, since it defines the set of bases used by these density estimators. any aspect of data cannot be modelled if these generated bases do not cover them.

 we have a pretty good idea of what kind of density estimator is suitable for various purposes. we have a pretty good idea what’s the best way to measure the similarity between two highly-complex, high-dimensional inputs (thanks, deep learning!) but, we cannot know what the right stochastic transformation generator should be, because it is heavily dependent on the problem and domain. for instance, the optimal transformation generator for static, natural images won’t be optimal for e.g. natural language text.

 so, my sense is that the success of using contrastive learning (or any self-supervised learning) for any given problem will ultimately boil down to the choice and design of stochastic transformation, since there’s a chance that we may find a near-optimal pair of the first two (density estimator and compatibility function) that works well across multiple problems and domains.

 Post navigation

 Ho-Am Prize & Lim Mi-Sook Scholarship (임미숙 장학금) at KAIST
 Supporting female researchers and researchers from under-represented groups, together with CIFAR

 Related Posts

 * [NeurIPS’22] Chasing reviewers

 * My opening statement at the ICML 2022 Debate

 * Reading others’ reviews

 Leave a Reply

 You must be logged in to post a comment.

 Search for:

 Recent Posts

 * [NeurIPS’22] Chasing reviewers
 * Donation to “청포도” 보호종료아동을 위한 커뮤니티 케어 센터
 * My opening statement at the ICML 2022 Debate
 * [TMLR] how to check the difference between revisions
 * [TMLR] how to set your availability as a reviewer/action editor
 * About me
 * Pro Bono
 * Blog
 * Talks
 * Courses
 * Research Group
 * Research Activites
 * Publications
 * Acknowledgement
 * Disclosure
 * Contact

 Blog Posts

 * News
 * Personal
 * Research
 * Uncategorized

 Meta

 * Log in
 * Entries feed
 * Comments feed
 * WordPress.org
 Proudly powered by WordPress | Theme: Gist by Candid Themes.
my inbox started to over-flow with emails that urgently require my attention, and my TODO list (which doesn’t exist outside my own brain) started to randomly remove entries to avoid overflowing. of course, this is perfect time for me to think of some random stuff. This time, this random stuff is contrastive learning. my thought on this stuff was sparked by Lerrel Pinto’s message on #random in our group’s Slack responding to the question “What is wrong with contrastive learning?” thrown by Andrew Gordon Wilson. Lerrel said,   My understanding is that getting negatives for contrastive learning is difficult.   Lerrel Pinto (2021)  i haven’t worked on the (post-)modern version of contrastive learning, but every time i hear of “negative samples” i am reminded me of my phd years. during my phd years, i’ve mainly worked on a restricted Boltzmann machine which defines a distribution over the observation space as $$p(x; W, b, c) \propto \exp(x^\top b) \prod_{j=1}^J (1+\exp(x^\top w_{\cdot, j} + c_j)),$$ where $W$, $b$ and $c$ are the weight matrix, visible bias and hidden bias. for simplicity, i’ll assume the visible bias is $0$, which is equivalent to saying that the input is on expectation an all-zero vector. This makes the definition above a bit simpler, and especially so when we look at the log-probability: $$\log p(x; W, c) = \sum_{j=1}^J \log (1+\exp(x^\top w_{\cdot, j} + c_j)) – \log Z,$$ where $\log Z$ is the log-partition function or log-normalization constant. the goal of learning with a restricted Boltzmann machine is then to maximize the log-probabilities of the observations (training examples): $$\max_{W, c} \mathbb{E}_{x \sim D} [\log p(x; W, c)],$$ using stochastic gradient descent with the stochastic gradient derived to be $$g_{\theta} = \sum_{j=1}^J \nabla_\theta \log (1+\exp(x^\top w_{\cdot,j} + c_j)) – \mathbb{E}_{x_- \sim p(x; W,c)} [\sum_{j=1}^J \nabla_\theta \log (1+\exp({x_-}^\top w_{\cdot,j} + c_j)].$$ the first term ensures that each hidden unit (or expert) $j$ is well aligned with the correct observation $x$ drawn from the data distribution (or training set.) not too surprising, since the alignment (dot product) between the expert weight $w_{\cdot, j}$ and a given observation gives rise to the probability of $x$. the second term corresponds to computing the expected negative energy (ugh, i hate this discrepancy; we maximize the probability but we minimize the energy) over all possible observations according to the model distribution. what this term does is to look for all input configurations $x_-$ that are good under our current model and to make sure the hidden units (or experts) are not well aligned with them. you can imagine this as playing whac-a-mole. we try to pull out our favourite moles, while we “whac” any mole that’s favoured by the whac-a-mole machine. in training a restricted boltzmann machine, the major difficulty lies with how to efficiently and effectively draw negative samples from the model distribution. a lot of bright minds at the University of Toronto and University of Montreal back then (somewhere between 2006 and 2013) spent years on figuring this out. unfortunately, we (as the field) have never got it to work well, which is probably not surprising since we’re talking about sampling from an unnormalized (often discrete) distribution over hundreds if not thousands of dimensions. if it were easy, we would’ve solved most of problems in ML already. Data augmentation creates a restricted Boltzmann machine let’s consider a stochastic transformation $T: \mathcal{X} \to \mathcal{X}$, where $\mathcal{X}$ is the input space. given any input $x \in \mathcal{X}$, this transformation outputs $\tilde{x} \sim T$ that highly likely maintains the same semantics as the original $x$. this is often used for data augmentation which has been found to be a critical component of contrastive learning (or as a matter of fact any so-called self-supervised learning algorithms). imagine a widely used set of input transformations in e.g. computer vision. $T$ would include (limited) translation, (limited) rotation, (limited) color distortion, (limited) elastic distortion, etc. we know these transformations often in advance, and these are often domain/problem-specific. what we will now do is to create a very large set of hidden units (or experts) by drawing transformed inputs from the stochastic transformation $T$ for one particular input $x$. that is, we have $J$-many $\tilde{x}_j \sim T(x)$. in the case of computer vision, we’ll have $J$-many possible distortions of $x$ that largely maintain the semantics of $x$. these hidden units then define a restricted Boltzmann machine and allow us to compute the probability of any input $x’$: $$\log p(x’ | \tilde{x}_1, \ldots, \tilde{x}_J) = \sum_{j=1}^J \log (1+\exp(s(x’,\tilde{x}_j))) – \log Z,$$ where i’m now using a compatibility function $s: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ instead of the dot-product for more generality. starting from here, we’ll make two changes (one relaxation and one restriction). first, we don’t want to only use $J$ many transformed copies of the input $x$. we want to in fact use all possible transformed versions of $x$ out of $T$. in other words, we want to relax our construction that this restricted Boltzmann machine has a finite number of hidden units. this turns the equation above to be: $$\log p(x’ | x, T) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log Z.$$ second, we will assume that the input space $\mathcal{X}$ coincides with the training set $D$ which has a finite number of training examples, i.e., $D=\left\{ x_1, \ldots, x_N \right\}$. this second change only affects the second term (the log-partition function) only: $$\log p(x’ | T(x)) = \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x’,\tilde{x})))\right] – \log \frac{1}{N} \sum_{n=1}^N \mathbb{E}_{\tilde{x} \sim T(x)}\left[ \log (1+\exp(s(x_n,\tilde{x})))\right].$$ to summarize what we’ve done so far: we build one restricted Boltzmann machine for a given input $x \in \mathcal{X}$ by drawing the hidden units (or experts) from the transformation distribution $\tilde{x} \sim T(x)$. the support of this restricted Boltzmann machine is restricted (pun intended) to be a training set. Contrastive learning trains N restricted Boltzmann machines what would be a good training criterion for one such restricted Boltzmann machine? the answer is almost always maximum likelihood! in this particular case, we want to ensure that the original example $x$ is most likely under the restricted Boltzmann machine induced by itself: $$\max_{\theta} \log p(x | T(x)),$$ where $\theta$ is the parameters for defining the compatibility function $s$ from above. we do so for all $N$ restricted Boltzmann machines induced from $N$ training examples: $$\max_{\theta} \frac{1}{N} \sum_{n=1}^N \log p(x_n | T(x_n)).$$ since it’s decomposed over the training examples, let’s consider only one example $x \in D$. we then train the induced restricted Boltzmann machine with stochastic gradient descent, following $$\frac{1}{M} \sum_{m=1}^M \nabla_{\theta} \log (1+\exp(s(x, \tilde{x}_m; \theta))) – \frac{1}{M} \sum_{m=1}^M \sum_{n=1}^N p(x_n|T(x)) \nabla_{\theta} \log (1+\exp(s(x_n, \tilde{x}_m; \theta))),$$ where we use $M$ transformed copies to approximate the two expectations over $T(x)$ but not $p(x_n|T(x))$. we probably should use another set of $M$ transformed copies to get the unbiased estimate. this does look quite similar to more recently popular variants of contrastive learning. we start from a training example $x$, generate a transformed version $\tilde{x}$, maximize the compatibility between $x$ and $\tilde{x}$, and minimize the compatibility between $\tilde{x}$ and all the training examples (including $x$). there are minor differences, such as the choice of nonlinearity, but at the high level, it turned out we can derive contrastive learning from the restricted Boltzmann machine. perhaps the only major difference is that this formulation gives us a clear guideline on how we should pick the negative examples. that is, according to this formula, we should either use all the training examples weighted according to how likely they are under this $x$-induced restricted Boltzmann machine or use a subset of training examples drawn according to the $x$-induced restricted Boltzmann machine without further weighting. of course, another alternative is to use uniformly-selected training examples as negative samples but weight them according to their probabilities under the $x$-induced restricted Boltzmann machine, à la importance sampling. so, yes, contrastive learning can be derived from restricted Boltzmann machines, and this is advantageous, because this tells us how we should pick negative examples. in fact, as i was writing this blog post (and an earlier internal slack message,) i was reminded of a recent workshop i’ve attended together with Yoshua Bengio. there was a talk on how to choose hard negative samples for contrastive learning (or representation learning) on graphs, and after the talk was over, Yoshua raised his hand and made this remark   That’s called Boltzmann machine learning!  Yoshua Bengio (2019, paraphrased)  Indeed… Data augmentation is what matters Based on this exercise of deriving modern contrastive learning from restricted Boltzmann machines, we can now have a meta-framework for coming up with a contrastive learning recipe. Any recipe must consist of three major ingredients:   A per-example density estimator: i used the restricted Boltzmann machine, but you may very well use variational autoencoders, independent component analysis, principal component analysis, sparse coding, etc. these will give rise to different variants of self-supervised learning. the latter three are particularly interesting, because they are fully described by a set of basis vectors and don’t require any negative samples for learning. i’m almost 100% certain you can derive all these non-contrastive learning algorithms by choosing one of these three.  A compatibility function $s$: this is the part where we design a network “architecture”, and how the output from this network is used to compute a scalar that indicates how similar a pair of examples is. it looks like the current practice is to use a deep neural net with a cosine similarity to implement this compatibility function.  A stochastic transformation generator: this generator effectively generates a density estimator for each example. this is very important, since it defines the set of bases used by these density estimators. any aspect of data cannot be modelled if these generated bases do not cover them.  we have a pretty good idea of what kind of density estimator is suitable for various purposes. we have a pretty good idea what’s the best way to measure the similarity between two highly-complex, high-dimensional inputs (thanks, deep learning!) but, we cannot know what the right stochastic transformation generator should be, because it is heavily dependent on the problem and domain. for instance, the optimal transformation generator for static, natural images won’t be optimal for e.g. natural language text. so, my sense is that the success of using contrastive learning (or any self-supervised learning) for any given problem will ultimately boil down to the choice and design of stochastic transformation, since there’s a chance that we may find a near-optimal pair of the first two (density estimator and compatibility function) that works well across multiple problems and domains.
Restricted Boltzmann machines or contrastive learning?
https://kyunghyuncho.me/restricted-boltzmann-machines-or-contrastive-learning/
https://www.geeksforgeeks.org/supervised-unsupervised-learning/
Supervised and Unsupervised learning - GeeksforGeeks
2017-10-01
GeeksforGeeks
<div><div><p><strong><u>Supervised learning</u></strong></p><p>Supervised learning, as the name indicates, has the presence of a supervisor as a teacher. Basically supervised learning is when we teach or train the machine using data that is well labelled. Which means some data is already tagged with the correct answer. After that, the machine is provided with a new set of examples(data) so that the supervised learning algorithm analyses the training data(set of training examples) and produces a correct outcome from labelled data.</p><p><strong>For instance</strong>, suppose you are given a basket filled with different kinds of fruits. Now the first step is to train the machine with all the different fruits one by one like this: </p><p></p><ul><li>If the shape of the object is rounded and has a depression at the top, is red in color, then it will be labeled as –<strong>Apple</strong>.</li><li>If the shape of the object is a long curving cylinder having Green-Yellow color, then it will be labeled as –<strong>Banana</strong>. </li></ul><p>Now suppose after training the data, you have given a new separate fruit, say Banana from the basket, and asked to identify it. <br /> </p><p></p><p>Since the machine has already learned the things from previous data and this time has to use it wisely. It will first classify the fruit with its shape and color and would confirm the fruit name as BANANA and put it in the Banana category. Thus the machine learns the things from training data(basket containing fruits) and then applies the knowledge to test data(new fruit). </p><p>Supervised learning is classified into two categories of algorithms: </p><ul><li><strong>Classification</strong>: A classification problem is when the output variable is a category, such as “Red” or “blue” , “disease” or “no disease”.</li><li><strong>Regression</strong>: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.</li></ul><p>Supervised learning deals with or learns with “labeled” data. This implies that some data is already tagged with the correct answer.</p><p><strong>Types:-</strong></p><ul><li>Regression</li><li>Logistic Regression</li><li>Classification</li><li>Naive Bayes Classifiers</li><li>K-NN (k nearest neighbors)</li><li>Decision Trees</li><li>Support Vector Machine</li></ul><p><strong>Advantages:-</strong></p><ul><li>Supervised learning allows collecting data and produces data output from previous experiences.</li><li>Helps to optimize performance criteria with the help of experience.</li><li>Supervised machine learning helps to solve various types of real-world computation problems.</li></ul><p><strong>Disadvantages:-</strong></p><ul><li>Classifying big data can be challenging.</li><li>Training for supervised learning needs a lot of computation time. So, it requires a lot of time.</li></ul><div><p></p><p>Steps</p></div><p><strong><u>Unsupervised learning</u></strong></p><p>Unsupervised learning is the training of a machine using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance. Here the task of the machine is to group unsorted information according to similarities, patterns, and differences without any prior training of data. </p><p>Unlike supervised learning, no teacher is provided that means no training will be given to the machine. Therefore the machine is restricted to find the hidden structure in unlabeled data by itself. <br /><strong>For instance</strong>, suppose it is given an image having both dogs and cats which it has never seen. <br /> </p><p></p><p>Thus the machine has no idea about the features of dogs and cats so we can’t categorize it as ‘dogs and cats ‘. But it can categorize them according to their similarities, patterns, and differences, i.e., we can easily categorize the above picture into two parts. The first may contain all pics having <strong>dogs</strong> in them and the second part may contain all pics having <strong>cats</strong> in them. Here you didn’t learn anything before, which means no training data or examples. </p><p>It allows the model to work on its own to discover patterns and information that was previously undetected. It mainly deals with unlabelled data.</p><p>Unsupervised learning is classified into two categories of algorithms: </p><ul><li><strong>Clustering</strong>: A clustering problem is where you want to discover the inherent groupings in the data, such as grouping customers by purchasing behavior.</li><li><strong>Association</strong>: An association rule learning problem is where you want to discover rules that describe large portions of your data, such as people that buy X also tend to buy Y.</li></ul><p>Types of Unsupervised Learning:-</p><p><strong>Clustering</strong></p><ol><li>Exclusive (partitioning)</li><li>Agglomerative</li><li>Overlapping</li><li>Probabilistic</li></ol><p><strong>Clustering Types:-</strong></p><ol><li>Hierarchical clustering</li><li>K-means clustering</li><li>Principal Component Analysis</li><li>Singular Value Decomposition</li><li>Independent Component Analysis</li></ol><p><strong>Supervised vs. Unsupervised Machine Learning</strong></p><figure><table><tbody><tr><td><strong>Parameters</strong></td><td>         <strong> Supervised machine learning</strong></td><td>           <strong> Unsupervised machine learning</strong></td></tr><tr><td>Input Data  </td><td>Algorithms are trained using labeled data.</td><td>Algorithms are used against data that is not labeled</td></tr><tr><td>Computational Complexity </td><td> Simpler method</td><td> Computationally complex</td></tr><tr><td>Accuracy</td><td>Highly accurate</td><td>Less accurate </td></tr><tr><td>No. of classes</td><td>No. of classes is known</td><td>No. of classes is not known</td></tr><tr><td>Data Analysis</td><td>Uses offline analysis</td><td>Uses real-time analysis of data</td></tr><tr><td>Algorithms used</td><td><p>Linear and Logistics regression, Random forest,</p><p>Support Vector Machine, Neural Network, etc.</p></td><td><p>K-Means clustering, Hierarchical clustering, </p><p>Apriori algorithm, etc.</p></td></tr></tbody></table></figure><p>This article is contributed by <a href="https://www.quora.com/profile/Shubham-Bansal-209">Shubham Bansal</a>. If you like GeeksforGeeks and would like to contribute, you can also write an article using <a href="http://write.geeksforgeeks.org/">write.geeksforgeeks.org</a> or mail your article to contribute@geeksforgeeksorg. See your article appearing on the GeeksforGeeks main page and help other Geeks.<br /> </p></div>||||I|||| Skip to content
 * Courses
 + For Working Professionals
 o DevOps(Live)
 o Data Structure & Algorithm Classes (Live)
 o System Design (Live)
 o Java Backend Developer (Live)
 o Full Stack Development with React & Node JS (Live)
 o Complete Data Science Program
 o Data Structure & Algorithm-Self Paced(C++/JAVA)
 o Data Structures & Algorithms in Python
 o Explore More Live Courses
 + For Students
 o Competitive Programming (Live)
 o Interview Preparation Course
 o Data Structure & Algorithm-Self Paced(C++/JAVA)
 o Data Structures & Algorithms in Python
 o Data Science (Live)
 o Full Stack Development with React & Node JS (Live)
 o GATE CS 2023 Test Series
 o OS DBMS CN for SDE Interview Preparation
 o Explore More Self-Paced Courses
 + Programming Languages
 o C++ Programming - Beginner to Advanced
 o Java Programming - Beginner to Advanced
 o C Programming - Beginner to Advanced
 o Python Foundation
 o JavaScript Foundation
 + Web Development
 o Full Stack Development with React & Node JS(Live)
 o Java Backend Development(Live)
 o React JS (Basic to Advanced)
 o JavaScript Foundation
 + Machine Learning and Data Science
 o Complete Data Science Program(Live)
 o Mastering Data Analytics
 + School Courses
 o CBSE Class 12 Computer Science
 o School Guide
 + All Courses
 * Tutorials
 + DSA
 + Data Structures
 o Arrays
 o Linked List
 o Stack
 o Queue
 o Binary Tree
 o Binary Search Tree
 o Heap
 o Hashing
 o Graph
 o Advanced Data Structure
 o Matrix
 o Strings
 o All Data Structures
 + Algorithms
 o Analysis of Algorithms
 - Asymptotic Analysis
 - Worst, Average and Best Cases
 - Asymptotic Notations
 - Little o and little omega notations
 - Lower and Upper Bound Theory
 - Analysis of Loops
 - Solving Recurrences
 - Amortized Analysis
 - What does 'Space Complexity' mean ?
 - Pseudo-polynomial Algorithms
 - Polynomial Time Approximation Scheme
 - A Time Complexity Question
 o Searching Algorithms
 o Sorting Algorithms
 o Graph Algorithms
 o Pattern Searching
 o Geometric Algorithms
 o Mathematical
 o Bitwise Algorithms
 o Randomized Algorithms
 o Greedy Algorithms
 o Dynamic Programming
 o Divide and Conquer
 o Backtracking
 o Branch and Bound
 o All Algorithms
 + Interview Corner
 o Company Preparation
 o Top Topics
 o Practice Company Questions
 o Interview Experiences
 o Experienced Interviews
 o Internship Interviews
 o Competitive Programming
 o Design Patterns
 o System Design Tutorial
 o Multiple Choice Quizzes
 + Languages
 o C
 o C++
 o Java
 o Python
 o JavaScript
 o PHP
 o C#
 o SQL
 o Scala
 o Perl
 o Go Language
 o Kotlin
 + Web Development
 o HTML
 o CSS
 o JavaScript
 o PHP
 o CSS Frameworks
 - Bootstrap
 - Tailwind CSS
 - Foundation CSS
 - Materialize CSS
 - Bulma
 - Pure CSS
 - Primer CSS
 - Blaze UI
 - Semantic UI
 o JavaScript Frameworks
 - AngularJS
 - Angular PrimeNG
 - Angular ngx Bootstrap
 - NodeJS
 - Express.js
 o JavaScript Libraries
 - jQuery
 - jQuery Mobile
 - jQuery UI
 - jQuery EasyUI
 - jQWidgets
 - ReactJS
 - React Bootstrap
 - React Rebass
 - React Desktop
 - React Suite
 - ReactJS Evergreen
 - ReactJS Reactstrap
 - Ant Design
 - BlueprintJS
 - p5.js
 - Lodash
 - TensorFlow.js
 - Moment.js
 - Collect.js
 o WordPress
 o JSON
 + School Learning
 o English Grammar
 o School Programming
 o Mathematics
 - Number System
 - Algebra
 - Trigonometry
 - Statistics
 - Probability
 - Geometry
 - Mensuration
 - Calculus
 o Maths Notes (Class 8-12)
 - Class 8 Notes
 - Class 9 Notes
 - Class 10 Notes
 - Class 11 Notes
 - Class 12 Notes
 o NCERT Solutions
 - Class 8 Maths Solution
 - Class 9 Maths Solution
 - Class 10 Maths Solution
 - Class 11 Maths Solution
 - Class 12 Maths Solution
 o RD Sharma Solutions
 - Class 8 Maths Solution
 - Class 9 Maths Solution
 - Class 10 Maths Solution
 - Class 11 Maths Solution
 - Class 12 Maths Solution
 o Physics Notes (Class 8-11)
 - Class 8 Notes
 - Class 9 Notes
 - Class 10 Notes
 - Class 11 Notes
 o Chemistry Notes
 - Class 8 Notes
 - Class 9 Notes
 - Class 10 Notes
 o Business Studies (Class 11th)
 o Business Studies (Class 12th)
 o Accountancy (Class 12th)
 o Statistics for Economics (Class 11th)
 + ML & Data Science
 o Machine Learning
 o Data Science
 + CS Subjects
 o Mathematics
 o Operating System
 o DBMS
 o Computer Networks
 o Computer Organization and Architecture
 o Theory of Computation
 o Compiler Design
 o Digital Logic
 o Software Engineering
 + GATE
 o GATE 2023 Test Series
 o GATE Computer Science Notes
 o Last Minute Notes
 o GATE CS Solved Papers
 o GATE CS Original Papers and Official Keys
 o GATE CS 2023 Syllabus
 o Important Topics for GATE CS
 o GATE 2023 Important Dates
 + Software Designs
 o Software Design Patterns
 o System Design Tutorial
 + GFG Sheets
 o Web Dev Cheat Sheets
 - HTML Cheat Sheet
 - CSS Cheat Sheet
 - Bootstrap Cheat Sheet
 - JS Cheat Sheet
 - jQuery Cheat Sheet
 - Angular Cheat Sheet
 o Company-Wise SDE Sheets
 - Facebook SDE Sheet
 - Amazon SDE Sheet
 - Apple SDE Sheet
 - Netflix SDE Sheet
 - Google SDE Sheet
 - Wipro Coding Sheet
 - Infosys Coding Sheet
 - TCS Coding Sheet
 - Cognizant Coding Sheet
 - HCL Coding Sheet
 o DSA Sheets
 - SDE Sheet
 - FAANG Coding Sheet
 - Love Babbar Sheet
 - Mass Recruiter Sheet
 - Product-Based Coding Sheet
 - Company-Wise Preparation Sheet
 - Array Sheet
 - String Sheet
 - Tree Sheet
 - Graph Sheet
 - DP Sheet
 + CS Exams/PSUs
 o ISRO
 - ISRO CS Original Papers and Official Keys
 - ISRO CS Solved Papers
 - ISRO CS Syllabus for Scientist/Engineer Exam
 o UGC NET
 - UGC NET CS Notes Paper II
 - UGC NET CS Notes Paper III
 - UGC NET CS Solved Papers
 + Student
 o Campus Ambassador Program
 o School Ambassador Program
 o Project
 o Geek of the Month
 o Campus Geek of the Month
 o Placement Course
 o Competitive Programming
 o Testimonials
 o Student Chapter
 o Geek on the Top
 o Internship
 o Careers
 + UPSC
 o Geography Notes
 o History Notes
 o Science & Tech. Notes
 o Ethics Notes
 o Polity Notes
 o Economics Notes
 o UPSC Previous Year Papers
 + SSC CGL
 o SSC CGL Syllabus
 o General Studies
 o English
 o Reasoning
 o Subjectwise Practice Papers
 o Previous Year Papers
 + Banking Exams
 o SBI Clerk
 - SBI Clerk Syllabus
 - General Awareness
 - English
 - Quantitative Aptitude
 - Reasoning Ability
 - SBI Clerk Practice Papers
 o SBI PO
 - SBI PO Syllabus
 - General Awareness
 - English
 - Quantitative Aptitude
 - Reasoning Ability
 - Previous Year Papers
 - SBI PO Practice Papers
 o IBPS PO
 - IBPS PO 2022 Syllabus
 - English Notes
 - Reasoning Notes
 - Previous Year Papers
 - Mock Question Papers
 o IBPS Clerk
 - IBPS Clerk Syllabus
 - English Notes
 - Previous Year Papers
 * Jobs
 + Apply for a Job
 + Apply through Jobathon
 + Post a Job
 + Hire through Jobathon
 * Practice
 + All DSA Problems
 + Problem of the Day
 + GFG SDE Sheet
 + Curated DSA Lists
 o Top 50 Array Problems
 o Top 50 String Problems
 o Top 50 Tree Problems
 o Top 50 Graph Problems
 o Top 50 DP Problems
 * Contests
 + GFG Weekly Coding Contest
 + Job-A-Thon: Hiring Challenge
 + BiWizard School Contest
 + All Contests and Events
 * 
 * 
 * 
 * Home
 * Saved Videos
 * Courses
 * 
 * GBlog
 * Puzzles
 * What's New ?
 Change Language
 * Data Structures and Algorithms
 * Interview Preparation
 * Data Science
 * Topic-wise Practice
 * C
 * C++
 * Java
 * Python
 * Latest Blogs
 * Competitive Programming
 * JavaScript
 * Machine Learning
 * Write & Earn
 * Web Development
 * Puzzles
 * Projects
 Related Articles
▲
 Open in App
 Not now

 Related Articles

 Write an Article
 * Write an Interview Experience
 * Analysis of test data using K-Means Clustering in Python
 * Gaussian Mixture Model
 * ML | Independent Component Analysis
 * ML | Spectral Clustering
 * Markov Decision Process
 * Reinforcement learning
 * Supervised and Unsupervised learning
 * ML | Types of Learning – Supervised Learning
 * ML | Types of Learning – Part 2
 * Implementation of K Nearest Neighbors
 * K-Nearest Neighbours
 * K means Clustering – Introduction
 * Clustering in Machine Learning
 * Different Types of Clustering Algorithm
 Write an Article
 * Write an Interview Experience
 * SDE SHEET - A Complete Guide for SDE Preparation
 * Naive Bayes Classifiers
 * Removing stop words with NLTK in Python
 * Linear Regression (Python Implementation)
 * Apriori Algorithm
 * Decision Tree
 * Software Engineering | Coupling and Cohesion
 * ML | Linear Regression
 * Decision Tree Introduction with example
 * What is Algorithm | Introduction to Algorithms
 * Python | Decision tree implementation
 * KDD Process in Data Mining
 * Getting started with Machine Learning
 * How to Run a Python Script using Docker?
 Write an Article
 * Write an Interview Experience
 * Analysis of test data using K-Means Clustering in Python
 * Gaussian Mixture Model
 * ML | Independent Component Analysis
 * ML | Spectral Clustering
 * Markov Decision Process
 * Reinforcement learning
 * Supervised and Unsupervised learning
 * ML | Types of Learning – Supervised Learning
 * ML | Types of Learning – Part 2
 * Implementation of K Nearest Neighbors
 * K-Nearest Neighbours
 * K means Clustering – Introduction
 * Clustering in Machine Learning
 * Different Types of Clustering Algorithm
 Write an Article
 * Write an Interview Experience
 * SDE SHEET - A Complete Guide for SDE Preparation
 * Naive Bayes Classifiers
 * Removing stop words with NLTK in Python
 * Linear Regression (Python Implementation)
 * Apriori Algorithm
 * Decision Tree
 * Software Engineering | Coupling and Cohesion
 * ML | Linear Regression
 * Decision Tree Introduction with example
 * What is Algorithm | Introduction to Algorithms
 * Python | Decision tree implementation
 * KDD Process in Data Mining
 * Getting started with Machine Learning
 * How to Run a Python Script using Docker?

 Supervised and Unsupervised learning

 Improve Article
 Save Article
 * Difficulty Level : Easy
 * Last Updated : 10 Jan, 2023
 * Read
 * Discuss
 Improve Article
 Save Article

 Supervised learning

 Supervised learning, as the name indicates, has the presence of a supervisor as a teacher. Basically supervised learning is when we teach or train the machine using data that is well labelled. Which means some data is already tagged with the correct answer. After that, the machine is provided with a new set of examples(data) so that the supervised learning algorithm analyses the training data(set of training examples) and produces a correct outcome from labelled data.

 For instance, suppose you are given a basket filled with different kinds of fruits. Now the first step is to train the machine with all the different fruits one by one like this:

 * If the shape of the object is rounded and has a depression at the top, is red in color, then it will be labeled as –Apple.
 * If the shape of the object is a long curving cylinder having Green-Yellow color, then it will be labeled as –Banana.

 Now suppose after training the data, you have given a new separate fruit, say Banana from the basket, and asked to identify it.

 Since the machine has already learned the things from previous data and this time has to use it wisely. It will first classify the fruit with its shape and color and would confirm the fruit name as BANANA and put it in the Banana category. Thus the machine learns the things from training data(basket containing fruits) and then applies the knowledge to test data(new fruit).

 Supervised learning is classified into two categories of algorithms:

 * Classification: A classification problem is when the output variable is a category, such as “Red” or “blue” , “disease” or “no disease”.
 * Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.

 Supervised learning deals with or learns with “labeled” data. This implies that some data is already tagged with the correct answer.

 Types:-

 * Regression
 * Logistic Regression
 * Classification
 * Naive Bayes Classifiers
 * K-NN (k nearest neighbors)
 * Decision Trees
 * Support Vector Machine

 Advantages:-

 * Supervised learning allows collecting data and produces data output from previous experiences.
 * Helps to optimize performance criteria with the help of experience.
 * Supervised machine learning helps to solve various types of real-world computation problems.

 Disadvantages:-

 * Classifying big data can be challenging.
 * Training for supervised learning needs a lot of computation time. So, it requires a lot of time.

 Steps

 Unsupervised learning

 Unsupervised learning is the training of a machine using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance. Here the task of the machine is to group unsorted information according to similarities, patterns, and differences without any prior training of data.

 Unlike supervised learning, no teacher is provided that means no training will be given to the machine. Therefore the machine is restricted to find the hidden structure in unlabeled data by itself.
 For instance, suppose it is given an image having both dogs and cats which it has never seen.

 Thus the machine has no idea about the features of dogs and cats so we can’t categorize it as ‘dogs and cats ‘. But it can categorize them according to their similarities, patterns, and differences, i.e., we can easily categorize the above picture into two parts. The first may contain all pics having dogs in them and the second part may contain all pics having cats in them. Here you didn’t learn anything before, which means no training data or examples.

 It allows the model to work on its own to discover patterns and information that was previously undetected. It mainly deals with unlabelled data.

 Unsupervised learning is classified into two categories of algorithms:

 * Clustering: A clustering problem is where you want to discover the inherent groupings in the data, such as grouping customers by purchasing behavior.
 * Association: An association rule learning problem is where you want to discover rules that describe large portions of your data, such as people that buy X also tend to buy Y.

 Types of Unsupervised Learning:-

 Clustering

 1. Exclusive (partitioning)
 2. Agglomerative
 3. Overlapping
 4. Probabilistic

 Clustering Types:-

 1. Hierarchical clustering
 2. K-means clustering
 3. Principal Component Analysis
 4. Singular Value Decomposition
 5. Independent Component Analysis

 Supervised vs. Unsupervised Machine Learning

 Parameters Supervised machine learning Unsupervised machine learning 
 Input Data Algorithms are trained using labeled data. Algorithms are used against data that is not labeled
 Computational Complexity Simpler method Computationally complex 
 Accuracy Highly accurate Less accurate 
 No. of classes No. of classes is known No. of classes is not known 
 Data Analysis Uses offline analysis Uses real-time analysis of data 
 Linear and Logistics regression, Random forest, K-Means clustering, Hierarchical clustering, 
 Algorithms used  
 Support Vector Machine, Neural Network, etc. Apriori algorithm, etc. 
   
 

 This article is contributed by Shubham Bansal. If you like GeeksforGeeks and would like to contribute, you can also write an article using write.geeksforgeeks.org or mail your article to contribute@geeksforgeeksorg. See your article appearing on the GeeksforGeeks main page and help other Geeks.

 My Personal Notes arrow_drop_up
 Save

 Please Login to comment...

 Like

 Previous
 Reinforcement learning
 Next
 ML | Types of Learning – Supervised Learning
 Related Articles
 1. Difference between Supervised and Unsupervised Learning
 2. ML | Types of Learning – Supervised Learning
 3. Customer Segmentation using Unsupervised Machine Learning in Python
 4. Unsupervised Machine Learning - The Future of Cybersecurity
 5. Regression and Classification | Supervised Machine Learning
 6. ALBERT - A Light BERT for Supervised Learning
 7. Self-Supervised Learning (SSL)
 8. ML | Semi-Supervised Learning
 9. ML | Unsupervised Face Clustering Pipeline
 10. What is Supervised Dataset in PyBrain?
 Article Contributed By :
 GeeksforGeeks
 Vote for difficulty
 Current difficulty : Easy


 Easy Normal Medium Hard Expert

 Improved By :
 * Akanksha_Rai
 * PratyakshaSrivastava
 * sharmaanushka
 * spodenten
 * adityapardeshi
 * marcosarcticseal
 * adityakumar129
 * farzams101
 * yashbargale72
 Article Tags :
 * Computer Subject
 * Machine Learning
 Practice Tags :
 * Machine Learning
 Report Issue
 What's New
 Master CS Subjects For SDE Preparation
 View Details
 Complete Interview Preparation Course
 View Details
 Master Data Science & ML
 View Details
 System Design- Live Classes
 View Details

 Improve your Coding Skills with Practice

 Try It!
 A-143, 9th Floor, Sovereign Corporate Tower,
 Sector-136, Noida, Uttar Pradesh - 201305
 feedback@geeksforgeeks.org
 * Company
 * About Us
 * Careers
 * In Media
 * Contact Us
 * Privacy Policy
 * Copyright Policy
 * Advertise with us
 * Learn
 * DSA
 * Algorithms
 * Data Structures
 * SDE Cheat Sheet
 * Machine learning
 * CS Subjects
 * Video Tutorials
 * Courses
 * News
 * Top News
 * Technology
 * Work & Career
 * Business
 * Finance
 * Lifestyle
 * Knowledge
 * Languages
 * Python
 * Java
 * CPP
 * Golang
 * C#
 * SQL
 * Kotlin
 * Web Development
 * Web Tutorials
 * Django Tutorial
 * HTML
 * JavaScript
 * Bootstrap
 * ReactJS
 * NodeJS
 * Contribute
 * Write an Article
 * Improve an Article
 * Pick Topics to Write
 * Write Interview Experience
 * Internships
 * Video Internship
 @geeksforgeeks , Some rights reserved
 We use cookies to ensure you have the best browsing experience on our website. By using our site, you acknowledge that you have read and understood our Cookie Policy & Privacy Policy Got It !

 Start Your Coding Journey Now!

 Login Register
Supervised learning   Supervised learning, as the name indicates, has the presence of a supervisor as a teacher. Basically supervised learning is when we teach or train the machine using data that is well labelled. Which means some data is already tagged with the correct answer. After that, the machine is provided with a new set of examples(data) so that the supervised learning algorithm analyses the training data(set of training examples) and produces a correct outcome from labelled data.  For instance, suppose you are given a basket filled with different kinds of fruits. Now the first step is to train the machine with all the different fruits one by one like this:     If the shape of the object is rounded and has a depression at the top, is red in color, then it will be labeled as –Apple. If the shape of the object is a long curving cylinder having Green-Yellow color, then it will be labeled as –Banana.   Now suppose after training the data, you have given a new separate fruit, say Banana from the basket, and asked to identify it.     Since the machine has already learned the things from previous data and this time has to use it wisely. It will first classify the fruit with its shape and color and would confirm the fruit name as BANANA and put it in the Banana category. Thus the machine learns the things from training data(basket containing fruits) and then applies the knowledge to test data(new fruit).  Supervised learning is classified into two categories of algorithms:    Classification: A classification problem is when the output variable is a category, such as “Red” or “blue” , “disease” or “no disease”.  Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.  Supervised learning deals with or learns with “labeled” data. This implies that some data is already tagged with the correct answer.  Types:-   Regression Logistic Regression Classification Naive Bayes Classifiers K-NN (k nearest neighbors) Decision Trees Support Vector Machine   Advantages:-   Supervised learning allows collecting data and produces data output from previous experiences. Helps to optimize performance criteria with the help of experience. Supervised machine learning helps to solve various types of real-world computation problems.   Disadvantages:-   Classifying big data can be challenging. Training for supervised learning needs a lot of computation time. So, it requires a lot of time.     Steps    Unsupervised learning   Unsupervised learning is the training of a machine using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance. Here the task of the machine is to group unsorted information according to similarities, patterns, and differences without any prior training of data.  Unlike supervised learning, no teacher is provided that means no training will be given to the machine. Therefore the machine is restricted to find the hidden structure in unlabeled data by itself.  For instance, suppose it is given an image having both dogs and cats which it has never seen.     Thus the machine has no idea about the features of dogs and cats so we can’t categorize it as ‘dogs and cats ‘. But it can categorize them according to their similarities, patterns, and differences, i.e., we can easily categorize the above picture into two parts. The first may contain all pics having dogs in them and the second part may contain all pics having cats in them. Here you didn’t learn anything before, which means no training data or examples.  It allows the model to work on its own to discover patterns and information that was previously undetected. It mainly deals with unlabelled data. Unsupervised learning is classified into two categories of algorithms:    Clustering: A clustering problem is where you want to discover the inherent groupings in the data, such as grouping customers by purchasing behavior.  Association: An association rule learning problem is where you want to discover rules that describe large portions of your data, such as people that buy X also tend to buy Y.  Types of Unsupervised Learning:-  Clustering   Exclusive (partitioning) Agglomerative Overlapping Probabilistic   Clustering Types:-   Hierarchical clustering K-means clustering Principal Component Analysis Singular Value Decomposition Independent Component Analysis   Supervised vs. Unsupervised Machine Learning       Parameters            Supervised machine learning              Unsupervised machine learning    Input Data   Algorithms are trained using labeled data. Algorithms are used against data that is not labeled   Computational Complexity   Simpler method  Computationally complex   Accuracy Highly accurate Less accurate    No. of classes No. of classes is known No. of classes is not known   Data Analysis Uses offline analysis Uses real-time analysis of data   Algorithms used  Linear and Logistics regression, Random forest, Support Vector Machine, Neural Network, etc.   K-Means clustering, Hierarchical clustering,  Apriori algorithm, etc.      This article is contributed by Shubham Bansal. If you like GeeksforGeeks and would like to contribute, you can also write an article using write.geeksforgeeks.org or mail your article to contribute@geeksforgeeksorg. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Supervised and Unsupervised learning - GeeksforGeeks
https://www.geeksforgeeks.org/supervised-unsupervised-learning/
https://www.languagehumanities.org/what-is-contrastive-linguistics.htm
What Is Contrastive Linguistics?
2023-01-27
Mark Wollacott
<div><section>
 
 <p>Contrastive <a href="//www.languagehumanities.org/what-is-linguistics.htm">linguistics</a> seeks to study and explain any two languages. This includes listing the differences and similarities between them. Contrastive linguistics has also been called differential linguistics and is a sub-section of comparative linguistics that is separated by its studying of only two languages at any one time.</p>
 
 
 <p>This area studies languages in order to discern how they developed as they did and what other languages they are historically related to. For example, comparative linguistic studies of Hungarian show its earliest links to Chinese and Korean, then how it was influenced by Mongolian, Turkish and other languages as the Magyars moved west across Siberia and eventually into Europe. Such studies have also shown how and when Hungarian split from its closest language partner, Finno-Estonian.</p>
 
 
 
 
 
 
 
 
 <div>
 
 <figure>
 <a href="#imagePopup">
 
 <figcaption>
 Contrastive linguistics seeks to study and explain any two languages. </figcaption>
 </a>
 </figure>
 
 <p>There are many subdivisions of comparative linguistics and, therefore, also of contrastive linguistics as they use similar techniques. The discipline is traditionally broken down into two main groups: general comparative linguistics and specialized comparative linguistics. General comparative linguistics is broken down into descriptive, typological and historical linguistics, while specialized comparative linguistics is broken down into generic comparative, the <a href="//www.allthescience.org/what-is-a-theory.htm">theory</a> of linguistic contact and areal linguistics. </p></div>
 
 
 
 
 
 
 
 
 <p>The theory of linguistic contact becomes more important during contrastive linguistic studies. It looks at the relationship of two languages. Not all languages studied in contrastive linguistics are related or have had contact with one another, but it allows the <a href="//www.languagehumanities.org/what-is-a-linguist.htm">linguist</a> to look at possible changes one language has influenced in another such as transfers and interference. This is known as the theory of bilingualism and includes the creation of creoles and translation.</p>
 
 
 
 
 
 
 
 
 <p>Both comparative and contrastive linguistics look at similar areas of a language. This includes the vocabulary or words used by the language and how those words are affected when they are pluralized or inflected. They also examine how a language uses <a href="//www.languagehumanities.org/what-is-syntax.htm">syntax</a> to form sentences, grammar to organize words and sentences, <a href="//www.languagehumanities.org/what-is-phonology.htm">phonology</a> and also how culture creates idioms.</p>
 
 
 
 
 
 
 
 
 <p>Contrastive linguistics was first developed in the 1950s. It was based on the ideas of linguistic <a href="//www.languagehumanities.org/what-is-structuralism.htm">structuralism</a> and was initially aimed not at linguistic studies, but at helping <a href="//www.languagehumanities.org/what-is-a-foreign-language.htm">foreign language</a> teachers. This aim was intended to make it easier to understand learning a <a href="//www.languagehumanities.org/what-is-a-second-language.htm">second language</a> and how to teach it most effectively. Contrastive linguistics led to large-scale linguistic projects across Europe in the 1970s before moving into academia.</p>
 
 
 
 
 
 
 
 
 <p>The study of contrastive linguistics follows four basic procedures. The first is to identify the two languages being studied. The second requires a full description of the characteristics of each language. Third, the scholar looks for juxtaposition; bonds between the two languages. In the fourth, the scholar compares the two languages to see how they correspond with one another.</p>
 
 
 
 
 
 
 
 
 
 </section>||||I|||| * Education
 * Language
 * Health
 * About
 * Contact Us
 * 
 * 
 * Categories
 * Education
 * Language
 * Health
 * Pages
 * More Articles
 * About Us
 * Contact Us
 * Privacy Policy
 * Terms and Conditions

 Join the Community

 Subscribe to our newsletter and learn something new every day.

 Language
 Fact Checked

 What Is Contrastive Linguistics?

 Mark Wollacott
 Last Modified Date: January 27, 2023
 Mark Wollacott
 Last Modified Date: January 27, 2023

 Contrastive linguistics seeks to study and explain any two languages. This includes listing the differences and similarities between them. Contrastive linguistics has also been called differential linguistics and is a sub-section of comparative linguistics that is separated by its studying of only two languages at any one time.

 This area studies languages in order to discern how they developed as they did and what other languages they are historically related to. For example, comparative linguistic studies of Hungarian show its earliest links to Chinese and Korean, then how it was influenced by Mongolian, Turkish and other languages as the Magyars moved west across Siberia and eventually into Europe. Such studies have also shown how and when Hungarian split from its closest language partner, Finno-Estonian.

 Contrastive linguistics seeks to study and explain any two languages.

 There are many subdivisions of comparative linguistics and, therefore, also of contrastive linguistics as they use similar techniques. The discipline is traditionally broken down into two main groups: general comparative linguistics and specialized comparative linguistics. General comparative linguistics is broken down into descriptive, typological and historical linguistics, while specialized comparative linguistics is broken down into generic comparative, the theory of linguistic contact and areal linguistics.

 The theory of linguistic contact becomes more important during contrastive linguistic studies. It looks at the relationship of two languages. Not all languages studied in contrastive linguistics are related or have had contact with one another, but it allows the linguist to look at possible changes one language has influenced in another such as transfers and interference. This is known as the theory of bilingualism and includes the creation of creoles and translation.

 Is Amazon actually giving you a competitive price? This little known plugin reveals the answer.

 Both comparative and contrastive linguistics look at similar areas of a language. This includes the vocabulary or words used by the language and how those words are affected when they are pluralized or inflected. They also examine how a language uses syntax to form sentences, grammar to organize words and sentences, phonology and also how culture creates idioms.

 Contrastive linguistics was first developed in the 1950s. It was based on the ideas of linguistic structuralism and was initially aimed not at linguistic studies, but at helping foreign language teachers. This aim was intended to make it easier to understand learning a second language and how to teach it most effectively. Contrastive linguistics led to large-scale linguistic projects across Europe in the 1970s before moving into academia.

 The study of contrastive linguistics follows four basic procedures. The first is to identify the two languages being studied. The second requires a full description of the characteristics of each language. Third, the scholar looks for juxtaposition; bonds between the two languages. In the fourth, the scholar compares the two languages to see how they correspond with one another.

 You might also Like

 Recommended

 AS FEATURED ON:

 AS FEATURED ON:

 Related Articles

 * What Is an Idiolect?
 * What Is Semantic Analysis?
 * What Is Double Discourse?
 * What Is Plurilingualism?
 * What Is Indirect Discourse?
 * What Is Romanization?
 * What Is a Conjunct?

 Discussion Comments

 Melonlity
 March 4, 2014

 One could actually do a pretty good comparison of American English vs. British English. Both variants come from the same roots, but spelling, slang and other things have differed over the years. For a quick example, a flashlight is called a torch in England, but a torch is something completely different in the United States.

 Post your comments

 Please enter the following code:

 Login:

 Forgot password?

 Register:

 * By: Focus Pocus LTD
 Contrastive linguistics seeks to study and explain any two languages.

 * Categories
 * Education
 * Language
 * Health
 * Copyright 2003 - 2023
 * Conjecture Corporation
 * Pages
 * About Us
 * Contact Us
 * Privacy Policy
 * Terms and Conditions

 Subscribe to our newsletter and learn something new every day.

Contrastive linguistics seeks to study and explain any two languages. This includes listing the differences and similarities between them. Contrastive linguistics has also been called differential linguistics and is a sub-section of comparative linguistics that is separated by its studying of only two languages at any one time.
 
 
 This area studies languages in order to discern how they developed as they did and what other languages they are historically related to. For example, comparative linguistic studies of Hungarian show its earliest links to Chinese and Korean, then how it was influenced by Mongolian, Turkish and other languages as the Magyars moved west across Siberia and eventually into Europe. Such studies have also shown how and when Hungarian split from its closest language partner, Finno-Estonian.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Contrastive linguistics seeks to study and explain any two languages. 
 
 
 
 There are many subdivisions of comparative linguistics and, therefore, also of contrastive linguistics as they use similar techniques. The discipline is traditionally broken down into two main groups: general comparative linguistics and specialized comparative linguistics. General comparative linguistics is broken down into descriptive, typological and historical linguistics, while specialized comparative linguistics is broken down into generic comparative, the theory of linguistic contact and areal linguistics.  
 
 
 
 
 
 
 
 
 The theory of linguistic contact becomes more important during contrastive linguistic studies. It looks at the relationship of two languages. Not all languages studied in contrastive linguistics are related or have had contact with one another, but it allows the linguist to look at possible changes one language has influenced in another such as transfers and interference. This is known as the theory of bilingualism and includes the creation of creoles and translation.
 
 
 
 
 
 
 
 
 Both comparative and contrastive linguistics look at similar areas of a language. This includes the vocabulary or words used by the language and how those words are affected when they are pluralized or inflected. They also examine how a language uses syntax to form sentences, grammar to organize words and sentences, phonology and also how culture creates idioms.
 
 
 
 
 
 
 
 
 Contrastive linguistics was first developed in the 1950s. It was based on the ideas of linguistic structuralism and was initially aimed not at linguistic studies, but at helping foreign language teachers. This aim was intended to make it easier to understand learning a second language and how to teach it most effectively. Contrastive linguistics led to large-scale linguistic projects across Europe in the 1970s before moving into academia.
 
 
 
 
 
 
 
 
 The study of contrastive linguistics follows four basic procedures. The first is to identify the two languages being studied. The second requires a full description of the characteristics of each language. Third, the scholar looks for juxtaposition; bonds between the two languages. In the fourth, the scholar compares the two languages to see how they correspond with one another.
What Is Contrastive Linguistics?
https://www.languagehumanities.org/what-is-contrastive-linguistics.htm