How to shape the weights or nodes during gradient training of neural network? Training with constraints? Gradient training changes indiscriminately all the weights and nodes of the neural network. But one can imagine the situations when the training should be shaped, e.g.:

One can put constraints on some of the weights. E.g. human brain contains regions whose inner connections are more dense that external connections with different regions. One can try to mimic this region-shaped structure in neural networks as well and hence one can require that inter-regional weights (in opposit to intra-regional weights) are close to zero (except, possibly, for some channels among regions);
One can put constraints on some of the weights in such manner that some layer of neurons have specific structure. E.g. consider the popular encoder-decoder architecture of neural machine translation e.g. https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html We can see that that the whole output of the encoder is expressed as a single layer of neurons which is forwarded to the input of the decoder. So - one can require that the set of all the possible outputs of the encoder (e.g. the possible values of this single layer of the neurons) forms some kind of structure, e.g. some grammar of some interlingua. This example is for illustration only, I have in mind more complex neural network which has one layer of neurons which indeed should output the encoded words of some interlingua. So, one is required to guid all the weights of the encoder in such manner that this single layer has only allowable values.

So - my question is - are there methods that guide the gradient descent training with additional information about the weights or about the nodes (i.e. about the whole subsets of weights that have some impact on specific layer of nodes)? E.g. about methods that impress the region structure on the neural network or that constrains the values of some nodes to be in specific range only?

Of course, it is quite easy to include such constraints in evolutionary neural networks - one can simply reject the neural networks with weights that violates the constraints. But is it possible to do this in gradient-like training?

I have a somewhat similar problem, there's this paper that I was supposed to extend (I can explain how): https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1984-2
What they do is that they have a connection network between input and second layer (giving some biological context to training) I don't know how you can maintain the zero edges during back propagation though.


Reading a value of a real gauge I have a simple gauge displaying analog values ranging from 0 to 4.
Here is an image of the gauge. Unfortunately there is no way to get a analog or digital signal for the value.

How do I read the value of the gauge?

My idea is to make an image every 5 minutes and get the value by analyzing it. I am thinking of manually generating reference images with the black needle in different positions and then compare it to the real image.

Since all the processing should be done on a raspberry pi without internet connectivity, a good approach would be a preconfigured docker image for image comparison which helps doing the image comparison locally, maybe supported by a python or php script.

How to proceed?

This is a simple digital image processing task. I am proposing a quite efficient solution without any error correction mechanism, so this is not a production quality approach. But as a first proof of concept it should do the trick.

The easiest approach would be splitting the image in two channels - one for the color red and one for the color black. You can neglect the rest outside a certain threshold. If the lighting is good, this should also deal with the background.

Step 1: Crop away everything outside the gauge to simplify your life. The easiest way to do this is just putting the camera close enough that it only captures the gauge. That way you don't have to implement it in your algorithm.

Step 2: The center of your image will be the red dot in the middle of the gauge. You can easily identify it with a filter looking for the largest concentration of red in the red channel of your image. Calculate the center and remember the coordinates.

Step 3: Do the same for the largest concentration of black in the black channel. This will be the thick end part of the black needle. Calculate the center as well and remember the coordinates.

Step 4: Calculate the vector between those two coordinates. The angle of the vector will tell you the angle of the needle (take care of the direction of the vector). Once you have the angle of the needle, you can easily derive the number it is pointing to.

There are obviously much more elegant approaches to solve this problem, but this quick and dirty algorithm should do the trick without a lot of hassle. You need to make sure that the camera is positioned in the right angle, because we haven't implemented any error corrections in our algorithm. You also need to make sure the lighting is sufficient and that there aren't any other significant influences on the image quality. As long as the orientation is stable and the image quality high, this should enable you to read the gauge with good precision.


Digital Rights and Agents talking to humans How does the legal question about agents talking to humans via telephone connection work? Recently Google gave a talk about Duplex, where an agent makes a call to a human to schedule a hairdresser.

I wonder if there are any regulations related to this type of scenario, if there are some limitations, if the human needs to know that he is talking to an AI.

Googling this throws a lot of debate on this issue of whether the call made to the restaurant for booking was legal or not. I found this article to put forward a lot of ideas for and against this. So it is for us to decide.

But one thing that I fully agree with you, which should be made into a regulation, is human should know that he/she is talking to a bot. This tweet by @traviskorte reasons it out why:

We should make AI sound different from humans for the same reason we put a smelly additive in normally odorless natural gas.


What should load_mask() return if an image doesn't have any objects? (Mask RCNN) I want to use Mask RCNN to do image segmentation. I need to override the load_mask function for the dataset class. I know this function should return mask tensors and class ids of objects in an image. But what should the function return if there is no object (only background) in this image?

It should return an empty mask. Like

mask = np.empty([0, 0, 0])
class_ids = np.empty([0], np.int32)


Is batch normalization not suitable for non-gaussian input? I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and   the other without BN.

I find that the model DNN with BN can't predict well.

The codes is shown as follow:

import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense,Dropout,Activation, BatchNormalization

np.random.seed(1)

# generate non-gaussian data
def generate_data():
    distribution = scipy.stats.gengamma(1, 70, loc=10, scale=100)
    x = distribution.rvs(size=10000)
    # plt.hist(x)
    # plt.show()
    print ('[mean, var, skew, kurtosis]', distribution.stats('mvsk'))

    y = np.sin(x) + np.cos(x) + np.sqrt(x)
    plt.hist(y)
    # plt.show()
    # print(y)
    return x ,y

x, y = generate_data()

x_train = x[:int(len(x)*0.8)]
y_train = y[:int(len(y)*0.8)]
x_test = x[int(len(x)*0.8):]
y_test = y[int(len(y)*0.8):]

def DNN(input_dim, output_dim, useBN = True):
    '''
    定义一个DNN model
    '''
    model=Sequential()

    model.add(Dense(128,input_dim= input_dim))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))

    model.add(Dense(50))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))

    model.add(Dense(output_dim))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('relu'))

    model.compile(loss= 'mse', optimizer= 'adam')
    return model

clf = DNN(1, 1, useBN = True)
clf.fit(x_train, y_train, epochs= 30, batch_size = 100, verbose=2, validation_data = (x_test, y_test))

y_pred = clf.predict(x_test)
def mse(y_pred, y_test):
    return np.mean(np.square(y_pred - y_test))
print('final result', mse(y_pred, y_test))

The input x is like this shape:

If I add BN layers, the result is shown as follows:

Epoch 27/30
 - 0s - loss: 56.2231 - val_loss: 47.5757
Epoch 28/30
 - 0s - loss: 55.1271 - val_loss: 60.4838
Epoch 29/30
 - 0s - loss: 53.9937 - val_loss: 87.3845
Epoch 30/30
 - 0s - loss: 52.8232 - val_loss: 47.4544
final result 48.204881459013244

If I don't add BN layers, the predicted result is better:

Epoch 27/30
 - 0s - loss: 2.6863 - val_loss: 0.8924
Epoch 28/30
 - 0s - loss: 2.6562 - val_loss: 0.9120
Epoch 29/30
 - 0s - loss: 2.6440 - val_loss: 0.9027
Epoch 30/30
 - 0s - loss: 2.6225 - val_loss: 0.9022
final result 0.9021717561981543

Anyone knows the theory about why BN is not suitable for non-gaussian data ?

So batch-normalization helps descent based learning have an easier time traversing the loss manifold, but in your case you use it along with a relu as a final activation is problematic, it means the output is relatively associated with the other samples in the batch.

Remove that last BN and you get better results, but also understand BN is inherently problematic for this task. Think of DNNs as featurizers, and BN in this case takes out the 2 batch-wide statistics which if they don't align to the initial distribution will cause error, which will lead to error in the output. In theory if BN gets the perfect statistics of the gaussian it should not matter too much, so one thing I tried with your code was remove last BN and increase N to 100,000 while increasing the batch size to 10000 and you see a huge boost in performance.


What is generalized policy iteration? I am reading Sutton and Barto's material now. I know value iteration, which is an iterative algorithm taking the maximum value of adjacent states, and policy iteration. But what is generalized policy iteration?

In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE, ...). However, in general, you don't have to follow this alternation strictly in order to converge (in the limit) to the optimal policy. For example, value iteration (VI) is an example of a truncated policy iteration that still converges to the optimal policy.

The term generalized policy iteration (GPI) refers to all algorithms based on policy iteration, such as value iteration, that alternate in some order PI and PE, and that are guaranteed to converge to the optimal policy, provided PE and PI are executed enough times.


Why do we minimise the loss between the target Q values and 'local' Q values? I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the qVal for the next state in our target network and the current state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:
for state, action, reward, next_state, done in minibatch:
    target_qVal = self.model.predict(state)

    # print(target_qVal)

    if done:
        target_qVal[0][action] = reward #done
    else:
        # predicted q value for next state from target model
        pred = self.target_model.predict(next_state)[0]
        target_qVal[0][action] = reward + self.gamma * np.amax(pred)

    # indentation position?
    self.model.fit(np.array(state),
                   np.array(target_qVal),
                   batch_size=batch_size,
                   verbose=0,
                   shuffle=False,
                   epochs=1)

I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future $s'$ (correct me if I'm wrong in my understanding) when following a given policy.
My fundamental misunderstanding is the loss equation:
$$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$
where $\theta'$ and $\theta$ are the weights of the target and online neural networks, respectively.
Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?
A bonus question would be, in order to collect $Q(s,a)$ values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the target_qVal[0] values during training and feed them into a list after each step to accumulate the Q values over time?

The loss function is designed in a way to approximate the bellman optimality for $Q^*(s,a)$. Given an optimal policy $\pi^*$, $Q^*(s,a)$ satisfies the equation $$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$
At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ and the max $Q$ values at the next state.
You can see in the loss function that the DQN tries to attain a $Q^*(s,a)$ value that closely approximates the equation above.
On a side note, in model based RL setting, the bellman optimality for $Q^*$ is expressed as $$Q^*(s,a) = r(s) + \gamma \sum_{s'}P(s'|s,a)V^*(s)$$ $V^*(s')$ is used in the model based case because $V^*(s)$ by definition represents the highest possible value attainable at state $s'$ following $\pi^*$. In model free setting, $V^*$ is replaced by $Q^*$ because computing $V^*$ is not useful in achieving model free control without a transition model $P(s'|s,a)$


Are there names for neural networks with a well-defined layer or neuron characteristics? Are there names for neural networks with a well-defined layer or neuron characteristics?

For example, a matrix that has the same number of rows and columns is called a square matrix.

Is there an equivalent for classifying different neural network structures. Specifically, I am interested if there is a name for a neural network with x number of layers, but each layer has the same number of neurons?

Neural networks (NNs) are usually classified into feed-forward (i.e. NNs with feedforward connections), recurrent (i.e. NNs with recurrent connections) and convolutional (i.e. NNs that perform a convolution or cross-correlation operation). The term multi-layer NN may also be used to refer to feed-forward neural networks or, in general, neural networks with multiple (hidden) layers. There are also the perceptrons, which do not have hidden layers (i.e. the inputs are directly connected to the outputs).

You may still divide neural networks into classifiers (i.e. the outputs and labels are discrete) and regressors (the outputs and labels are numerical). Furthermore, you may classify them into generative models (e.g. the VAE) or discriminative models.

By analogy with linear algebra concepts, each layer of a feedforward neural network (FFNN) can be seen as a linear operation followed by an element-wise application of a linear or non-linear activation function.

$$
\mathbf{o}^{l} = \sigma \left(\mathbf{a}^{l} \mathbf{W}^{l} + \mathbf{b}^{l}\right)
$$

where $\sigma$ is the activation function, $\mathbf{a}^{l}$ the inputs to the layer $l$ and $\mathbf{W}^{l}$ the parameters of the layer $l$ (similar to the parameters or coefficients in linear regression) and $\mathbf{b}^{l}$ is the bias vector (a scalar bias for each neuron) of layer $l$. $\mathbf{o}^{l}$ will then be $\mathbf{a}^{l+1}$ (i.e. the input to the next layer).

A recurrent neural network (RNN) performs a slightly more complex operation.

$$
\mathbf{o}^{l}_t = \sigma \left(\mathbf{a}^{l}_t \mathbf{W}^{l} + \mathbf{o}^{l}_{t-1} \mathbf{R}^{l} + \mathbf{b}^{l}\right)
$$

where, in addition to the matrix $\mathbf{W}^{l}$ associated with the feedforward connections, it also uses another matrix $\mathbf{R}^{l}$ associated with the recurrent connections (i.e. cyclic or loopy connections of the neurons), which is multiplied by the output of the same layer but at the previous time step. $\mathbf{o}^{l}_t$ may actually just be the state of the recurrent layer, which is then used to compute the actual output of the layer, but, for simplicity, you can ignore this. Furthermore, note that there are more complex recurrent architectures, but this is the basic idea.

A convolutional neural network (CNN) performs the convolution (or cross-correlation) operation. If you are familiar with signal processing, e.g. kernels, convolution, etc., then you can view a CNN as performing a convolution (or cross-correlation) operation. It's actually possible to view the convolution as a matrix multiplication operation, but the details can easily become cumbersome and tedious to explain in an answer. A CNN may also perform other types of operations (such as downsampling) and it can also be composed of a feedforward part (usually, the last layers of a CNN are feedforward layers), but a CNN is a CNN because it performs the convolution (or cross-correlation).

In all cases, the matrices do not necessarily take any particular form (e.g. they are not necessarily square matrices). The dimensionality of the matrices depends on the number of layers and connections in the network, which can vary depending on many factors (e.g. the need for recurrent connections because they are useful for sequence modeling). These matrices (along with the biases) are the learnable parameters of the networks, but the learned values of these matrices highly depend on your data, the way you initialize them before learning, the architecture of the network, the learning algorithm, etc.

In the case of the FFNN, if the previous layer $l-1$ has the same number of neurons as the current layer $l$, then $\mathbf{W}^{l}$ is a square matrix. AFAIK, there is no name for a neural network with the same number of neurons for each layer. It could be called a rectangular neural network (but this is a name I've just come up with).

To conclude, there are many different neural networks and taxonomies for neural networks, so it is impossible to list or discuss them all in an answer, but, nowadays, the subdivision into feedforward, recurrent and convolution is the most common and general. See e.g. the paper A Taxonomy for Neural Memory Networks if you are interested in a more detailed taxonomy for memory-based neural networks (e.g. recurrent neural networks).


How would an AI learn idiomatic phrases in a natural language? After an AI goes through the process described in How would an AI learn language?, an AI knows the grammar of a language through the process of grammar induction. They can speak the language, but they have learned formal grammar. But most conversations today, even formal ones, use idiomatic phrases. Would it be possible for an AI to be given a set of idioms, for example,

  Immer mit der Ruhe

Which, in German, means 'take it easy' but an AI of grammar induction, if told to translate 'take it easy' to German, would not think of this. And if asked to translate this, it would output

  Always with the quiet

So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?

Short answer: Yes.

TL;DR

In the presence of good datasets this can be accomplished with a pipeline.

Long Answer

In reality an idiom is a series of words which is supposed to have a semantic meaning that is not denoted by the literal reading (source). This means that any system that is used must be capable of considering multiple words at a time. Additionally, some idioms are context dependent. Example:

The fisherman broke the ice with his tool.

Are we to believe that this is a very suave fisherman?

  So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?

Observe that humans do not come linguistically "pre-loaded" with idioms. So we can safely assume that idiom usage is a learning task and that the only way for them to keep up is for them to keep learning. So if we solve the idiom learning task we just need to keep our agent online or periodically retrain it on nascent corpora.

One difficulty is that, in the absence of a label, a metaphor could be easily mistaken for an idiom and vice versa. So semantic outlier (sorry it's not free) approaches  may suffer from precision issues. Example:

She's a thorny wildflower (metaphor - could easily be an idiom)
She's a diamond in the rough (idiom - could easily be a metaphor)

Though, idioms will most likely be repeated if a dataset is large whereas a "custom metaphor" is less likely to repeat.

Additionally, some idioms (eg bite the bullet or break a leg) do not have readily available "interpretable semantics" that allow us to extract their intended meaning. For example, if one did not know the idiom "cut me some slack" one could think:

"Slack implies loosening or to make less tight/taut. I was being very uptight. They probably want me to loosen up and not be so critical."

Of course the human understanding of it might happen in a flash and not follow such a delineated path. The idea is that some NLP pipeline might be constructible that satisfactorily handles idioms in some specific use cases (example of a pipeline). For example, one module might attempt to process outliers like "diamond in the rough" which have said interpretable semantics. Though, something like "bite the bullet" may have to be labelled with the correct semantics.

I've only scratched the surface of this. Natural language understanding is already a hard problem - and idioms are thus a tough task in a tough task. I hope that this motivates the reading of some more thorough articles. I have gathered some articles that can be used as a springboard into the literature.

Here's a source that uses a dictionary type approach to train the model to recognize idioms. Excerpt:

  For identification, we assume data of the form ${(⟨p_i,d_i⟩,y_i) : i = 1...n}$ where $p_i$ is the phrase associated with definition $d_i$ and $y_i ∈ \{literal, idiomatic\}$.

This source provides pseudo-code for idiom extraction.

This source describes a dataset to help solve the idiom difficulties.


Which model should I use to determine the similarity between predefined sentences and new sentences? The Levenshtein algorithm and some ratio and proportion may handle this use case.

  Based on the pre-defined sequence of statements, such as "I have a dog", "I own a car" and many more, I must determine if an another input statement such as "I have a cat" is the same or how much percentage does the input statement is most likely equal to the pre-defined statements.

For Example:

  Predefined statements: "I have a dog", "I own a car", "You think you are smart"

Input statements and results:

  I have a dog - 100% (because it has exact match), I have a cat - ~75% (because it was almost the same except for the animal, think - ~10% (because it was just a small part of the third statement), bottle - 0% (because it has no match at all)

The requirement is that TensorFlow be used rather than Java, which is the language I know, so any help with what to look at to get started would be helpful.

My plan was to use the predefined statements as the train_data, and to output only the accuracy during the prediction, but I don't know what model to use. Please, guide me with the architecture and I will try to implement it.

If this is a simple syntax comparison, neural networks is not the best way to achieve this.
If it's semantic comparison, then you can take a look at models used in the SNLI dataset for example.
From your question, it looks like just a syntax comparison.
Consider the 2 sentences :

She likes playing guitar
She likes listening guitar

The 2 sentences have almost the same words, but the meaning is different.
Now consider these 2 sentences :

The bird is taking a bath in the fountain
Birdie wash himself with water in public place

These 2 sentences have almost no words in common, but the meaning are very similar.

So if your use case need to return a high-score for the first example, give up neural network (it is possible, but pointless).
If your use case need to return a high-score in the second example, take a look at SNLI leaderboards, there is plenty of models that can works.


Why is no activation function used at the final layer of super-resolution models? I'm trying to implement some image super-resolution models on medical images. After reading a set of papers, I found that none of the existing models use any activation function for the last layer.
What's the rationale behind that?

I am not into the field of super-resolution, but I think this question applies to general neural network construction.
Usually, you try to solve a classification problem or a regression problem with your neural network.

For classification, you try to predict probabilities that a specific output corresponds to a specific class. Therefore, every output value should be a probability and therefore have a range between 0 and 1. To achieve this, you usually use a softmax or sigmoid function as your last layer to squash the output between 0 and 1. In addition to this (which is wanted in classification tasks), these functions raise the probability output of likely classes while decreasing the probability of all other unlikely classes (therefore enforcing the network to choose for one specific class over the others).

For the regression task, you are not looking for probability values as your output values but instead for real-valued numbers. In such a case, no activation function is wanted, since you want to be able to approximate any possible real value and not probabilities.

So, in the case of super-resolution, I think the generated output is a map where each value corresponds to a pixel value of the super-resolution image. In that case, your pixels are real number values and no probabilities. So, you are solving a regression problem.
But you could also go with a classification approach, where you have 256 output maps that give probability to each possible pixel value between $0$ and $255$.


Where do the feature extraction and representation learning differ? Feature selection is a process of selecting a subset of features that contribute the most.
Feature extraction allows to get new features that are not actually present in the given set of features.
Representation learning is the process of learning a new representation that contributes the most.
I can see no difference between the feature extraction and representation learning.
Is feature extraction the same as representation learning? If no, where do they differ? Do they differ at the application level only?

Feature extraction (FE) is not the same as representation learning (RL), but they are similar and related.
You describe accurately what feature extraction typically refers to, i.e. the process of extracting (new) features from existing ones or raw data (e.g. images). For example, let's say you have a dataset associated with a car. You have only two features in your dataset: distance and velocity. However, from these two, you can extract a third feature, e.g. the acceleration. So, feature extraction can be performed with a fixed algorithm (e.g. PCA for dimensionality reduction or SIFT) or manually.
Representation learning is the collection of all techniques that extract features automatically from the data (i.e. they learn the features or representations, hence the name representation/feature learning). So, for example, a convolutional neural network trained on ImageNet can (and/or needs to) learn general features in order to solve the corresponding classification task. (Chapter 9 of the book Deep Learning by Goodfellow et al. talks more about this topic.) These features are learned from the data (and that's why CNNs are data-driven), and they can later be exploited for transfer learning (TL), i.e. TL is based on the idea that neural networks learn general representations (which is thus a synonym for features) of data that can be exploited to solve other tasks (sometimes known as downstream tasks, especially in the context of self-supervised learning).
Yoshua Bengio et al. define representation learning as follows

learning representations of the data that make it easier to extract useful information when building classifiers or other predictors

So, RL is a subset of FE, given that RL also extracts features, but RL emphasizes the extraction of features automatically.


Can AI solve jumbled words? My question is that is there any general idea on how humans solve jumbled words? I know many people will say we match it against a commonly used words checklist mentally, but it is kind of vague. Is there any theory on this and how might an AI learn to do the same?

I remember a problem similar to this  https://vladris.com/puzzles/facebook/puzzle_master/snack/breathalyzer.html
this is given as a problem in facebook engineering puzzles.

But i doubt that it is not much of a machine learning/AI problem. you could implement an algorithm that converts each word into a set of its characters , then pick the word in your master list with minimum distance based on its character list .

Even when humans solve jumble we do it in a systematic/algorithmic way , if the jumbled word is not present in our memory we can't do anything otherwise we can solve the jumble.

But human brain can simply recognize scrabled words if some of the structure is retained and not fully scrambled like an anagram .

We generally index words based on the first syllable/character in our brain , like  or even this

In the second picture we recognised the words even if they are not made of english characters because our brain doesn't scan the text character by character like "7-H " but treats it like an image.

So the model should not immediately classify the segmented characters but should find the "nearest" characters in every class that optimises the combined probability of characters of the word being one of the classes of words we have in our dictionary.


How to perform back-propagation in Decoupled Neural Interfaces? I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (2017) by Max Jaderberg et al. As in the paper, the DNI is able to produce a synthetic error gradient that reflects the error with respect the output:
$$
\frac{\partial L}{\partial h_{i}}
$$
I can then use this to update the current layer's parameters by multiplying by the parameters to get the loss with respect to the parameters:
$$
\frac{\partial L}{\partial \theta}=\frac{\partial L}{\partial h_{i}} * \frac{\partial h_{i}}{\partial \theta}
$$
In the paper, the layer's model is then updated based on the next layer sending the true error backwards.
My question is, given that I am able to calculate the error with respect to the current output, how do I use this to calculate the loss with respect to the previous layer's output?

I kid you not when I tell you I just started with decoupled networks this morning. But the issue is terribly worded by most computer scientists. The way a decoupled neural interface works is not by sending the error backwards, but by storing it in a form of learned error inside a separate network. So not only do we have our nodes and weights, but also the Synths. The Synths modify the weight tables using past error. So instead of passing error back like the article suggests, it actually is passing it forward in a kind of "hey don't do this." What's really fun is that to you never have to mix up your thinking to reverse a network for backprop if you're using a decoupled network. The amazing part is that it is actually easier to code this decoupled network, than it is to write a "simple" ANN, even better still is that it functions a lot like an LSTM network without all the programming mumbo-jumbo. You can see for yourself two different forms of the same network from here.

The code from the demo in the repository is from https://towardsdatascience.com/only-numpy-implementing-and-comparing-combination-of-google-brains-decoupled-neural-interfaces-6712e758c1af it was a fundamental resource for me, and it may be for you. It also helps you with loss, or cost.


Are there any DeepQA-based computers other than Watson? My understanding is that Watson is the name of the computer, and DeepQA is the name of the software or technology. They are both correlated.

Are there any computers/technologies other than Watson which are using DeepQA? Or is Watson the only computer which implements that software/technology?

I cannot say for certain, but I know of no such other uses (I work at the building where Watson is developing but do not directly work with it).

The DeepQA team's page (https://www.research.ibm.com/deepqa/deepqa.shtml) only ever references Watson as the implementation, and based on the structure of the FAQ there I would imagine they'd be eager to list any interesting other uses it would have, but no such entries exist there.

I would, however, also note that while DeepQA is IBM's proprietary implementation of QA using deep-learning methods, that approach is becoming more popular and is almost certainly being worked on by other companies and could soon be on other computers, if not already so in some form.


Isn't a simulation a great model for model-based reinforcement learning? Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having a good model of the environment allows to use planning and thus drastically improves the sample efficiency!

Why is the simulation not used for planning in these cases? It is a sampling model of the environment, right? Can't we try multiple actions at each or some states, follow the current policy to look several steps ahead and finally choose the action with the best outcome? Shouldn't this allow us to find better actions more quickly compared to policy gradient updates?

In this case, our environment and the model are kind of identical and this seems to be the problem. Or is the good old curse of dimensionality to blame again? Please help me figure out, what I'm missing.

I will give one perspective on this from the domain of robotics. You are right that most RL agents are trained in simulation particularly for research papers, because it allows researchers to in theory benchmark their approaches in a common environment. Many of the environments exist strictly as a test bed for new algorithms and are not even physically realizable, e.g. HalfCheetah. You could in theory have a separate simulator say running in another process that you use as your planning model, and the "real" simulator is then your environment. But really that's just a mocked setup for what you really want in the end, which is having a real-world agent in a real-world environment.
What you describe could be very useful, with one important caveat: the simulator needs to in fact be a good model of the real environment. For robotics and many other interesting domains, this is a tall order. Getting a physics simulator that faithfully replicates the real-world environment can be tricky, as one may need accurate friction coefficients, mass and center of mass, restitution coefficients, material properties, contact models, and so on. Oftentimes the simulator is too crude an approximation of the real-world environment to be useful as a planner.
That doesn't mean we're completely hosed though. This paper uses highly parallelized simulators to search for simulation parameters that approximate the real-world well. What's interesting is it's not even necessarily finding the correct real-world values for e.g. friction coefficients and such, but it finds values for parameters that, taken together, produces simulations that match the real-world experience. The better the simulation gets at approximating what's going on in the real world, the more viable it is to use the simulator for task planning. I think with the advent of GPU-optimized physics simulators we will see simulators be a more useful tool even for real-world agents, as you can try many different things in parallel to get a sense of what is the likely outcome of a planned action sequence.


Which algorithm can I use to solve a problem with multiple objectives and constraints? Consider a problem with many objectives. In my case, these are school grades for different courses (or subjects). To be more concrete, suppose that my current grade for the math course is $12/20$ and for the philosophy course is $8/20$. My objective is to get $16/20$ for the math course and $15/20$ for the philosophy course.
I have the possibility to take different courses, but I need to decide which ones. These courses can have a different impact depending on the subject. Let's say that the impact factor is in the range $[0, 1]$, where $0$ means no impact and $1$ means a big impact. Then the math course could have a big impact (e.g. $0.9$) on the grade, while maybe a philosophy course may not have such a big impact.
The overall goal is to increase all the grades as much as possible while taking into account the impact of their associated course. In my case, I can have more than two courses and subjects.
So, which algorithms can I use to solve this problem?

From your description, the problem you want to solve is a linear optimization problem: suppose we use the indices $i$ and $j$ to denote the $i-$th class and the $j-$th grade. Also, let us call $y_j$ the current value of the $j-th$ grade, $g_j$ the goal value in grade $j$, $c_{ij}$ the impact factor of taking class $i$ in grade $j$, and $x_i$ the binary variable that indicates if you go to class $i$ or not. Now, suppose you decided to take certain classes (which is equivalent to choosing the values $x_i$ for each $i$) and measure the new grades $\tilde g_j$. If your impact factors are accurate, then the new grades should be $\tilde g_j = \sum_i(x_ic_{ij}+y_j)$. Clearly, what you would like is for this new values to be as close as possible to the goal values. So, a possible way to express your optimization problem is that you want to minimize the addition of all these differences:

$$\min_{x_0,\cdots,x_n} \sum_j g_j - \tilde g_j = \sum_j g_j - \sum_i(x_ic_{ij}+y_j), \\ \text{s.t. } x_i\in\{0,1\}, \forall\hspace 2pt i.$$

Most probably, you will have certain time restrictions. For example, maybe each class takes $t_i$ hours from your time and your total available time is just $T$ hours. Or maybe you have a limit of classes $N$ you can take, irrespective of their time duration. With these two constraints, your problem would look like this:

$$\min_{x_0,\cdots,x_n} \sum_j g_j - \sum_i(x_ic_{ij}+y_j), \\ \text{s.t. } x_i\in\{0,1\}, \forall\hspace 2pt i,\\ \sum_ix_it_i\leq T,\\ \sum_i x_i\leq N.$$

Apart from the constraint that $x_i$ should be 0 or 1, this is a linear optimization problem with linear constraints. Integer programming is an area of optimization that studies this type of problems so probably it's a good direction for your research.


Use Reinforcement Learning instead of genetic algorithm for optimization I want to use RL instead of genetic or any other evolutionary algorithm in order to find the best parameter for a function.
Here is the problem:
Given a function $$f(x,y,z,data)$$
x,y and z are some integeres from 1 to 50. So I can say I have a 3-dimensional array which is a way to save fitness values:
$$parameters=[[1..50], [1..50], [1..50]]$$ The $$data$$ is another input which is the f needed to do some calculation on.
Currently, I am optimizing it using a genetic algorithm with $$Cost(fitness) = f(x,y,z,data)$$ which is customized cost function.
any value for x,y, and z will result in a cost for example:
$$f(1, 5, 8, X) = 15$$
$$
parameters: [1, 5, 8] = 15$$
or
$$ parameters: [2, 9, 11] = 30
$$
In the provided example 2, 9, and 11 is a better set of parameters.
So I run a genetic algorithm and make some children with a sequence of x,y, and z. Then I calculate the cost(fitness) and then select them and so on.
I want to know is there any alternative or method in reinforcement learning which I can use instead of a genetic algorithm? If yes, please provide the name or any helpful link.
Note that F is completely defined by the user and should be changed in other contexts.
Thanks.

In order to have anything resembling reinforcement learning you must at the very least have a set of states $S$ and a set of actions $A$.
In your formulation I can vaguely identify the set of states $S$ as all possible $(x,y,z)$ triplets. But don't see anything in your description that could be interpreted as a set of actions $A$. You either oversimplified the description of your problem or reinforcement learning is not applicable here by lack of very basic ingredients for it.


Unsupervised alien natural language learning While conducting research, I recently stumbled upon the deep learning and natural language processing concepts. In this question they say that the ‘grammar induction’ is a ‘supervised learning’ mode. So I was wondering:

Let’s say that there’s a way-more-than-human intelligent alien probe orbiting our planet. It can receive, decode and analyze all the broadcasting signals leaving the Earth. For all we know at this moment, how could it learn the basics of a language with nothing else but our broadcasting signals, so without the help from a ‘supervisor’? How would an artificial intelligence human engineer (theoretically?) face the problem?

I’m interested on the more “technical” side of the issue.

I think I understand where your confusion comes from, and the reason for your question.

When we talk about a supervised problem, it means that there is some feedback of how well the machine did at a certain task. Of course, we always need this kind of feedback in some way or another (otherwise, the machine would never learn, because it would have no incentive to change anything that it does), but the feedback can be classified into "supervised" if we have a clear way of telling when something was right/wrong, or "unsupervised" if we just roll out with the results and try to make it better each time.

For example, an AI performing translation like in grammar induction in the other question will take as input both the phrase and the syntactic tree. After performing the prediction of what the syntactic tree for the phrase should be, it can be compared against the real syntactic tree given as input and based on the accuracy of the prediction, the weights (think of it as "knobs" that adjust the result) can be tweaked a little bit to give better predictions next time.

This type of learning is considered supervised, not because of the existence of a supervisor, but rather because there is labeled data that we can use to test predictions and better them next time.

Unsupervised problems don't have this kind of labeled data, and just work with what they have. They cannot tell what's right and what's wrong in terms of predictions, they only have raw data and try to make sense out of it by extracting correlations or common properties.

For the case of a language, I will go on a limb here but I'll say that this is mostly likely a task for supervised algorithms. Unsupervised techniques could properly analyze language and determine that they actually have a particular structure, but this is also true for lots of other sources of data that are not language. An alien race, knowing nothing about humanity, could end up deducting that cars honking are some sort of language too, albeit much simpler than human-produced noises.

Also, the natural flexibility that languages have make it extra complex, because they don't follow a correct structure completely, and to make it worse, they are constantly changing.

As leitasat mentioned in their comment, if even human scientists (which we certainly deem intelligent) could not decipher ancient Egyptian on their own without any context, it's very unlikely that a machine would do it.

Finally, notice that translator machines or inference machines don't actually derive meaning in a way that's meaningful. Without going much in much detail, you should think of them as "correlation detection machines" -- so a machine might notice a high enough correlation between how "Hola" is used in Spanish as to how "Hi" is used in English, and by giving you that information, we call it a translator. However, the internal learnt structure is actually just a probability distribution for an output given a set of inputs.

Not to say that any of this is not useful -- it definitely is -- but with something as separate as an alien race of super-human intelligence and without any way to derive meaning from just a bunch of sounds, it's very unlikely that a machine would prove useful in such correlations.

And still, I cannot find any hard stop to your idea -- if these aliens have a language and they express the same concepts that human express, then there is correlation. There just needs to be a way to find it.

Sorry for the wall of text and the huge rambling, I hope I provided some context on the points you were looking clarification for.


How can I keep context in my chatbot I have created a chatbot by Keras based on movie dialog. I used RNN more specifically GRU . My bot can reply well. But the problem is , it can't hold the context . As an example if I say Tell me a joke, the bot will reply something , and then if I say one more , the bot simply doesn't understand that I was asking for another joke and many more similar cases, like if I used a slang against the bot , the bot will reply me with something similar , but if I just say something romantic or good immediately after using slang , the bot will reply to me with something good . I want to keep context or environment . How can I do so . Any lead would be helpful .

This is an idea that I used for my model -
Try using two RNN (GRU) Networks, one of them to manage current output state and the other to maintain context

Say we are at timestamp $t$ and the two GRUCells are represented as $GRU_c$ and $GRU_s$ for GRU context network and state network. (Your output coming from the state network)

At time stamp $t$ , the input $GRU_s(t) = concat(input, att(all~GRU_c~from ~[0, ~t-1]))$ where $att$ is an attention mechanism to give importance to specific parts of the conversation uptil that point (This is what maintains context)  and input $GRU_c(t) = learned~representation~of~GRU_s(t)$ , hence updating $GRU_c$ for that timestamp, which along with the historical information can be used for $GRU_s(t+1)$

Hope this helped!


How to deal with the time delay in reinforcement learning? I have a question regarding the time delay in reinforcement learning (RL).
In the RL, one has state, reward and action. It is usually assumed that (as far as I understand it) when the action is executed on the system, the state changes immediately and that the new state can then be analysed (influencing the reward) to determine the next action. However, what if there is a time delay in this process. For example, when some action is executed at time $t_1$, we can only get its effect on the system at $t_2$ (You can imagine a flow: the actuator is in the upstream region and the sensor is in the downstream region, so that there will be a time delay between the action and the state). How do we deal with this time delay in RL?

This problem has been formally termed as Delayed MDP (Katsikopoulos & Engelbrecht, 2003)[1] - the actions generated are not instantly applied to the environment and/or the captured observations are not immediately seen by the agent, as expected in an MDP. The delay can either be:
a) A constant delay - Constant Delayed MDP (CDMDP) (Learning and planning in en-
vironments with delayed feedback, 2008)[2]

The delay introduced by the environment is constant and known. Like the example you provide in the question.

b) A random delay - Random Delayed MDP

We don't know what delay to expect, or when the delay will occur. This is more realistic for a real-world case.

TRIED APPROACHES
1. Just ignore the delay
This assumes the CDMDP is an MDP. We then attempt to search for the policy that best ignores the delay
$$
\pi(I_k)={\pi^∗(s)|I_k=(s, a_1,···,a_k)}
$$
$k$ is the number of time-steps between acting on the current state, and receiving feedback. This is simple and can give a reasonable result if the delay is small compared to the state transition magnitude [2].
2. Reconstruct the MDP from the CDMDP (Augmented approach)
The corresponding optimal policy for the reconstructed MDP will then the optimal policy for the CDMDP. This is however made intractable by the size of the action buffer growing with the delay length. It's therefore limited to small constant delays. [1] defines how to do the re-construction.
3. Predicting the delayed observation
This tries to "undelay" the environment by using a predictive model $P$ to approximate what the delayed observation would be, instead of keeping the agent patient to the end of the delay (Learning and planning in envs with delayed feedback, (2008)[5], At Human Speed: Deep RL with action delay, (2018)][6]).
e.g., For a delayed action:
$$
s_{t+i} = P(s_{t + i -1}, a_{t + i - delay})
$$
If the delay is on the observation, the most recent K actions can be used with the most recent observation to approximate the current state[2]. However, this also means the delays are part of the state - it's possible to have a better approach, as next described in the random delays part.
Estimating the current state enables the agent to act conditioned an estimate of the true state, on which the action is executed. The limitation is that it assumes a constant delay.

All the above approaches assume a constant delay
Handling random delays*
4. Partial Trajectory resampling (PTR)
A method that recursively resamples actions in the buffer, replacing them with on-policy actions. It uses the delay dynamics to simulate their effect on the current policy.
If random delays exist on an MDP, some actions present in the off-policy replay buffer will not influence the later delayed observations and rewards, for a number of timesteps. This will allow generation of on-policy sub-trajectories from the off-policy samples by recursively resampling the most recent actions from the action buffer. It will not invalidate the sub-trajectory.
The resampled action will be valid as long as the observation delay $\omega$ and the action delay $\alpha$ are both greater than the current time step $t$ i.e., No delayed observation depends on a resampled action.
$$
\omega_t + \alpha_t >= t
$$

Illustration of PTR, (3)
The benefits of this method:

Allows discarding of outdated information. e.g If the delay is such that the are no observations for $5$ timesteps, which then all arrive at step $5$, we can safely discard the transitions of steps $(1,2, 3,4)$. This is because this info will be compressed in the most recent observation.

Provides information on the "age" of an observation and the actions applied next

Efficient credit assignment

These factors give partial trajectory resampling better performance compared to concatenating the past $K$ actions with the most recent observation to estimate the current delayed state.
RL with random delays (2020)[3] describes and applies Partial Trajectory Resampling in RL. It's the only work I've managed to come across that handles both random and constant delayed MDPs.
What of the reward delay?
Reward delay, in this context, is referring to the rewards attributed to a delayed observation and action.
A solution to a delayed MDP will involve tackling the credit assignment problem. In this case, this is deciding what to do with the delayed rewards and how to give credit to the delayed actions and states.
This has had different approaches. For instance:

" ...we chose to accumulate the rewards corresponding to the [excessively delayed transitions we drop]. When an observation gets repeated
because no new observation is available, the corresponding reward is 0, and when a new observation arrives, the corresponding reward contains the sum of intermediate rewards in lost [dropped] transitions." [3]

Real time sample efficint RL for robots, (2013)[9] on the other hand, uses a decision tree to assign credit to an MDP reconstructed from a CMDP. The tree learns which delayed actions are relevant.

Related reads:

Real-time Reinforcement Learning (2019)
Thinking while moving: Deep reinforcement learning with concurrent control, (2020)


How to build a DQN agent with state and action being arrays? I have a Reinforcement-Learning environment where the state is an array of 0s and 1s with length equals to the number of users the agent must satisfy (11 users).
The agent must choose one of 12 resources for the 11 users according to the state array. If state[0] == 1, that means that user0 needs a resource, so the agent must choose a resource out of the 12 resources it has. So, the action array's first element would be, for example: action[0] = 10, which means that resource 10 was allocated to user0.
If the next user (user1) is asking for a resource as well, then the number of resources to choose from is 12 - 1, in other words, because resource10 was already allocated to user0, it cannot be allocated to another user.
If state[X] == 0, it means that userX is not asking for a resource, therefore it must not be allocated any resource.
An example of a state array:
[1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0]

An example of an action array according to the state array example: (resource count starts at 0 | -1 indicates no resource was allocated)
[10, 2, -1, -1, -1, 3, 11, 5, -1, -1, -1]

I'm new to Reinforcement Learning and Deep Learning, and I have no idea how to translate that into a neural network.

Let's assume you would like to work with a classic DQN. You need to train the Q-network where inputs are the states and actions. The DQN is a function of Q(states, actions). The network is supposed to predict Q-value. The agent must pick up the action that produces the highest Q by giving the all possible actions to the network, in your case. Let's assume the current state is
[1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0]

You must pick up an item for the user which state is 1? you need to generate the all possible actions, for example,
[1, 2, -1, -1, -1, 3, 4, 5, -1, -1, -1]
[12, 1, -1, -1, -1, 2, 3, 4, -1, -1, -1]
[11, 12, -1, -1, -1, 1, 2, 3, -1, -1, -1]
...

You will need to generate all of these possible actions into the Q-network. Your worst case would be around 479 million possible actions to calculate in DQN.
If you would like to implement DQNs for this problem, I do recommend you to check out REINFORCE -> DQN -> Advantage Actor-Critic (A2C), which is the combination of policy network and DQN. With A2C, you can produce continuous actions with a policy network, action in range a[i]∈(0,1)*12, then probably round the generated actions. Then, the next problem is the policy network may produce an impossible action. So, you might skip to the nearest possible item.
For further reading on continuous action domains, I recommend to check out DDPG, PPO


Where can I find the proof of the universal approximation theorem? The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper does not include the proofs of the theorem. Does anybody know where the proof can be found?

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used). Note that these proofs tell you that neural networks can approximate any continuous function, but they do not tell you exactly how you need to train your neural network so that it approximates your desired function. Moreover, most papers on the topic are quite technical and mathematical, so, if you do not have a solid knowledge of approximation theory and related fields, they may be difficult to read and understand. Nonetheless, below there are some links to some possibly useful articles and papers.
The article A visual proof that neural nets can compute any function (by Michael Nielsen) should give you some intuition behind the universality of neural networks, so this is probably the first article you should read.
Then you should probably read the paper Approximation by Superpositions of a Sigmoidal Function (1989), by G. Cybenko, who proves that multi-layer perceptrons (i.e. feed-forward neural networks with at least one hidden layer) can approximate any continuous function. However, he assumes that the neural network uses sigmoid activations functions, which, nowadays, have been replaced in many scenarios by ReLU activation functions. Other works (e.g. [1, 2]) showed that you don't necessarily need sigmoid activation functions, but only certain classes of activation functions do not make neural networks universal.
The universality property (i.e. the ability to approximate any continuous function) has also been proved in the case of convolutional neural networks. For example, see Universality of Deep Convolutional Neural Networks (2020), by Ding-Xuan Zhou, which shows that convolutional neural networks can approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough.
See also page 632 of Recurrent Neural Networks Are Universal Approximators (2006), by Schäfer et al., which shows that recurrent neural networks are universal function approximators. See also On the computational power of neural nets (1992, COLT) by Siegelmann and Sontag. This answer could also be useful.


How to load an image into tensorflow.js code which reads handwritten numbers and clasify them I'm new to machine learning, so i figured I should look into google's tensor flow guides and I know how to code in JS so that's why I'm using tensorflow.js, there's and example in the guide that trains itslef to recognize handwritten numbers from the  MNIST handwriting dataset, I sort of understand what's going on in the code but since I'm very new to ML it's not a lot, I went through the code and saw that it didn't took image by image to train itself but it requests one sprite which contains all the images and then cuts it into what it needs, this makes sense from a performance point of view, but as this process is kind of abstract I don't understand what's really going on, I want to upload an image of my own and call the predictor of the model but I don't know how to do it, any help?

I was thinking that drawing in a canvas of 28x28 a number might be very interesting as well instead of uploading an image, but I need to know how to test the model once it's trained with my own data.

The tutorial: https://js.tensorflow.org/tutorials/mnist.html

To test the model, you have to achieve a Tensor with the same dimensions as the training data. This Tensor should be the same for the same image, no matter how many times you remake the Tensor with the same image. That being said, you can attempt to reverse-engineer how the Tensors are being made. Hint: The dimensions of the input tensor should be (28, 28, 1).


What does Brooks mean by Representation? So for a class I'm reading Brooks' "Intelligence without representation". The introduction is dedicated to slating Representation as a focus for AI development.

I've read that representation is the problem of representing information symbolically, in time for it to be useful. It's related to the reasoning problem, which is about reasoning about symbolic information.

But I don't feel like I really understand it at any practical level. I think the idea is that when an agent is given a problem, it must describe this problem in some internal manner that is efficient and accurately describes the problem. This can then also be used to describe the primitive actions that can be taken to reach the solution. I think this then relates to Logic Programming eg Pascal?

Is my understanding of Representation correct? Just what does representation look like in practice, are there any open source codebases that might make a good example?

First, Pascal is not a logic programming language. Logic programming refers to languages like Prolog where you have a declarative style of programming compared to a imperative style like you have in Pascal. Maybe you mean if-statements which are typical for imperative languages.

Second, representation means a certain level of abstraction. For instance, a model represents a certain part of the reality. Imagine a cup on a table. If the agent has a representation of this situation, it has a symbol table and a symbol cup which represents the things in the real world. Now it can have a relation on(cup, table) which represents the situation that the cup is on the table. This type of abstraction can be easily represented in a logic language like first order logic. Therefore, one uses logic programming languages like Prolog or other types of languages like OWL to represent knowledge and perform reasoning. So the important term to which Brooks refers is Knowledge Representation and Reasoning.

Third, if your agent only have sensor data like video or sonar data, then it knows only distances or pixels from the real world. That is not meant with representation. Brooks' Creatures have only this information and calculate with this data directly to perform an action without reasoning. In that sense also artificial neural networks have no representation.

Finally, for an open source project to understand representation I would recommend the above mentioned OWL. You can look at the Protégé editor for working with OWL. In an OWL ontology you can represent relations between things and reason about them.


If an heuristic is not admissible, can it be consistent? I am solving a problem in which, according to the given values, the heuristic is not admissible. According to my calculation from other similar problems, it should be consistent, as well as keeping in mind the values, but the solution says it's not consistent either. Can someone tell why?

For a heuristic to be admissible, it must never overestimate the distance from a state to the nearest goal state.

For a heuristic to be consistent, the heuristic's value must be less than or equal to the cost of moving from that state to the state nearest the goal that can be reached from it, plus the heurstic's estimate for that state.

What this means is that, as you move along the sequence of nodes from start to goal that the heuristic recommends, a consistent heuristic should monotonically decrease in value. A consistent heuristic is thus also always admissible.

Notice that this means that if a heuristic is not admissible (like yours), it is also not consistent (by the contrapositive).

Therefore, if you already know your heuristic is not admissible, you should not be surprised that it is not consistent.

It seems most likely that you may have confused the definition of consistent for monotone. A consistent heuristic is both monotone and admissible.

As Neil Says, if you want to know why your specific heuristic is inadmissible, you should post another question about it, or modify this one.


What does it mean to have epochs=30 in Keras' fit method given certain data? I have read a lot of information about several notions, like batch_size, epochs, iterations, but because of explanation was without numerical examples and I am not native speaker, I have some kind of problem of understanding still about those terms, so I decided to work with data. Let us suppose we have the following data

Of course, it is just subset of original data, and I want to build a neural network with three hidden layers, the first layer contains 500 nodes, it takes input three variable and on each node, there is sigmoid activation function, next layer contains 100 node and sigmoid activation, the third one contains 50 node and sigmoid again, and finally we have one output with sigmoid to convert the result into 0 or 1 that classify whether a person with those attributes is female or male.

I trained the model using Keras Tensorflow with the following code

model.fit(X_train,y_train,epochs=30)

With this data, what does mean epochs=30? Does it mean that all 177 rows  (with 3 input at times) will go to the model 30 times? What about batch_size=None in model.fit parameters?

Batch size and epochs are independent parameters - they serve very different purposes. Your main question as I understand it (and for general, non-library specific consumption) is what is an epoch and how is the data used for each epoch?

Simply put, an epoch is a single iteration though the training data. Each and every sample from your training dataset will be used once per epoch, whether it is for training or validation. Therefore, the more epochs, the more the model is trained. The key is to identify the number of epochs that fits the model to the data without overfitting.

Your explaination of how batch size affects the training process is correct but not relevant to the question since it has no relation to the epoch training iterations. That is not to say that these values should be considered independently since they have similar effects on the model training process.


Why is image recognition a key function of AI? Image recognition, in the context of machine vision, is the ability of software to identify objects, places, people, writing and actions in images. Computers can use machine vision technologies in combination with a camera and artificial intelligence software to achieve image recognition. So, why is image recognition a key function of AI?

Image recognition is an important application of AI techniques, as images usually act as sensory input for further problems to be solved.

For example, a self-driving car needs to take into account its environment; it needs to recognise the path/road it is driving on, obstacles, other traffic, traffic signs, etc. All this is visual input which needs to be recognised. Without image recognition a self-driving car would not be possible.

A machine that operates in a warehouse picking goods: in order to handle the goods, eg with a robotic arm, it needs to recognise where the goods are and what they look like, so that the arm can be controlled.

A system controlling a CCTV camera should be able to distinguish an intruder from an animal that happens to just walk past. Again, with image recognition and classification this could be achieved.

An AI system will react to sensory input from the 'outside' world, and act on it. The type of sensory input depends on the application: it could be a numerical value (like a bank balance), language (text or speech), tactile (eg a robotic hand), or visual.  If it is a stock brokering system, image recognition is not needed. Machine translation would also not need it. But for many real-world interactions, having 'eyes' is either necessary or at the very least useful.


In the context of importance sampling ratio, how is the equation $\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$ derived? When reading the book by Sutton and Barto, I came across the importance sampling ratio.

The first equation, I believe, describes the probability a particular sequence is obtained given the current state, and the policy.

\begin{align}
&\operatorname{Pr}\left\{A_{t}, S_{t+1}, A_{t+1}, \ldots, S_{T} | S_{t}, A_{t: T-1} \sim \pi\right\} \\
&=\pi\left(A_{t} | S_{t}\right) p\left(S_{t+1} | S_{t}, A_{t}\right) \pi\left(A_{t+1} | S_{t+1}\right) \cdots p\left(S_{T} | S_{T-1}, A_{T-1}\right) \\
&=\prod_{k=t}^{T-1} \pi\left(A_{k} | S_{k}\right) p\left(S_{k+1} | S_{k}, A_{k}\right)
\end{align}

The next part takes the ratio between the probabilities of the two trajectories:

$$\rho_{t: T-1} \doteq \frac{\prod_{k=t}^{T-1} \pi\left(A_{k} | S_{k}\right) p\left(S_{k+1} | S_{k}, A_{k}\right)}{\prod_{k=t}^{T-1} b\left(A_{k} | S_{k}\right) p\left(S_{k+1} | S_{k}, A_{k}\right)}=\prod_{k=t}^{T-1} \frac{\pi\left(A_{k} | S_{k}\right)}{b\left(A_{k} | S_{k}\right)}$$

I don't understand how this ratio could lead to this:

$$\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$$

The $G_t$ rewards are obtained through the $b$ policy, not the $\pi$ policy.

I think there is something to do with Bayes rule, but I could not derive it. Could someone guide me through the derivation?

It has nothing to do with Bayes rule, like you said, $G_t$ is the return we get by following the policy $b$ , so the value of the state we get that return in is equal to the expected value of that return. On page 59 of the book, equation 3.14 derives the value of the state as expected value of the return but in this case we are following the behaviour policy $b$ so you would replace $\pi (a \mid s)$ with $b (a \mid s)$ and $v_\pi (s')$ with $v_b (s')$ and you would get the value of the state following the policy $b$, that is $v_b (s)$.
We are actually interested in the value of the state when we follow target policy $\pi$ , but we got the value by following behaviour policy, so we will multiply that expression with importance sampling ratio. If we do that $b (a \mid s)$ terms will cancel out and you are only left with term $\pi (a \mid s)$ from importance sampling ratio so we will get the value by following the target policy $\pi$ that is $v_\pi (s)$


How to use speaker's information as well as text for fine-tuning BERT? I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways to improve it, and a rough idea came up, but I don't know what to do it exactly.

Currently, I simply put chat text into the model, but don't use the speaker's information (who said the text, the speaker's ID in our DB). The idea is if I can use the speaker's information, the model might better understand the text and classify it better.

The question is, are there any examples or prior researches similar to what I want to achieve? I googled for a few hours, but couldn't find anything useful. (Maybe the keywords weren't good.)

Any advice would be appreciated.

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper.
Generally, when working with mixed data such as continuous and categorical features, the first step is to simply concatenate all the inputs into one long vector. In your case, you would concatenate a one-hot encoding of speaker ID to the BERT [CLS] output for each example. If you're using tensorflow, you'll need to use the functional API to create a multiple input model, as I outline below.
import tensorflow.keras as keras

# bert_input is the wordpiece tokenized input text
# BERT is the BERT model, such as a tensorflow_hub.KerasLayer with a pretrained model from tfhub.dev
bert_out = BERT(bert_input)

# spkr_input is the one-hot speaker ID encoding.
spkr_input = keras.Input(shape=(num_speakers,), name='spkr_input')
dense_input = keras.layers.concatenate([bert_out, spkr_input])
scores = keras.layers.Dense(num_classes, activation='softmax')(dense_input)
model = keras.Model(inputs=[bert_input , spkr_input], outputs=[scores])

You could also try feeding the one-hot encoded speaker ID vectors through a separate fully-connected network first to obtain a continuous representation (i.e. a speaker embedding) and then concatenate that to the BERT [CLS] output and feed the result into your classification layer. Modifying my example above,
spkr_emb = keras.layers.Dense(spkr_emb_size, activation="relu")(spkr_input)
dense_input = keras.layers.concatenate([bert_out, spkr_emb])

You can find a more detailed guide to mixed input models with Keras at https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/


Can Alpha–Beta be used on symmetric zero sum games? This question was asked in an AI exam. How would you answer such question?

If the game is not sequential, there would be no game tree and no need for pruning.  Alpha-beta is a technique applied to look-ahead search.  Alpha-beta has demonstrated utility in algorithms that play combinatorial games.

(Even in iterated dilemmas, it doesn't really branch because it's simultaneous, more of a vine than a tree.  Decisionmaking would be based on mathematical analysis of the payoff matrix and statistical analysis of competitor behavior over time.)


How can I model a problem as an MDP if the agent does not follow the successive order of states? In my problem, the agent does not follow the successive order of states, but selects with $\epsilon$-greedy the best pair (state, action) from a priority queue. More specifically, when my agent goes to a state $s$ and opens its available actions $\{ a_i \}$, then it estimates each $(s,a)$ pair (regression with DQN) and stores it into the queue. In order for my agent to change to state $s'$, it picks the best pair from the queue instead of following one of the available actions $\{ a_i \}$ of $s$. I note that a state has a partially-different action set from the others.
However, in this way, how can I model my MDP if my agent does not follow the successive order of states?
More specifically, I have a focused crawler that has an input of a few seeds URLs. I want to output as many as possible relevant URLs with the seeds. I model the RL framework as follows.

State: the webpage,
Actions: the outlink URLs of the state webpage,
Reward: from external source I know if the webpage content is relevant.

The problem is that, while crawling, if the agent keeps going forward by following the successive state transition, it can fall into crawling traps or local optima. That is the reason why a priority queue is used importantly in crawling. The crawling agent does not follow anymore the successive order of state transitions. Each state-action pair is added to the priority queue with its estimated action value. For each time, it selects the most promising state-action pair among all pairs in the queue. I note that each URL action can be estimated taking into account the state-webpage where it was extracted.

Your problem fundamentally is that you are confusing what the state and actions are in this setting. Webpages are not your states; your state is the entire priority queue of (website-outlink) pairs + the (new_website-outlink) pairs. Your action is which pair you select.
Now this is a variable sized state-space and variable sized action-space problem setting at the same time. To deal with this lets start by noting that state==observation need not be (in general). So what is your observation? Your observation is a variable-sized batch of either:

(website-outlink) pairs or
next_website (where each next_website is determined by its corresponding pair)

Both of these observations could work just fine, choosing between one or the other is just a matter of whether you want your agent to learn "which links to open before opening them" or "which links are meaningful (after opening them)".
What your priority queue is essentially doing is just adding a neat trick that:

Saves the computational complexity of keeping the state ordered (remember that your state is not a website, but the list/batch of website-outlink)
Avoids needlessly recomputing the Q-values for each of your actions (remember that an action is not selecting an outlink from new_website, but selecting an outlink from all available choices in the queue)

Note however that to actually have the second saving it is crucial to store the Q-values for each pair!!!
Last important thing to note is that in a scenario where you use a Replay Buffer (which I guess is likely given that you chose a DQN), you can't use the priority queue whilst learning from the RB. To see why (and to see in detail how your learning process actually looks like), start by remembering that your Q-value updates are given by the formula here; your state s_t is a (quasi-ordered1) batch of pairs. Q(s_t, a_t) is just the output of running your DQN regression on just the best website/pair in this batch (you have to add an index to denote the best choice when adding transitions to the RB, in order to be consistent about which action was taken from this state). To compute the estimate of the optimal future value however you will have to recompute the Q-value of every single website/pair in the next state. You CANNOT use the priority queue when training from the RB.
1 You have the priority queue ordered for all websites you had in it whilst looking at the last website, but all the new_website-outlink pairs you are now adding are not ordered yet. You still have to run the agent on them and then you can order them with the rest of the priority queue to generate the next state (which still will not be ordered because you will have new_new_website-outink pairs).


Move blocks to create a designed surface I am new to machine learning and AI, so forgive me if this is obvious. I was talking with a friend on how to solve this problem, and neither of us could figure out how to do it.

Say I have a grid area of 100x100 blocks, and I want a robot to build a horizontal 100x100 grid, and 3 blocks high. I am given a random, but known starting surface, always 100x100 but the height of the random surface can vary from 1 to 5 blocks. I have an extra reserve of blocks i can pick up, so dont have to worry about running out. The robot can move in any direction, even diagonally at some cost penalty. The robot can obviously move a 4 high block to fill in a 2 high, so each is at the design height of 3.
This sounds like a reinforcement learning problem, but would any one be able to explain more detail how I would do this, to a) minimize the amount of moves, and b) to get to the design surface.

Essentially, you could do something like have the robot randomly make moves (moving around and moving blocks) for some number of steps. Repeat this a bunch of times, and record the 'score' at the end (how close you are to a perfect result grid). Tell your algorithm to act more like the best scoring runs (Optimize a loss function), and start it over. Hopefully, you'll eventually get a robot that manages the task - the whole 'optimal path' thing will come around by the computer telling itself to learn from the lowest cost examples.

Remember, you're letting the machine do the hard thinking about the best way to do this or that. All you have to do is give it the framework to learn.


What is the time complexity of the upsampling stage of the U-net? I am trying to determine the complexity of the neural network we use. The neural network is a U-net generator with an input shape of NxN (not an image but image-like data) and output of the same shape. There is 7x downsampling and 7x upsampling. Downsampling is a simple convolutional layer, where I have no problem to determine complexity as stated here:
$$
O\left(\sum_{l=1}^{d} n_{l-1} \cdot s_{l}^{2} \cdot n_{l} \cdot m_{l}^{2}\right)
$$
I however cannot find what is big O complexity for the upsampling stage, where the UpSampling2D layer is used before convolution.
Any idea what is the time complexity of the upsampling convolutional layer, or where I might find information? Thanks in advance!

After further investigating the problem I have found the answer:
U-net generators' up-sampling stage consists of two steps:

Use UpSampling2D layer
Apply convolution on the output

The UpSampling2D layer is in the keras documentation described as:
Repeats the rows and columns of the data by size[0] and size[1] respectively.
From this information, we can calculate the time cost for UpSampling2D alone. Lets set size to (2,2), as is set in basic configuration of the U-net generator. The output of the UpSampling2D is then doubled. In case we started with (4,4,3), where the last index corresponds to number of channels, the output shape will be 8,8,3. We can see that each row and column need to be copied twice in each channel. From this we can define time complexity of a single up-sampling as:
$$
O\left(2 \cdot c \cdot n \cdot s\right)
$$
Where c corresponds to number of channels, n corresponds to input length (one side of a matrix) and s is equal to filter size. Assuming that length and filter size have square shape, the complexity is multiplied by 2. Since in this case the the filter size is known, equal to (2,2), the notation can be simplified to:
$$
O\left(4 \cdot c \cdot n \right) = O\left(c \cdot n \right)
$$
In my case, with only 1 channel, the complexity is simply
$$
O\left(n \right)
$$
Which means the up-sampling stage is linear, and the only important feature is input size, which is negligible to the complexity of the following convolutional layer and can be ignored.


Why does off-policy learning outperform on-policy learning? I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.
I saw this in a book:

Q-learning is an off-policy learner. An off-policy learner learns the value of an optimal policy independently of the agent’s actions, as long as it explores enough.

An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps.

However, I am not quite understanding the difference. Secondly, I came across that off-policy learner works better than the on-policy agent. I don't understand why that would be i.e. why off-policy would be better than the on-policy.

This post contains many answers that describe the difference between on-policy vs. off-policy.
Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-X, R2D2, Agent57 are technically "off-policy", since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number of benefits, such as reusing experience and not forgetting important experiences.
Another benefit is that you can collect a lot of experience distributedly. Since RL is typically not bottlenecked by the computation for training but rather from collecting experiences, the distributed replay buffer in Ape-X can enable much faster training, in terms of seconds but not sample complexity.
However, it's important to emphasize that these replay-buffer approaches are almost on-policy, in the sense that the replay buffer is constantly updated with new experiences. So, the policy in the replay buffer is "not too different" from your current policy (just a few gradient steps away). Most importantly, this allows the policy to learn from its own mistakes if it makes any...
Off-policy learning, in general, can also refer to batch RL (a.k.a. offline RL), where you're provided a dataset of experiences from another behavior policy, and your goal is to improve over it. Notably, you don't get to rollout your current policy in any way! In this case, algorithms that worked well with a replay-buffer (like DQN, SAC) fail miserably, since they over-estimate the value of actions when they extrapolate outside the "support" of the dataset. See the BCQ paper which illustrates how a lot of "off-policy" algorithms like DQN fail when the "distance between the two policies is large". For this task, SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR).
It's also worth noting that importance sampling can correct off-policy gradients to be on-policy; but the farther away your target policy is, the larger the variance. This is especially deadly for long horizon tasks (often called curse of horizon).
To summarize, using replay-buffer (which makes the algorithm off-policy), especially a distributed one, can offer a lot of benefits over pure on-policy algorithms. However, this is a very special class of off-policy algorithms, where the behavioral policy is close to your policy.
But in general, off-policy is a lot harder than on-policy; you'll suffer from extrapolation bias if you use DQN-based approaches, and exponential variance blow-up if you use importance sampling to correct for it.


How to make training data for AI Right now I'm planning to make a DNN for classifying taste of crystals with their molecular structure which include information like no of atoms mass of each atoms atomic no of atoms. How should I make a data set for training testing and validation?

You can cluster all your features in one matrix X, in which each line would be one element of the data set you want to construct, and each column would be a different feature of this element.

You construct then a Y vector containing the different target classes, where the i-th element will be the target class of the i-th X element.

For the following I suppose you use python

Create train and test sets using sklearn wrapper train_test_split on X and Y just built, in which you can shuffle your X and Y (non-avoidable in this case) :

from sklearn.model_selection import train_test_split

BTW, I recommend you to rescale your data with sklearn with for example one of :

from sklearn.preprocessing import MinMaxScaler, StandardScaler

It allows better learning and so better generalization


Are there clever (fitness-based) crossover operators for binary chromosomes? While studying genetic algorithms, I've come across different crossover operations used for binary chromosomes, such as the 1-point crossover, the uniform crossover, etc. These methods usually don't use any "intelligence".
I found methods like the fitness-based crossover and Boltzmann crossover, which use fitness value so that the child will be created from better parents with a better probability.
Is there any other similar method that uses fitness or any other way for an intelligent crossover for binary chromosomes?

It's not obvious what you mean by "intelligent crossover".

However, it is common to use fitness-based selection of parents: individuals in the current population who have higher fitness are assigned a higher probability of being selected to mate and produce offspring.  This will increase the likelihood that "good" combinations of genes in members of the current population will be passed along to the next generation, and that independent "good" combinations will be combined in some members of the next generation.

The "best" crossover operator depends dramatically on the structure of the problem being solved, and on the mapping of gene "vectors" to the salient features of a solution.

Edit #1:
In some cases it is important to increase diversity in order to avoid convergence to a local optimum.  In that case, an "intelligent" GA might for example select a first parent for its high fitness, and a second parent at random.  In "Generator", a GA I sold for a while about 25 years ago, mate selection worked that way, and it was often very effective.  Generator also replaced any duplicate individuals in the population with entirely random individuals.  I have also structured genetic algorithms specifically to evolve multiple separate populations of individuals, with minimum gene flow between the populations, in order to evolve multiple solutions corresponding to "regional" fitness optima.

Edit #2:
In genetic algorithms it is not common to directly seek the best combination of genes from both parents. The assumption is that higher-fitness parents are more likely to produce even higher-fitness offspring than lower-fitness parents.  Sometimes there is a local search (hill climbing) operation where the offspring of two parents is mutated in various ways and the best of the mutants is put in the next generation.

And, sometimes a crossover operation involves producing a larger number of offspring than the parent generation, followed by culling low-fitness individuals to keep the population size constant.  This is vaguely analogous to a local search via mutation, but uses random crossover instead of random mutation for its search.


Does a quantum computer resolve the halting problem and would that advance strong AI? Have there been proposed extensions to go beyond a Turing machine that solve the halting problem and if so, would those proposed extensions have value to advance strong Artificial Intelligence?  For example, does quantum computing go beyond the definition of a Turing machine and resolve the halting problem, and does that help in creating strong AI?

It depends a bit on what you mean by 'quantum computer'. The 'conventional' notion is that quantum computation buys a (in some cases, exponential) speedup - it doesn't change what can be computed, just how quickly.

In contrast, advocates of hypercomputation claim that quantum effects may make it possible to do infinite computations in finite time. Note, however, that this is not a mainstream belief - the reknowned logician Martin Davis has written an article claiming that hypercomputation is a myth.

Roger Penrose has also claimed that quantum vibrations in neural microtubules may be responsible for consciousness.


Can analog quantum computer implement real-valued neural networks and hence do hypercomputation? It is said, that the essence of https://www.springer.com/us/book/9780817639495 "Neural Networks and Analog Computation. Beyond the Turing Limit" is that the continuous/physical/real-valued weights for neural networks can induce super-Turing capabilities. Current digital processors can not implement real-valued neural networks, they can only approximate them. There are very little efforts to build analog classical computers. But it is quite possible that quantum computers will be analogue. So - is there research trend that investigates true real-valued neural networks on analog quantum computers?

Google is of no use for my efforts, because it does not understand the true meaning of "true real-valued neural network", it gives just real-value vs complex valued neural networks articles, which are not relevant to my question.

Digital and Analog

The question about analog computing is important.

Digital circuitry gained popularity as a replacement for analog circuitry during the four decades between 1975 to 2015 due to three compelling qualities.

Greater noise immunity
Greater drift immunity (accuracy)
No leakage of stored values

This quickly led to digital signaling standards, architecture of a general purpose computing, and central processing units on a chip. The later, combined with an array of registers to perform elementary operations is the meaning of the word microprocessor.

Quanta and Computing

Regarding quantum computing, there have been some interesting proposals to pack digital gates into much smaller volumes, but the notion that a computer can be made of transistors the size of electrons might be a bit fantastic. That's what the term quantum computing implies. That degree of miniaturization would have to defy principles of particle physics that are very strongly supported by amassed empirical evidence. Among them is Heisenberg's uncertainty principle.

All computing involves quanta, but statistically. For a transistor in a digital circuit to be statistically stable, there must be a sufficient number of Si atoms with at least 0.1% molar concentration of the atoms used to dope the Si to create a PN junction. Otherwise the transistor will not switch reliably.

The lithographic limit of most mass produced VLSI chips is around 7 nm as of this writing. Crystalline Si, nucleus to nucleus, is about .2 nm, so the miniaturization of a stable transistor is near its quantum limit already. Exceeding that limit by a considerable amount destabilizes the digital circuitry. That's a quantum physics limitation, not a lithographic limitation.

Projections, Models, and Techniques to Push Limits

Moore's law was simply an approximate model for the chip industry during the period between the invention of the integrated circuit to the physical limitation of the atomic composition of transistors, which we are now approaching.

Field effect transistors (FETs) can take the miniaturization only slightly further than the mechanics of PN junctions. 3-D circuitry has theoretical promise, but no repeatable nanotechnology mechanisms have yet been developed to form complex circuitry in the third dimension.

Returning to the Primary Question

Placing aside the magical idea that quantum computing will somehow completely revolutionize circuitry, we have a question that is both feasible and predictable.

  Can an analog computer implement real-valued neural networks and hence do artificial network computation better?

If we define better in this context as cheaper and faster, while maintaining reliability and accuracy, the answer is straightforward.

It definitely takes fewer transistors to create the feed forward part of an artificial network using an analog approximation of the closed forms resulting from the calculus of artificial networks than a digital one. Both are approximations. Analog circuitry has noise, and drift and digital circuitry has rounding error. Beyond rounding, digital multiplication is much more complex in terms of circuitry than analog, and multiplication is used quite a bit in artificial network implementations.

Limitation Interplay of Gödel and Turing

The idea from the tail end of the title of the book this question referenced, "Beyond the Turing Limit," is also a little fantastic. The thought experiment of Alan Turing leading to the Turing machine and the associated computability theory (including Turing completeness) was not developed to be a limit. Quite the opposite. It was an answer to Gödel's incompleteness theory. People in Turing's time saw the work Gödel's genius as the annoying but indismissable limit threatening the centuries-old vision of using machines to automatically expand human knowledge. To summarize this work, we can state with assurance this.

  The theory limiting what computers can do is not related to how the numbers are represented in electronic circuit implementations. It is a limitation of information mechanics.

These principles are important but have little to do with the limitation.

Analog computing
Miniaturization
Parallel computing
Ways that stochastic injection can help
How algorithms can be defined in programming languages

The above has to do with the feasibility of a project for which some person or corporation must pay and the intellectual capacities required to completing it, not the hard limit on what is possible.

Defining what a super-Turing capability might be would be a dismissal or a dismantling of what mathematicians consider to be well constructed theory. Dismantling or shifting the contextual frame of some computability theory is plausible. Dismissing the work that has been done would be naive and counterproductive.

Real Numbers are Now Less Real Than Integers

The compelling idea contained in the question is the reference to continuity, physicality, and the real valued nature of parameters that acquire a learned state during the training of artificial networks.

To multiply a vector of digital signals by a digital parameter matrix requires many transistors and can require a significant number of clock cycles, even when dedicated hardware is used. Only a few transistors per signal path are required to perform the analog equivalent, and the throughput potential is very high.

To say that real values cannot be represented in digital circuits is inaccurate. The IEEE standards for floating point numbers, when processed in a time series, represent real valued signals well. Analog circuits suffer from noise and drift as stated above. Both analog and digital signals only appear to be comprised of real number values. Real numbers are not real except in the world of mathematical models. What we call quantities in the laboratory are essentially measurements of means of distributions. Solidifying and integrating the probabilistic nature of reality into science and technology may be the primary triumph of the twentieth century,

For instance, when dealing in milli-Amps (mA), electric current seems to be a continuous phenomenon, but when dealing with nano-Amps (nA), the quantum nature of electric current begins to appear. This is much like what happens with the miniaturization of the transistor. Real numbers can only be represented in analog circuits through the flow of discrete electrons. The key to the advantage of an analog forward feed in artificial networks is solely that the density of network cells can be considerably higher, reducing the cost of the network in its VLSI space.

In summary, real numbers received the name for their type prior to the emergence of quantum physics. The idea that quantities formerly considered real and continuous were actually statistical averages of discrete activities at a quantum level revolutionized the field of thermodynamics and microelectronics. This is something that disturbed Einstein in his later years. In essence, mathematics using real numbers is effective in engineering because it simplifies what physicist now believe are distributions of a large numbers of quantum phenomena occurring in concert.

Summarizing the Probable Future of Analog Computing

This phrase from the question is not precisely scientific, even though it points to a strong likelihood.

  It is quite possible that quantum computers will be analogue.

This modified version is more consistent with scientific fact in the way it is phrased, and is also factual.

  It is possible that computers dealing with signals at a near quantum level of miniaturization will have a higher proportion of analog circuitry.

This question and its answers have many links to papers and research work regarding analog computing: If digital values are mere estimates, why not return to analog for AI?.


Are there Python packages for recent Bayesian optimization methods? I want to try and compare different optimization methods in some datasets. I know that in scikit-learn there are some corresponding functions for the grid and random search optimizations. However, I also need a package (or multiple ones) for different recent Bayesian optimization methods.

Are there any good and stable ones to use? Which packages do you recommend? (If any recent for grid/random search, it is also okay.)

Apart from the Scikit-Optimize package related to Scikit-Learn, following are some of the packages related to Bayesian optimization:

GPyOpt
pyGPGO
Hyperopt
bayesian-optimization
safeopt
RoBO