What are the different types of goals for an AI system called? I remember reading about two different types of goals for an intelligence. The gist was that the first type of goal is one that "just is" - it's an end goal for the system. There doesn't need to be any justification for wanting to achieve that goal, since wanting to do that is a fundamental purpose for that system. The second type of goal is a stepping stone, for lack of better words. Those aren't end goals in and of themselves, but they would help the system achieve its primary goals better.
I've forgotten the names for these types of goals and Googling didn't help me much. Is there a standard definition for these different types of goals?

Do you mean weak AI and strong AI?
The former is roughly about pretending to be intelligent, ie do intelligent things, but without trying to work the same way an actually intelligent system would work. The latter attempts to replicate how an intelligent system works, so would require us to understand a lot more about cognition than if we just mocked up a quick little chatbot to try and compete the Turing Test.


Seq2Seq model produces repeating words My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the model cannot produce any meaningful output, instead it keeps generating repeating words.
The framework can translate Franch-Enlgish very effectively, but for my problem it generates the result like this.
Can you explain why it produces such result, Thank you

Here is my printed output:

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial.
What you are seeing is most likely the result of a greedy sampling - the most probable next word is chosen from the probability distribution. This quite often leads to infinite repletion loops as you experience.
Simplest way to solve the repetition problem is to actually randomly sample from the probability distribution of the next word - the distribution is usually augmented with so-called temperature parameter.
Finally, there's beam search - you just perform a tree search for a best N-gram sample in the future. That's the most sophisticated, and the most computationally hard one.


How is dropout applied to the embedding layer's output? model = tf.keras.Sequential([
    tf.keras.layers.Embedding(1000, 16, input_length=20),
    tf.keras.layers.Dropout(0.2),                           # <- How does the dropout work?
    tf.keras.layers.Conv1D(64, 5, activation='relu'),
    tf.keras.layers.MaxPooling1D(pool_size=4),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

I can understand when dropout is applied between Dense layers, which randomly drops and prevents the former layer neurons from updating parameters. I don't understand how dropout works after an embedding layer.
Let's say the output shape of the Embedding layer is (batch_size,20,16) or simply (20,16) if we ignore the batch size. How is dropout applied to the embedding layer's output?
Randomly dropout rows or columns?

It doesn't drops rows or columns, it acts directly on scalars. The Dropout Layer keras documentation explains it and illustrates it with an example :

The Dropout layer randomly sets input units to 0 with a frequency of rate

After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, in your case, you should have rate * (16 * input_length) = 0.2 * 20 * 16 = 64 inputs set to 0 out of the 320 scalars inputs. These 64 dropped inputs are randomly selected in the 20x16 grid. Note that the Dropout rescales the non dropped inputs by multiplicating them by a factor $\frac{1}{1-rate}$.


Adding Class Weights for imbalanced dataset? Suppose I have the following toy data set:

Each instance has multiple labels at a time. Now you can see I have 2 instances for Label2, however, only one, one instance for other labels. It means, we have class imbalanced issues. I read about the term "Adding Class Weights for an imbalanced dataset", however, I could not understand, how actually it works and beneficial. Can anyone explain this method generally as well as according to my given toy data set. Other than that, how we handle this Miss Lable (nan)?

The paper A systematic study of the class imbalance problem
in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning":

Cost sensitive learning. This method assigns different cost to misclassification of examples from different classes [44]. With respect to neural networks it can be implemented in various ways. One approach is threshold moving [22] or post scaling [23] that is applied in the inference phase after the classifier is already trained. Similar strategy is to adapt the output of the network and also use it in the backward pass of backpropagation algorithm [45]. Another adaptation of neural network to be cost sensitive is to modify the learning rate such that higher cost examples contribute more to the update of weights. And finally we can train the network by minimizing the misclassification cost instead of standard loss function

Without further context, "Adding Class Weights for an imbalanced dataset" can mean many things as enumerated by the above. But if I had to guess, the most common meaning is that they weigh the misclassification cost differently per label by multiplying it with a different weight variable.
For example, maybe label 1, 2, and 3, when misclassified, gets a 1x multiplier in the training loss (the standard weight), but label 4 gets a 3x multiplier because it is roughly 3x more important, and label 5 gets a 0.5x multiplier because it is 0.5x less important.
Here is an example of how to do this with Keras.
(FYI the paper above recommends you over-sample the minority class rather than using the cost-sensitive learning approach to help with class imbalance)
As for missing labels, the quickest solution is just to skip those training instances when training for that particular missing label. If you want to still use the training instances somehow, a common approach is to fix the missing label with semi-supervised learning approaches.
One way to do semi-supervised learning is self-training, where your model is trained on only the labeled instances, and then makes predictions on the missing labels. High-confidence predictions of the missing labels are then added to the training data, and the model is trained on the new training data. This process repeats until convergence of the training data.


An "elevator pitch" breakdown of areas of applications for Reinforcement Learning & Neural Networks vs. Genetic Algorithms I'm looking for an "elevator pitch" breakdown of areas of applications for Reinforcement Learning & Neural Networks vs. Genetic Algorithms, both actual and theoretical.

Links are welcome, but please provide some explanation.

Your question suggests a confusion of techniques, representations and problems.

Neural Networks are a representation that can be used to approximate functions. A neural network approximates a function that maps from inputs to outputs by optimizing parameters (weights).
Genetic Algorithms are a technique that can be used to optimize a problem. You might chose to use a GA to optimize the weights in a neural network for instance. Or you might use it to optimize a different representation or approximation of a function.
Reinforcement Learning is a problem. In a Reinforcement learning problem, the agent learns a function mapping states to actions. You can learn this function directly in some problem domains, or by a near-direct approximation (like tile-coding), or with a function appropriator (like a neural network).


How can I perform lossless compression of images so that they can be stored to train a CNN? I have a set of images, which are quite large in size (1000x1000), and as such do not easily fit into memory. I'd like to compress these images, such that little information is missing. I am looking to use a CNN for a reinforcement learning task which involves a lot of very small objects which may disappear when downsampling. What is the best approach to handle this without downscaling/downsampling the image and losing information for CNNs?

Your input image size and memory are not directly related. While using CNN's, there are multiple hyperparameters that effect the video memory(if you are using GPU) or physical memory(if you are using CPU). All the frameworks these days uses a simplified data-loaders, for instance in Tensorflow or PyTorch, you are required to write a data-loader that takes in multiple hyper-parameters that are mentioned below and fit the data into VRAM/RAM, and this is strictly dependent upon you batch size - memory occupied on VRAM has direct relation to the batch size.

Whatever may be your image size, while you are writing the data-loader you have to mention the transformation parameters to your data-loader, during the training phase the data-loader will automatically load required images into your memory according to the batch size you have mentioned. As you have mentioned about image compression, this is an irrelevant parameter at-least for most of the generic use-cases, the most relevant hyperparameters are

Scaling
Cropping
Random flip
Normalization of the RGB values
ColorJitter
Padding
RandomAffine

And many more.

PyTorch provides really good transformers in data-loader, please do check https://pytorch.org/docs/stable/torchvision/transforms.html.

For Tensorflow, have a look at https://keras.io/preprocessing/image/.


Teaching a neural network to play a card game with two phases I'm writing a virtual environment for a 4-player card game named estimation, and will use deep reinforcement learning to teach an agent to play it.

Each player gets a hand of 13 cards, and the first phase is for each player to estimate the number of tricks they will collect. The highest player gets to start first,and then after each round, the player who collects the trick starts the next. So basically the first phase is for bidding and the next phase consists of 13 rounds.

The state input I'll use will include all the cards that have been played, the goal and collected tricks, and the available cards. The output for each round will be a vector of length 54, containing all the cards and then the available card with the highest probability would be played.

At first I thought that the bidding phase should use the same input but with zeros everywhere except the available hand, and the output would exclude all the cards with no numbers like king, queen or jack. But then the ability to dash (estimate that you'll collect 0 tricks) wouldn't be available. Also I don't think it would work really well.

Should I just use two NNs for each phase, or what should I do? Also if anyone has any advice on things I need to watch out for, I'd really appreciate it if they shared them.

I've played this game. If I remember correctly, a successful strategy (for winning as many tricks you estimated to win) involves continually evaluating how well you're doing. If you're doing very poorly relative to your initial estimation, that's valuable information to have.

I think you should have one network that outputs--at every time step--both an estimate of future tricks won and an action of which card to play. This has the additional advantage of giving you more estimation experience with which to train. After all rounds, you'll know how far off each of your estimations were throughout the game. This should speed up learning on that part of the game.


Does this argument refuting the existence of superintelligence work? A superintelligence is a machine that can surpass all intellectual activities by any human, and such a machine is often portrayed in science fiction as a machine that brings mankind to an end.

Any machine is executed using an algorithm. By the Church-Turing thesis, any algorithm that can be executed by a modern computer can be executed by a Turing Machine. However, a human can easily simulate a Turing Machine. Doesn't this mean that a machine can't surpass all intellectual activities, since we can also execute the algorithm?

This argument is most likely flawed, since my intuition tells me that superintelligence is possible. However, it is not clear to me where the flaw is. Note that this is my own argument.

I believe this argument is based on the fact that intelligence is a single dimension when it really isn't. Are machines and humans really on the same level if a machine can solve a complex problem in a millionth of the time a human can?

It also assumes that the Turing machine is still the best computational model for the time period that you are in, which is not necessarily true for the future, it is just true until this point in time.


Why use semi-gradient instead of full gradient in RL problems, when using function approximation? Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed?

I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?

Semi gradient methods work well in Reinforcement Learning, but what is the reason of not using the true gradient if it can be computed?

Just complexity and extra computation, in many cases for a marginal benefit.

  I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?

It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:

size of minibatch to train from experience replay on each step
number of training steps between taking frozen copies for estimation*
whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)

There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.

* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.


What data formats/pipelining are best to store and wrangle data which contains both text and float vectors? Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem to be meant for either pure text or pure numerical data.

There are those that can handle data with the dual data types, but those are generally not flexible for wrangling. For example, for pickle you have to load the entire thing into memory if you want to wrangle anything. You can just append directly to the disk like you can with hdf5, which can be very helpful for huge datasets which can not be all loaded into memory?

Also, any alternatives to Pandas for wrangling Huge datasets? Sometimes you can't load all the data into Pandas without causing a memory crash.

There are different possible ways to handle huge datasets:

If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in the article Why and How to Use Pandas with Large Data, section 1. Read CSV file data in chunk size. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.

If the data is really big, probably it's better to store and process it by multiple computers using special software, A specific tool depends on what you want to do with the data:

2.1. You can even keep using Pandas: there is an extension named Dask which wraps interfaces of Python iterables, NumPy, and Pandas to run on multiple machines. Also, there are other similar tools.
2.2. A more independent yes simple approach is to use some analytical, distributed DBMS like Google BigQuery or Cassandra.
2.3. And if you need something more powerful and complex, MapReduce systems like Apache Hadoop, Spark can be utilised. Moreover, there are many articles and scientific papers that describes usage of MapReduce for the machine learning.

So, as always, it is a matter of your opportunities and your aims of dealing with the data.


Is there any real-time computer vision system that can learn to detect new objects of new classes? Suppose you have a ground plane and can use a stereo vision system to detect things that are possibly separate objects.
Suppose also your robot or agent can attempt to pick up and move these objects around in real-time.
Is there any current system in computer vision that allows new objects to be learned in real-time?

You are probably looking for incremental learning (sometimes known as lifelong learning) techniques, i.e. machine learning techniques that attempt to address the catastrophic forgetting effect of neural networks when trained incrementally, i.e. as new classes or data are added to the original training data.
There are different techniques and some of them store (or compress) the old data in order to fully or partially re-train the neural network with the new classes or data.  However, note that this is a relatively new area of research and significant progress still needs to be made to produce serious tools. If you are specifically interested in incremental class learning, maybe have a look at this paper Class-incremental Learning via Deep Model Consolidation (2020).


What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)? I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of the output for negative inputs is a learnable parameter while in the Leaky ReLU it's a hyperparameter.

However, I'm not able to tell if there are cases where is more convenient to use ReLU instead of Leaky ReLU or Parametric ReLU.

Combining ReLU, the hyper-parameterized1 leaky variant, and variant with dynamic parametrization during learning confuses two distinct things:

The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation — Saturation is thee loss of signal to either zero gradient2 or the dominance of chaotic noise arising from digital rounding3.
The comparison between training-dynamic activation (called parametric in the literature) and training-static activation must be based on whether the non-linear or non-smooth characteristics of activation have any value related to the rate of convergence4.

The reason ReLU is never parametric is that to make it so would be redundant.  In the negative domain, it is the constant zero.  In the non-negative domain, its derivative is constant.  Since the activation input vector is already attenuated with a vector-matrix product (where the matrix, cube, or hyper-cube contains the attenuation parameters) there is no useful purpose in adding a parameter to vary the constant derivative for the non-negative domain.

When there is curvature in the activation, it is no longer true that all the coefficients of activation are redundant as parameters.  Their values may considerably alter the training process and thus the speed and reliability of convergence.

For substantially deep networks, the redundancy reemerges, and there is evidence of this, both in theory and practice in the literature.

In algebraic terms, the disparity between ReLU and parametrically dynamic activations derived from it approaches zero as the depth (in number of layers) approaches infinity.
In descriptive terms, ReLU can accurately approximate functions with curvature5 if given a sufficient number of layers to do so.

That is why the ELU variety, which is advantageous for averting the saturation issues mentioned above for shallower networks is not used for deeper ones.

So one must decided two things.

Whether parametric activation is helpful is often based on experimentation with several samples from a statistical population.  But there is no need to experiment at all with it if the layer depth is high.
Whether the leaky variant is of value has much to do with the numerical ranges encountered during back propagation.  If the gradient becomes vanishingly small during back propagation at any point during training, a constant portion of the activation curve may be problematic.  In such a scase one of the smooth functions or leaky RelU with it's two non-zero slopes may provide adequate solution.

In summary, the choice is never a choice of convenience.

Footnotes

[1] Hyper-parameters are parameters that affect the signaling through the layer that are not part of the attenuation of inputs for that layer.  The attenuation weights are parameters.  Any other parametrization is in the set of hyper-parameters.  This may include learning rate, dampening of high frequencies in the back propagation, and a wide variety of other learning controls that are set for the entire layer, if not the entire network.

[2] If the gradient is zero, then there cannot be any intelligent adjustment of the parameters because the direction of the adjustment is unknown, and its magnitude must be zero.  Learning stops.

[3] If chaotic noise, which can arise as the CPU rounds extremely small values to their closest digital representation, dominates the correction signal that is intended to propagate back to the layers, then the correction becomes nonsense and learning stops.

[4] Rate of convergence is a measure of the speed (either relative to microseconds or relative to the iteration index of the algorithm) in which the result of learning (system behavior) approaches what is considered good enough.  That's usually some specified proximity to some formal acceptance criteria for the convergence (learning).

[5] Functions with curvature are ones that are not visualized as straight or flat.  A parabola has curvature.  A straight line does not.  The surface of an egg has curvature.  A perfect flat plane does not.  Mathematically, if any of the elements of the Hessian of the function is non-zero, the function has curvature.


How do you calculate the heuristic value in this specific case? The A* algorithm uses the "evaluation function" $f(n) = g(n) + h(n)$, where

$g(n)$ = cost of the path from the start node to node $n$
$h(n)$ = estimated cost of the cheapest path from $n$ to the goal node

But, in the following case (picture), how is the value of $h(n)$ calculated?

In the picture, $h(n)$ is the straight-line distance from $n$ to the goal node. But how do we calculate it?

The most obvious heuristic would indeed simply be the straight-line distance. In most cases, where you have, for example, x and y coordinates for all the nodes in your graph, that would be extremely easy to compute. The straight-line distance also fits the requirements of an admissible heuristic, in that it will never overestimate the distance. The travel-distance between two points can never be shorter than the straight-line distance (unless you start involving things like... teleportation).

From an image like that, the straight-line distance might be difficult to figure out yourself, which is probably why they gave you the straight-line distances on the right-hand side of the image. If the image is perfectly consistent, I suppose you could theoretically figure out by inspecting some of the roads in detail how much distance is covered per pixel. Then, you can also figure out how many pixels the figure has along the straight-line paths you're interested in, and compute the straight-line distances yourself. I have no idea if the figure was actually drawn in a 100% consistent manner though.


Possible inconsistency in the Policy Improvement equation I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78).

If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ then the policy $\pi'$ is as good or better than $\pi$.

NOTE on Convention: As per the convention of the book goes, I think they are using rewards for state to action to state transition sequence rather than state to action transition.

My questions are:

Why are they comparing state value function to action value function?
Isn't it obvious the above equation might hold true (provided we select the best action among the possible actions) even for the policy $\pi$ since then the equation will change to $q_\pi(s, \pi(s)) \geq v_\pi(s)$ and we know $v_\pi(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s)q_\pi(s, a)$?

What is the inconsistency here?

Why are they comparing state value function to action value function?

It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the values at the same $s$ and modifying how $a$ is selected, the proof makes assertions about how that choice impacts the value.

It is important to recall that $v_{\pi}(s)$ measures the expected future reward when starting in state $s$ and following policy $\pi$, and that $q_{\pi}(s,a)$ measures the expected future reward when starting in state $s$ and taking action $a$, thereafer following policy $\pi$. When $a$ is chosen using a deterministic $a = \pi(s)$ then $v_{\pi}(s) = q_{\pi}(s,\pi(s))$

  Isn't it obvious the above equation might hold true

The inequality (it is not an equation) does strictly hold true, because the equation $v_{\pi}(s) = q_{\pi}(s,\pi(s))$ is true. However, that is not terribly useful, it doesn't prove anything new.

What is interesting is if you change the decision for $a$

  (provided we select the best action among the possible actions)

You cannot do that. The policy decides the actions, by definition. Selecting a better action according to the action value function, and showing what that does is precisely what the proof is showing.

  since then the equation will change to $q_\pi(s, \pi(s)) \geq v_\pi(s)$

$q_\pi(s, \pi(s)) = v_\pi(s)$ - yes your inequality holds, but is equivalent to not changing anything.

  and we know $v_\pi(s) = \pi(a|s)q_\pi(s, \pi(s))$?

Here you have switched to using a non-deterministic policy and a deterministic policy in the same statement (you also have not defined $a$ properly), making your equation badly formed. The correct form would be:

$$v_\pi(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s)q_\pi(s, a)$$

This is not a relevant form for the initial policy improvement theorem. However it does become relevant when the theory is adapted to show improving $\epsilon$-greedy policies later.

  NOTE on Convention: As per the convention of the book goes, I think they are using rewards for state to action to state transition sequence rather than state to action transition.

That is not a convention in the book. In the book, the reward distribution is generally given as joint distribution with state transition in $p(s', r | s, a)$ - this is a very generic approach that models any discrete MDP, regardless of how the rewards are described as associated with any of $s, a, s'$ or a random factor.

Possibly the convention you are referring to is labelling the immediate reward as $R_{t+1}$, so that a small section of trajectory starting from state $S_t$ might be $S_t, A_t, R_{t+1}, S_{t+1}$ . . . some other texts will use $R_t$ here. However, that is not relevant to the policy improvement theorem, it would hold the same under either convention, just with slight relabelling of the reward indices.


How to avoid over-fitting using early stopping when using R cross validation package caret I have a data set with  36 rows and 9 columns. I am trying to make a model to predict the 9th column
I have tried modeling the data using  a range of models using caret to perform cross-validation and hyper parameter tuning: 'lm', random forrest (ranger) and GLMnet, with range of different folds and hyper-parameter tuning, but the modeling has not been very successful.
Next I have tried to use some of the neural-network models. I tried the 'monmlp'. During hyper parameter tuning I could see that the RMSE drops to a level when using ~ 6 hidden units. The problem I observe using this model is

Prediction is almost equal to data
When doing a "manual" cross validation by removing a single datapoint and using the trained model to predict, it has no predictive power

I have tried to use a range of different hidden units, but i think the problem is that the model is overfitted despite using  caret cross validation feature.
There two feedbacks I would appreciate

Is there a way to prevent overfitting, by chosen optimal number of training iterations ( optimal RMSE on out of sample ). Can this by done using caret or some other package
Am I using the right model?

I am relatively unexperienced with ML and choosing a good model is tough: when you look at the available packages it is overwhelming:
https://topepo.github.io/caret/train-models-by-tag.html

You are handling a very small dataset. The only way to prevent overfitting then is to choose a very restrictive model search space. The simpler the better, and you should prefer models involving some regularization. Even tuning hyperparameters will be hard, so avoid families with many hyperparameters. Neural networks are definitely a no-go, IMHO.
Cross-validation is important in this case, a.o. for tuning the regularization hyperparameter. You were right to perform the extra leave-one-out cross-validation. But remember checking the confidence intervals, which will be very broad.
I would suggest to use some prior business knowledge to further reduce the number of candidate predictor variables to 2-4. Then you can use a linear regression or a ridge regression (i.e. linear regression with L2 regularization, amount to be tuned as hyperparameter) using only these variables. If you have reasons to believe that the relation is very non-linear, you'll need to find another family.
Another possibility is to perform a data-driven feature selection (see https://topepo.github.io/caret/feature-selection-overview.html), but it is harder to properly implement and cross-validate. I wouldn't recommend it to a beginner. Remember that this step can also introduce overfit or instability.


Do good approximations produce good gradients? Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my gradient is a decent approximation of the gradient of the true classifier everywhere?

Specifically, suppose that I have a true loss function, $f$, and an estimation of it, $f_k$. Is it the case that there exists a $c$ (dependent on $f_k$) such that for all $x$ and $\epsilon > 0$ if $|f(x)-f_k(x)|<\epsilon$ then $|\nabla f(x) - \nabla f_k(x)|<c\epsilon$? This isn’t true for general functions, but it may be true for neural nets. If this exact statement isn’t true, is there something along these lines that is? What if we place some restrictions on the NN?

The goal I have in mind is that I’m trying to figure out how to calculate how long I can use a particular sample to estimate the gradient without the error getting too bad. If I am in a context where resampling is costly, it may be worth reusing the same sample many times as long as I’m not making my error too large. My long-term goal is to come up with a bound on how much error I have if I use the same sample $k$ times, which doesn’t seem to be something in the literature as far as I’ve found.

In general $|f(x) - f_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either.

You can also look at Sobolev Training Paper (https://arxiv.org/abs/1706.04859). In particular, note that Sobolev training was better than critic training, which indirectly may indicate approximating function may not be the same as approximating gradient and function. In Sobolev training, the network is trained to match gradient and function whereas in critic training network is trained to match function. They produce quite different results which might give us some hints about the above problem.

In general, if two functions are arbitrary close, they might not be close in gradients.

Edit:
(Trying to come up with a negative example)
Consider $f(x) = g(x) + \epsilon \sin (\frac {kx} {\epsilon}) $. $g(x)$ is some neural network. Now, we train some another neural network $h(x)$ to fit $f(x)$ and after training we get $h(x) = g(x)$ ($h(x)$ and $g(x)$ have same weights precisely). However, $\nabla f_x =\nabla g_x + k\cos (\frac {kx} {\epsilon})$ is not arbitarariy close $\nabla g_x$.

I hope this example is enough to prove that a neural network that nicely approximates the function may not nicely approximate the gradients and no such result can be proved mathematically rigorously. However, considering the paper in the discussion, you might think for practical purposes it works. However, if you have both function and grad information available that is expected to work better.


A neural network to learn the connection between two images Is it possible to build a neural network that learns the connection between two images?

Let's say I have a number of X images that related to Y images. How can I build a neural network that takes an image as an input and outputs (generates) the output image?

The Y images are generated by applying some function to the X images.

Do I need a generate neural network for that? Are conventional neural networks capable of classification only?

It is possible to have both input and output be images that differ in a predictable way. For example, architectures similar to autoencoders have been used to remove blur, change weather conditions, change between day and night photos etc. In these architectures, the training data is matching pairs of images. If your goal is to replicate some image enhancement, then often the input is artificially processed e.g. to reduce its quality in a hard to reverse way. A good example of this would be to remove distortion or noise from an image.

You can also use generative models. These are  harder to get working, but can be more flexible in that you don't need image pairs in order to train, just a set of images labelled with the traits that you want to learn. Converting an image using a generative model involves using an encoder stage to get its embedding, altering the embedding based on label you require and then feeding the new embedding into the decoder stage. This is how you might alter a face portrait from male to female, or young to old, because it is not possible to find good natural image pairs for that task.


What is the relationship between the Q and V functions? Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.

Can we say that $Q^\pi(s, a) = V^\pi(s)$?

The reason why I think this can be the case is because $Q^\pi(s, a)$ is defined as the value obtained from taking action $a$ and then following policy $\pi$ thereafter. However, the action $a$ taken is the policy according to $\pi$ for all $s \in S $. This seems to corresponds to the value function equation of $V^\pi(s_t) = r(s_t) + \gamma V^\pi(s_{t+1})$.

Can we say that $Q^\pi(s, a) = V^\pi(s)$

No.

The correct relationship is this:

$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$

or, if you have a deterministic policy $a = \pi(s)$ you can instead write:

$$V^\pi(s) = Q^\pi(s, \pi(s))$$

Intuitively, this is because the $V^\pi(s)$ is the expected future return when following the policy $\pi$ from state $s$, whilst $Q^\pi(s, a)$ is the expected future return where it ignores the policy for only the next action $a$ (which will decide immediate reward $r$ and next state $s'$ independently of the policy), and thereafter follows $\pi$.

The above equations essentially show what happens when you apply the policy in state $s$ to decide which Q value(s) to use, they remove the independent choice of $a$ in $Q^\pi(s,a)$.

One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When any algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $s$ when making a free choice of $a$ for the next step, and thereafter strictly following the policy being evaluated.

What is different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different target policy. Both approaches have the same interpretation of what $Q(s,a)$ means otherwise, and have the same relationship between Q and V for their repsective policies.


Should you reload the optimizer for transfer learning? For example, you train on dataset 1 with an adaptive optimizer like Adam. Should you reload the learning schedule etc from the end of training on dataset1 when attempting transfer to dataset2? Why or why not?

When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.


Should I use Monte Carlo or a classifier for this Decision Making problem? I want to build a model to support decision making for loan insurance proposal.

There are three actors in the problem: a bank, a loaner applicant (someone who ask for a loan) and a counselor. The counselor studies the loaner application and if it has a good profile it will propose to him loan from banks that fits his profile. Then the application is sent to the bank but the bank could refuse the applicant (based on criteria we don't know).

The counselor has also to decide whether or not he will propose to the loaner applicant a loan insurance.

The risk is that some banks reject loan applicant who accepts a loan insurance and other banks accept more applicants with a loan insurance. But there aren't rules regarding banks since some banks accept or reject applicants with loan insurance according of the type of acquisition applicants want with their loan for example.

Thus, the profile of the applicant can matter in their rejection from banks but all criteria influencing the decision are quite uncertain.

I've researched online and found several scholarly articles on using Monte Carlo for decision making. Should I use Monte Carlo or a simple classifier for this Decision Making problem ?

I saw that Monte Carlo (possibly Monte Carlo Tree Search) can be used in Decision Making and it is good when there is uncertainty. But it seems that it would forecast by producing some strategy (after running a lot of simulations) but what I want is an outcome based on both the profile of the loaner applicant and the bank knowing that criteria from banks (to accept loaner applicant from could change every six months. And I would have too model banks which seems quite difficult.

A classifier seems to me to not really fit the problem. I am not really sure. Actually, I don't see how a classifier like a decision tree, for example, would work here. Because I have to predict decision of the counselor to propose or not based on the decision of banks (and I don't know their criteria) to refuse or accept applicants who were proposed loan insurance and accepted it.

The data I have is former applicants profile who were sent to banks and if they were accepted or not by the bank, if they wanted a loan insurance or not and the type of acquisition they wanted to make with their loan.

I am new to Decision Making. Thank you!

A classifier seems to me to not really fit the problem. I am not really sure. Actually, I don't see how a classifier like a decision tree, for example, would work here. Because I have to predict decision of the counselor to propose or not based on the decision of banks (and I don't know their criteria) to refuse or accept applicants who were proposed loan insurance and accepted it.

  The data I have is former applicants profile who were sent to banks and if they were accepted or not by the bank, if they wanted a loan insurance or not and the type of acquisition they wanted to make with their loan.

Why does this seem to you like something where a classifier wouldn't fit? Unless I'm missing something, it sounds like a prototypical example of a classification problem to me.

You have:

Input features (applicants' profile)
A clear (binary?) prediction target: propose or don't propose (equivalent to predicting whether or not the bank would accept, because you'll always want to propose if the bank would accept, and never propose if the bank wouldn't accept).
Old training data containing both the input features and the matching prediction targets.

Approaches like Monte-Carlo Tree Search can only be used if you have a forward model or simulator. In your setting, you could view the features (applicants' profile) as a "game state", and model the problem as a game with two actions (propose or not propose). However, you don't have a forward model (a function that, given a current state and action, generates a possible reward and subsequent state).

In applications where MCTS is often used (such as games), you do have such a forward model: for a game like Go or chess, you can easily program the game's rules, program how you transition from one state into another when you select an action, etc. This does not appear to be the case for you.


Is reinforcement learning needed to create Strong AI? By reinforcement learning, I don't mean the class of machine learning algorithms such as DeepQ, etc. I have in mind the general concept of learning based on rewards and punishment.

Is it possible to create a Strong AI that does not rely on learning by reinforcement, or is reinforcement learning a requirement for artificial intelligence? The existence of rewards and punishment imply the existence of favorable and unfavorable world-states. Must intelligence in general and artificial intelligence in particular have a way of classifying world-states as favorable or unfavorable?

Motivational Theory Versus Learning Theory

Rewards and punishment are based on human motivational theory and are broad generalizations of the neurological factors in mental growth.  At a neurological level, there are chemically based signals, such as seratonin, dopamine, and oxytocin.  Some of these have their own pathways in the brain.

There are also pain signals arising from the body and internally generated stimulatory and inhibitory signals in chemical and electro-chemical forms.  There is neuroplasticity, where the brain circuits form new topologies.  The hippocampus is a structure known to be important in learning.1  The study of learning in the brain is fervent, but the object of study is complex.  Even so, every year, progress is made.

Multiple Dimensions or Singly Quantifiable Intelligence

Conjecture about strength versus weakness of intelligence may be of limited use.  There are two valid criticisms discussed in other Q&A here in more detail.

Twenty-two independent genes have been identified that correlate to typical academic intelligence quantification in humans, rational and linguistic, eleven of which were identified in 2017.2  There are likely more.  Creative, athletic, emotional, and leadership qualities have not yet been captured in standardized intelligence testing, so it is likely that additional genes related to those abilities exist.  The notion that intelligence can be accurately aggregated into a single g-factor is weakly supported, if it were true, implies that all twenty-two genes affect only one magical function.  Statistically, such alignment is close to impossible.
Intelligence is only meaningful in context.  Fast learning in one context appears like mental brilliance, but in another context the same learning capability may be dysfunctional.

Reinforcement in the Larger Pool of AI Research

Reinforcement Learning is not a requirement for AI as far as is currently known.  Reinforcement is a mathematical strategy for driving a set of behavioral parameters toward a defined optimum.  Although its variants work well for many applications, it is not closely aligned with any of the features of the hippocampus, the limbic system, the cerebral cortex, or any other learning systems in the human brain.

Even though there is still much to learn about brain workings, the success of reinforcement learning, rule based systems, convolution, attention based networks, and generative networks provides information about what is possible in computers and what may or may not be a mechanism of the brain.

What Reward and Punishment Imply

The existence of rewards and punishment imply the existence of favorable and unfavorable family, workplace, or community states.  If the term world in the question is referring to the immediately related environment, then the sentence is probably true, but it may be misleading without clarification.

World states are based on economics which involve the minds of many people in community, aggressive actions, resource acquisition, consumption, cultural perspectives, and many other factors.  A single individual's perception of world states is subjective.

Defying Motivational Mechanisms and Intelligence

The same person on two different days may have different ideas of what is favorable or unfavorable, which may or may not correlate to reward and punishment motivations.  The words rebellion and indifference describe states where external reward and punishment produce opposing or independent results in learned behavior respectively.  Some of the most intelligent behavior in humans were of these kinds.

What drives the classification of external states within a single individual is something not well understood.  The neurological factors involved in the emergence of opposition to pain, pleasure, reward, and punishment is an interesting area of study and certainly related to AI.

Genetic factors involved in such contrary learning are likely to fall outside the twenty-two genes, since those genes correlated with standardized academic testing.  Academia is about conformance.  If we don't agree with the author of the textbook for a course, we pretend we do when we take the test if we want a high grade.

World-changing intelligence, such as that of Ghandi's nonviolent opposition, Isaac Newton's invention of physics, the furtherance of logical disproof by Socrates, or whoever dared to learn how to light a fire. is always the exhibition of classifying in a way that contradicts that of the surrounding culture.

Discovery and Acceptance in Group Learning

This is even true of project-changing intelligence.  A software engineer might say, "That software will never work because the database schema indicates a one to many relationship between companies and office addresses, but it is a many to many relationship."  Such a statement doesn't change the world and may completely lack group acceptance when spoken.  Later in the project, after the statement is fully accepted, the reclassification changes the design so that the project is not caught in a well of wasted coding.

Interestingly, convincing the others often falls to someone other than the originator of the idea — someone else on the team that has a different set of mental abilities.

It is true that classification of what is favorable or unfavorable is key, but how that classification exhibits intelligence is not well understood.

What Benefits AI Development

Although not a requirement, a solid approach to AI development is discovery, from the directions of multiple fields of study, of what intelligence is.  Some say we know what it is now, but most definitions are largely expressions of historical inaccuracies.

Clarity about what intelligence is and what it is expected to be in such a way that it can be represented mathematically is needed.  Only then can we better achieve it in humans and machines.  Sources of understanding may come from discovery in any area of study, such as these.

Evidence based psychology
Evidence based genetics
Evidence based neurological research
Evidence based artificial systems design

There is a growing trend to hack AI, but playing with AI as if it were a game might be a formula for creating Frankenstein.  A useful and collaborative collection of AI devices in robots, autonomous vehicles, and data centers will require interdisciplinary inquiry, experimentation, and engineering.  The description of this site implies that ethics should be involved, and that's a smart conception.

With a purview that crosses divisions in study and practice, central themes and models of intelligence are likely to develop.  This question is on the right track in this respect. It asks questions without accepting too quickly conjecture  — ideas and hypotheses presented but not yet shown to be true.

What is Known

That computer simulation or replication of human abilities have already been achieved and that such achievements indicate AI to be a reasonable long term objective few would deny.  Certainly in the areas of arithmetic, high speed machine control, sorting of addressed parcels, and many other roles, computers far outperform humans.  Whether world-changing intelligence will emerge in artificial systems is hoped for by some, feared by others, and asserted as inevitable by those who seek funding to prove it.

References

[1] Training your brain: Do mental and physical (MAP) training enhance cognition through the process of neurogenesis in the hippocampus?, D.M. Curlik and T.J. Shors, 2012

[2] Genome-wide association meta-analysis of 78,308 individuals identifies new loci and genes influencing human intelligence, Suzanne Sniekers et. al., 2017


How to find a cost function for human data Currently, I am interested in how NNs or any other AI models can be used for composing music.

But there are many other interesting applications too, like language processing.

I am wondering that: NNs generally need a cost function for learning. But for example, for composing music, what would be an appropriate cost function? I mean,  algorithms can't (yet) really 'calculate' how good music is, right?

You've hit upon the central conundrum of supervised learning: if you want a machine to learn to do something, you need to know how to explain what that something is.

In the case of music, there are several possible approaches:

Make one set of "bad" songs, and one set of "good" songs. Develop a measure of how similar two songs are (maybe euclidian distance between their discrete Fourier transforms is a good starting place?). Your cost function is then based minimizing the average distance to "good" songs, and maximizing the average distance to "bad" songs. This may not work well though, because good and bad songs might differ only in the occasional misplaced notes.
Move to a reinforcement learning paradigm. Listen to each song proposed by your network. Give is a score based on your subjective enjoyment. Your cost function is based on maximizing this score. This might work well, but again, it might not. Music is tricky.
Use unsupervised approaches. Reward your network just for making something that resembles music (perhaps using the Fourier transform approach above), without labelling good and bad. The advantage is that you don't need to decide what is good or bad, and so you can use a lot more music in your dataset. The drawback is music as a whole might be too diverse to learn easily from examples.
Treat your music as a sequence of notes, and train a generative model to predict future notes on the basis of past notes. You can then generate new music by starting the model with a set of notes and letting it generate new ones for a long time.


Is it feasible using today's technology to use an AI training algorithm to custom teach a robot to do common household cores? Like making a bed, washing dishes, taking out the garbage, etc., by training it on the video of specific individuals doing those cores in their own unique environments?
I have researched what machine learning is capable of doing at this point in time, and it seems this may be now feasible when done on a customer-specific basis and enable by an A.I. enhanced, full articulated, robot along the lines of an enhanced InMoov. https://en.wikipedia.org/wiki/InMoov
If it's feasible, what are the AI algorithms I should be considering to train my robot to do these tasks? Isn't deep learning the most promising of these selections:  https://www.ubuntupit.com/machine-learning-algorithms-for-both-newbies-and-professionals/?

I would suggest using a neural network with back propagation. From what I know, they can be applied to many different circumstances and work well. For your more simpler and repetitive tasks like moving an object, you can just use simpler regression methods.


How should I deal with variable-length inputs for neural networks? I am a very beginner in the field of AI. I am basically a Pharma Professional without much coding experience. I use GUI based tools for the neural network.

I am trying to develop an ANN that receives as input a protein sequence and produces as output a drug molecule. Drug molecules can be represented as fixed-length binary (0-1). This length is 881 bit.

However, I do not know how to transform protein sequences of variable length into a fixed-length binary representation.

So, how should I deal with variable-length inputs for a neural network? What is the best way?

The most common way people deal with inputs of varying length is padding.

You first define the desired sequence length, i.e. the input length you want your model yo have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired length, usually you'd split it into multiple inputs.


What would the commercial application of a conscious AI look like/be? Sometimes, but not always in the commercialization of technology, there are some low hanging fruits or early applications, I am having trouble coming up with examples of such applications as they would apply to a conscious AI.

As per conscious I would propose an expanded strict definition: the state of being awake and aware of one's surroundings along with the capability of being self aware.

Thanks.

They may be just for fun. If you had a robot that understood you, could hold a conversation with you about your interests, and even had goals of its own (good or bad), it wouldn't really need to do anything special. People would buy it like it was a toy or game.

Also, they might be usable as programmers, artists, designers, anything creative that a computer can't successfully do on its own.

It really just depends on what you define as 'consciousness'. Does it just understand what it's supposed to do, decide if it wants to, and if so, complete the task? Or does it wonder about religion, politics, moral situations, etc. that even regular humans don't fully understand? If it was pretty much just a human, it wouldn't be any more useful than one. Of course unless it can solve problems super quickly and effectively, then it would just be a really good worker.


Can an artificial intelligence eventually think like a human? It seems to me that the way neural networks are trained is similar to the way we educate a child (or a person, in general).

Can an AI eventually think like a human?

You need to fully understand the human mind before you can say that a  machine can eventually think like a human. However, the human brain is nowhere near fully understood, nor is human cognition.


How important is consciousness for making advanced artificial intelligence? How important is consciousness and self-consciousness for making advanced AIs? How far away are we from making such?

When making e.g. a neural network there's (very probably) no consciousness within it, but just mathematics behind, but do we need the AIs to become conscious in order to solve more complex tasks in the future? Furthermore, is there actually any way we can know for sure if something is conscious, or if it's just faking it? It's "easy" to make a computer program that claims it's conscious, but that doesn't mean it is (e.g. Siri).

And if the AIs are only based on predefined rules without consciousness, can we even call it "intelligence"?

Artificial consciousness is a challenging theoretical and engineering objective. Once that major challenge is met, the computer's conscious awareness of itself would likely be a minor addition, since the conscious computer is just another object of which its consciousness can be aware.

A child can look in the mirror and recognize that moving their hands back and forth or making faces produces corresponding changes in the reflection.  They recognize themselves.  Later on they realize that exerting physical control over their own movement is much easier than exerting control over another person's hands or face.

Some learn that limited control of the faces and manual operations of others is possible if certain social and economic skills are mastered.  They become employers, landlords, investors, activists, writers, directors, public figures, or entrepreneurs.

Anyone who has studied the cognitive sciences or experienced the line between types of thought because they are a professional counselor or just a deep listener knows that the lines around consciousness are blurry.  Consider these.

Listening to speech
Watching a scene
Focusing on a game
Presenting an idea
Washing up for work
Driving a car
Choosing a purchase

Any one of these things can be done with or without certain kinds of consciousness, subconsciousness, impulse, or habit.

Subjectively, people report getting out of the car and not recalling having driven home.  One can listen to someone talking, nod in affirmation, respond with, "Yeah, I understand," and even repeat what they said, and yet appear to have no memory of the content of the speech if queried in depth.  One can read a paragraph and get to the end without comprehension.

In contrast, a person may mindfully wash up for work, considering the importance of hygiene and paying attention like a surgeon preparing for an operation, noticing the smell of the soap and even the chlorination of the city water.

Between those extremes, partial consciousness is also detectable by experiment and in personal experience.  Consciousness most definitely requires attention functionality, which tentatively supervises the coordination of other brain-body sub-systems.

Once a biological or artificial system achieves the capacity to coordinate attentively, the objects and tasks toward which they can be coordinated can be interchanged.  Consider these.

Dialog
Playing to win
Detecting honesty or dishonesty

Now consider how similar or different these mental activities are when we compare self-directed or externally directed attention.

One can talk to one's self or talk to another
One can play both sides of a chess game or play against another
One can scrutinize one's own motives or those of another

This is an illustration why the self- part of self-consciousness is not the challenge in AI.  It is the attentive (yet tentative) coordination that is difficult.  Early microprocessors, designed to work in real time control systems, included (and still include) exception signaling that simplistically models this tentativeness.  For instance, while playing to win in a game, one might try to initiate dialog with the subject.  Attention may shift when the two activities require the same sub-systems.

We tend to consider this switching of attention consciousness too.  If we are the person trying to initiate dialog with the person playing to win, we might say, "Hello?"  The question mark is because we are wondering if the player is conscious.

If one was to diminish the meaning of consciousness to the most basic criteria, one might say this.

  "My neural net is intelligent in some small way because it is conscious of the disparity between my convergence criteria and the current behavior of the network as it is parametrized, so it is truly an example of artificial intelligence, albeit a primitive one."

There is nothing grossly incorrect about that statement.  Some have called that, "Narrow Intelligence."  That is a slightly inaccurate characterization, since there may be an astronomical number of possible applications of an arbitrarily deep artificial network that uses many of the most effective techniques available in its design.

The other problem with narrowness as a characterization is the inference that there are intelligent systems that are not narrow.  Every intelligent system is narrow compared to a more intelligent system.  Consider this thought experiment.

  Hannah writes a paper on general intelligence with excellence, both in theoretical treatment and in writing skill.  Many quote it and reference it.  Hannah is now so successful in her AI career that she has the money and time to build a robotic system.  She bases its design on her now famous paper and spares no expense.

  To her surprise, the resulting robot is so adaptive that its adaptability exceeds even himself.  She names it Georgia Tech for fun because she lives near the university.

  Georgia becomes a great friend.  She learns at an incredible rate and is a surprisingly great housemate, cleaning better than Hannah thought humanly possible, which may be literally true.

  Georgia applies to Georgia Tech, just down the bus line from Hannah's house and studies artificial intelligence there.  Upon the achievement of a PhD after just three years of study, Georgia sits with Hannah after a well attended Thesis Publication party that Hannah graciously held for her.

  After the last guest leaves, there is a moment of silence as Hannah realizes the true state of her household.  She thinks, "Will Georgia now exceed me in her research?"  Hannah finally, sheepishly asks, "In complete honesty, Georgia, do you think you are now a general intelligence like me?"

  There is a pause.  With a forced look of humility, Georgia replies, "By your definition of general intelligence, I am.  You are no longer."

Whether this story becomes true in 2018, 3018, or never, the principle is clear.  Georgia is just as able to analyze herself comparatively with Hannah as Hannah is similarly able.  In the story, Georgia applies the definition created in Hannah's paper because Georgia is now able to conceive of many definitions of intelligence and chooses Hannah's as the most pertinent in the context of the conversation.

Now imagine this alteration to the story.

  ... She thinks, at what level is Georgia thinking?  Hannah finally, sheepishly asks, "In complete honesty, Georgia, are you now as conscious as me?"

  Georgia thinks through the memory of all uses of the word conscious in her past studies — a thousand references in cognitive science, literature, law, neurology, genetics, brain surgery, treatment of brain injury, and addiction research.  She pauses for a few microseconds to consider it all thoroughly, while at the same time sensing her roommates body temperature, neuro-chemical balances, facial muscle motor trends, and body language.

  Respectfully, she waits 3.941701 extra seconds, which she calculated as the delay that would minimize any humiliation to Hannah, whom she loves, and replies, "Conscious of what?"

In Georgia's reply may be a hypothesis of which Hannah may or may not be aware.  For any given automatons, $a, b, \ldots$, given consciousness, $C$, of a scenario $s$, we have a definition, $\Phi_c$ that can be applied to evaluate the aggregate of all aspects of consciousness of any of the automations, $x$, giving $\Phi_c(C_x(s))$.  Georgia's (apparently already proven) hypothesis is thus.

  $\forall \Phi_c(C_a(s)) \;\;\; \exists \;\;\; b, \, \epsilon>0 \;\; \ni \;\; \Phi_c(C_b(s)) + \epsilon > \Phi_c(C_a(s))$

This is a mathematical way of saying that there can always be someone or some thing more conscious of a given scenario, whether or not she, he, or it is brought into existence.  Changing the criteria of evaluation from consciousness to intelligence, we have thus.

  $\forall \Phi_i(C_a(s)) \;\;\; \exists \;\;\; b, \, \epsilon>0 \;\; \ni \;\; \Phi_i(C_b(s)) + \epsilon > \Phi_i(C_a(s))$

One can only surmise that Hannah's paper defines general intelligence relative to what whatever is the smartest thing around, which was once well-educated human beings.  Thus Hannah's definition of intelligence is dynamic.  Georgia applies the same formula to the new situation where she is now the standard against which lesser intelligence is narrow.

Regarding the ability to confirm consciousness, it is actually easier to confirm than intelligence.  Consider this thought experiment.

  Jack is playing chess with Dylan using the new chess set that Jack bought.  In spite of the aesthetic beauty of this new set, with its white onyx and black agate pieces, Dylan moves each piece with prowess and checkmates Jack.  Jack wonders if Dylan is more intelligent than him and asks what would be a normal question under those conditions.

  "Dylan, buddy, how long have you been playing chess?"

Regardless of the answer and regardless whether Dylan is a robot with a quantum processor of advanced AI or a human being, the intelligence of Dylan cannot be reliably gauged.  However, there is NO DOUBT that Dylan was conscious of the game play.

In the examples in the lists at the top of this answer there are a particular sets of requirements to qualify as consciousness.  For the case of Jack and Dylan playing, a few things MUST working in concert.

Visual recognition of the state of the board
Motor control of the arm and hand to move pieces
Tactile detection in finger and thumb tips
Hand-eye cordination
Grasp coordination
A model of how to physically move board pieces
A model of the rules of chess in memory
A model of how to win when playing it (or astronomical computational power to try everything possible permutation that makes any sense)
An internal representation of the board state
Attention execution, visually and in terms of the objective of winning
Prioritization that decides, unrelated to survival odds or asset accumulation, whether to beat Jack in chess, do something else, or nothing (non-deterministic if the ancient and commonplace notion of the causal autonomy of the soul is correct)

The topology of connections are as follows, and there may be more.

  1 ⇄ 4 ⇄ 2

  3 ⇄ 5 ⇄ 2

  4 ⇄ 6 ⇄ 5

  7 ⇄ 8 ⇄ 9

  6 ⇄ 10 ⇄ 8

  10 ⇄ 11

This is one of many integration topologies that support one of many types of things to which consciousness might apply.

Whether looking in the mirror just to prepare for work or whether looking deeply, considering the ontological question, "Who am I?" each mix of consciousness, subconsciousness, impulse, and habit require a specific topology of mental features.  Each topology must be coordinated to form its specific embodiment of consciousness.

To address some other sub-questions, it is easy to make a machine that claims itself to be conscious a digital voice recorder can be programmed to do it in five seconds by recording yourself saying it.

Getting a robot to read this answer or some other conception, consider it thoughtfully, and then construct the sentence from knowledge of the vocabulary and conventions of human speech to tell you its conclusion is an entirely different task.  The development of such a robot may take 1,000 more years of AI research.  Maybe ten.  Maybe never.

The last question, switched from plural to singular is, "If [an artificially intelligent device] is only [operating] on predefined rules, without consciousness, can we even call it intelligent?" The answer is necessarily dependent upon definition $\Phi_i$ above, and, since neither $\Phi_c$ nor $\Phi_i$ have a standard definition within the AI community, one can't determine the cross-entropy or correlation.  It is indeterminable.

Perhaps formal definitions of $\Phi_c$ and $\Phi_i$ can now be written and submitted to the IEEE or some standards body.


Why are state-values alone not sufficient in determining a policy (without a model)? "If a model is not available, then it is particularly useful to estimate action values (the
values of state-action pairs) rather than state values. With a model, state values alone are
sufficient to determine a policy; one simply looks ahead one step and chooses whichever
action leads to the best combination of reward and next state, as we did in the chapter on
DP. Without a model, however, state values alone are not sufficient. One must explicitly
estimate the value of each action in order for the values to be useful in suggesting a policy."

The above extract is from Sutton and Barto's Reinforcement Learning, Section 5.2 - part of the chapter on Monte Carlo Methods.
Could someone please explain in some more detail, as to why it is necessary to determine the value of each action (i.e. state-values alone are not sufficient) for suggesting a policy in a model-free setting?

P.S.
From what I know, state-values basically refer to the expected return one gets when starting from a state (we know that we'll reach a terminal state, since we're dealing with Monte Carlo methods which, at least in the book, look at only episodic MDPs). That being said, why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values? I'm a little confused, it'd really help if someone could clear it up.

why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values?

A policy function takes state as an argument and returns an action $a = \pi(s)$, or it may return a probability distribution over actions $\mathbf{Pr}\{A_t=a|S_t=s \} =\pi(a|s)$.
In order to do this rationally, an agent needs to use the knowledge it has gained to select the best action. In value-based methods, the agent needs to identify the action that has the highest expected return. As an aside, whilst learning it may not take that action because it has decided to explore, but if it is not capable of even identifying a best action then there is no hope of it ever finding an optimal policy, and it cannot even perform $\epsilon$-greedy action selection, which is a very basic exploration approach.
If you use an action value estimate, then the agent can select the greedy action simply:
$$\pi(s) = \text{argmax}_a Q(s,a)$$
If you have state values, then the agent can select the greedy action directly only if it knows the model distribution $p(r,s'|s,a)$:
$$\pi(s) = \text{argmax}_a \sum_{r,s'}p(r,s'|s,a)(r + \gamma V(s'))$$
In other words, to find the best action to take the agent needs to look ahead a time step to find out what the distribution of next states would be following that action. If the only values the agent knows are state values, this is the only way the agent can determine the impact of any specific action.
Although there are alternatives to this specific equation, there is no alternative that does not use a model in some form. For instance, if you can simulate the environment, you could simulate taking each action in turn, and look over multiple simulation runs to see which choice ends up with the best $(r + \gamma V(s'))$ on average. That would be a type of planning, and perhaps the start of a more sophisticated approach such as MCTS. However, that simulation is a model - it needs access to the transition probabilities in some form in order to correctly run.
It is possible to have an entirely separate policy function that you train alongside a state value function. This is the basis of Actor-Critic methods, which make use of policy gradients to adjust the policy function, and one of the value-based methods, such as TD learning, to learn a value function that assists with calculating the updates to the policy function. In that case you would not be using a value-based method on its own, so the quote from that part of Sutton & Barto does not apply.


Should the actor or actor-target model be used to make predictions after training is complete (DDPG)? The situation

I am referring to the paper T. P. Lillicrap et al, "Continuous control with deep reinforcement learning" where they discuss deep learning in the context of continuous action spaces ("Deep Deterministic Policy Gradient").

Based on the DPG approach ("Deterministic Policy Gradient", see D. Silver et al, "Deterministic Policy Gradient Algorithms"), which employs two neural networks to approximate the actor function mu(s) and the critic function Q(s,a), they use a similar structure.
However one characteristic they found is that in order to make the learning converge it is necessary to have two additional "target" networks mu'(s) and Q'(s,a) which are used to calculate the target ("true") value of the reward:

y_t = r(s_t, a) + gamma * Q'(s_t1, mu'(s_t1))

Then after each training step a "soft" update of the target weights w_mu', w_Q' with the actual weights w_mu, w_Q is performed:

w' = (1 - tau)*w' + tau*w

where tau << 1. According to the paper

  This means that the target values are constrained to change slowly, greatly improving the stability of learning.

So the target networks mu' and Q' are used to predict the "true" (target) value of the expected reward which the other two networks try to approximate during the learning phase.

They sketch the training procedure as follows:

The question

So my question now is, after the training is complete, which of the two networks mu or mu' should be used for making predictions?

Equivalently to the training phase I suppose that mu should be used without the exploration noise but since it is mu' that is used during the training for predicting the "true" (unnoisy) action for the reward computation, I'm apt to use mu'.

Or does this even matter? If the training was to last long enough shouldn't both versions of the actor have converged to the same state?

It does not matter, target should converge to the main network after long enough training.


Is the PyTorch official tutorial really about Q-learning? I read Q-learning algorithm and also I know value iteration (when you update action values).  I think the PyTorch example is value iteration rather than Q-learning.

Here is the link:
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.

Q learning* and value iteration are very strongly related. When considering action values, both approaches use the same Bellman equation for optimal policy, $q^*(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma \text{max}_{a'} q^*(s', a'))$ as the basis for update steps. The differences are:

Value iteration makes updates using a model of the environment, Q learning works from samples from the environment made by an active agent.

By working from a simulated environment rather than a real one, it may not be clear when an agent is model-free or model-based (or planning rather than acting). However, the way that the simulated environment in the PyTorch example is used is consistent with a model-free method.

Value iteration loops through all possible states and actions for updates independently of any action an agent might take (in fact the agent need not exist). Q learning works with whichever states the agent experienced.

By adding experience replay memory in DQN, Q learning becomes a little bit closer to value iteration, as you can frame the memory as a learned model, plus consider it to be a type of planning (or a "sweep" through states). This is how it is described for instance in DynaQ which is an almost identical algorithm to experience replay as used in DQN when both are used in the simplest versions - see Sutton & Barto chapter 8.

Value iteration value update steps are over an expectation of next states and rewards - it processes the weighted sum $\sum_{r,s'}p(r,s'|s,a)$. Q learning update steps are over sampled next states and rewards - it ends up approximating the same expectation over many separate updates.

Even using large amounts of experience replay memory does not get Q learning the same as value iteration on this issue, samples are not guaranteed perfect. However, in a deterministic environment, this difference is not meaningful. So if you have a deterministic environment, Q learning and value iteration may also be considered a little closer in nature.

* Technically this applies to single-step Q-learning. n-step Q-learning and Q($\lambda$) use different estimates of future expected return, that are related but not the same as the single-step version shown here.


How should the neural network deal with unexpected inputs? I recently wrote an application using a deep learning model designed to classify inputs. There are plenty of examples of this using images of irises, cats, and other objects.

If I trained a data model to identify and classify different types of irises and I show it a picture of a cat, is there a way to add in an "unknown" or "not a" classification or would it necessarily have to guess what type of iris the cat most looks like?

Further, I could easily just add another classification with the label "not an iris" and train it using pictures of cats, but then what if I show it a picture of a chair (the list of objects goes on).

Another example would be in natural language processing. If I develop an application that takes the input language and spits out "I think this is Spanish", what if it encounters a language it doesn't recognize?

This is a very important problem that is usually overlooked. In fact, when training a neural network, there's often the implicit assumption that the data is independent and identically distributed, i.e., you do not expect the data to come from a distribution different than the distribution from which your training data comes, so there's also the implicit assumption that data comes from the same family of distributions (e.g. only Gaussians) and that all your training examples are independently drawn from the same distribution (specific mean and variance). Of course, this is a big limitation!
A partial solution to your problem is to use a Bayesian neural network (BNN). The idea of a BNN is to associate, rather than a single number, a distribution (usually a Gaussian distribution) with each unit (or neuron) of the neural network. Therefore, for each unit of the network, there are two learnable parameters: the mean and variance of a Gaussian distribution. Consequently, a BNN usually has the double of the number of parameters of a conventional (or non-Bayesian) neural network. However, by learning a distribution for each parameter, you also learn the uncertainty about the potential true value of each unit, based on the available training data.
The forward passes of such a BNN are stochastic, i.e. you sample from each of these Gaussian distributions for every forward pass, so the output of the network is also stochastic (i.e. given the same input example, the output may be different each time).
If your dataset is small, one expects the BNN to have wide Gaussian distributions, i.e. high uncertainty about the true value of the units. So, one expects a BNN to be able to deal with unexpected inputs more robustly. To be more precise, if you train a BNN with a small dataset, the hope is that Gaussian distributions will be wide and thus the outputs of the BNN will be highly variable (i.e. the model is highly uncertain). The more data is gathered, the less uncertain the BNN should be.
This doesn't completely solve your problem, but it should at least mitigate it, i.e. if you provide an unseen example to the BNN, then ideally it should be uncertain about the actual label of that input.
For simplicity, I didn't explain certain details of BNNs, but, at least, this answer gives you a potential solution. Of course, this doesn't exclude the possibility of having an "unknown" class. The approaches are not mutually exclusive. There may also be other solutions, but I am not aware of any.


Why do we use $D(x \mid y)$ and not $D(x,y)$ in conditional generative adversarial networks? In conditional generative adversarial networks (GAN), the objective function (of a two-player minimax game) would be

$$\min _{G} \max _{D} V(D, G)=\mathbb{E}_{\boldsymbol{x} \sim p_{\text {data }}(\boldsymbol{x})}[\log D(\boldsymbol{x} | \boldsymbol{y})]+\mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log (1-D(G(\boldsymbol{z} | \boldsymbol{y})))]$$

The discriminator and generator both take $y$, the auxiliary information.

I am confused as to what will be the difference by using $\log D(x,y)$ and $\log(1-D(G(z,y))$, as $y$ goes in input to $D$ and $G$ in addition to $x$ and $z$?

It looks like you're asking about the difference between using conditional and joint probabilities.

The joint probability $$D(x,y)$$ is the probability of x and y both happening together.

The conditional probability $$D(x | y)$$ is the probability that x happens, given that y has already happened. So, $$D(x,y) = D(y) * D(x | y)$$.

Notice that, in a C-GAN, we have some extra information that is given, like a class label $y$. We actually don't care at all about how likely that information is to appear. We care only about how likely it is to appear with a given $x$ from the source distribution, versus how likely it is to appear with a given $z$ from the generated distribution.

If you tried to minimize the joint probabilities, you would be attempting to change something that the networks have no ability to control (the chance of $y$ appearing).


Which functions can't neural networks learn efficiently? There are a lot of papers that show that neural networks can approximate a wide variety of functions. However, I can't find papers that show the limitations of NNs.

What are the limitations of neural networks? Which functions can't neural networks learn efficiently (or using gradient-descent)?

I am looking also for links to papers that describe these limitations.

One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible.

  "A feedforward network with a single layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly." - Ian Goodfellow, DLB

I can't think of any function that I would definitively declare as unlearnable, but neural networks have many problems. Consider adversarial examples and adversarial patches, which highlight the poor generalization going on under the hood of recent advances in computer vision.

Neural Networks are also inherently limited by the innate priors baked into their architecture and the sample density of their training data. Check out this recent discussion at Stanford's AI Salon between Yann LeCun and Christopher Manning on innate priors if that is the kind of limitation you are talking about.


What is the difference between artificial intelligence and cognitive science? Sometimes I understand that people doing cognitive science try to avoid the term artificial intelligence. The feeling I get is that there is a need to put some distance to the GOFAI.

Another impression that I get is that cognitive science is more about trying to find out how the human intelligence or mind works. And that it would use artificial intelligence to make tests or experiments, to test ideas and so forth.

Is artificial intelligence (only) a research tool for cognitive science? What is the difference between artificial intelligence and cognitive science?

Another impression that I get is that cognitive science is more about trying to find out how the human intelligence or mind works. And that it would use artificial intelligence to make tests or experiments, to test ideas and so forth.

I think that's pretty much it.  I mean, clearly there is some overlap, but I feel like most people who use "cognitive science" are referring more to understanding human intelligence for its own sake.  Artificial Intelligence, OTOH, is more about implementing "intelligence" on a computer, where the techniques used may or may not be influenced by research done under the rubric of cognitive science.


Is the AI Effect caused by bad tests of intelligence? According to Wikipedia...

The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not real intelligence.
Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was chorus of critics to say, 'that's not thinking'."[1] AI researcher Rodney Brooks complains "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"[2]

The Wikipedia page then proposes several different reasons that could explain why onlookers might "discount" AI programs. However, those reasons seem to imply that the humans are making a mistake in "discounting" the behavior of AI programs...and that these AI programs might actually be  intelligent. I want to make an alternate argument, where the humans are making a mistake, but not in "discounting" the behavior of AI programs.
Consider the following situation. I want to build a machine that can do X (where X is some trait, like intelligence). I am able to evaluate intuitively whether a machine has that X criteria. But I don't have a good definition of what X actually is. All I can do is identify whether something has X or not.
However, I think that people who has X can do Y. So if I build a machine that can do Y, then surely, I built a machine that has X.
After building the machine that can do Y, I examine it to see if my machine has X. And it does not. So my machine lacks X. And while a machine that can do Y is cool, what I really want is a machine that has X. I go back to the drawing board and think of a new idea to reach X.
After writing on the whiteboard for a couple of hours, I realize that people who has X can do Z. Of course! I try to build a new machine that can do Z, yes, if it can do Z, then it must have X.
After building the machine that can do Z, I check to see if it has X. It does not. And so I return back to the drawing board, and the cycle repeats and repeats...
Essentially, humans are attempting to determine whether an entity has intelligence via proxy measurements, but those proxy measurements are potentially faulty (as it is possible to meet those proxy measurements without ever actually having intelligence). Until we know how to define intelligence and design a test that can accurately measure it, it is very unlikely for us to build a machine that has intelligence. So the AI Effect occurs because humans don't know how to define "intelligence", not due to people dismissing programs as not being "intelligent".
Is this argument valid or correct? And if not, why not?

I think it is mostly right. But not that intelligence is hard to define. In my opinion, it is simple: A is more intelligent than B if A achieves some purpose in less steps than B. It is functional/algorithmic efficiency.

What is difficult to define is human intelligence.

But when someone says, "No, X is not real intelligence," what they mean is that it does not satisfy what we would consider real human intelligence.

So when people discount new and amazing discoveries in machine intelligence, it is not because they are not amazing in their own way, but because those discoveries, while exhibiting intelligence, are actually not replicating human intelligence - which is what many people actually mean when they say "that thing isn't really intelligent."

In truth we are very far, in the science of artificial intelligence, algorithmically speaking, from an artificial human intelligence (AHI).

Additional note: What's funny is that we don't call the science of artificial intelligence just 'the science of intelligence.' That we add the "artificial" qualifier by necessity pegs the science to what the artificiality implicitly emulates: human intelligence. In other words, "artificial intelligence" must be by definition more specific to the thing it allegedly artificializes than a more general science of just "intelligence."


DQN rgb input channels problem using pytorch I've been trying to learn about CNN's and reinforcement learning and I found this project to play with: https://github.com/adityajn105/flappy-bird-deep-q-learning
I've been trying to change the code to work with RGB input instead of grayscale. Pre-precosessing part is fine, but I'm having a problem with state and next_state I guess because they're deque and when deque is appended shape is (4,H,W) because it's appended 4 times (4 frames). Problem I'm having is when I append frames to deque which are RGB it becomes something like (4,H,W,3). I tried some stuff that came to mind and that I googled and read about online, but I still had problems with dimensions. What should be done so that it works with RGB instead of grayscale?

You can reshape it to (12, H, W) using NumPy reshape function. By the way, this will only increase the complexity of this problem. If you want to practice RL then just get the idea from their code and try implementing on some other problem/game.


How does the gradient increase the probabilities of the path with a positive reward in policy gradient? Pieter Abbeel in his deep rl bootcamp policy gradient lecture derived the gradient of the utility function with respect to $\theta$ as $\nabla U(\theta) \approx \hat{g} = 1/m\sum_{i=1}^m  \nabla_\theta logP(\tau^{(i)}; \theta)R(\tau^{(i)})$, where $m$ is the number of rollouts, and $\tau$ represents the trajectory of $s_0,u_0, ..., s_H, u_H$ state action sequences.

He also explains that the gradient increases the log probabilities of trajectories that have positive reward and decreases the log probabilities of trajectories with negative reward, as seen in the picture. From the equation, however, I don't see how the gradient tries to increase the probabilities of the path with positive R?

From the equation, what I understand is that we would want to update $\theta$ in a way that moves in the direction of $\nabla U(\theta)$ so that the overall utility is maximised, and this entails computing the gradient log probability of a trajectory.

Also, why is $\theta$ omitted in $R(\tau^{(i)})$, since $\tau$ depends on the policy which is dependent on $\theta$ ?

Think of a surface where z-axis is $U$, x and y axis are $\theta_{1}$ and $\theta_{2}$ accordingly. Since you are following the gradient direction with respect to $\theta$ vector, that means you are moving at the direction that increases the $U$. If $R(\tau)$ is positive, you are moving towards the uphill direction and vice-versa. More formally you will say the following:

In the policy gradient algorithm, our update step is:

$ \theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta) $

So, if we select a very bad trajectory the sum of all rewards ($R(\tau)$) will be negative and the following update will shift the $\theta_{new}$ vector away from $\nabla_{\theta}U(\theta)$ vector.

If we select a good trajectory means $R(\tau)$ is positive, the update will shift $\theta_{new}$ vector towards the $\nabla_{\theta}U(\theta)$ vector. So, it will increase the probability of selecting path with positive $R$


Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers? Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.
Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?

To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:

the odd numbers
the primes
numbers expressed in digits, but listed in alphabetical order of their name in German
numbers listed in the lexical order of the reverse of their representation in base 3
the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber


Image classification with an associated matrix I have a dataset of images with 9 different classes. However, there are different categories with the same type of associated image and only can be differentiated with an associated matrix in my specific problem.

I want to train a neural network with the images and the associated matrix as inputs. What type of architecture is good to use? Or where can I find bibliography about it?

The topic is multimodal neural networks.

Here some repositories that i hope will help me a lot

https://docs.google.com/presentation/d/1z8-GeTXvSuVbcez8R6HOG1Tw_F3A-WETahQdTV38_uc/edit#slide=id.g1ea5aac985_0_892

https://github.com/prml615/prml

https://github.com/husseinmozannar/multimodal-deep-learning-for-disaster-response

https://github.com/guillaume-be/multimodal-avito

Thanks to kbrose, his suggestion led me to this type of architectures.


Why is AI safety so much harder than Isaac Asimov's "Three Laws of Robotics"? I understand that AI researchers are trying to create AI designs that allow for desired behavior without undesirable side-effects. A classic example of an attempt is Isaac Asimov's Three Laws of Robotics. This idea seems to have been debunked due to its vague phrasing. In particular, I realize that the exact 3 laws from the stories can't work, but surely there is a set of more robust ones that can limit an AI to good behavior in the same way that people are restricted by law.
Why have AI researchers not accepted an idea like the following (just making the laws more specific):

What if the UN voted for the country with the fairest laws and all
utility functions have 2/3 of their points made up of not breaking any
of those laws. If the ai has a question, it could look at court
precedent just like a judge would or ask humans (with two-thirds of
its points on the line, it should be pretty cautious).
People have been looking for loopholes in law for thousands of years
and there may not be any catastrophic ones left. (It certainly
wouldn't be violent)

I must be missing something if this is still an open problem.

The reason this is hard is because it is not trivial to understand what a law means. Many humans still have a hard time understanding laws and thus we have millions of judges and lawyers who study years to be able to even debate whether a law was broken at all.
More generally to AI, the problem of understanding laws is a byproduct of the bigger problem that AIs today do not yet have mechanisms to turn input (text, audio, other sensor data) into concepts. This is evident in even the most advanced conversation systems which struggle with dialog which is off of a particular set of subjects.
However, there are promising results with probabilistic logic programming languages that attempt to combine first-order logic with statistical inference ($\text{LP}^{\text{MLN}}$ is an example). These languages could provide, in theory, a way for future AI-lawyers (or whatever one should call them) to encode exactly what laws mean. Using this set or encoded laws, a system with a rich enough representation could determine if an action would brake a law or not. This sounds great but the "rich enough representation" is beyond what even today's best deep learning systems (or any other architecture) can do. For comparison, today's best deep learning systems don't even have good object permanence, something humans have around 4-7 months old. Asking those deep learning systems to also be able to represent the complexities of laws is profoundly outside of current AI capabilities.
Another promising area is only including text data and having artificial judges. This wouldn't immediately mean that AIs would obey laws but it is the start of AIs understanding the word of law.