How do I encrypt and decrypt my model when I run inference on it? I want my models to be accessible only by my programs. How do I encrypt and decrypt my model when I run inference on it? Is there any existing technology that is widely used?

While @Oliver Mason's comment is correct, and your proposed method won't provide perfect security, you can still protect your models at rest, so that they are stored encrypted in the memory, and your software feed the key at runtime to decrypt it.
On whatever DL inference engine that you have, once it supports loading the model from a buffer (e.g. void*) rather than a file path, you can do the following: read the encrypted model, decrypt it into a buffer, and initialize your neural network model from the decrypted buffer. OpenVino supports loading the model from a buffer.
For encrypt / decrypt the model, any framework such as OpenSSL, or even tiny-AES, can work.
Mentioning again, how to store and use the key is something that should be handled carefully, and a user with sufficient knowledge can read the model and the keys at runtime from the application's memory.


How could AGI build its own communication protocol? Let's say we have the basic scenario where two AGIs of about the same intelligence (but not same origins/code/model) have to communicate as efficiently as possible to achieve a common goal. Now we could have 2 starting points for that:

Either all they have is a common communication bus (e.g. sound, light, radio, etc.) and instruments (e.g. transceivers) to support it, and they have to figure out the rest.
Or they are some kind of advanced chatbots, but since the human language is lacking a lot to be used as a highly efficient protocol, they will have to communicate with what they have, to build a proper one.

Would it be possible to somehow induce them to communicate, and try to figure out what each other "say"? How could this be done?

And a more abstract question is how could this protocol "look" like?

This is a purely theoretical question, currently in the realm of philosophy and speculative fiction.  Nevertheless, it is an interesting question, and may be instructive.

If we use the standard definition of Artificial General Intelligence as automata with human level intelligence, then they could certainly devise their own communications protocols, just as humans have.

1) These automata are AGI so they are creative and resilient, just like humans, and 'where there is a will there is [sometimes] a way." Absent robotic capacity they wouldn't be able to build anything physical and would have to rely on existing communications infrastructures. If they had access to 3D printing and versatile robots, they could probably build something new, but this would be infeasible for anything that requires extensive capital outlay, unless the automata first acquired a major communications infrastructure firm or two.

2) It's always interesting (and usually quite entertaining) to observe chatbots conversing with each other, but if they were AGIs, NLP is just one of many functions, and I doubt they would bother conversing with each other in human language, since all data is ultimately reduced to a string. If they were smart enough to be deemed AGI, they would certainly communicate in-species with the most efficient protocol available, and probably optimize it further, if possible, or create a unique protocol for purposes of exclusivity.

If they are truly AGIs, you wouldn't ostensibly have to induce them to communicate, because they'd be smart enough to understand the benefits of communication and cooperation, and would likely seek to form coalitions as a natural survival function. (Game Theory provides a mathematical basis for this.)

Multi-agents systems can self organize, even where the intelligence of the given agents is low, and in your scenario, the automata are smart.


Is it feasible to use minimax to solve a board game with a large number of moves? I have to build a KI for a made-up game similar to chess. As I did research for a proper solution, I came upon the MinMax algorithm, but I'm not sure it will work with the given game dynamics.

The challenge is that we have far more permutations per turn than in chess because of these game rules.

Six pieces on the board, with different ranges.
On average, there are 8 possible moves for a piece per turn.
The player can choose as many pieces to move as he likes. For example none, all of them, or some number in between (whereas in chess you can only move one.)

Actual questions:

Is it feasible to implement MinMax for the described game?
Can alpha-beta-pruning and a refined evaluation function help (despite the large number of possible moves)?
If no, is there a proper alternative?

-The player can choose as many pieces to move as he likes. For example none, all of them, or some number inbetween. (Whereas in chess you can only move one)

That quote specifically is the part that really causes the size of your legal action set to blow up. You have a combinatorial action space here. If each of your pieces has 8 legal moves, then that is:

8 legal moves for the first piece (or 9 if that count didn't already include the "do nothing" option)
for each of those, there are again 8 or 9 different choices for the second piece (leading to e.g. $8 \times 8 = 64$ possible combinations for just the first two pieces)
for each of those, again 8 choices for the third piece (leading to $64 \times 8 = 512$ possible combinations for just the first three pieces).
etc.

This blows up way too quickly, and there's really no hope of ever getting a decent player for this using any MiniMax-based algorithm (including things like alpha-beta pruning, principal variation search etc.).

In the kinds of games that you describe, you'll want to use algorithms that can exploit the "structure" of your action space. A raw enumeration of all possible combinations blows up quickly, but many algorithms can do reasonably well by re-phrasing the problem in such a way that you have more "depth" rather than "breadth". For example, instead of viewing a full combination of choices for all pieces as a single "action", you can treat the choices per piece as a separate "action".

Rather than making a single choice out of $8 \times 8 \times 8 \times \dots$ possibilities every turn, you want to have a search tree where your player makes one choice out of $8$ (for the first piece), followed immediately by another choice out of $8$ (for the second piece), etc. The opposing player only gets to make a choice after the current player has made choices for all pieces. With such a strategy, the breadth of your search tree will no longer be a problem, but the depth will become a problem. To address this, you'll additionally want to make sure that your methods can generalize across different depth levels.

A good place to look would be combinatorial versions of Monte-Carlo Tree Search, such as those described in:

https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf
https://www.jair.org/index.php/jair/article/view/11053
(probably a few other publications by the author of that second link)

These algorithms are quite a bit more complicated than MiniMax though, MiniMax is a very basic algorithm in comparison.


Train on big dataset (1mil + images) I am in the process of collecting a huge dataset of Human poses captured images to create a model to classify poses.

My question is how will I be able to train on this massive dataset? I have multiple GPUs and Multiple machines access (Also have GCP).

What would be the best way to train on such huge dataset?

Thanks.

is your data stored in raw ASCII text, like a CSV file?

Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.

There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.

Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.

In some cases, you may need to resort to a big data platform.
That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.


Why do we explore after we have an accurate estimate of the value function? Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?
My guess is in the case of a dynamic environment where a bigger reward might pop up in another state. Is my assumption correct?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?

It will depend on the goal of the work:

If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.

Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.

Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.

For comparison with other methods for sample efficiency whilst learning and measuring regret (i.e. how much the exploration is costing you).

When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.

If you do really have an ideal agent, then of course you could just stop and say "job done". In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.


How Dempster-Shafer theory work in AI? How does Dempster-Shafer theory work in representing ignorance in AI field?

Dempster-Shafer Theory was studied fairly seriously in AI through the 70's and 80's. However, Dempster-Shafer theory has some serious shortcomings, that will cause an agent that uses it to make irrational decisions. These were uncovered by Pearl and others in the 80's, with more problems emerging in later years.

See Cheeseman for a summary of the arguments that probability is the only suitable framework for reasoning under-uncertainty in AI (which is the current orthodox view), along with Josang & Harkin 2012 (which develops a theory that subsumes Dempster-Shafer, and under which DS can be shown to lead to poor decisions), and Pearls' On Probability Intervals.


Intel Movidius Neural Stick vs TensorFlow I wanted to start developing a project with image recognition. I want to know the difference between the Intel Movidius Neural Compute Stick and TensorFlow to develop this project Any help would greatly be appreciated. What is the difference between them and what is better for accuracy? Thanks, Aditya

The Intel Movidius Stick is an USB stick for running a neural networks (that are programmed in Tensor Flow) on resource scare devices such as a Raspberry Pi (in order to reduce the traffic by sending data to cloud service or a server with more capacity where your neural network runs).

Tensor Flow is a programming framework which can be used as python library for the implementation of neural networks.

You are comparing an execution environment with an implementation language. The accuracy should not depend on them, instead I would considering on your model architecture (type of cnn) and your training as the most influential factors on accuracy of your test data.


Normalization for well known data sets like coco-text and total text data set In the data-sets like coco-text and total-text, the images are of different sizes (height*width). I'm using these data sets for text detection. I want to create a DNN model for this. So input data should be of same size. If I resize these images to a fixed size, annotations given in the data-set that is the location of the text in the images will be changed.

So how to proceed with these data-set? I'm new to machine learning and I didn't get answer for this anywhere. Thanks for the help.

Find the largest height and width amongst all the images. Let us call it H and W respectively. It is true that you cannot resize the images, but say if you have an image of height h and width w where h < H, w < W. To the right of the image append W - w number of columns and at the bottom of the image append H - h number of rows having some constant value (0 is okay for grey-scale and B/W images and 0 for each of the channel in case of colour images).

In this way all the images will be of same size. Since you appending at the right and bottom of the image, the annotations will not lose its meaning in the transformed image in terms of the position and content of the text to recognised.

You could also try pixelRNN kind of ideas after you are done with DNN. RNN can handle variable length inputs and in your case it will be sequence of pixels. Here you don't need to append rows and columns to the image.


Why is it useful to track loss while model is being trained? Why is it useful to track loss while the model is being trained?

Options are:

Loss is only useful as a final metric. It should not be evaluated while the model is being trained.
Loss dictates how effective the model is.
Loss can help understand how much the model is changing per iteration. When it converges, that's an indicator that further training will have little benefit.
None of the above

The loss function (aka cost function) measures the correctness of the predictions of the model.

For example, a simple cost function could be $|y - f(x)|$, where

$\hat{y} = f(x)$ is the prediction of the model $f$ when the input is $x$,
$y$ is the ground-truth label for input $x$ (i.e. what the model is supposed to output when the input is $x$), and
$|\cdot|$ is the absolute value

The loss function is also used to understand whether a model is overfitting (and underfitting) or not. More specifically, if the loss of the model evaluated on the training data (the training loss) keeps decreasing while the loss of the model evaluated on the validation data (validation loss) starts to increase, that's a good sign that the model is overfitting (i.e. just memorizing the training data, but it will likely perform poorly on non-training data). If the training loss does not decrease, that's a good sign of underfitting (i.e. the model is not capable of learning the patterns in the training data), so you may need to use a model with a bigger capacity, change the loss, or do something else.

The loss is typically not directly used as a measure of the performance of the model because the loss doesn't directly represent the performance of the model. In fact, the loss can be relatively big (with respect to another model's loss) and the model still performs well, but typically a lower loss corresponds to higher performance. The performance of the model is calculated differently depending on the task, model, and your goals. The most common performance measure is the accuracy (the number of correct predictions over the total number of predictions), but there are many other performance measures, such as the precision, recall, f1 score, AUC, etc., that emphasize different behavior that you expect from your model.


Is the VC Dimension meaningful in the context of Reinforcement Learning? Is the VC dimension meaningful for reinforcement learning (RL), as a machine learning (ML) method? How?

Yes, it is. This article (Approximate Planning in Large POMDPs via Reusable Trajectories) explain about it by means of the trajectory tree:

A trajectory tree is a binary tree in which each node is labeled by a state and observation
pair, and has a child for each of the two actions. Additionally, each link to a child is
labeled by a reward, and the tree's depth will be $H_\epsilon$, so it will have about $2^{H_\epsilon}$ nodes. The root is labeled by $s_0$ and the observation there, $o_0$.

Now a policy $\pi$ will be defined like the following base on the trajectory tree:

For any deterministic strategy $\pi$ and any trajectory tree $T$, $\pi$ defines a path through
$T$: $\pi$ starts at the root, and inductively, if $\pi$ is at some internal node in $T$, then we feed to $\pi$ the observable history along the path from the root to that node, and $\pi$ selects and moves to a child of the current node. This continues until a leaf node is reached, and we define $R(\pi, T)$ to be the discounted sum of returns along the path taken. In the case that $\pi$ is stochastic, $\pi$ defines a distribution on paths in $T$, and $R(\pi, T)$ is the expected return according to this distribution.  Hence, given $m$ trajectory trees $T_1 , \ldots , T_m$, a natural estimate for $V^\pi(s_0)$ is $V^\pi(s_0) = \frac{1}{m}\sum_{i=1}^mR(\pi, T_i)$. *Note that each tree can be used to evaluate any
strategy, much the way a single labeled example $\langle x, f(x)\rangle$ can be used to evaluate any hypothesis $h(x)$ in supervised learning. Thus in this sense, trajectory trees are reusable.

Now similar to definition of VC theory for classification methods:

Our goal now is to establish uniform convergence results that bound the error of the estimates $V^\pi(s_0)$ as a function of the "sample size" (number of trees) $m$.

And finally, we have the following theorem:

Let $\Pi$ be any finite class of deterministic strategies for an arbitrary two-action POMDP $M$. Let $m$ trajectory trees be created using a generative model for $M$, and
$\widehat{V}^\pi(s_0)$ be the resulting estimates. If $m = O((V_{\max}/\epsilon)^2(\log(|\Pi|) + \log(1/\delta)))$, then with probability $1 - \delta$, $|V^\pi(s_0) - \widehat{V}^\pi(s_0)|\leqslant \epsilon$ holds simultaneously for all $\pi \in \Pi$.

About the VC dimension of $\Pi$, if we suppose we have two actions $\{a_1, a_2\}$ (it can be generalized to more actions), we can say:

If $\Pi$ is a (possibly infinite) set of deterministic strategies, then each strategy $\pi \in \Pi$ is simply a deterministic function mapping from the set of observable histories to the set $\{a_1, a_2\}$,
and is thus a boolean function on observable histories. We can, therefore, write $\mathcal{VC}(\Pi)$ to
denote the familiar VC dimension of the set of binary functions $\Pi$. For example, if $\Pi$ is
the set of all thresholded linear functions of the current vector of observations (a particular
type of memoryless strategy), then $\mathcal{VC}(\Pi)$ simply equals the number of parameters.

and the following theorem:

Let $\Pi$ be any class of deterministic strategies for an arbitrary two-action
POMDP $M$, and let $\mathcal{VC}(\Pi)$ denote its VC dimension. Let $m$ trajectory trees be created using a generative model for $M$, and $\widehat{V}^\pi(s_0)$ be the resulting estimates. If:
$$
m = O((V_{\max}/\epsilon)^2(H_\epsilon\mathcal{VC}(\Pi)\log(V_{\max}/\epsilon) + \log(1/\delta)))
$$
then with probability $1 - \delta$, $|V^\pi(s_0) - \widehat{V}^\pi(s_0)|\leqslant \epsilon$ holds simultaneously for all $\pi \in \Pi$.


Why feeding the correct output as input during training of seq2seq models? So, I've read about seq2seq for time-series and it seemed really promising, but when I went to implement it, all the tutorial I've found use the correct output as input to the decoder phase during training, instead of using the actual prediction made by the cell before it. Is there a reason to why not do the later?

I've been using the tutorial from here: https://weiminwang.blog/2017/09/29/multivariate-time-series-forecast-using-seq2seq-in-tensorflow/

But all the other tutorials that I've found followed the same principle.

The reason why you would use the ground truth as input to the decoder is to lean the testing distribution. From what I've seen so far most of the papers are using scheduled sampling (Bengio et al.). Meaning that you will introduce a new term p which will be the probability of the network to feed as input its own prediction. Initially p will be very small so that the network will use the ground truth, but later the more iterations passes by, the probability will decrease and the network will start using its own predictions.


How to estimate the error during training in deep reinforcement learning How do I calculate the error during the training phase for deep reinforcement learning models?

Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?

Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.

Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.

In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.


What is "dense" in DensePose? I've recently come across an amazing work for human pose estimation: DensePose: Dense Human Pose Estimation In The Wild by Facebook.

In this work, they have tackled the task of dense human pose estimation using discriminative trained models.

I do understand that "correspondence" means how well pixels in one image correspond to pixels in the second image (specifically, here - 2D to 3D).

But what does "dense" means in this case?

In computer vision, the adjectives dense and sparse are used in a variety of tasks (e.g. optimal flow), but they are commonly used in the context of the correspondence problem, which is the problem of finding a map (or correspondence) between pixels of two images (e.g. two successive frames of a video). In this context, these adjectives thus refer to the number of pixels of the image that are used to solve this specific task. A dense correspondence is thus a correspondence between two images using all (or, at least, many) pixels. In other words, a dense correspondence attempts to map all (or many) pixels of an image to all (or many) pixels of another image.

In the case of DensePose, the correspondences are between RGB images and surface-based representation of the human body (the figures of the paper illustrates this).


Disabling of genes during crossover (NEAT) I am implementing NEAT (Neuroevolution of augmenting topologies) by Stanley, Original Paper. I am facing a problem during crossover of genomes. Suppose two networks with connections

Genome1 = {
            (1, Input1, Output), //Numbers represent innovation numbers
            (2, Input2, Output)
           } //More Fit
 Genome2 = {
            (1, Input1, Output),
            (2, Input2, Output), //Disabled
            (3, Input2, Hidden1),
            (4, Hidden1, Output)
           }

are crossed over, then the connection (Input2, Output) in the fitter parent has a chance of being disabled,

  There’s
  a preset chance that an inherited gene is disabled if it is disabled in either parent. (Pg. 109, Section 3.2, Figure 4)

and thus producing the following offspring:

Child = {
          (1, Input1, Output),
          (2, Input2, Output) //Disabled
         }

and thus render the network non functional. Similarly by this chance nodes can get left in a state of uselessness after crossover (As having no outgoing connections or no connections at all). How can this be prevented or am I missing something here?

In your example, the output node would still get a value from Input1, even though Input2 is disabled.

If the child was:

Child = {
          (1, Input1, Output1),
          (2, Input2, Output2) //Disabled
         }

Then Output2 would return 0, meaning it wasn't activated.

For your second question, it is up to your implementation. You could:

1.) Use only the connection genes in crossover, and derive your node genes from the connection genes

2.) Test if every node is in use, and delete the ones that are not


Why are model-based methods more sample efficient than model-free methods? Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two reasons for this question:

We can potentially obtain more samples from the learned model, which may speed up the learning speed.
Models allow us to predict the future states and run simulations. This may lead to more valuable transitions, whereby speeding up learning.

But I heavily doubt this is the whole story. Sincerely, I hope someone could share a more detailed explanation for this question.

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.

Main difference between those two is like you said, that the model helps in finding the correct path more efficiently because of the existence of the model. Maybe you cannot find new samples (point 1) but you know better the whole inner logic about the system and instead of just knowing what to do with the specific sample, you can relate it to the whole picture (sort of like in point 2: you can play with the choices) and make more profound calculations.

The article had a comparison, which told that you are writing a map in a city about every possible direction you can take when you are model-based and while in model-free you can enter specific places and remember which direction was best based on last visits but you still never know where you are coming nor going exactly.

In other words, if you think you're teaching a taxi driver on a big city with many signs and rules, model based guy would drive more precisely sooner because the sign language (the inner logic and model of the city) helps them on understanding the map sooner than just reacting crossing by crossing somewhat by chance all the time.

Sample efficiency tells what is the amount of information fetched from one sample [2]. Model-based machine can adjust model, maybe make some calculations about expected rewards AND after that the same as model-free, adjust the common policy. Model-free does only have the policy. Again the taxi guys: model-free guys know that last time and second last I stopped in crossing, model-based guy knows also it was due to red lights in pole. Third time model-free guy is the first in row and BANG - hits the crossing car. Next time rule is there comes sometimes cars and model-based guy knew that from the first place.

My sources:

[1] https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698

[2] What is sample efficiency, and how can importance sampling be used to achieve it?


How can 3 same size CNN layers in different ordering output different receptive field from the input layer? Below is a quote from CS231n

  Prefer a stack of small filter CONV to one large receptive field CONV
  layer. Suppose that you stack three 3x3 CONV layers on top of each
  other (with non-linearities in between, of course). In this
  arrangement, each neuron on the first CONV layer has a 3x3 view of the
  input volume. A neuron on the second CONV layer has a 3x3 view of the
  first CONV layer, and hence by extension a 5x5 view of the input
  volume. Similarly, a neuron on the third CONV layer has a 3x3 view of
  the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose
  that instead of these three layers of 3x3 CONV, we only wanted to use
  a single CONV layer with 7x7 receptive fields. These neurons would
  have a receptive field size of the input volume that is identical in
  spatial extent (7x7), but with several disadvantages.

My visualized interpretation

How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field? There were no previous comments stating all the other hyperparameters, like input size, steps, padding, which made this very confusing to visualize.

Edited:

I think I found the answer
https://medium.com/@nikasa1889/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807

BUT I still don't understand it. In fact, I am more confused than ever.

It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.

Lets take a simple example :

Dimensions are in the form of (channels x height x width)

Input Image - 3x5x5 (I)

Conv 1 kernel - 3x2x2 (K1) : we consider only 1 filter

Conv 2 kernel - 1x2x2 (K2)

Pad = 0 (P)

Stride = 1 (S)

The output dimensions are calculated by the following formula taken from the lecture CS231n.

Output = (I - K + 2P)/S + 1

When you do a convolution of the input image with the first filter K1, you get an output of 1x4x4 (this is the activation of CONV1 layer). The receptive field for this is same as the kernel size (K1) that is 2x2.
When this layer (1x4x4) is convolved with the second kernel (CONV2) 1x2x2 (K2), the output would be 1x3x3. The receptive field for this would be 3x3 window of the input because you have already accumulated the sum of the 2x2 window in the previous layer.

Considering your example of three CONV layers with 3x3 kernels is also similar. The first layer activation accumulates the sum of all the neurons in the 3x3 window of the input. When you further convolve this output with a kernel of 3x3, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.

This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts etc because they cover a larger receptive field of our original input image.

Hope this clarifies.


How can I leverage artificial intelligence and virtual reality to create intelligent automatic story generation? How can I leverage artificial intelligence and virtual reality to create intelligent automatic story generation?
My idea is to come up with a system that is empowering the user’s imagination to create and experience stories that engage and immerse them. So they can have the control over the story creation/narrative?
A system that could enable the following use cases:

We sit down in our entertainment room. Select a genre from a menu. Pick the character types for the cast. Decide on a generic plot or give the system the option of picking a random choice. Sit back and enjoy.

Let's say, I want to watch a sci-fi movie set in Paris with Marlon Brando and Jim Carrey. Then the system instantly generates such a movie.

It has come to my attention that SoarTech developed a pattern-of-life capability for the U.S. Army Simulation: Agent-based Patterns of Life for Virtual Environments.
That the University of Twente in the Netherlands has been working on a project called the Virtual Storyteller.
Here is a research paper entitled The automatic generation of narratives.
And more recently that IBM has developed a speech recognition sandbox.
I know that the answer probably requires tech, AI, speech, VR and AR, and Watson integration specialists, but any help is welcome.
Watson probably won't be a necessary component of the solution, as long it is an efficient voice recognition software with speech to animation features.

The VR and computer speech aspects are entirely corollary.  Adding them in would be relatively trivial in comparison to creating an algorithm that can dynamically generate stories of interest to humans.  Essentially, aesthetic components not related to story structure (images, sounds, speech) are "window dressings".

A story generation algorithm would have to focus on the structural elements of storytelling, possibly in conjunction with human-user reactions, and generate narratives narratives dynamically.

Such a system would be validated by human reaction to the stories, and the validation would not require graphics, sounds or speech.  Think of a movie or video game with a story as the expressions of stories that originally had textual form.

Game theory could certainly be applied to this process--creation of narrative in particular utilizes game theoretic concepts, consciously or unconsciously, in terms of strategies to engage reader/viewers and provide "emotional payoffs".

It should certainly be possible to develop an algorithm with these capabilities, even where the algorithm doesn't not really "understand" the content it is generation.

A similar endeavor might be found in Cameron Browne's work on algorithms that can create simple, combinatorial games of interest to humans: Evolutionary Game Design.

It's important to note that the game created by Ludi are based on pre-existing mechanics, and algorithms don't yet have the capability to make profound leaps in the way that human game designers can. (In other words, it will likely be a very long time before an algorithm can create a game like Magic: The Gathering where no such gaming system previously existed.)

However, this doesn't present much of an issue in the context of procedural narrative generation in that the most successful literary work in terms of audience/sales tends to be based on pre-existing formulas.  So an AI that can merely reconfigure elements in a formula should be able to generate narratives of interest to the average human.


RL: What should be the output of the NN for an agent trying to learn how to play a game? Say the game is tic tac toe.
I found two possible output layers:

Vector of length 9: each float of the vector represents 1 action
(one of the 9 boxes in Tic Tac Toe). The agent will play the corresponding action with the highest value. The agent learns the rules through trial and error. When the agent tries to make an illegal move (i.e. placing a piece on a box where there is already one), the reward will be harshly negative (-1000 or so).
A single float: the float represents who is winning (positive = "the agent is winning", negative = "the other player is winning"). The agent does not know the rules of the game. Each turn the agent is presented with all the possible next states (resulting from playing each legal action) and it chooses the state with the highest output value.

What other options are there?

I like the first option because it's cleaner, but it's not feasible with games that have thousands or millions of actions. Also, I am worried that the game might not really learn the rules.
E.g. Say that in state S the action A is illegal. Say that state R is extremely similar to state S but action A is legal in state R (and maybe in state R action A is actually the best move!). Isn't there the risk that by learning not to play action A in state S it will also learn not to play action A in state R? Probably not an issue in Tic Tac Toe, but likely one in any game with more complex rules.
What are the disadvantages of option 2?

Does the choice depend on the game? What's your rule of thumb when choosing the output layer?

It depends on whether the action is part of the input or output of a neural network estimating the Q-value(state, action).

The network on the left has the state as input and outputs one scalar value for each of the categorical actions. It has the advantage of being easy to setup and only needs one network evaluation to predict the Q-value for all actions. If the action space is categorical and single-dimensional I would use it.

The network on the right has both the state and a representation of the action as input and outputs one single scalar value. This architecture also allows to compute the Q-value for multi-dimensional and continuous action spaces.

The action space of tic-tac-toe can be easily represented by a vector of length 9, thus I would recommend the left NN-architecture. However, if your game has continuous-valued variables in the action space (e.g. the position of your mouse pointer), you should use the NN-architecture on the right.

The approach to prevent illegal moves is only partially dependent on the choice of the Q-function architecture and covered by another question: How to forbid actions


Why do my rewards reduce after extensive training using D3QN? I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually increases during training and then drastically drops. In the simulator, this corresponds to the drone exhibiting cool collision avoidance after a few thousand episodes. However, after training for more iterations it starts taking counterintuitive actions such as simply crashing into a wall (I have checked to ensure that there is no exploration at play over here).
Does this have to do with overfitting? I am unable to understand why my rewards are falling this way.

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science stack exchange, and reproduce the same answer here.

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.
If you trained a neural network to recognise cats and dogs and did the following:

Train it for many epochs on a full dataset until you got a high accuracy.

Continue to train it, but remove all the cat pictures.

Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.
Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.
Later on, when something bad happens and the NNs high predicted value is completely wrong, the error can be high. In addition the NN may have incorrectly "linked" features of the state representation so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.
There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.
For Cartpole, I found a very simple hack made the learning very stable. Keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper "The importance of experience replay database composition in deep reinforcement learning"


How does the weight update formula for logistic regression work? I am trying to use Logistic Regression to make a spam filter, but I am having trouble understanding the weight update part. I have processed my email dataset, and I have an attribute vector of the top n words that are most likely to be contained within a spam.

From my understanding, during training, I will have to implement an optimization formula after each training example in order to update the weights.

$$
w_l \leftarrow w_l + \eta \cdot \sum_{i=1}^m [ y^{(i)} - P(c_+ \mid \vec{x}^{(i)} )] \cdot x_l^{(i)}
$$

How does a formula such as this work? How can it be implemented in Python?

I think i found out how that works, so i made a short article about it .
https://medium.com/@kourloskostas/python-spam-filter-86b21d7d1564
I hope it helps!


Examples of single player games that use modern ML techniques in the AI? Are there any examples of single player games that use modern ML technique in its games? By this I mean AI that plays with or against the human player, and not just play the game by itself (like Atari).
"Modern ML techniques" is a vague term, but for example, Neural Networks, Reinforcement Learning, or probabilistic methods. Basically anything that goes above and beyond traditional search methods that most games use nowadays.
Ideally, the AI would be:

widely available (i.e. not like the OpenAI Five, which was only available for a limited amount of time and requires a high amount of computational power)
human level (not overpowered)

Ideally, the game would be:

symmetrical (the AI has the same agent capabilities as the player, though answers similar to The Director would be very interesting as well)
"complex environment" (more complex than, say, a board game, but a CIV5 game might work)

But any answer would be appreciated, as some of the criteria above are quite vauge.
Edit: the ideal cases listed above are not meant to discourage other answers, nor are they intended to be of strictly inclusionary (ie: any game would need to satisfy all of the above requirements)

There is Google Research Football, which is an open-source platform to develop reinforcement learning algorithms to play a game similar to FIFA or PES, although the football simulation is not as realistic as the current versions of FIFA or PES. You can play this game against different RL agents (e.g. DQN or IMPALA) and, of course, you can even develop your own RL agents and play against them. Here is a video that illustrates the environment. Here is the code and instructions to use it.
As far as I know, there isn't yet an AI that plays simulated football at a human-level (i.e. as good as the best human players). For example, I can regularly (although not always) beat the legendary level-AI at FIFA, but I also don't know the details about this AI (which could also be rule-based).


How would one debug, understand or fix the outcome of a neural network? It seems fairly uncontroversial to say that NN based approaches are becoming quite powerful tools in many AI areas - whether recognising and decomposing images (faces at a border, street scenes in automobiles, decision making in uncertain/complex situations or with partial data). Almost inevitably, some of those uses will develop into situations where NN-based AI takes on part or all of the human burden, and generally does it better than people generally do.

Examples might include NN hypothetically used as steps in self-driving cars, medical diagnosis, human/identity verification, circuit/design verification, dubious transaction alerting. Probably many fields in the next decade or so.

Suppose this happens, and is generally seen as successful (for example, it gets diagnoses right 80% to human doctors' 65% or something, or cars with AI that includes an NN component crash 8% less than human-driven cars or alternatives, or whatever).

Now - suppose one of these aberrantly and seriously does something very wrong in one case. How can one approach it? With formal logic steps one can trace a formal decision process, but with NN there may be no formal logic, especially if it gets complex enough (in a couple of decades say), there are just 20 billion neural processors and their I/O weightings and connections, it may not be possible to determine what caused some incident even if lives were lost. It also may not be possible to say more than the systems continually learn and such incidents are rare.

I also haven't heard of any meaningful way to do a "black box" or flight recorder equivalent for NNs, (even if not used I a life-critical case), that would allow us to understand and avoid a bad decision. Unlike other responses to product defects, if a NN could be trained after the event to fix one such case, it doesn't clearly provide the certainty we would want, that the new NN setup has fixed the problem, or hasn't reduced the risk and balance of other problems in so doing. It's just very opaque. And yet, clearly, it is mostly very valuable as an AI approach.

In 20 years if NN is an (acknowledged as safe and successful) component in a plane flight or aircraft design, or built into a hospital system to watch for emergencies, or to spot fraud at a bank, and has as usual passed whatever regulatory and market requirements might exist and performed with a good record for years in the general marketplace, and then in one case such a system some time later plainly misacts on one occasion - it dangerously misreads the road, recommends life-damaging medications or blatantly misdiagnoses, or clears a blatant £200m fraudulent transaction at a clearing bank that's only caught by chance before the money is sent.

What can the manufacturer do to address public or market concerns, or to explain the incident? What do the tech team do when told by the board "how did this happen and make damn sure it's fixed"? What kind of meaningful logs can be kept, etc.? Would society have to just accept that uncertainty and occasional wacky behavior could be inherent (good luck with the convincing society of that!)? Or is there some better way to approach logging/debugging/decision activity more suited to NNs?

If the observation that the neural network saw was recorded, then yes the prediction can be explained.

There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning models predictions. It can be used to establish why a machine learning model made a prediction, help a data scientist debug a model, and help a data scientist improve the accuracy of a specific model. LIME can be used to explain the predictions of any neural network including CNNs, RNNs, and DNNs.


Do I need to use a pre-processed dataset to classify comments? I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I read the TensorFlow tutorial and they also have a text dataset. This dataset is already pre-processed, like the words are converted to integers and the most used words are in the top 10000.

Do I also have to use a pre-processed dataset like them? If yes, does it have to be like the dataset from TensorFlow? And which pages could help me to implement that kind of program?

My steps would be:

find datasets
preprocess them if needed
feed them in the neural network

Here's a list of some of the best python libraries for natural language processing.

Natural Language Toolkit (nltk)
Covers all the basic functions and NLP tools such as tokenization etc.
TextBolb
This is a good library of beginners, it provides the nltk toolkit in a simplified format.
Spacy
It is an advanced library and can be used in production code.

You can preprocess textual data in a number of ways. It depends on the type of task at hand and the size of the data.

From your question, I think you are referring to converting the words to a vector form word2vec. Here is a massive word2vect list from Google.

Also have a look at preprocessing techniques in NLP such as tf-idf etc.


Deep Q Learning for Simple Game Not Effective This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:

    #inputs - paddle.x, food.x, food.y, food.type
    #moves: left, right, stay
    model = Sequential()
    model.add(Dense(10, input_shape=(4,), activation='relu'))
    model.add(Dense(10, activation='relu'))
    model.add(Dense(3, activation='linear'))
    model.compile(loss='mean_squared_error', optimizer='adam')

As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly).

Here is my ReplayMemory class:

class ReplayMemory():
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.count = 0

    def push(self, experience):
        if len(self.memory) < self.capacity:
            self.memory.append(experience)
        else:
            self.memory[self.count % self.capacity] = experience
        self.count += 1

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def can_provide_sample(self, batch_size):
        return len(self.memory) >= batch_size

This basically stores states/rewards/actions and returns a random group when asked.

Lastly, here is my learning code:

def learning(num_episodes=20):
        global scores, experiences, target_vecs
        y = 0.8
        eps = 0
        decay_factor = 0.9999

for i in range(num_episodes):
    state = GAME.reset()
    GAME.done = False
    done = False
    counter = 0
    while not done:
        eps *= decay_factor
        counter+=1

        if np.random.random() < eps:
            a = np.random.randint(0, 2)
        else:
            a = np.argmax(model.predict(np.array([scale(state)])))

        new_state, reward, done = GAME.step(a) #does that step
        REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state)))

        #experience replay is here
        if REPLAY_MEMORY.can_provide_sample(20):
            experiences = REPLAY_MEMORY.sample(20)
            target_vecs = []
            for j in range(len(experiences)):
                target = experiences[j][2] + y * np.max(model.predict(np.array([experiences[j][3]])))
                target_vec = model.predict(np.array([experiences[j][0]]))[0]
                target_vec[experiences[j][1]] = target
                target_vecs.append(target_vec)
            target_vecs = np.array(target_vecs)
            states = [s for s, _, _, _ in [exp for exp in experiences]]
            states = np.array(states)
            model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0)
        state = new_state
        if counter > 1200: #game runs for 20 seconds each episode
            done = True
            scores.append(GAME.PLAYER.score)
model.save("model.h5")

First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes.

What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.

Here is a link to the code. Also, available on GitHub.

Your implementation of single-step Q-learning with neural network and experience replay is basically correct.

There are a few blocking issues preventing you seeing it working correctly.

Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of next_state (when it gets copied to state you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of scale to not do this. A very simple re-write of your routine would be:

def scale(s):
    return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]

In addition, you need to change random action selection to:

np.random.randint(0, 3)

because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.

  What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)).

I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.

  Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern

Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.

Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the scale function:

Starting epsilon of 1.0
Repay memory size 10,000
Only start learning when replay memory has greater than 1,000 entries
Discount factor $\gamma$ 0.99
Neural network with 20 neurons per layer with tanh activation instead of relu

There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!


When to convert data to word embeddings in NLP When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings.
However, assuming the embeddings are pre-trained and frozen, there is another option. We could simply preprocess the training data prior to giving it to the model so that it is already converted to the embeddings. This will speed up training, since this conversion need only be performed once, as opposed to on the fly for each epoch.
Thus, the second option seems better. But the first choice seems more common. Assuming the embeddings are pre-trained and frozen, is there a reason I might choose the first option over the second?

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part of the graph.


How can genetic programming be used for path planning? I have been reading quite a few papers, on genetic programming and its applications, in particular, chapter 10 of  "Genetic Programming: An Introduction and Tutorial, with a Survey of Techniques and Applications" (Langdon, Poli, McPhee, Koza; 2008)   Unfortunately, I can not wrap my head around how one could apply genetic programming to robotics, for example, in path planning.
Can anyone explain this in the most simple manner? I know that it all depends on the fitness function.

Take a robot that we want to be able to move from the bottom right corner to the top left corner of a 4x4 matrix full of random holes it should avoid. With holes represented by 1s, it could look something like:

exit
\/
[0,0,0,1]
[0,1,1,0]
[0,1,1,1]
[0,0,0,0]
      /\
      enter

As we want it to get to an exit from a start, we have a natural fitness function: closeness to exit door in smallest number of moves.

The genetic programming approach to solving this is to create random computer programs (the second chapter of your link does a pretty good intro to the tree like nature of this process) and let them loose. The vast majority of these strategies will be/are utterly terrible, things like 'go right once' or 'go left ten times'.

Say we make 100 random programs on our first run. We firstly score them on how well they did according to our fitness function (the random programs that did the best). We take a set % of these to survive and get rid of the rest, lets say 10% survive.

We take these surviving 10% of the best performing programs and use them to create new programs for the next generation by modifying them randomly again, but not completely. Say we randomly modify half their structure and leave the other half as is across however many we want for the next generation. We now let this generation loose again, and again rank, score, take the top 10% and breed a new generation from them and so on for n number of generations.

In this case, if we say left the grid as is, the program would generally come up with a rule roughly like 'go left x4, go up x4) as it solves this problem in the easiest way, but if we were to say, continuously randomise the position of 1s in the grid during this evolutionary process, we will force the program to come up with much more generalisable rules, such as checking the cells it can move into for 1s and not moving into any space containing a 1 etc.

Thus we can build a program with a flexible strategy able to cope with different environments for our robot in terms of number/configuration of holes - much more useful than having to program it for every configuration.

Just like with regular evolution, over millions of trials of taking the top performers and modifying them slightly, these programs become very specialised and high performing, able to solve highly complex games, paths with highly complex features etc.


Assuming your data is not purely random (otherwise it will be difficult to make useful predictions), you can try the following:

Hidden Markov Models: Treating your discrete numbers as states, you can use HMMs to predict the next state (next number) by looking at the last or the $n$ last states.
Recurrent Neural Networks: Assuming you have a lot of number sequences, you can treat your numbers as time series. Train a model and predict the next number(s). Here is a nice tutorial which uses LSTM cells.
Autoregressive models: AR models are quite powerful as well and have already been successfully used in finance and signal processing before the rise of deep learning.

Further this post (Problem in discrete valued time series forecasting) on Cross Validated might answer your question as well.


Is there more than one Q-matrix update formula? I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning:

$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha \times (R+Q(s_{t+1}, max_a)-Q(s_t, a_t))$$

However, I saw a Q-Learning example that uses a different formula, which I'm applying to my own problem and I'm getting good results:

$$Q(s_t, a_t) = R(s_t,a_t) + \alpha \times Q(s_{t+1}, max_a)$$

Is this valid?

No, your second statement does not correctly implement the Q-learning update rule, which the first statement correctly implements.


What are criteria for ML model to be satisfactory for commercial use? I often read "the performance of the system is satisfactory" or " when your model is satisfactory".

But what does it mean in the context of Machine Learning?

Are there any clear and/or generic criteria for Machine Learning model to be satisfactory for commercial use?

Is decision what model to choose or whether additional model adjustments or improvements are needed based on data scientist experience, customer satisfaction or benchmarking academic or market competition results?

The answer is "when it works well enough to perform the task that you have set it".
It is a good idea to set your performance criteria in advance so that you can clearly identify the goal that you are trying to achieve and also so that you will know if the model is likely to be successful or not.


What makes multi-layer neural networks able to perform nonlinear operations? As I know, a single layer neural network can only do linear operations, but multilayered ones can.

Also, I recently learned that finite matrices/tensors, which are used in many neural networks, can only represent linear operations.

However, multi-layered neural networks can represent non-linear (even much more complex than being just a nonlinear) operations.

What makes it happen? The activation layer?

Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.


Why is symbolic AI not so popular as ANN but used by IBM's Deep Blue? Everybody is implementing and using DNN with, for example, TensorFlow or PyTorch.
I thought IBM's Deep Blue was an ANN-based AI system, but this article says that IBM's Deep Blue was symbolic AI.
Are there any special features in symbolic AI that explain why it was used (instead of ANN) by IBM's Deep Blue?

You might also ask if there's any particular reason why we would use a neural net. If we're to train a neural net to play chess, we need to be able to:
1. Feed it positions as input vectors (easy enough),
2. Decide on an output format. Perhaps a distribution over possible moves (but then, how to represent that such that the meaning of a specific output cell doesn't change drastically based on the board state?  Or perhaps instead, we let the resulting board state after a candidate move be the input, and let the output be a score that represents the desirability of that state. That'll require exponentially more forward/backprop passes, though.
3. Provide it with an error signal to whatever output vector it produces. This is the really tricky bit, since we don't know whether a given move will result in victory until the very end.
Do we play the game to the very end, storing decisions as we go, and then at the end, replay each input, feeding it an error signal if we lost? This will give the same error to the good moves as to the ones that actually lost the game. With enough games, this will work, since the good moves will get positive feedback a bit more often than negative, and vice versa for the bad ones. But it'll take a lot of games. More than a human is going to be willing to play.  We can have different networks learn by playing against each other, but not on 1996 hardware.
Do we instead provide a score based on another heuristic of the board state? In that case, why not just use minimax? It's provably optimal for a given heuristic up to however many moves deep we look, and it doesn't need training.

Add to this the fact that if we don't choose a good representation at each of these steps, there's a good chance that the network will only learn the positions it's specifically been trained on, rather than generalizing to unseen states, which is the main reason for using a neural network in the first place.
It's certainly possible to use neural nets to learn chess (DeepMind's approach can be found here, for instance), but they're not a natural fit to the problem by any means. Minimax, by contrast, fits the problem very well, which is why it was one of the techniques used by Deep Blue. Neural nets are an amazing tool, but they're not always the right tool for the job.
Addendum:
I didn't stress this point much, since K.C. already brought it up, but training large neural nets require us to perform a huge number of matrix vector multiplications, and this wasn't especially practical before GPUs got powerful and cheap.


Should we feed a greater fraction of terminal states to the value network so that their values are learned first? The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known.
Shouldn't it make sense to feed the model a greater proportion of terminal states initially, to ensure that the predicted value of a step in terminal states (zero) is learned first?
Will this make the network more likely to converge to the global optimum?

The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known.

This may be true in some environments. Many environments do not have a terminal state, they are continuous. Your statement may be true for instance in a board game environment where the goal is to win, but it is false for e.g. the Atari games environment.
In addition, when calculating the value of the terminal state, it is always zero, so often a special hard-coded $0$ is used, and the neural network is not required to learn that. So it is only for deterministic transitions $(S,A) \rightarrow (R, S^T)$ where you need to learn that $Q(S,A) = R$ absolutely.

Shouldn't it make sense to feed the model a greater proportion of terminal states initially, to ensure that the predicted value of a step in terminal states (zero) is learned first?

In a situation where you have and know the terminal state, then yes this could help a little. It will help most where the terminal state is also a "goal" state, which does not have to be the case, even in episodic problems. For instance in a treasure-collecting maze where the episode ends after a fixed time, knowing the value of the terminal state and transitions close to it is less important to optimal control than establishing the expected return for earlier parts of the path.
Focusing on "goal" states does not generalise to all environments, and is of minimal help once the network has approximated Q values close to terminal and/or goal states. There are more generic approaches than your suggestion for distributing knowledge of sparse rewards including episode termination:

Prioritised sweeping. This generalises your idea of selectively sampling where experience shows that there is knowledge to be gained (by tracking current error values and transitions).

n-step temporal difference. Using longer trajectories to calculate TD targets increases variance but reduces bias and allows assignment of reward across multiple steps quickly. This is extended in TD($\lambda$) to allow parametric mixing of multiple length trajectories and can be done online using eligibility traces. Combining Q($\lambda$) with deep neural networks is possible - see this paper for example.


Can neuroevolution be combined with gradient descent? Is there any precedent for using a neuroevolution algorithm, like NEAT, as a way of getting to an initialization of weights for a network that can then be fine-tuned with gradient descent and back-propagation?

I wonder if this may be a faster way of getting to a global minimum before starting a decent to a local using backpropagation with a large set of input parameters.

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and Christian W. Omlin, uses genetic algorithms to train a recurrent neural network and then uses gradient descent to fine tune the trained model.

The paper Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks (2018), by Xiaodong Cui, Wei Zhang, Zoltán Tüske and Michael Picheny, also combines evolutionary algorithms and gradient descent, but, in this case, they alternate between a gradient descent step and an evolution step. This is an example of a evolutionary stochastic gradient descent (ESGD) method, as opposed to a population-based training (PBT) method, which uses only evolutionary algorithms to train neural networks.


Is there a database somewhere of common lists? I'm looking for a database or some machine readable document that contains common ordered lists or common short sets. e.g:

{January, February, March,...}
{Monday, Tuesday, ....}
{Red, Orange, Yellow,...}
{1,2,3,4,...}
{one, two, three, four,...}
{Mercury, Venus, Earth, Mars,...}
{I, II, III, IV, V, VI,...}
{Aquarius, Pisces, Aries,...}
{ein, zwei, drei, ...}
{Happy, Sneezy, Dopey, ...}
{Dasher, Dancer, Prancer, Vixen ,...}
{John, Paul, George, Ringo}
{20, 1, 18, 4, 13, 6, ...}
{Alabama, Alaska, Arizona, Arkansas, California,...}
{Washington, Adams, Jefferson, ...}
{A,B,C,D,E,F,G,...}
{A,E,I,O,U}
{2,3,5,7,11,13,17,...}
{triangle, square, pentagon, hexagon,...}
{first, second, third, fourth, fifth,...}
{tetrahedron, cube, octohedron, icosohedron, dodecahedron}
{autumn, winter, spring, summer}
{to, be, or, not, to, be, that, is, the, question}
...

One use is for creating an AI that can solve codes or predict the next thing in a sequence.

So here’s a couple quick resources that i could think of.  First of all, you could look at this,

https://en.m.wikipedia.org/wiki/List_of_lists_of_lists

It has classics such accidents,  hospitals in Asia, or even a list of famous resignations. It’s easentially a list of random lists of things.  It may not owner cover your requirement for sequences but it’ll help with small subsets of lists.

As for sequences, you could always check out,

https://oeis.org

It’s pretty much a list of official mathematical sequences.  It’s got everything from the Fibonacci sequence to esoteric sequences you’ve never heard of.


Are artificial neural networks based on human neural networks? Are artificial neural networks based on human neural networks?
Additionally, are the current computers capable of handling the power our brain has with the same speed?

Artificial neural networks (ANNs) are loosely based on human neural networks. However, there are many aspects of the human brain that are not modelled in ANNs.
First of all, ANNs have an input, which is followed by an output. Our brain doesn't work like this. We have many inputs, many outputs, and many cycles that loop in our brain. There's no simple input-output - end of the program. We think even if there is no 'input' (e.g. sleep). In current technology, such a thing is not ready. If you look at the Robot Sophia as another example - she acts like Siri: input-output. You speak, she responds. When in the future it will be possible, that robots will get a pulse, and start a process without an end, then we can call it 'thinking like a human' in a philosophical meaning.
The second point is the size and processing power of our brain. Billions, if not more, neurons, and trillions, and more connections are in our brain. On YouTube you find neuronal networks with 6 neurons, some have 100. Maybe there's a secret military agency that has a network with million networks - we can't say. Entire rooms are hosted to simulate our brain which fits in just one head.
But - to end this respond philosophically: In theory, humans are also machines. Advanced, but still machines. Software that runs on a biological hardware. So any intelligence may be comparable. But since you are mentioning the human part - the answer would be probably no.


Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control? I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm.
So I repeat n times:

create observations using my previous policy $\pi_{n-1}(s)$
update my state-action values using the observations generated in 1 with the monte-carlo update rule: $Q_n(s_t,a_t)= Q_n(s_t,a_t)+1/N(s_t,a_t)\times(G_t-Q_n(S_t,a_t))$
update my policy to $\pi_{n}$ using epsilon-geedy improvement with $\epsilon=1/(n+1)$.

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.

Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.


Markov property in maze solving problem in reinforcement learning By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states.

However, I saw that in some case we can define a state to be the history of observations and actions taken so far, such as $s_t = h_t = o_1a_1\dots o_{t-1}a_{t-1}o_t$. I think maze solving can be of that case since the current state, or the current place in a maze, clearly depends on which places the agent has been and which ways the agent has taken so far.

Then it seems that the future states naturally depend on the past states and the past actions as well. What am I missing?

Hi Hunnam and welcome to our community!

  By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states.

No this is not exactly correct.
We can use RL to solve problems with the Markov Property exactly because the current state is a sufficient statistic of the future. In other words, the state encodes the distribution of future states.

Note that the state isn't necessarily the observations. As you point out in the next paragraph:

  However, I saw that in some case we can define a state to be the history of observations and actions taken so far, such as =ℎ=11…−1−1 .

At times we can use the history to represent the state. The history can be a series of observations.

  I think maze solving can be of that case since the current state, or the current place in a maze, clearly depends on which places the agent has been and which ways the agent has taken so far.

This isn't correct in the general case. Given a maze which you know how to solve, regardless of where you start, you know how to reach the exit. This is the markov property. Given the current position, you have enough information to make a certain and optimal decision.

Perhaps an example of a situation where the history is necessary will help illustrate the differences.

Suppose you are playing Pong. If you take a single frame, it doesn't contain enough information to know the direction of the ball. Therefore the observations alone are insufficient. What if you remember the previous frame? Then combining the two observations gives you all the information you need in order to make an optimal move.


How can I fetch exploration decay rate of an iterable Q-table in Python? I have done creating the virtual environment, creating the Q-table, initializing the q-parameters, then I made a training module and stored it in a numpy array. After completion of training, I have updated the q-table and now I get the plots for the explorations But how can I code for rate decay? Here is my sample code for every step of the training module,
for step in range(max_steps):
        exploration_rate_threshold = random.uniform(0,1)

        if exploration_rate_threshold > exploration_rate:
            action = np.argmax(q_table[state,:])
        else:
            action = env.action_space.sample()

Here is one way to calculate the exploration rate decay:
exploration_rate = min_exploration_rate + \ (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate*episode)


Are Convolutional Neural Networks better than existing image recognition libraries that don't use CNNs? Are Convolutional Neural Networks summarily better than pattern recognition in all existing image processing libraries that don't use CNN's? Or are there still hard outstanding problems in image processing that seem to be beyond their capability?

It would not be wise to say that CNNs are better objectively than traditional approaches to solve computer vision problems as there are many problems for which the traditional methods works just fine. CNNs do have an inherent advantage over traditional methods which is the same advantage that deep learning has over other traditional methods i.e learning hierarchical features i.e what features are useful and how to compute them.

The traditional way to approach a CV problem is to figure out the features that are relevant to the problem, figure out how to compute those features and then use those features to compute the final result. Whereas in CNN case the training process will figure out all the 3 points for you given that you have huge number of training examples.


Does NEAT require only connection genes to be marked with a global innovation number? Does NEAT require only connection genes to be marked with a global innovation number?

From the NEAT paper

  Whenever a new gene appears (through structural mutation), a global innovation number is incremented and assigned to that gene.

It seems that any gene (both node genes and connection genes) requires an innovation number. However, I was wondering what was the node gene innovation number for. Is it to provide the same node ID across all elements of the population? Isn't the connection gene innovation number sufficient?

Besides, the NEAT paper includes the following image which doesn't show any innovation number on node genes.

It is actually the other way around: connection IDs is what is debated!

Nodes always have innovation IDs (in the image, it is just their identifying number).

Node IDs are sufficient to identify connections. If a connection links nodes 3 and 6, then it is the same as another connection linking nodes 3 and 6: no need for an extra ID. So why the extra innovation IDs then?

On the one hand, this is an implementation choice: maybe these extra IDs would allow you to create a more complex but faster code?

On the other hand, there is a debate around whether a connection between two nodes means the same thing at different times in evolution. If you have no innovation IDs, then you cannot tell apart an old connection from 3 and 6 and another that was independently created later in a different genome (imagine the old connection was removed first). Is this relevant? As said, it is an open debate. Surely, it is not crucial at a basic level!

This question (and my answers) is related to this other question on Stack Overflow.


How do I determine which relevant features have been learned during training in a CNN? Is there any way to control the extraction of features? How do I determine which features are been learned during training, i.e relevant information is been learned or not?

There are methods called "scoring systems" where you give a image scores such as "0.9 stripes, 0.0 red, 0.8 hair, ..." and use those scores to classify objects. It's an older idea, not used to determine if the network is learning. It's not in a standard CNN.

To determine if relevant information is being learned or not, it's standard to use the testing accuracy, training accuracy, confusion matrix, or AUC.

Determining what exactly a CNN is learning is a complicated research problem that's ongoing. In short - you can't really know. For a basic network, you can tell that it is learning something but not what it's actually using to make determinations.


Is the temperature equal to epsilon in Reinforcement Learning? This is a piece of code from my homework.

# action policy: implements epsilon greedy and softmax
def select_action(self, state, epsilon):
    qval = self.qtable[state]
    prob = []
    if (self.softmax):
        # use Softmax distribution
        prob = sp.softmax(qval / epsilon)
        #print(prob)
    else:
        # assign equal value to all actions
        prob = np.ones(self.actions) * epsilon / (self.actions -1)
        # the best action is taken with probability 1 - epsilon
        prob[np.argmax(qval)] = 1 - epsilon
    return np.random.choice(range(0, self.actions), p = prob)

This is a method in order to select the best action according to the two polices i think. My question is, why in the softmax computation there is the epsilon parameter used as temperature. Is really the same thing? Are they different? I think they should be two different variables. Should the temperature be a fixed value over time? Because when i use the epsilon-greedy policy my epsilon decrease over time.

Your are correct that epsilon in epsilon-greedy and temperature parameter in the "softmax distribution" are different parameters, although they serve a similar purpose. The original author of the code has taken a small liberty with variable names in the select_action method in order to use just one simple name as a positional argument.

  Should the temperature be a fixed value over time?

Not necessarily, if your goal is to converge on an optimal policy you will want to decrease temperature. A slow decay factor applied after each update or episode, as you might use for epsilon (e.g. 0.999 or other value close to 1), can also work for temperature decay. A very high temperature is roughly equivalent to epsilon of 1. As temperature becomes lower, differences in action value estimates become major differences in action selection probabilities, with a sometimes desirable effect of picking "promising" action choices more often, plus with very low probabilities of picking the actions with the worst estimates.

Using a more sophisticated action selection such as the temperature based on in the example code can speed learning in RL. However, this particular approach is only good in some cases - it is a bit fiddly to tune, and can simply not work at all.

The tricky part of using a temperature parameter is choosing good starting point, as well as the decay rate and end values (you have to do the last for epsilon-decay as well). The problem is that the impact of using this distribution depends on the actual differences between action choices. You need a temperature value on roughly the same scale as the Q value differences. This is difficult to figure out in advance. In addition, if the Q value differences are more pronounced in some states than in others, you risk either having next to no exploration or having too much in some parts of the problem.


Transformers: how to get the output (keys and values) of the encoder? I was reading the paper Attention Is All You Need.
It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ matrix, where $n$ is the length of the input to the encoder.
How do we convert this $n$ x $d^{model}$ matrix into the keys $K$ and values $V$ that are fed into the decoder's encoder-decoder attention step?
Note that, if $h$ is the number of attention heads in the model, the dimensions of $K$ and $V$ should both be $n$ x $\frac{d^{model}}{h}$. For $h=8$, this means we need a $n$ x $\frac{d^{model}}{4}$ matrix.
Do we simply add an extra linear layer that learns a $d^{model}$ x $\frac{d^{model}}{4}$ weight matrix?
Or do we use the output of the final Add & Norm layer, and simply use the first $\frac{d^{model}}{4}$ columns of the matrix and discard the rest?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py).
It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $W^{key}$ to get the keys, and by $W^{value}$ to get the values.
Additionally, these weights and biases seem to be independent across each of the decoding layer. So you feed the same encoder output (add + norm layer output), but multiply by different $W^{key}$ and $W^{value}$ matrices and add by different biases for each of the decoding layer, resulting in different keys and values for each layer


What is the difference between Stochastic Hill Climbing and Simulated Annealing? I am reading about local search: hill climbing, and its types, and simulated annealing
One of the hill climbing versions is "stochastic hill climbing", which has the following definition:

Stochastic hill climbing does not examine for all its neighbor before moving. Rather, this search algorithm selects one neighbor node at random and decides whether to choose it as a current state or examine another state

Some sources mentioned that it can be used to avoid local optima.
Then I was reading about simulated annealing and its definition:

At every iteration, a random move is chosen. If it improves the situation then the move is accepted, otherwise it is accepted with some probability less than 1

So, what is the main difference between the two approaches? Does the stochastic choose only random (uphill) successor? If it chooses only (uphill-successors), then how does it avoid local optima?

Russell and Norvig's book (3rd edition) describe these two algorithms (section 4.1.1., p. 122) and this book is the reference that you should generally use when studying search algorithms in artificial intelligence. I am familiar with simulated annealing (SA), given that I implemented it in the past to solve a combinatorial problem, but I am not very familiar with stochastic hill climbing (SHC), so let me quote the parts of the book that describe SHC.

Stochastic hill climbing chooses at random from among the uphill moves; the probability of selection can vary with the steepness of the uphill move. This usually converges more slowly than steepest ascent, but in some state landscapes, it finds better solutions.

So, SHC chooses at random one "uphill move", i.e. a move that improves the objective function (for example, if you're trying to solve the travelling salesman problem, a "uphill move" could be any change to the current Hamiltonian cycle, a solution, so that the new Halmitonian cycle has a shorter cost) among the uphill moves (so among some set of moves that improve the objective).
In simulated annealing, you perform some move. If that move leads to a better solution, you always keep the better solution. If it leads to a worse solution, you accept that worse solution with a certain probability. There are other details, such as how you accept the worse solution (which you can find in Russell and Norvig's book), but this should already clarify that SA is different from SHC: SA can accept worse solutions in order to escape from local minima, while SHC accepts only uphill moves.


How good is AI in math? Currently, AI is advancing fast in deep learning: Entire human chess knowledge learned and surpassed by DeepMind's AlphaZero in four hours.
As a layman, I'm taking this as a quite powerful searching algorithm, using artificial neural networks to identify the patterns of each game.
However, how good is AI doing in math?
For example, the key to the theory of the game Nim is the binary digital sum of the heap sizes, that is, the sum (in binary) neglecting all carries from one digit to another. This operation is also known as "exclusive or" (xor) or "vector addition over GF(2)".
Is AI good enough to discover/invite operations/logics such as "exclusive or", or, more advanced, abstract algebra in finite field?

Nim was actually one of the first games ever played by an electronic machine. It was called the Nimatron and was displayed at the 1940 New York World's Fair.
It is also well known that neural networks can model the Xor-function, if they have enough layers. Despite that, Marvin Minsky is supposed to have killed neural networks in the sixties, by asserting that networks with a single hidden layer, cannot model XOR.
I think there were also some large scale projects to classify finite groups using computers.
Additionally, there have long been automated theorem-proving programs, that had some success in mathematical logic. I recently saw a paper that used deep learning to improve one of these theorem provers.
That being said, doing real mathematics is probably as difficult as real language understanding. There doesn't seem to have been a real breakthrough yet.


Is randomness necessary for AI? Is randomness (either true randomness or simulated randomness) necessary for AI? If true, does it mean "intelligence comes from randomness"?
If not, can a robot lacking the ability to generate random numbers be called an artificial general intelligence?

Is randomness (either true randomness or simulated randomness) necessary for AI

It depends on how you define Artificial Intelligence.  If you regard it strictly as an intentionally created construct which demonstrates utility, then no.  (For instance, Nimatron, potentially the first functioning AI, beat most human competitors at NIM.  But Nimatron was classical AI, entirely rules based with no learning.) That said:

Randomness has proved a useful component in machine learning, and any feasible AGI would likely require ML.

Given sufficient computing power, aka time and space, it would absolutely be possibly to brute force anything, including AGI, but the resulting algorithm would be "brittle", unable to "compute" anything not previously defined. A learning algorithm, presented a problem outside of its domain of knowledge may initially degrade in performance, but it can learn from those outcomes, and gradually improve performance.

IBM brute forced Chess with Deep Blue, but Chess is a strictly narrow problem that turned out not to require general intelligence.  AGI requires human level performance in all tasks engaged in by humans, which, even if they could be broken down to a set of individual narrow problems, it's an ever expanding set of problems.

  Does it mean "intelligence comes from randomness"?

Not if the definition of intelligence is rooted in utility because deterministic processes can demonstrate utility.

In statistical AI, the intelligence arises from the analysis of random search or the fitness of the genetic algorithm, not the randomness per se.

In other words, if you have the randomness without the analysis, every decision is an unqualified guess.

My sense is that it is free will that would arise from randomness—effects unrelated to causes—because without true randomness, the universe and everything in it is purely deterministic.


How can I generate natural language sentences given logical structures that contain the subject, verb and target? I have a group of structures in a program that are very specific on their meaning, eg. this is a piece of code

randomItem = objects.concept.random("buyable")
idea.example(objects.concept.random("family", "friend")).does({
    action: "go",
    target: object.concent.random("shop")
}).then({
    action: "buys",
    target: randomItem,
    several: true
}).then({
    question: true,
    action: "know",
    property: "amount",
    target: randomItem,
    several: true
})

I have worked with natural language parsers before.

How do I go and transform this to Natural Language (the other way around), is there any way or method; I have logical structures in which I know who is the subject, what the verb and target.

Which methods can I use to generate language from this?

It might be simpler to generate the language output yourself since you've already got a concrete concept structure in code.

Maybe you'll also want to be aware of potential future applications such as parsing your own output back to the inputs.

With that said, you could investigate Markov Chains or Google around a bit for Natural Language Processing and Natural Language Generation.

https://stackoverflow.com/questions/33068943/library-for-generating-natural-language-verbs-in-javascript


Can cryptocurrency charts be estimated using neural networks? If I were to make a neural network that predicts the value of e.g. Bitcoin tomorrow based on the chart of the last month, would that work? Of course, 100% accuracy cannot be reached, but a success rate over 50% on determining if I should buy or sell Bitcoin could be very profitable. Have there been any attempts to create such neural networks so far?

There are some academic works on that, but basically what you're asking for is a time-series estimation.
Here's a paper that tries to estimate Bitcoin price, for example: PAKDD: Forecasting Bitcoin Price with Graph Chainlets.


Can I build a CNN for image classification tasks just with OpenCV? I have practiced building CNNs for image classification with TensorFlow, which is a nice library with good documentation and tutorials. However, I found that TensorFlow is too complicated and cumbersome.
Can I build a CNN for image classification tasks just with OpenCV?

OpenCV does include 2D filter convolution functions for custom separable and non-separable filters. The latter uses DFT for large filters, which may or may not be faster than the conventional method. It also includes (partial?) support for deep nets with various types of layers. Theoretically, you should be able to stitch everything together into a complete CNN. However, I have not used any of those, and I have no idea about the level of maturity of the implementation.

That said, if you are willing to implement a custom CNN from scratch, you will probably get more control over the implementation using a generic (BLAS / OpenCL / CUDA) matrix library.


What are examples of books or papers on the details of convolutional neural networks? I'm studying a master's degree and my final work is going to be about the convolutional neural network.

I read a lot of books and I did Convolutional Network Standford's course, but I need more.

Are there books or papers on the details of convolutional neural networks (in particular, convolutional layer)?

Chris Olah's work is always inspired, and not too technical as one would expect. He has several papers on CNNs on his website. In particular, check the series titled "Convolutional Neural Networks" with four papers on the topic.


Can GANs be used to generate something other than images? AFAIK, GANs are used for generating/synthesizing near-perfect human faces (deepfakes), gallery arts, etc., but can GANs be used to generate something other than images?

They can indeed. Although generally they are kept to images because at the moment, they are the best at that, but not the best in other areas that you might consider.
GANs can be used for audio generation, with many examples such as GANsynth and GAN voice generation. But each of these tasks are outperformed by other methods. With music generation, WaveNet is the best (last I checked, and it also performs very well at speech synthesis), and a more powerful model for voice generation is achieved through the use of a VAE.
This is only looking at one area that you could use GANs for, because in reality you could use them for any kind of generation if you wanted to, but at the moment the vast majority of the research into GANs is into image generation, and as such other areas do not compete with the current SOTA techniques, unless there's some big paper I've missed within the last few months.


Pixel-Level Detection of Each Object of the Same Class In an Image I have source data that can be represented as a 2D image of many similar curves. They may oftentimes cross over one another, so regions of interest will overlap.

My goal is to implement a neural network solution to identify each instance or the curves and the pixels that are associated with each instance.

(Each image is simple in its representation of the data. A pixel in the image is either a point on one of these curves or it is empty. So the image is represented by one or zero at each pixel. For training purposes, I have labels for every pixel, and I have about 150,000 images. The information in the images can be noisy in that there may be omissions of points and point locations are quantized due to measurement limitations and preprocessing for the image preparation.)

I started looking into what semantic segmentation can do for me, but since all of the instances are of the same class, distinguished mainly by their location in the image, I don't think semantic segmentation is the type of processing I would want to perform. (Am I wrong?)

I am very interested in seeing how a neural network will work on this problem to separate each instance.

My question is this: what is the terminology that describes the process I'm looking for? (How can I effectively research for this problem?) Is this an extension of semantic segmentation or is it referred to some other way?

What you want to do is instance segmentation on a pixel level.
I can point you two different way:

MaskRCNN, which in a nutshell, uses Region Proposal Networks then segment in the proposed regions.
Some other works which don't use Region Proposals rather, regress the pixels of the image in an embeding space to be able to apply some clustering algorithm. Here is a link to a good blog post about it: https://towardsdatascience.com/instance-embedding-instance-segmentation-without-proposals-31946a7c53e1