What is the most suitable AI technique to use for path planning? I am making a firetruck using Arduino Uno with flame sensors and ultrasonic sensors to detect how to move and where to go. As this is a project for my university, I am asked to implement AI in it for path planning.

I am not sure whether to use something like A* technique or ID3 decision tree or if there is something better than both to implement path planning for my robot. Any suggestions?

If I had to implement a path exploration/finding algorithm on a robot, I would follow these steps:

Make sure you can detect your position. You need to be able to record your position otherwise you have no reference for the exploration. You don't need a global positioning system (like GPS), a local one is more than enough in your case. This means that the robot must know that it has been switched on in position (0,0). If you go straight for 5 meters, you'll update it to (5,0) or something like that.
Knowing how much you moved and towards where is the difficult thing.
Once you can know where you are, it is time to record it. As you want to explore the environment, you might want to create a tree with states on the nodes. The node can be open if you can still explore around it, closed if the exploration has been done.
To get the path from a node to another, A* works more than enough.
To know what is around you, you can use the sensors to explore the surroundings and know the position of the obstacles.

This is the general idea:

Tree tree = new Tree()
Node first = new Node(Here)
tree.Add(first)

Node current = first

while(true) {
    ExploreAround(current)
    var nodes = CreateSurroundingNodes()
    tree.Add(nodes)
    current.State = closed
    var next = PickNearestOpenNode()
    MoveTo(next)
    WaitUntilRobotIsOn(next)
    current = next
}

In my bachelor thesis I was implementing a very similar algorithm on a drone on Unity3D. You can find the package here. You can get an idea out of it.

Here is a video of how it worked: https://youtu.be/Xrh9-4Bfcew


How do I statistically evaluate a ML model? I have a model that predicts sentiment of tweets. Are there any standard procedures to evaluate such a model in terms of its output?

I could sample the output, work out which are correctly predicted by hand, and count true and false positives and negatives but is there a better way?

I know about test and training sets and metrics like AUROC and AUPRC which evaluate the model based on known data, but I am interested in the step afterwards when we don't know the actual values we are predicting. I could use the same metrics, I suppose, but everything would need to be done by hand.

There are a lot of ways to evaluate the performance of an ML model.  You mentioned AUROC and AUPRC.  Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc.  You can see a good outline of them here.

It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data.  You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives.  I recommend you read Neil Slater's answer at https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228.  He gives some good advice on the subjectivity of sentiment analysis classification and points out a labeled data set of Tweets which you might be able to use to test your classifier.

I also found this Kaggle competition which has a test set that might be of help to you: Angry Tweets


Must all CNNs and RNNs not have a fully connected layer in order to be considered as such? In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers cannot be part of the RNN.
So, must all Convolutional Neural Networks and Recurrent Neural Networks not have a fully connected layer in order to be considered CNNs and RNNs, respectively? If yes, should we consider CNNs and RNNs with an FC layer "hybrid models"?

Not quite sure about RNN & LSTM (and it always depends on the task), but for CNN the answer is clearly no; CNN routinely include FC layers. Quoting from the highly popular (and recommended) Stanford course CS231n: Convolutional Neural Networks for Visual Recognition:

ConvNet Architectures
We have seen that Convolutional Networks are commonly made up of only
three layer types: CONV, POOL (we assume Max pool unless stated
otherwise) and FC (short for fully-connected).

You can easily verify that this is the case for practically all popular CNN models for computer vision in any relevant exposition (e.g. Illustrated: 10 CNN Architectures).
In fact, the all-convolutional NN (i.e. without FC layers) is considered a special case: see Striving for Simplicity: The All Convolutional Net.


What is the intuition behind variational inference for Bayesian neural networks? I'm trying to understand the concept of Variational Inference for BNNs. My source is this work. The aim is to minimize the divergence between the approx. distribution and the true posterior
$$\text{KL}(q_{\theta}(w)||p(w|D) = \int q_{\theta}(w) \ log \frac{q_{\theta}(w)}{p(w \mid D)} \ dw$$
This can be expanded out as $$- F[q_{\theta}] + \log \ p(D)$$ where $$F[q_{\theta}] = -\text{KL}(q_{\theta}(w) || p(w)) + E[\log p(D \mid w)]$$
Because $log \ p(D)$ does not contain any variational parameters, the derivative will be zero. I really would like to summarize the concept of VI in words.
How can one explain the last formula in words intuitive and with it on the fact that one approximates a function without really knowing it / able to compute it?
My attempt would be: Minimizing the KL between the approximate distribution and the true posterior boils down in minimizing the KL between the approximate distribution and the prior (?) and maximizing the log-likelihood that the parameters of the approximate distribution resulted in the data. Is this somehow correct?

Your description of what is going on is more or less correct, although I am not completely sure that you have really understood it, given your last question.
So, let me enumerate the steps.

The computation of the posterior is often intractable (given that the evidence, i.e. the denominator of the right-hand side of the Bayes' rule, might be numerically expensive to approximate/compute or there's no closed-form solution)

To address this intractability, you cast the Bayesian inference problem (i.e. the application of Bayes' rule) as an optimization problem

You assume that you can approximate the posterior with another simpler distribution (e.g. a Gaussian), known as the variational distribution

You formulate this optimization problem as the minimization of some notion of distance (e.g. the KL divergence) between the posterior and the VD

However, the KL divergence between the posterior and the VD turns out to be intractable too, given that, if you expand it, you will find out that there's still an evidence term

Therefore, you use a tractable surrogate (i.e. equivalent, up to some constant) objective function, which is known as the evidence lower bound (ELBO) (which is sometimes known as the variational free energy), which is the sum of 2 terms

KL divergence between the VD and the prior
the likelihood of the parameters given the data

To address your last doubt/question, the ELBO does not contain the posterior (i.e. what you really want to find), but only the variational distribution (you choose this!), the prior (which you also define/choose), and the likelihood (which, in practice, corresponds to the typical usage of the cross-entropy; so the only thing that you need more, with respect to the traditional neural networks, is the computation of the KL divergence): in other words, you originally formulate the problem as the minimization of the KL divergence between the posterior and the VD, but this is just a formulation.


Can a neural network be trained on a dataset containing only values for true output for a classification problem? I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points is 1 i.e, probability of life existing there is 1. If I train my neural network on this data, and test it on data points with concentrations other than these, can I expect to get probability values at the output? Asking because the model knows no false labelled value.

Yes, you can and the answer is One-Class Classification. A well-written resource to understand is this.


Are search engines considered AI? Are search engines considered AI because of the way they analyze what you search for and remember it? Or how they send you ads of what you've searched for recently?

Is this considered AI or just smart?

I believe it would be more accurate to say that (some) search engines use AI.  Broadly saying "search engines are AI" is not really correct. At the core, most search engines are nothing more than an inverted text index using something like tf–idf scoring. That's a very mechanical/simple thing that nobody would really call AI.

But more sophisticated search engines may use AI or AI techniques to do things like semantic analysis - so they can actually "answer questions" instead of just looking up words in an index.


What are sim2sim, sim2real and real2real? Recently, I always hear about the terms sim2sim, sim2real and real2real. Will anyone explain the meaning/motivation of these terms (in DL/RL research community)?

What are the challenges in this research area?

Anything intuitive would be appreciated!

The abbreviations sim2sim, sim2real and real2real refer to techniques that can be used to transfer knowledge from one environment (e.g. in simulation) to another one (e.g. in the real world).

sim2sim stands for simulation-to-simulation,
sim2real stands for simulation-to-real, and
real2real stands for real-to-real.

In sim2sim, knowledge acquired during one simulation is transferred to an agent (or robot) in another simulation. Similarly, in sim2real, knowledge acquired during the simulation is used in a real-world problem (or environment). Finally, in real2real, knowledge acquired in a real-world problem can be transferred to another agent in another real-world problem.

The main challenges are related to the differences that exist between one environment and the other (either in simulation or in the real-world). For example, in sim2real, the simulation is almost never a perfect model of the real-world environment, so an agent trained in a simulation will probably not behave optimally in the real-world environment, which is often a lot more complex than the simulated environment. However, it is often the case that a robot needs to be trained in simulation, given that a robot trained in a real-world environment is subject to crashes.


How do I perform object detection if there is only one type of object? How do I do object detection (or identify the location of an object) if there is only one kind of object, and they are more of less similar size, but the picture does not look like standard scenes (it is detection of drops on a substrate in microscopic images)?  Which software is good for it?

The hardest part is image annotation: here the difference between object recognition and object detection becomes important. If you just want to answer the question "does this image contain object X?", then you just need to provide as many images that contain object X as possible, together with as many images  that don't contain object X (but are otherwise similar). However if you want to answer the question "Where exactly object X is located in this image?" then you will need to manually provide a bounding box for each instance of object X in each image. Obviously, the second scenario is a lot more labor intensive.

After you've done this part, train either a binary image classifier (typically this will be a convolutional neural network) on your annotated images (split them into train and test partitions), or an object detector (googling "custom object detection" produces lots of code examples how to train it (e.g. https://towardsdatascience.com/tutorial-build-an-object-detection-system-using-yolo-9a930513643a start with Step 2B).


What kind of simulated environment is complex enough to develop a general AI? Imagine trying to create a simulated virtual environment that is complicated enough to create a "general AI" (which I define as a self aware AI) but is as simple as possible. What would this minimal environment be like?

i.e. An environment that was just a chess game would be too simple. A chess program cannot be a general AI.

An environment with multiple agents playing chess and communicating their results to each other. Would this constitute a general AI? (If you can say a chess grand master who thinks about chess all day long has 'general AI'? During his time thinking about chess is he any different to a chess computer?).

What about a 3D sim-like world. That seems to be too complicated. After all why can't a general AI exist in a 2D world.

What would be an example of a simple environment but not too simple such that the AI(s) can have self-awareness?

I think this is one of the best AGI related questions I've see in this forum.

I will skip all thematic about "what is an AGI", "simulation game", ... These topics have been discussed during decades and nowadays they are, in my opinion, a dead end.

Thus, I can only answer with my personal experience:

It is a basic theorem in computing that any number of dimensions, including temporal one, in a finite size space, can be reduced to 1D.

However, in practical examples, the 1D representation becomes hard to analyze and visualize. It is more practical work with graphs, that can be seen as an intermediate between 1D and 2D. Graphs allows representation of all necessary facts and relations.

By example, if we try to develop an AGI able to work in the area of mathematics, any expression (that humans we write in a 2D representation with rationals, subscripts, integrals, ...) can be represented as 1D (as an expression written in a program source) but this 1D must be parsed to reach the graph that can be analyzed or executed. Thus, the graph that results after the parsing of the expression is the most practical representation.

Another example, if we want an agent that travels across a 3D world, this world can be seen as an empty space with objects that have some properties. Again, after the initial stage of scene analysis and object recognition (the equivalent to the parser in previous example), we reach a graph.

Thus, to really work in the area of AGI, I suggest skip the problems of scene analysis, object recognition, speech recognition (Narrow AI), and work directly over the representative graphs.


Is the singularity equivalent to the creation of God? The definition of God

  a superhuman being or spirit worshipped as having power over nature or human fortunes

I don't believe any of the Gods of the present or past are real in any way shape or form. However, when the singularity happens, theoretically speaking, isn't this essentially the birth of a God, given that it will be the creation of a superhuman being?

infinite being, eternal, supernatural and existing by itself, necessary cause and ultimate end of all that exists

As the Portuguese dictionary says, the problem of calling God is the connection with spirituality, it is even strange to call a machine of God.

Now, we can put Google as an example. They know what you do, where you are, what you like and what you do not want anyone to know, and he's everywhere listening to you. The only way to not be under Google's vision is to stop using a cell phone, computer and living a boring life so carefully that you would have to have to not be watched.

It's funny the relationship. According to the Bible (I am an atheist), the man departed from God in Babylon (cities).

But can we call Google God? If Google does not have a backup of its core scattered through space and other planets, I think the human can still destroy the world and bring Google together. But following rational ideology, he is closer to being God than anything on earth, including us.


Building ML to finding the closest matching image based on users drawing I am trying to build a ML Agent to find the closest matching image from a given set. The user will draw something and the agent should list the closest matching images.

Very similar to these examples

https://sketchx.eecs.qmul.ac.uk/
Emoji search in Android keyboard

One unique problem I've is, each image will represent a category. Imagine we have product images and user will draw something and we have to find the products close to the drawing. So category in my case will be product Id.

I would like to evaluate the approach before trying out.

There are lot of examples to classify images, however if I use the item identifier instead of category it should work. But am trying to find the best approach for this problem.

There is a good project you can research to find information on this specific task, it is Google quickdraw. It has 50 million drawings across 345 categories and it was used in Google autoDraw. AutoDraw guesses what you’re trying to draw.

You can use quickdraw dataset to train a neural network (Sequence-to-Sequence Variational Autoencoder) in order to get draws from your Sketches. for more info on this read this papper

Classified Sketches into categories is a difficult task because Sketches are a high-level representation that does not always convey enough information to distinguish between different categories.

The the best approach will depend on your specific needs


How to build a test set for a model in industry? Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what comes most important is the building of our test set. We are not given a large corpus first, but we are discussing how to build the test set first.
The most resource-saving method is just sampling(simple random sampling) cases from the log and then having them labeled, which represents the population. Perhaps we are concerning that some groups should be more important than others, then we do stratified sampling.
Are there any better sampling methods?
What to do when we are releasing a new feature and we cannot find any cases of that feature from the user log?

I am not sure whether that solves your problem at hand, but one approach you could look into is k-fold Cross Validation (CV).
In this approach, you split your combined train, development, and test data into $k$ randomized and equally-sized partitions.
Afterwards, you train and evaluate your model $k$ times. In the $i^{th}$ iteration, you train your model on all but the $i^{th}$ partition. After training on the $k-1$ partitions is done, you evaluate your model on the $i^{th}$ partition. You repeat this process for all $i \in {1, 2, ..., k}$.
To be clear, you keep your initially randomized partitions fixed during k-fold CV.
Then, you take the average performance over all $k$ performed test runs to assess the quality of your model based on the averaged test performance. Afterwards, you could train your model on all train, development, and test data and deliver the resulting model as your final one.
In the most extreme case, you would perform Leave-One-Out-CV, where you set $k$ equal to the number of all the data points at your disposal. That is the most expensive approach, but yields the most accurate performance estimates.
For more information, see this website.
Generally, using that approach, you don't waste any data by reserving it exclusively for development/testing. Also, it might be important to mention that this approach is compatible with other sampling techniques as well. For example, in the $i^{th}$ iteration of your CV algorithm, you could apply stratified sampling to your $k-1$ partitions used for training during that ($i^{th}$) iteration.
I am not entirely sure whether I get the second part of your question right.
If it is about how to later introduce the new feature to a given model, I would say the following.
When it comes to introducing new features, I think you are pretty much out of luck with respect to recycling your old model. Of course, (assuming that the introduction of new features to the existing model is technically possible) there might be types of models which allow continual learning under certain circumstances, but in the worst case that might cause catastrophic forgetting since you change the distribution of your underlying training data when adding new features, which not all models might be able to deal with.
An example of this case is when you add more diverse training images for a given Convolutional Neural Net (CNN), which the CNN then has to learn to map to an already existing set of classes.
In other cases, introducing new features might not even be technically possible if their introduction would require adding new input (or output) nodes to an existing model.
However, when your second part of the question asks for how to fill gaps in your older data, caused by missing data, there are different strategies you could try for imputing the missing data, some of which are briefly mentioned here.


Is there a notion of generalization in unsupervised learning? I've been learning a little bit about generalization theory, and in particular, the PAC (and PAC-Bayes) approach to thinking about this problem.

So, I started to wonder if there is an analogous version of "generalization" in Unsupervised Learning? I.e., is there a general framework that encapsulates how "good" an unsupervised learning method is? There's reconstruction error for learning lower dimensional representations, but what about unsupervised clustering?

Any ideas?

In the paper Generalization in Unsupervised Learning (2015), Abou-Moustafa and Schuurmans develop an approach to assess the generalization of an unsupervised learning algorithm $A$ on a given dataset $S$ and how to compare the generalization ability of two unsupervised learning algorithms $A_1$ and $A_2$, for the same learning task.

They first provide a more abstract and general definition of an unsupervised learning algorithm and loss function. Then they define the expected risk, empirical risk and generalization gap in a similar way to the case of supervised learning. Finally, they derive an upper bound on $A$'s expected loss.

Of course, you should read the paper for more details. Specifically, section 2 (page 3) describes their setting in detail.


What does "off-the-shelf" mean? I encountered the phrase/concept off-the-shelf CNN in this paper in which authors used off-the-shelf CNN representation, OverFeat, with simple classifiers to address different recognition tasks.

As I understand it correctly, it literally means something is ready to be used for a task without alteration.

Can somebody explain in simple words what off-the-shelf CNN technically means in the context of AI and convolutional neural networks?

The dictionary's definition of off-the-shelf is

used to describe a product that is available immediately and does not need to be specially made to suit a particular purpose

The same dictionary provides several examples

You can purchase off-the-shelf software or have it customized to suit your needs.
If you have complex needs, we don't recommend that you buy software off the shelf.
For this, off-the-shelf algorithms included in the robot's programming libraries are used.

So, your intuition is correct! An off-the-shelf model, software, product, etc., is any model, software or, respectively, product that would be easily (or immediately) available or applicable to the specific context, but, at the same time, it may also be applicable to many other contexts or problems.
An off-the-shelf convolutional neural network is thus a typical or standard CNN that can be applied immediately in that context (but that is potentially applicable to many other contexts or problems). Examples of CNNs that could be used as off-the-shelf models are AlexNet or LeNet-5, but the actual choice depends on the context and needs.
An off-the-shelf model can also be a baseline model (i.e. a very simple model that is used in experiments as the model that every other model should outperform), but not necessarily.
Alex Graves uses this term/expression in his paper Practical Variational Inference for Neural Networks.


Why AlphaGo didn't use Deep Q-Learning? In the previous research, in 2015, Deep Q-Learning shows its great performance on single player Atari Games. But why do AlphaGo's researchers use CNN + MCTS instead of Deep Q-Learning? is that because Deep Q-Learning somehow is not suitable for Go?

$Q$-learning (and also its deep variant, and most of the other well-known reinforcement learning algorithms) are inherently learning approaches for single-agent environments. The entire problem setting that these algorithms are developed for (Markov decision processes, or MDPs) is always framed in terms of a single agent situated in some environment, where that agent can take actions that have some level of influence over the states that they lead to, and rewards may be observed.
If you have a problem that is, in reality, a multi-agent environment, there is a way to translate this environment to a single-agent setting; you simply have to assume that all other agents (i.e. your opponent in Go) are an inherent part of "the world" or "the environment", and that all the states in which these other agents make moves are not really states (not visible to your agent), but just intermediate steps where these part-of-the-environment-agents cause the environment to change and, as a result, create state transitions.
The primary issue with this approach is; we still need to model the decision-making of these agents in order to implement this new view of "the world", where our opponents are really a part of the world. Whatever implementation we give them, that is what our single-agent RL algorithm will learn to play against. We can just implement our opponents to be random agents, and run a single-agent RL algorithm like DQN, and then we will likely learn to play well against random agents. We'll probably still be very bad against strong opponents though. If we want to use a single-agent RL algorithm to learn to play well against strong opponents, we need to have an implementation for those strong opponents first. But if we already have that... why even bother with learning at all? We've already got the strong Go player, so we're already done and don't need to learn!
MCTS is a tree search algorithm, one that actively takes into account the fact that there is an opponent with opposing goals, and tries to model the choices that this opponent can make, and can do so better the more computation time we give it. This algorithm, and learning approaches built around it, are inherently designed to tackle the multi-agent setting (with agents having opposing goals).


How does YOLO handle non-class objects? I have been reading more about computer vision and I'm bothered by YOLO and similar deep learning architectures.
The thing I am confused about is how non-class image sections are dealt with. In particular, it's not clear to me at all why YOLO doesn't consider every part of an image a possible class.
What actually sets the cutoff for detection and then classification?

The output of YOLO is (x,y,w,h,confidence,class). The confidence value presents whether the rectangle holds an object, the rectangle is non-classed when confidence is low.
The class value will be used, only when confidence is high.


Can you learn parameters in nonlinear function? In the paper Nonlinear Interference Mitigation via Deep Neural Networks, the the following network is illustrated.

The network structure is

The network parameters are $\theta = \{W_1^{1},...,W_1^{l-1},W_2^{1},...,W_2^{l-1},W^{l},\alpha_1,...,\alpha_{l-1}\}$, where $W_1$ and $W_2$ are linear matrices and $\rho^{(i)}(x)=xe^{-j\alpha_i|x|^2}$ is element-wise nonlinear function ($i$ is the index of layer).

Where should I add this $\rho^{(i)}(x)$? Is it possible to learn the parameter $\alpha$? I don't think it is the same idea as activation function since it is positioned in the middle of two linear matrices... Or can it be added as embedding layer?

In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $\rho$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $\alpha$.

In this paper, $\rho$ is a non-linear function (that is, a function that is not linear, e.g. the sigmoid function) that applies element-wise to its input. So, if you pass a vector to this $\rho$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $\rho$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $\rho$ is also followed by a matrix. In general, this is not forbidden, even though it is not common.

In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections.


In DQN, updating target network every N steps or slowly update every step is better? The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and with experience replay the training could diverge.

Target network is a slow changing network designed to slowly track the main value network. In Mnih 2013, it was designed to match the main network every $N$ steps. There is another way which slowly updates the weight in the direction to match the main network every step. To someone, the latter is called Polyak updates.

I have done some very limited experiments and seen that with the same update rate, e.g. $N=10$, Polyak update would update with the rate of 0.1, I usually see Polyak updates to give smoother progress and converge faster. My experiments are by no means conclusive.

I would thence ask if it is known which one to perform better, converge faster or has smoother progress, in a wider range of tasks and settings?

Most of papers and my experience support hard update, once per N steps. N is usually very bug, range in between 10^4 to 10^6. DQN training is slow. But that depend on the problem. If you DQN converge with soft update with weight ~.1 (N~10) your problem could be very simple.


Why is this ResNet50 misclassifying objects? I'm new to Deep Learning, and I have some conceptual problems. I followed a simple tutorial here, and trained a model in Keras to do image classification on 10 classes of logos. I prepared 10 classes with each class having almost 100 images. My trained Resnet50 model performs exceptionally great when the image is one of those 10 logos, with 1.00 probability. But the problem is if I pass a non-logo item, a random image totally unrelated visually, still it marks it as one of those logos with close to 1.00 probability!

I'm confused. Am I missing anything? Why is this happening? How to find a solution? I need to find logos in video frames. But right now, with a high possbility each frame is marked as a logo!

Here is my simple training code:

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = False

    x = base_model.output
    x = Flatten()(x)
    for fc in fc_layers:
        # New FC layer, random init
        x = Dense(fc, activation='relu')(x)
        x = Dropout(dropout)(x)

    # New softmax layer
    predictions = Dense(num_classes, activation='softmax')(x)
    finetune_model = Model(inputs=base_model.input, outputs=predictions)
    return finetune_model
finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
adam = Adam(lr=0.00001)
finetune_model.compile(adam, loss='categorical_crossentropy', metrics=['accuracy'])
filepath="./checkpoints/" + "ResNet50" + "_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["acc"], verbose=1, mode='max')
callbacks_list = [checkpoint]

history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8,
                                       steps_per_epoch=steps_per_epoch,
                                       shuffle=True, callbacks=callbacks_list)

plot_training(history)

Your problem is a classification problem. If you are following the tutorial, it is using the ResNet50 network, which is a convolutional neural network with one fully connected layer at the end. At the end the activation function is softmax. Detailed description of the activation function can be found here: Softmax function explained

Basically, softmax increases the difference between the higher probability and lower probability. It also limits the output between 0 and 1.

Problem Origin

Due to the nature of the softmax function, it always chooses the best one and enlarge the value to be a value near one, even if the range of the output predictions is very small like 0-0.1. Also you training data only have the data of the 10 logos labelled, so if the network see unseen images with no logo it recognizes, it predicts the one with the most similarity. If you want to classify images with no logos in it, you should add an extra class in the training dataset and also the code to train the network to learn to classify images which is not in the 5 class into a separate maybe called unlabeled. Hope I can help you and have a nice day.


What are the most instructive movies about artificial intelligence? The field of AI has expanded profoundly in recent years, as has public awareness and interest.  This includes the arts, where fiction about AI has been popular since at least Isaac Asimov.  Films on various subjects can be good teaching aids, especially for younger students, but it can be difficult for a non-expert to know which films have useful observations and insights, suitable for the classroom.

What are insightful films about AI?

Listed films must be suitable for academic analysis, providing insight into theory, methods, applications, or social ramifications.

Here are my suggestions

Her, the AI part (movie spoiler):

   The AI can define a user's profile just by hearing his short story, and "act" based on the user's profile. The AI makes the user comfort with it (her). It shows a very advance user profiling.

Ex Machina the AI part (movie spoiler):

   This movie will show you how an AI learn to trick someone. The AI can express her feelings, and make you trust their feeling.

Eagle Eye, the AI part (movie spoiler):

   A movie about a general story "AI that want to kill". This movie can show you how The AI can compile a lot of information for its purpose.

Big Hero 6, the AI part (movie spoiler):

   Baymax is a very good example of a very complex expert system, he has a "knowledge chip" and a very smart way to diagnose people


How can policy gradients be applied in the case of multiple continuous actions? Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.
When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:
$$L(\theta) = \log(P(a_1)) A,$$
where $A$ is the advantage of rewards, $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in the Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py.
The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).
Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment?
Is following approach correct for policy gradient loss function?
$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.

There are 2 solutions to this problem:
The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and here is a paper which describes this approach with 3 actor-critic agents working together to move a robotic arm.

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.

This is a more general answer but should help you and others on their related problems.


M.c.Escher and abstract thought The lithographs of dutch artist M.C. Escher have been used in the study of artificial Intelligence. How can the human mind Incorporate these optical illusions into abstract thought? Is this reverse artificial intelligence?

It is not reverse-artificial-intelligence .not only escher , almost all paintings can be interpreted into abstract thought , in fact that is what the brain does. Even when you read something your brain processes this into its mental language ("hermaneutics").

for more on "mental language" and "abstract thought" you should check https://drive.google.com/file/d/0B8i61jl8OE3XdHRCSkV1VFNqTWc/view (geoff hinton is a noted researcher in this field (ai))


How does the region proposal method work in Fast R-CNN? I read so many articles and the Fast R-CNN paper, but I'm still confused about how the region proposal method works in Fast R-CNN.
As you can see in the image below, they say they used a proposal method, but it is not specified how it works.
What confuses me is, for example, in the VGGnet, the output of the last convolution layer is feature maps of shape 14x14x512, but what is the used algorithm to propose the regions and how does it propose them from the feature maps?

Yes, it is not specified because the region proposal algorithm did not change from R-CNN (the previous version from Fast R-CNN, however, in the next verion, Faster R-CNN, this algorithm is replaced by a CNN).
The region proposal algorithm you are looking for is called selective search. You can find in the R-CNN paper that the algorithm is described in "Selective Search for Object Recognition", I found a copy here.
The algorithm is based on a series of segmentation and aggregation techniques of the input image for generating the proposed regions. Check it out 4 iterations of segmentation & aggregation over the same input image to build the proposed regions.

All the algorithm is doing is just iterating over 4 steps:

Initial regions based on segmentation by pixel light intensity are obtained by applying a segmentation algorithm described in the paper. For example, given a picture of a shepherd with his sheep in the mountain, it is segmented by light intensity, and the image of Figure (a) is obtained.
Different regions are proposed based on the previous segmentation, Figure (e)
The similarity between the proposed regions is calculated using the formula proposed in equation 6
in Section 3.2 in the paper which is nothing more than an aggregate metric of the similarity of two regions based in 4 metrics: similarity in color, texture, size and fill (measures how well a region within another)
Add the regions based on similarity and get Figure (b). Then return to step two.

That is how iteratively you get all the images depicted.


How do I write a good evaluation function for a board game? I'm currently writing the Alpha-Beta pruning algorithm for a board game. Now I need to come up with a good evaluation function. The game is a bit like snakes and ladders (you have to finish the race first), so for a possible feature list I came up with the following:

field index should be high
in the lower fields my fuel should be high, when coming to the end it should be low (maximum of '10' required to enter the goal)
all 'power-ups' must be spent to enter the goal, so prioritize them
if it is possible to enter the goal (a legit move), do it!

There could be some more for some special cases.
I've read somewhere that it is the best (and easiest) to combine them in a linear function, for example:
$$0.75 * i - 5 * p - 0.25 * |(f - \text{MAX_FIELD_INDEX}/i)|,$$
where

$i$ = field index
$p$ = power-ups
$f$ = fuel

Since I can't ask an expert and I'm not an expert by myself, I have nobody to ask if those parameters are good, if I've forgotten something or if I've combined the factors correctly.
The parameters aren't that big of a deal because I could use a genetic algorithm or something else to optimize them.
My problem and question is: What do I have to do to find out how to put together my features optimally (how can I optimize the function/parameter arrangement itself)?

Based on your description, I'd maximize the following terms:

i
-max(f - 10 - (MAX_FIELD_INDEX - i), 0) - assuming consumption of one fuel per field; this becomes negative when you have too much fuel
a similar function of p, as spending them gets more important when approaching the goal

As having fuel is probably a good thing in the beginning, you could use a term like f. Similarly for the "power packs" (or are they rather "weakness packs"?).

I'd combine the terms using a linear function like you did and let it optimize. You may need more such terms. Maybe it's simpler to get rid of the power packs when you have enough fuel? Then something like -max(p-f, 0) may help.

You may generate some ad-hoc expressions or add some products of your terms as new terms. You may want to do this after the coefficients of the simpler terms have already been optimized (so you help the more complex optimization with a good staring point).


Searching an AI to recognize and locate persons in a factory TLDR : Is there an AI available that can recognize employees in a factory and tell when they entered and left pre-defined areas?

I work in a factory where we gather cycle time data from various inputs (computer interfaces, bar code readers and RFID tags). We follow both parts moving along the production line and persons working on those parts because they move from stations to stations during their day and can be more than one at the same place to help one another.

Our goal is to know how many people are working on each part at any given time, but the problem is that it takes time for people to check themselves in when they get to their workstation and they can also forget to do it.

Many employees asked me if I could find a way to track them automatically so they wouldn't need to check in everywhere they go. And then I stumbled upon Microsoft Workplace Safety Demo and Yitu System (this one is scary) but they both seem to be a little overkill for my needs.

After learning about these, here is the ideal AI features that I'd need :

Can use video feeds to detect new people in the workspace and prompt someone (via e-mail or text message) for identification.
Can use video feeds to recognize persons that are already known and document every time someone has entered or left a pre-defined zone (workstation) in the video feed.
Won't force me to have a camera for each workstation.
Allows one workstation to span on more than one video feed.

As of now, I found nothing available that does this, but I may have missed lots of things because this is not what I am used to work with. So maybe you know something that could help me with that?

I am definitely not the most qualified person in order to answer this question, but I might be able to give you a couple of buzz words to go and do some more research on your own. A convolutional neural network is usually used for analyzing anything visual. (you can read about it here: https://en.wikipedia.org/wiki/Convolutional_neural_network)

There are a lot of frameworks that allow you to build recognition classifiers that can be loaded into a real-time video feed and give you feedback when it gets a hit.

Some of the ones that I know of are:

https://caffe2.ai/ or http://caffe.berkeleyvision.org/
https://pjreddie.com/darknet/
https://docs.opencv.org/2.4/modules/ml/doc/neural_networks.html

With Caffe being the most popular.

Some articles on how these work:

https://pjreddie.com/darknet/yolo/
http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

Hope this is at least able to get you on the right path.


Why don't we perform classification of crowd density? For the case of crowd density estimation using CNN, using datasets like shanhaiTech or UCF, why there hasn't been attempts to tackle this type of task as a classification problem? All current papers I've seen are related to regression, not classification.

For example, if I have the crowd images labeled based on their crowd density (low, moderate, high), and I'm not interested in the count, but the density class (low, moderate, high), can't I train the network to classify the data based on these classes as a classification network?

For example, if I have the crowd images labeled based on their crowd density (low, moderate, high), and I'm not interested in the count, but the density class (low, moderate, high), can't I train the network to classify the data based on these classes as a classification network?

Yes you can, all you need is enough correctly labelled training data.

A good rule of thumb is if a human expert can assign the correct label from an image (and purely from the image, not using extra information) then it is a realistic goal to train a CNN to perform the same labelling.

  why there hasn't been attempts to tackle this type of task as a classification problem.

Probably because there are no natural, and likely no widely accepted, classes in this case (I may be wrong, maybe some international society has defined classes you could use). If you use a regression, you can map it to a particular problem case - e.g. sending an alert to someone responsible for traffic and safety when crowd density hits some threshold - by setting numerical boundaries to your classes. Using classification and mapping back the other way is harder.


Does the image is logistic regression or SVM, and why? Does the image is logistic regression or SVM, and why?

The straight dashed-line shows the typical decision line in logistic regression or any linear classifier. The dashed-circle shows the decision line from SVM.
Obviously, since the data is not linearly separable in the original 2D feature space, if someone makes a higher dimension space by taking into account non-linear interaction of the original 2 features then they can discriminate between x and o data using a linear discriminator applied in higher dimensions. This shows the beauty of kernel methods that can make a linear yet high-dimensional (infinite dimensions indeed) problem from a non-linear low-dimensional problem (finite dimensions actually).


How to go about classifying 1000 classes? I am trying to find research paper with theory(preferably implementation) that is about classifying 1000 (or more) classes. I have heard of an implementation, that initially clustering needs to be done then classification with something like softmax. Does anyone know of any research paper that implements 1000+ class classification.

If you are asking for arbitrary ML task dealing with 1000+ classes the most straigtforward thing that comes to mind is the ImageNet - https://en.wikipedia.org/wiki/ImageNet#cite_note-nytimes_2012-2. It has more the 20k categories in the present time.
In order to perform such classification task you need

A large enough dataset, such that each class occurs frequently enough or efficient data augmentation
A powerful model, such that can adapt to variety of different problems

The first neural network to tackle this problem was AlexNet and since then plethora of architectures : VGG's, ResNet's, EfficientNet got better and better quality.
They have a lot of filters that react on different patterns.


Neural network architecture for line orientation prediction Imagine that a line divides an image in two regions which (slightly) differ in terms of texture and color. It is not a perfect, artificial line but rather a thin transition zone. I want to build a neural network which is able to infer geometrical information on this line (orientation and offset). The image may also contain other elements which are not relevant for the task. Now, would a classical CNN be suitable for this task? How complex should it be in terms of number of convolutions (and number of layers, in general)?

A long shot: these guys have worked on a problem that might be relevant. They define "semantic" lines as lines delimiting significant regions or objects in an image. To detect such lines, they use the conv layers from a pre-trained VGG16 net and then add their own layers on top. The cool thing about their approach is that they run both classification and regression in parallel on the same network.

You might be able to adopt a similar technique to determine where the line is, and then run some simple analysis on the extracted line to determine the offset and the orientation.


Do algorithms like OpenAI's "think up strategies"? I was discussing with a friend whether current AI does anything remotely similar to 'thinking' and he argued that AIs that play games must think up strategies.

While thinking may not be precisely defined, my understanding of algorithms like OpenAI was that they just minimize a very non-convex objective, but still play the game based on examples, and not by coming up with intentional strategies. Is my understanding incorrect?

The key here is think up strategies. If we define this as examining, creating a hypothesis, and testing it as strategizing then yes AI has the ability to strategize. It can examine other users' games, quantifies actions that correlated with victory then test if it gains victory by doing those actions.

Strategy by definition is: a plan of action or policy designed to achieve a major or overall aim.

AI can not classically plan a series of actions designed to achieve a major victory. Instead, it learns the right strategy by testing simulated scenarios, like someone who thinks about the consequences before doing them, but the AI actually has the opportunity to play the game hundreds of times, in order to learn the correct strategy. Similar to Bill Murray in the movie Groundhog Day, learning the ideal day to live. The AI can strategize by experiencing the game over and over until it fine-tunes what an ideal game should be and has seen enough examples of games to not be outwitted by gimmicky strategies.

To summarize, AI can strategize, just in a way fairly different than people.


Single AI-person in software firm about to adopt AI, good idea? I have been offered a job with a software firm which does  high level custom software. I believe i am the first person there with some AI background and that very soon now many of their projects will require some AI.
I am not a lead, i don't think i have the skills, technical or managerial. Is it possible for a software development team with one person with some AI background, to learn to implement various AI techniques succesfully?

My own background is that i studied AI but with a focus on cognition (ie: use AI to better understand cognition, human or otherwise). Afterwards i learned the craft of software development (Java webapps) and worked in this field until i found a PhD position.
During my PhD i dabbled a bit with machine learning, academically. Soon i focussed again on the more cognition oriented approach, and looked for ways of formalising knowledge from literature so it could be used inside a custom classifier to better analyse a new dataset.

The funding ran out before i could finish and i returned to software development.
Ideally i would love a job where i could join a AI-oriented team, and learn more about production-level AI on the job.

The job i have been offered is with a firm that makes custom software. They are starting a new project which should employ AI to help human operators fulfil a complex task. Another project uses very costly calculation and one ML component. The data it works on is public and google has announced it will have a go at the data. I expect that when google is done, there will be a scramble to use ML for all the components.

In general i think that 5 years from now all their software projects will require some form of AI. If i were running this software company i would go and look for a lead developer who had proven themselves in AI. Then i would task this person with interviewing to find 2 'luitenants'.

  What are the chances of a software development team including one
  member who knows about machine learning in an academic context,
  getting up-to-speed and delivering production level software
  incorporating AI?

I think you should be fine if you are using other people's tried and tested algorithms and have a reasonable understanding of machine learning best practices - how to clean and balance your dataset and how to effectively measure performance. Maybe if you are trying to come up with new algorithms / DNN architectures you will struggle.


How to plot Loss Landscape with more than 2 weights in the network For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):

But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.

And also, the 3D space can't be used to plot more than 3 dimensions.

However, with a deep network with lots of weights, these can be plotted:

Loss value against any pair of 2 weights
Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):


Regression with more than one output, neural network Currently in my country, there is a system in which certain groups of researchers upload information on products of scientific interest, such as research articles, books, patents, software, among others. Depending on the number of products, the system assigns a classification to each group, which can be A1, A, B and C, where A1 is the highest classification and C is the minimum. According to the classification of the groups, they can compete to receive monetary incentives to make their research.

At the momen, I am working in an application that takes the data of the system that I mentioned previously. I am able to say what classification the group currently has because we develop a scraper that counts the products and there is another service that is in charge of implementing all the mathematical model that the system has to calculate the category of the group.

But what I want to achieve is that my application would be able to give an estimate of how many products a research group should have to improve its category and i Want to know if I can do that using neural networks.

For example, if there is a category C group, I want the application to tell the user with how many articles and books it would make its category go up to B.

From what I have seen in some web resources, I could insert a training set into the neural network and have it learn to classify the groups, but I think it is unnecessary, because I can do that mathematically.

But I do not understand if it is possible for a neural network to process the current category that the group has and be able to give suggestions of how many products it needs to improve its category.

I think it must be a neural network with several outputs, so that in each one it throws the total for each one of the products, although it is not necessary to list all the products that the measurement model contemplates. But it is necessary for the network to learn which products are handled by a certain group, for example if a group does not write books, avoid suggestions that contemplate the production of books for the improvement of the category that the group has.

If you read the whole problem thanks in advance, I would appreciate any suggestion of what kind of subject I must research or any tutorial. I must say that I am a totally newbie in this subject.

Welcome to AI-SE! Please use shorter description next time because it makes it hard to understand what you ask :)

I believe you want a neural network that can predict future values of multiple variables given multiple inputs. This belongs to the general time series forecasting problem.

One of the best neural network architectures that can handle this problem is the LSTM Recurrent Neural Network (LSTM-RNN). The reason I say "best" is because of their architecture that allows them to develop memory of what they have seen in the past and use it for future predictions. In other words they can cross-correlate in a linear/nonlinear fashion several past steps of multiple variables to future values of multiple other variables, like a black box.

No other neural network type does that. They are the only type of network that can learn the dynamics of a system.

An extremely useful tutorial for your purposes is this, please take a look.


How to find AI specialists interested in additive Manufacturing We recently founded a company in the area of additive manufacturing. Our development focuses on making the process easier and liberate time to the user, which I personally believe in as the ultimate promise of any computerised process, including AI, for the public.

We have gone through the usual channels including regular job offers and personal contacts, visiting universities and meetups. However there appears to be a lack of participation of real specialists in most of those gatherings as the ones interested in learning about AI are doing just that, and don't show up somewhere where we can find them, although we believe to have a very interesting task at hand.

It appears to be remarkably hard to find specialists in the field, as AI specialists are either employed by huge multinational companies (instead of startups), have no interest in additive manufacturing (because it implies something else than only code) and the challenge of machines (making machines) or just do not pop up anywhere.

How would a specialist of AI look for someone else in the specific field of additive manufacturing/motion planning/creative strategy and its employment? (Or course in the case they do not already know someone.)

You should probably compile a list of Universities/Instituitions that specialise in Additive Manufacturing, and target graduating students or researchers looking for a new challenge. This may prove more fruitful than targeting the normal channels. You may have to search worldwide, but of course this adds the issue of work visa etc.
There is no easy way to solve this....

Yo can try posting on the Academia forum too.


What is the difference between actor-critic and advantage actor-critic? I'm struggling to understand the difference between actor-critic and advantage actor-critic.

At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an asynchronous mechanism that uses multiple worker agents interacting with their own copy of the environment and reports the gradient to the global agent.

But what is the difference between the actor-critic and advantage actor-critic (A2C)? Is it simply with or without advantage function? But, then, does the actor-critic have any other implementation except for the use of advantage function?

Or maybe are they synonyms and actor-critic is just a shorthand for A2C?

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "bootstrapping" by using some sort of prediction of value.

Advantage Actor-Critic specifically uses estimates of the advantage function $A(s, a) = V(s) - Q(s, a)$ for its bootstrapping, whereas "actor-critic" without the "advantage" qualifier is not specific; it could be a trained $V(s)$ function, it could be some sort of estimate of $Q(s, a)$, it could be a variety of things.

In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict $V(s)$. Combined with an empirically observed reward $r$, they can then compute the advantage estimate $A(s, a) = r + \gamma V(s') - V(s)$.


Why is the denominator ignored in the Bayes' rule? The naïve Bayes' generative algorithm is often represented by the following formula:
$$\text{argmax}_{y} p(y|x) = \text{argmax}_y \frac{p(x|y)p(y)}{p(x)} \approx \text{argmax}_y p(x|y)p(y)$$
Why do we have $p(x)=1$ which allows the approximation $\text{argmax}_y \frac{p(x|y)p(y)}{p(x)} \approx \text{argmax}_y p(x|y)p(y)$?

The answer from @kiner_shah in the comments has solved it:

  They have eliminated it because in comparing probability for best outcome, it would just introduce additional division and the divisor p(x) is same for all candidates. For example, if I tell you to compare the largest of the following numbers: 3/2, 5/2, 1/2, you will without thinking pick 5/2 since the divisors are same and 5 is greatest of all.


Why is the effective branching factor used for measuring performance of a heuristic function? For search algorithms with heuristic functions, performance of heuristic functions are measured by effective branching factor ${b^*}$ which involves total nodes expanded ${N}$ and depth of the solution ${d}$. I'm not able to find out how different values of ${d}$ affect the performance keeping same ${N}$. Put another way, why not use just the ${N}$ as the performance measure instead of ${b^*}$?

As you found $N$ is the number of nodes that are expanded. The cost of expansion of each node is equal to the number of children of that node. Hence, we use $b^*$ for each node. In other words, the total number of nodes that are involved in the expansion process is $N \times b^*$.


Does stochasticity of an environment necessarily mean non-stationarity in MDPs? Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a certain probability $p_1$ of transitioning you into $s_2$, and a probability $1-p_1$ of keeping you in $s_1$. There is also a similar probability for taking $a_2$ in $s_1$, and taking either action in $s_2$. Let's also say that there is a reward $r$ given only when a transition occurs from either state, and 0 otherwise. This is a stochastic environment. But isn't this non-stationary in one sense and stationary in another? I think it is stationary because the expected return from taking a particular action in a particular state converges to a constant value. But it is non-stationary in the sense that the reward obtained from taking a certain action in a given state may change at a given time. Which is really the case?

Is a stochastic environment necessarily also non-stationary?

No.
A stochastic environment (i.e. an MDP with a transition model $p(s', r \mid s, a)$) can be stationary (i.e. $p$ does not change over time) or non-stationary ($p$ changes over time). Similarly, a deterministic environment, i.e. the probabilities are $1$ or $0$, can also be either stationary or not. To emphasize that an MDP may be non-stationary, you could write $p$ as a function of time, i.e. $p_t$ (you also do the same thing for the reward function if it's separate from the transition function).
The same idea applies to a stochastic/deterministic policy, which can either be stationary or not.
A non-stationary environment may lead to a non-stationary policy (or may require you to relearn a model of the environment, if you need to learn a model of the environment) [1]. However, note that a stochastic environment (i.e. an MDP) does not necessarily imply a stochastic policy (actually, under some conditions, stationary and stochastic MDPs are known to have a deterministic optimal policy [1]).
In general, if something (e.g. environment, policy, value function or reward function) is non-stationary, it means that it changes over time. This can either be a function or a probability distribution. So, a probability distribution (the stochastic part of an MDP) can change or not over time. If it changes over time, then it makes the MDP non-stationary.

But it is non-stationary in the sense that the reward obtained from taking a certain action in a given state may change at a given time

Informally, you could say that the empirical reward obtained is non-stationary because it changes over time, due to the stochasticity of the reward function, behaviour policy, etc., but the dynamics (transition function and reward function) would still be fixed, so the environment would still be stationary. So, there's a difference between the environment and the experience that you collected so far (with some behaviour policy).


Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward? I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
Knowing that, my main question concerns Q-learning and off-policy methods:
For me, there is something weird in updating a Q value based on the max Q for a state and a reward value that was not from the action taken? How does this make learning better and makes you learn the optimal policy?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?

The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not affect whether the state has the Markov property. All that matters is that knowing the current state is enough to be determine the next state and reward in principle. You do not need to explicitly know the state transition model, or the reward function.
An example of a non-Markov state for CartPole would be if one of the features was missing - e.g. the current position of the cart, or the angular velocity of the pole. It is still possible to have agents attempt to solve CartPole with such missing data, but the state would no longer be Markov, and it would be a harder challenge.

For me, there is something weird in updating a Q value based on the max Q for a state and a reward value that was not from the action taken? How does this make learning better and makes you learn the optimal policy?

To recap, you are referring to this update equation, or perhaps some variation of it:
$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha(R_{t+1} + \gamma\text{max}_{a'}Q(S_{t+1},a') - Q(S_t, A_t))$$
Another correction here, the reward value is related to the action taken and the experience that happened. This is important - the values of $R_{t+1}$ and $S_{t+1}$ are from the experience that was observed after taking action $A_t$, they do not come from anywhere else. In fact this is the only data from experience that is inserted into the update, everything else is based on action value estimates in a somewhat self-referential way (a.k.a. "bootstrapping").
How does this converge to an optimal policy? It follows a process known as generalised policy iteration (GPI), which works as follows:

The target policy is to always take the best action according to your current estimates

In off policy methods only (i.e. does not apply to all GPI), there is a conversion needed from experience in a behaviour policy to value estimates for the target policy. In single-step Q learning, this is where the $\text{max}_{a'}Q(S_{t+1},a')$ comes from. It is calculating the future value from following the target policy*. No interim values are needed, because you are estimating a new value for $Q(S_t,A_t)$, there are no other time steps where you need to care about difference between behaviour and target policy (there is just "now" and "all the future, which is bootstrapped")

Once you update the action value estimates in Q table, that automatically updates the target policy because it always takes the best estimated action.

Each update improves your value estimates, due to injecting some real experienced data in the form of $R_{t+1}$ and $S_{t+1}$.

Each time the estimates improve, the target policy can also improve and become closer to the optimal policy.

When the target policy changes, this makes the estimates less accurate, because they were made for the previous policy. However, the next time the same states and actions are visited they will be updated based on the new improved policy.

With a stochastic environment, it is possible for estimates to get worse due good or bad luck for the agent when it tries different actions. However, over time, the law of large numbers will win, and the value estimates will converge to close to their true values for the current policy, which will make policy improvement inevitable if it is possible (caveat, it is only close to inevitable for the tabular Q-learning method which can be proven to converge eventually on an optimal policy).
There is proof that always taking the maximising action is a strict improvement to a policy when estimates are accurate. It is called the Policy Improvement Theorem.

* To be clear, there is nothing particularly clever about taking the max here. It is literally what the target policy has been chosen to do, so the term $\text{max}_{a'}Q(S_{t+1},a')$ is just the same as $V_{\pi}(S_{t+1})$ - the value of being in state $S_{t+1}$ for the current target policy $\pi$.


What does the agent in reinforcement learning exactly do? What is an agent in reinforcement learning (RL)? I think it is not the neural network behind. What does the agent in RL exactly do?

The agent in RL is the component that makes the decision of what action to take.

In order to make that decision, the agent is allowed to use any observation from the environment, and any internal rules that it has. Those internal rules can be anything, but typically in RL, it expects the current state to be provided by the environment, for that state to have the Markov property, and then it processes that state using a policy function $\pi(a|s)$ that decides what action to take.

In addition, in RL we usually care about handling a reward signal (received from the environment) and optimising the agent towards maximising the expected reward in future. To do this, the agent will maintain some data which is influenced by the rewards it received in the past, and use that to construct a better policy.

One interesting thing about the definition of an agent, is that the agent/environment boundary is usually considered to be very close to the abstract decision making unit. For instance, for a robot, the agent is typically not the whole robot, but the specific program running on the robot's CPU that makes the decision on the action. All the relays/motors and other parts of the physical body of the robot are parts of the environment in RL terms. Although often loose language is used here, as the distinction might not matter in most descriptions - we would say that "the robot moves its arm to achieve the goal" when in stricter RL terms we should say that "the agent running on the robot CPU instructs the arm motors to move to achieve the goal".

  I think it is not the Neural Net behind?

That is correct, the agent is more than the neural network. One or more neural networks might be part of an agent, and take the role of estimating the value of a state, or state/action pair, or even directly driving the policy function.


Why is the E step in expectation maximisation algorithm called so? The E step on the EM algorithm asks us to set the value of the variational lower bound to be equal to the posterior probability of the latent variable, given the data points and parameters. Clearly we are not taking any expectations here, then why is it called the Expectation step? Am I missing something here?

In expectation step, firstly we calculate the posterior of latent variable $Z$ and then the $Q(θ | θ^{(t)})$ is defined as the expected value of the log likelihood of $θ$, with respect to the current conditional contribution of $Z$ given $X$ and the current estimates of $θ^{(t)}$. In maximization step, we update $θ$ using the argmax on $Q$, with respect to $θ$.
$$Q(θ | θ^{(t)}) = E_{Z|X,θ^{(t)}}[logL(θ;Χ,Z)]$$
To be more intuitive, think of k-means as a special case of EM, where in expectation step the $Z$ variables are defined, that is the latent variables indicating the membership in a cluster, and calculated in a hard assignment way. In maximization step the $μ$s of the clusters are updated. If you want to see the corresponding relation for $Q$ in k-means, I suggest you read the chapter 9.3.2 in C.Bishop's book: Pattern Recognition and Machine Learning.


Clarifications on "Prioritized Experience Replay" (Deepmind, 2015) Paper link : Prioritized Experience Replay

About the blind cliffwalk setup:

Why is the number of possible action sequences equal to 2^N? I cant think of sequences more than (N + 1) where one sequence is the sequence of all right actions and the other N sequences are due to wrong actions at each state.

Generally for prioritized experience replay:

The replay memory consists of some transitions which are repeated.In the priority queue I feel that there should only be a single priority for each transition to speed up learning. Is there any advantage of having priority values for each repeated instance of the transition?

Edit for 2nd question:

Consider algorithm 1 on page 5 of the article.

Lets consider one of the transitions to be repeated in the replay memory. If one of them is sampled (line 9) and the priority updated (line 12). Will the priority update on the other instance of the same transition?

for 1) i think your confusing elements touched vs sequences. At each point for N turns you have 2 possible options, therefore you have $\prod_{i=1}^N 2$ or $2^N$ possible sequences.

for 2) The priorities are updated based on the expected rewards. They do not add new elements each time, they update


Purpose of using actor-critic algorithms under deterministic MDP dynamics? One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from episode to episode). Therefore it is perfectly reasonable to use a critic to reduce variance and this is what for example Deep Deterministic Policy Gradient (DDPG) does.

Now, let's assume that we're given an MDP with completely deterministic dynamics. In that case, if we start from a specific state and follow a certain deterministic policy we will always obtain the exact same return (therefore we have zero bias and zero variance). If we follow a certain stochastic policy the return will vary depending on how much we explore, but under an almost-deterministic policy our variance will be quite small. In any case, there's no contribution to the variance from the deterministic MDP dynamics.

In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely deterministic (if historical prices are used as states) and there's no real sequentiality involved. Consequently, all return variance stems from the stochasticity of the policy itself. However, most researchers still use DDPG as a variance reduction mechanism.

What's the point of using DDPG for variance reduction when the underlying MDP used for training has deterministic dynamics? Why not simply use the Reinforce algorithm?

In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely deterministic (if historical prices are used as states) and there's no real sequentiality involved.

Whilst I cannot comment on the specific financial model, I think it unlikely that these researchers would apply RL without there being a sequence.

More likely in my opinion, the historic data feed forms a major part of the environment, but that there are still time steps and a state which depends on an agent's actions. For instance, in a trading simulation, provided the values of trades are below levels that would significantly alter the market itself, it may be a reasonable approximation to use the history of prices and other factual information that progress like a recording, plus have state include the agent's current portfolio of investment and working funds.

  What's the point of using DDPG for variance reduction when the underlying MDP used for training has deterministic dynamics? Why not simply use the Reinforce algorithm?

Variance in returns occurs due to stochastic dynamics (if they are present) and the behaviour policy. You cannot use any RL control algorithm with a deterministic behaviour policy*. It would never gain any data that allowed it to assess alternative behaviour.

So in REINFORCE, which is on-policy (the behaviour policy and target policy are the same), and typically starts with near equiprobable action choices, there is high variance. It could be very high when measured over a long episode with many action choices. In basic REINFORCE, the variance is not controlled for, and training uses individual Monte Carlo style returns.

In DDPG, which is off-policy (the target policy is deterministic**, the behaviour policy is stochastic), there is still variance, but it is much reduced with the actor-critic mechanism, plus can be constrained by choice of noise function that relates the behaviour policy to the target policy. In addition, updates to policy and value functions can be made independently of episode ends, which can significantly speed up learning.

To determine what difference this makes for any experiment, you would need to compare the two algorithms on the same task. In practice DDPG will significantly out-perform REINFORCE on many tasks, including those with deterministic environments. However, there might be specific combinations where the simplicity of REINFORCE wins out, if only because there are less hyperparameters to tune.

On one point:

  under an almost-deterministic policy our variance will be quite small

That's true, but how do you get to that stage of training with REINFORCE? It is by testing and working through more stochastic policies, which is what will take time. Your statement only applies to REINFORCE when the control problem is nearly complete, or if you take a short-cut and force the policy function into what you hope is a near-optimal policy. In which case you are engaging in a form of variance reduction - it may even work for some scenarios, but is likely not as general as applying an Actor-Critic algoirthm.

* Actually technically you can if the environment is stochastic in the right way so that you effectively explore all state/action combinations. But we are talking about deterministic environments here, and obviously the stochastic environment would introduce variance in returns.

** The target policy changes over time, and this introduces non-stationarity and bias for the critic component to deal with, but not technically variance.


How do I compute log-likelihood for training set in supervised learning? I am building a supervised learning model and I wish to compute the log-likelihood for the training set at the point of the minimum validation error.

Initially, I was computing the sum of all the probabilities with maximum value obtained after applying softmax for each example in the training set at the point of minimum validation error but that doesn't look correct.

What is the correct formula for the log-likelihood?

The log-likelihood function for the training set (in general, not for deep learning in particular) will depend on your choice of loss function.

I'm guessing you're using something like a quadratic loss function for a binary classification problem, since this is a common approach. In that case, the log-likelihood is the sum of logs of the squared differences between the target label and the softmax value produced by your model. If you want to compute the log likelihood under a particular set of parameters (say, those that minimize validation error), then you just use those parameters in the model when generating softmax values.


What is the Turing test? I'm looking for intuition in simple words but also some simple insights (I don't know if the latter is possible). Can anybody shed some light on the Turing test?

The Turing test is a test proposed by Alan Turing (one of the founders of computer science and artificial intelligence), described in section 1 of paper Computing Machinery and Intelligence (1950), to answer the question

Can machines think?

More precisely, the Turing test was originally framed as an interactive quiz (denoted as the imitation game by Turing) where a human interrogator $C$ asks multiple questions to two entities, $A$ (a computer) and $B$ (a human), which stay in different rooms than the room of the interrogator, so the interrogator cannot see them, in order to figure out which one is $A$ (the computer) and which one is $B$ (the human). $A$ and $B$  can only communicate in written form or any form that avoids them being easily recognized by $C$. The goal of the computer is to fool the interrogator and make him/her believe that it is a human and the goal of $B$ is to somehow help him and make him believe that he/she is the actual human.
If the computer is able to fool the interrogator and make him/her believe that it is a human, then that would be an indication that machines can think. However, note that even Turing called this game the imitation game, so Turing was aware of the fact that this game would only really show that a machine can imitate a human (unless he was using the term "imitation" differently than its current meaning).
Nowadays, there are different variations of the Turing test and some people use the term Turing test to refer to any test that attempts to tell humans and computers apart. For example, some people consider the CAPTCHA test a Turing test. In fact, CAPTCHA stands for "Completely Automated Public Turing Test To Tell Computers and Humans Apart".
The Turing test also has different interpretations and meanings. Some people think that the Turing test is sufficient to test that a machine can actually think and possesses consciousness, other people think that this only tests human-like intelligence (and there could be other intelligences) and some people (like me) think that this test is limited and only tests the conversational skills (and maybe other properties too) of the machine. Even Turing attempted to address these issues in the same paper (section 2), where he discusses some advantages and disadvantages of his imitation game. In any case, we can all agree that, if machines (in particular, programs like Siri, Google Home, Cortana, or Alexa) were always able to pass the Turing test, they would be a lot more useful, interesting and entertaining than they are now.


How AlphaGo Zero is learning from $\pi_t$ when $z_t = -1$? I have questions on the way AlphaGo Zero is trained.
From original AlphaGo Zero paper, I knew that AlphaGo Zero agent learns a policy, value functions by the gathered data $\{(s_t, \pi_t, z_t)\}$ where $z_t = r_T \in \{-1,1\}$.
However, the fact that the agent tries to learn a policy distribution when $z_t = -1$ seems to be counter-intuitive (at least to me).
My assertion is that the agent should not learn the policy distribution of when it loses (i.e, gets $z_t=-1$), since such a policy will guide it to lose.
I think I have missed some principles and resulted in that assertion. Or is my assertion reasonable, either?

Intuitively I think there's definitely something to be said for your idea, but it's not a 100% clear case, and there are also some arguments to be made for the case that we should also be training the policy from data where $z_t = -1$.
So, first let's establish that if we do indeed choose to discard any and all data where $z_t = -1$, we are in fact discarding a really significant part of our data; we're discarding 50% of all the data we generate in games like Go where there are no draws (less than that in games like Chess where there are many draws, but still a significant amount of data). So this is not a decision to be made lightly (it has a major impact on our sample efficiency), and we should probably only do it if we really believe that policy learning from any data where $z_t = -1$ is actually harmful.

The primary idea behind the self-play learning process in AlphaGo Zero / AlphaZero may intuitively be explained as:

When we run an MCTS search biased by a trained policy $\pi_t$, we expect the resulting distribution of visits to be slightly better than what was produced by $\pi_t$ alone.
According to the expectation from point 1., we may use the visit counts of MCTS as a training target for the policy $\pi_t$, and hence we expect to get a slight improvement in the quality of that trained policy.
If we were to now run a new MCTS search biased by the updated policy in the same situation again, we would expect that to perform even better than it previously did because it is now biased by a new policy which has improved in comparison to the policy we previously used.

Of course, there can be exceptions to point 1. if we get unlucky, but on average we expect that to be true. Crucially for your question, we don't expect this to only be true in games where we actually won, but still also be true in games that we ultimately end up losing. Even if we still end up losing the game played according to the MCTS search, we expect that we at least put up a slightly better fight with the MCTS + $\pi_t$ combo than we would have done with just $\pi_t$, and so it may still be useful to learn from it (to at least lose less badly).
On top of this, it is important to consider that we intentionally build in exploration mechanisms in the self-play training process, which may "pollute" the signal $z_t$ without having polluted the training target for the policy. In self-play, we do not always pick the action with the maximum visit count (as we would in an evaluation match / an important tournament game), but we pick actions proportionally to the MCTS visit counts. This is done for exploration, to introduce extra variety in the experience that we generate, to make sure that we do not always learn from exactly the same games. This can clearly affect the $z_t$ signal (because sometimes we knowingly make a very very bad move just for the sake of exploration), but it does not affect the policy training targets encountered throughout that game; MCTS still tries to make the best that it can out of the situations it faces. So, these policy training targets are still likely to be useful, even if we "intentionally" made a mistake somewhere along the way which caused us to lose the game.


Should I use exploration strategy in Policy Gradient algorithms? In policy gradient algorithms the output is a stochastic policy - a probability for each action.

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.

Is it beneficial or is it common to use extra exploration strategies, like UCB, Thompson sampling, etc. with such algorithms?

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.

Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C.

  Is it beneficial or is it common to use extra exploration strategy like UCB, Thompson sampling etc. n such algorithms?

It can be, but needs to be done carefully, as the gradient sampling for the policy function is different. Many policy gradient methods are strictly on-policy and will not work if you simply add extra exploration. It is relatively straightforward to adjust the critic part of actor-critic methods by using e.g.  Q learning update rules for it. However, the gradient of the policy function is trickier.

There are some policy gradient methods that do work with a separate, tunable, exploration function. Deep Deterministic Policy Gradient (DDPG) may be of interest to you - as per the title, it works with a deterministic policy function, and exploration is achieved by adding a separate noise function on top. The sampling for policy gradient is then corrected for being off-policy.


How to use a VAE to reconstruct an image starting from an initial image instead of starting from a random vector? Is it possible to use a VAE to reconstruct an image starting from an initial image instead of using K.random_normal, as shown in the “sampling” function of this example?
I have used a sample image with the VAE encoder to get z_mean and z_logvar.
I have been given 1000 pixels in an otherwise blank image (with nothing in it).
Now, I want to reconstruct the sample image using the decoder with a given constraint that the 1000 pixels in the otherwise blank image remain the same. The remaining pixels can be reconstructed so they are as close to the initial sample image as possible. In other words, my starting point for the decoder is a blank image with some pixels that don’t change.
How can I modify the decoder to generate an image based on this constraint? Is it possible? Are there variations of VAE that might make this possible? So we can predict the latent variables by starting from an initial point(s)?

The thing is, the decoder samples from a latent mu and sigma, so you cant sample from a raw image directly. But if you’re trying to put a random image into the encoder of a trained VAE to match it to some sample image (via reconstruction loss), then your random input image will converge to the target sample.

This will work when the following VAE architecture constraints are satisfied:

The target sample is contained in the previously used training distribution.
The parameters of the VAE are frozen after training.
The input image values are “backpropagate-able”. (Interpret the input image as optimizable parameters.)


Is my Neural Network program fully connected? I have the following program for my neural network:

n_steps = 9
n_inputs = 36
n_neurons = 50
n_outputs = 1
n_layers = 2
learning_rate = 0.0001
batch_size =100
n_epochs = 1000#200
train_set_size = 1000
test_set_size = 1000
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
         for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:]

I want to know whether my network is fully connected or not?
When I try to see the variables, I see:

multi_layer_cell.weights

The output is:

[<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref>]

I didn't understood whether each layer is getting the complete inputs or not.
I want to know whether the following figure is correct for the above code:

If this is not then what is the figure for the network? Please let me know.

Is figure = code?

No.
Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.

Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

Why size 86?

The input is concatenated with the hidden state:
n_inputs + n_neurons = 36 + 50 = 86.
Why size 100 in the second layer?

The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50).
Therefore you get 50 + 50 = 100.
Why size 200?

There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (layer0/kernel).
Why size 50?

The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.


Can we automate the choice of the hyper-parameters of the evolutionary algorithms? Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually need to be defined or specified by a human. Could also these definitions be automated? Could we also mutate the fitness function or automatically decide the size of the offspring generation?

Yes, you can also automate the choice of certain hyperparameters of the evolutionary algorithm. In this context, this process is called self-adaptation. There are different ways of performing self-adaptation (depending on the hyper-parameter that needs to self-adapt). See e.g. the chapter Self-Adaptation in Evolutionary Algorithms (by Silja Meyer-Nieberg and Hans-Georg Beyer) of the book Parameter Setting in Evolutionary Algorithms (2007).


Is it possible to make a 'forked path' neural network? I want to make a network, specifically a CNN for image recognition, that takes an input, processes it the same way for several layers, and then at some point splits before coming to two different outputs. Is it possible to create a network such as this? It would look something like this:

Input -> Conv -> Pool -> Conv -> Pool ---------> Dense -> Output 1

                                  ||

                                     ----> Dense -> Output 2

I.E. it splits off after the second pooling layer into separate fully connected layers. Of course, it has to train to both outputs, so that it is producing minimal error on both separate outputs using these common convolutional layers. Also, I am using Python Keras, and it would help if there was some way to do this using Keras in some way. Thank you!

Keras Functional APIs can help you define complex models.

You can find the documentation here :
https://keras.io/getting-started/functional-api-guide/.

For example:

# prev_layer is the layer you want to be forked
fork1 = Dense(32, activation='relu')(prev_layer)
fork2 = Dense(32, activation='relu')(prev_layer)

# you do some operations on fork1 to get output1
# and on fork2 to get output2

model = Model(inputs=input_layer, outputs=[output1, output2])