How do you program fear into a neural network? If you've been attacked by a spider once chances are you'll never go near a spider again.

In a neural network model, having a bad experience with a spider will slightly decrease the probability you will go near a spider depending on the learning rate.

This is not good. How can you program fear into a neural network, such that you don't need hundreds of examples of been bitten by a spider in order to ignore the spider. And also, that it doesn't just lower the probability that you will choose to go near a spider?

There are a lot of approaches you could take for this. Creating a realistic artificial analog for fear as implemented biologically in animals might be possible, but there is quite a lot involved in a real animal's fear response that would not apply in simpler AI bots available now. For instance, an animal entering a state of fear will typically use hormones to signal changes throughout its body, favouring resource expenditure and risk taking ("fight or flight").

In basic reinforcement learning, the neural network would not need to directly decide switch on a "fear mode". Instead, you can make use of some design in the agent and learning algorithm to help learn from rare but significant events. Here are a few ideas:

Experience replay. You may already be doing this in the Pacman scenario, if you are using DQN or something similar. Storing the state transition and reward that caused a large positive or negative reward, and repeatedly learning from it should offset your concern
Prioritised sweeping. You can use larger differences experienced between predicted and actual reward to bias sampling from your replay memory towards significant events and those linked closely to them.
Planning. With a predictive model - maybe based on sampled transitions (you can re-use the experience replay memory for this), or maybe a trained state transition prediction network - then you can look multiple steps ahead by simulating. There is a strong relation between RL and look-ahead planning too, they are very similar algorithm. The difference is which states and actions are being considered, and whether they are being simulated or experienced. Experience replay blurs the line here - it can be framed as learning from memory, or improving predictions for planning. Planning helps by optimising decisions without needing to repeat experiences as much - a combination of planning and learning can be far more powerful than either in isolation.
Smarter exploratory action selection. Epsilon-greedy, where you either take a greedy action or take a completely random action, completely ignores how much you may have already learned about alternative actions and their relative merit. You can use something like Upper Confidence Bound with a value-based agent.
In a deterministic world, increase the batch size for learning and planning, as you can trust that when a transition is learned once, that you know everything about it.

You will need to experiment in each environment. You can make learning agents that are more conservative about exploring near low reward areas. However, if the environment is such that it is necessary to take risks in order to get to the best rewards (which is often the case in games) then it may not be optimal in terms of learning time to have a "timid" agent. For instance in your example of Pacman, sometimes the ghosts should be avoided, sometimes they should be chased. If the agent learned strong aversion initially, it might take a long time to overcome this and learn to chase them after eating a power-up.

For your example of the spider, as the constructor of the experiment then you know that the bite is bad every time and that the agent must avoid it as much as possible. To most RL algorithms, there is no such knowledge, except gained through experience. An MDP world model does not need to match common sense, it may be that a spider bite is bad (-10 reward) 90% of the time and good 10% of the time (+1000 reward). The agent can only discover this by being bitten multiple times . . . RL typically does not start with any system to make assumptions about this sort of thing, and it is impossible to come up with a general rule about all possible MDPs. Instead, for a basic RL system, you can consider modifying hyperparameters or focusing on key events as suggested above. Outside of a basic RL system there could be merit in replicating other things, such as "instinctive" fear.


What are non-held-out data or non-held-out classes? I'm Spanish and I don't understand the meaning of "non-held-out". I have tried Google Translator and online dictionaries like Longman but I can't find a suitable translation for this term.

You can find these term using this Google Search, and in articles like this one:

"computing SVD on the non-held-out data" from here.
"The training set consists all the images and annotations containing non-held-out classes while held-out classes are masked as background during the training" from Few-Shot Semantic Segmentation with Prototype Learning.
"A cross-validation procedure is that non held out data (meaning after holding out the test set) is splitted in k folds/sets" from here.

What is non-held-out data and held-out data or classes?

Held-out simply means "not included" particularly in the sense of:

This part of the data was not included in this specific training run.

Depending on the context of all of these text non-held-out data/classes means the data that actually was included in a particular modeling exercise.

Consider this excerpt from your first example:

  For instance, Owen and Perry (2009) show a method for holding out data, computing SVD on the non-held-out data, and selecting k so as to minimize the reconstruction error between the held-out data and its SVD approximation.

It actually means:

  For instance, Owen and Perry (2009) show a method for excluding data, computing SVD on the remaining data, and selecting k so as to minimize the reconstruction error between the excluded data and its SVD approximation.

So it simply talks about a particular way of train-test-validation splitting the data.


Is GAIL applicable if the expert's trajectories are for the same task but are in a different environment? Is the GAIL applicable if the expert's trajectories (sample data) are for the same task but are in a different environment (modified but will not be completely different)?
My gut feeling is, yes, otherwise we can just simply adopt behavioural cloning. Furthermore, since the expert's trajectories are from a different environment, the dimension/length of state-action pairs will most likely be different. Will those trajectories still be useful for GAIL training?

The authors of the paper Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2018, published in ICRL), which introduced the inverse RL technique AIRL, argue that GAIL fails to generalize to different environment's dynamics. Specifically, in section 7.2 (p. 7), they describe an experiment where they disable and shrink the two front legs of the ant, then, based on the results, they conclude

GAIL learns successfully in the training domain, but does not acquire a representation that is suitable for transfer to test domains.

On the other hand, according to their experiments, AIRL is more robust to changes in the environment's dynamics.


How does the regression layer in the localization network of a spatial transformer work? I am trying to understand the spatial transformer network mentioned in this paper https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf. I am clear about the last two stages of the spatial transformer i.e. the grid generator and sampler. However I am unable to understand the localization network which outputs the parameters of the transformation that is applied to the input image. So here are my doubts.

Is the network trained on various affine/projective transforms of the input or only the standard input with a standard pose?
If the answer to question 1 is no, then how does the regression layer correctly regress the values of the transformation applied to the image? In other words how does the regression layer know what transformation parameters are required when it has never seen those inputs before?

Thanks in advance.

Localization network is not trained separately on special transform of input. It's just a part of feed forward network which is trained as a whole, with normal backpropagation.
It's just part of the network which affect the loss function. As in any backpropagation loss function propagate back gradient, which backpropagate through final part of the network, backpropagate through differentiable sampler(that is non-trivial part, which use transfromation produced by localization subnetwork) and after that backpropagate into localization part
Whole approach of spatial transformers could be in doubt now. There were some anecdotal evidences that it was not working on less trivial tasks (private communications, was not working for me either)


If the transition model is available, why would we use sample-based algorithms? Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model.
Assuming I do have access to a transition model, are there any reasons one might want to use sample-based methods instead of performing a full Bellman update?

A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if your state space is large this will also be intractable.


What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning? I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four hidden layers with 500 neurons in each layer, and sigmoidal activation functions in each layer.

The network appears to be unable to learn, and has high loss even on the training data. What might cause this to happen? Is there something wrong with the design of my network?

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.

The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25.

The backpropigation algorithm, which is used to train a neural network, will propagate an error signal backwards. At each layer, the error signal will be multiplied by, among other things, the derivative of the activation function.

It is therefore the case that by the 4th layer your signal is at most $0.25^4 = \frac{1}{256}$ the size that it was at the start of the network. In fact, it is likely much smaller than this. With a smaller signal, your learning rates at the bottom of the nextwork will effectively be much smaller than the learning rates at the top, which will make it very difficult to pick a learning rate that is effective overall.

This problem is known as the vanishing gradient.

To fix this, if you want to use a deep architecture, consider using an activation function that does not suffer from a vanishing gradient. The Rectified Linear activation function, used in so-called "ReLU" units, is a non-linear activation that does not have a vanishing gradient. It is common to use ReLUs for the earlier layers in a network, and a sigmoid at the output layer, if you need outputs to be bounded between 0 and 1.


Deep learning model training and processing requirement for Traffic data I am a newbie in deep learning and am looking for advice on predicting traffic congestion events. I have a table for vehicle travel times data, another table with the road length segmented based on stop locations. I am thinking to derive the time-wise route-specific speed details based on stop locations. After initial data cleansing and messaging, my input parameters are the time and stop location with actual speed details. I train my model with the training dataset and validate it as per the deep learning recommended approach.

So my questions are:

Is this approach correct or how can I improve it? I am not sure if
the number of inputs can be increased for better results.
Which activation method will be best to utilize to get a range of conditions/event types rather than binary 1 or 0?
This will require dealing with a bigger dataset of at least over a few GBs. This will evolve into around 200GBs in the final product. Can I use my professional grad laptop to process this data or if I should consider going to Big Data Processing power?

Please advise. Thanks in advance for your help.

Adjustments

There are a few considerations that may help with problem analysis:

Traffic data alone will produce limited predictive results.  For instance, the onset of precipitation is critical input for any production-ready predictive model.
Traffic congestion is highly chaotic in that events cascade, thus what is colloquially called the butterfly effect, is a dominant challenge.  What you can realistically achieve are what chaoticians call attractors.  They are morphological features in phase spaces and auto-correlations that are predictable under specific conditions.
Modelling traffic requires further segmentation beyond stop locations to include turn options and merge points, both of which are critical factors in free flow and congestion.
Feedback from the field will DRASTICALLY improve prediction.  Expecting the system to predict start to finish scenarios longer than a few minutes of potential congestion dynamics is unrealistic.  Inputs will need to be continuous and the closer to real time the better.
The physical relationships between position, velocity, and acceleration must be built into the parameterized model in which parameters are continuously updated.

We can assume that the system design predicts congestion for some purpose, which was not stated in the question.  It is reasonable to assume, because of the economics of transportation, that the objective is to reduce the area under the probability distribution curve for undesirable events resulting from congestion.

Lost time
Accident occurrence
Fatalities

The mechanism of control through which such objectives could be achieved can be any of these.

Notifying drivers of traffic conditions via cell towers or WiFi
Notifying drivers via dynamic sign display
Adjusting remotely configurable traffic signaling devices
Controlling gates

These controls can be to human drivers or pilots or to automated driving or piloting systems.  The context of the system is key to designing it properly, since the use cases are entirely driven by this context.

The Questions

Is this approach correct, [and] how can I improve it? — suggestions above
I am not sure if the number of inputs can be increased for better results. — absolutely yes
Which activation method will be best to utilize to get a range of conditions/event types rather than binary 1 or 0? — A ReLU or one of its derivatives may produce the best results for off line training, however it may be advisable to look into real time approaches such as Q-learning.  In either case, it is advisable to finish requirements definition, decide upon approach, and produce a general architectural diagram of the system before attending to such minutia as activation functions.
[Given the likely data set size of 200GB] can I use my professional grad laptop to process this data or if I should consider going to Big Data Processing power? — Using VLSI hardware acceleration from vendors such as Intel or NVidia is advisable.  Over the course of time, the initial expense and learning curve may be absorbed by not having to pay for additional bandwidth for PasS or SaaS services.

Regarding computing resources, it is inadvisable to begin with some of the most congested places.

Cities Definitely Too Congested

It is likely that these cities would require a super-computing platform of significant size and a multinational corporation sized budget.

Beijing
Dubai
Tokyo
Los Angeles
Chicago
London
Hong Kong
Shanghai
Paris
Amsterdam
Dallas

Commuter locations such as Melbourne, Australia; Palm Beach, Florida; or New Haven, Connecticut are examples of good choices to start.


How to train the NN of simple agents given a reward system? I'm not an expert in AI or NN, I gathered most of the information I have from the internet, and I'm looking for advice and guidance.
I'm trying to design a NN that is going to be used by all the agents of my simulation (each agent will have its own matrix of weights). This is what I plan to have:

The NN will have 1 input layer and 1 output layer (no hidden layers).
The number of inputs will always be greater than the number of outputs.
The outputs represent the probability of an action being taken by the agent (the output node with the highest value will identify the action that will be taken). Which means there are as many output nodes are there are actions.

When an agent takes an action it receives a reward: a number that represents how well the agent performed. This happens "online" that is, the agent is trained on the fly.
What I would like to know if how to best train the NN: that is, how to update the weights of my matrix to maximise the rewards long term.
From the research I made it seems this is close to the concept of Reinforcement Learning, but even if it was, it's not clear to me how to apply it to such a simple NN shape.

You might be able to glean what you want from Chapter 13 or Sutton & Barto's Reinforcement Learning: An Introduction, which deals with policy gradient algorithms, and includes pseudocode for a variety of agents based on linear approximation using softmax regression. From your description, you appear to be using - or should consider - softmax regression for the policy function.
Probably a good bet using your setup will be REINFORCE with baseline, which does not add requirements to track anything more than your current weights, sum of rewards seen after each state & action pair, and mean rewards (or other baseline). It does require an episodic environment though, such as a game that ends.
Essentially REINFORCE and its variants require you to figure out the return (sum of rewards received after each action to the end of the episode) and perform a gradient step update multiplied by that return (minus a baseline which can help with stability).
With REINFORCE and a discrete distribution, the gradient due to action selection is the same as supervised learning for multi-classification evaluated as if whichever action the agent took was the correct one. When you multiply this gradient by the return as well, it will make steps in the direction of better action choices larger. Regardless of the fact that you may train towards selecting many different actions, the better actions will win the race in the end due to this weighting, and have the highest probability of being selected after a large number of training episodes.
A multi-agent problem may bring you some problems, such as instability due to dependency of performance on what the other agents are doing. It is not possible to say how much of an issue this will be. However, I would advise not jumping straight into your main problem, and try some reinforcement learning methods with a single agent on toy problems first. If you build up to solving the problem you are currently facing, then you will be much more confident in how you have implemented the learning. There are plenty of stepping-stone projects and pre-built environments to solve if you search for them. A good starting point is OpenAI's Gym, which presents a standardised set of toy - and not so toy - problems that you can test any learning algorithm against.


Why is image classification tasks are dominated by minimizing cost function instead of maximizing ones? I was watching a video of policy gradient by Andrej Karpathy at 10:00 there shows an equation for supervised learning for image classification.

$max\sum _{i}log \:p(y_i|x_i)$

I have worked with image classification models before but I always used a minimizing cost function aka loss function. I also never seen some one using maximizing cost function for image classification in the wild.

Why image classification tasks are dominated by loss functions?
What is the advantages of a minimizing loss function  over a maximizing
loss function in image classification?
Other than RL where else maximizing cost functions are used?

There is really no difference between minimizing a cost function and maximizing a value function.  One can be the reciprocal of the other, or the negative of the other, for example.


How can I calculate the shortest path between two 2d vector points in an environment with obstacles? I have a 2D plane, with a fixed height and width of 10M. The plane has an agent (or robot) in the point $(1, 2.2)$, and an electric outlet in the point $(8.2, 9.1)$. The plane has a series of obstacles.

Is there an algorithm to find the shortest path between the agent and the goal?
And if the point has a fixed wingspan? For example, that the space between O and N is smaller than the agent, and then the agent cannot cross?

The usual way to solve this kind of problem is to construct a configuration space: extruding all the polygonal obstacles by sliding the polygon corresponding to the robot around them (some slides).

The exterior vertices of the configuration space can then be used as input to a path-planning algorithm, such as A*.


How artificial intelligence will change the future? AI is the emerging field and biggest business opportunity of the next decade. It's already automating manual and repetitive tasks. And in some areas, it can learn faster than humans, if not yet as deeply.
From the Forbes article

In the AI-enabled future, humans will be able to converse and interact with each other in the native language of choice, not having to worry about miscommunicating intentions.

I would like to know more about how artificial intelligence will change the future?

Here is something I've noticed about humans: We're bad at projecting the future with all of its 2nd, 3rd ... N order effects, and we're REALLY bad a projecting and quantifying risk. So, I'm not sure that you'll get an answer that is anything more than either trivially true ("Chatbots will be commonplace") or a correct but wasn't justified ("We'll explore the stars by sending out swarms of AI-enhanced satellites that will use their computing power to improve themselves since they are always on but have copious amounts of down-time").
But if you want to limit the timeframe to the next decade and don't mind the previous caveat, here goes:

Chatbots will be commonplace. This will include ones that can handle conversation, like Gridspace but can also handle multiple languages and code-switching.
In professional use, AI will augment professionals (such as lawyers, physicians, teachers, and accountants) at the upper end (who will add value with strategy, experience, and research) and replace them at the lower end.
In military use, they will be used for target identification, automated assessment of damage, threat notification, and forecasting.
We'll explore the stars by sending out swarms of AI-enhanced satellites that will use their computing power to improve themselves since they are always on but have copious amounts down-time.

I don't hold a lot of hope out for communication without error of to speakers of different languages as you can have two native fluent speakers who have known each other for years speak to each and still have them miscommunicate.


What kind of problems require more than 2 hidden layers? I've read that the most of the problems can be solved with 1-2 hidden layers.

How do you know you need more than 2? For what kind of problems you would need them (give me an example)?

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal Approximation Theorem.

Finding the best topology for a given problem is an open research problem. As far as I know, there are few universal 'rules of thumb' for this.

For a given problem, one option is to apply a neuroevolutionary approach such as NEAT, which attempts to find a topology that works well for the problem at hand.


What is the difference between multi-head and normal output? Let's say that I have a neural network with 2 heads. The first consists of X neurons. The second consists of Y neurons. I have these 2 heads because I want to predict 2 different variables. And I can see the loss for each head during training.
Now, let's say that I have only one head that consists of X+Y neurons. I can interpret the output because I know that the first X neurons describe some variable and the latter Y neurons describe the second variable.
I want to know if there is any difference between these 2 methods (maybe in performance or something). What are the pros and cons? Are there any advantages of one method over another for some particular tasks?

It depends on what your outputs are. For example, if both outputs are similar then you can use one output branch. However, what if the two outputs are different? With two output branches you can used two different loss functions. Now your model will optimize the two branches separately.
Imagine if you have a model that has to output a class label for the input and a real value describing something in the input. One is a classification task and the other is a regression task. And the two  will require different loss functions, so you would use two branches.


How many training example text classifier needs to be trained? I want to train text classifier (using https://www.uclassify.com) with 12 classes/categories. I will be training it to classify news/articles (I know that there are existing classifier but I want to train my own).

uclassify uses following algorithm (directly copied from their site):

  The core is a multinominal Naive Bayesian classifier with a couple of
  steps that improves the classification further (hybrid complementary
  NB, class normalization and special smoothing). The result of
  classifications are probabilities [0-1] of a document belonging to
  each class. This is very useful if you want to set a threshold for
  classifications. E.g. all classifications over 90% is considered spam.
  Using this model also makes it very scalable in terms of CPU time for
  classification/training.

I was wondering how many examples I will need to train such classifier? It is possible to estimate the number? Let's assume that one article will "fit" 2 categories by average.

As a general rule of thumb I typically use 10*(# of features) for shallow machine learning models such as Naive Bayes with only 2 classes.

So it all depends on the number of features you will be using. However, the more output classes the more data you will need for proper discrimination. The addition of more classes is not linear but I think you can get away with: 10*(# of features)*(# of output classes)


Why does Simulated Annealing not take worse solution if the energy difference becomes higher? In Simulated Annealing, a worse solution is accepted with this probability:
$$p=e^{-\frac{E(y)-E(x)}{kT}}.$$
If that understanding is correct: Why is this probability function used? This means that, the bigger the energy difference, the smaller the probability of accepting the new solution. I would say the bigger the difference the more we want to escape a local minimum. I plotted that function in Matlab in two dimensions:

Note that you can't really predict whether your escape from a local minimum will work or not - you might just wind up in another, worse local minimum. The probability function you describe increases the likelihood of this happening. By upweighting the likelihood of allowing small energy differences, you allow for the possibility of escaping local minima, while ensuring that whatever new minimum you find can't be that much worse that where you started. If you make the acceptance of large energy differences more likely, you can escape local minima more often, but you increase the likelihood that you'll just wind up in a region with an even higher local minimum.


What are the most common feedforward neural networks? What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?

What is a neural network?
Many neural networks can be defined as a function $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$, where $n, m \geq 1$.
Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) $f^i$ that receive some input and produce output, i.e. $f^i(\mathbf{x}^i) = y^i$, where $\mathbf{x}^i \in \mathbb{R}^k$. The actual function $f^i$ is variable and depends on the application or problem you want to solve. For example, $f^i$ can just be a linear combination of the inputs, i.e. $f^i(\mathbf{x}^i) = \sum_{j} \mathbf{w}_j^i \mathbf{x}_{j}^i$, where $\mathbf{w}^i \in \mathbb{R}^k$ is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.
If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).
What are the most common feedforward neural networks?
Perceptron
The simplest (non-binary) FFNN is the perceptron, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.
Before the perceptron, McCulloch and Pitts had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.
Multi-layer perceptron
An FFNN with more layers (of units) between the input and the output is often called a multi-layer perceptron (MLP). The layers in the middle are often denoted as hidden layers. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.
Convolution neural network
You can have other forms of FFNNs that perform other operations.
For example, a convolution neural network (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the convolution operation (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)
However, note that CNNs can also have recurrent connections, but this is not usually the case.
Residual neural network
There are also residual neural networks, i.e. neural networks where a node in a certain layer $l$ can be connected to other nodes in layers $l+j$, for $j \geq 1$, as opposed to being connected only to the nodes in layer $l+1$, which is the typical case.
Auto-encoders
Auto-encoders are neural networks that compress the input and then decompress it. The answers to this question may help you to understand why AEs would be useful.
What kind of inputs do they receive?

What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?

In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).
How can you further classify the NNs?
Based on chapter 2 of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas, you can also divide neural networks into other categories

Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)
Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)
Neural networks that store states vs NNs that don't store states

You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.


Do these FOL formula both represent "You can fool some of the people all of the time"? You can fool some of the people all of the time.

This can be represented in FOL as follows
$$\exists x \; \forall t  \; (\text{person}(x) \land \text{time}(t)) \Rightarrow \text{can-fool}(x,t) \tag{1}\label{1}$$
Is $\exists x \; \forall t \; \text{can-fool}(\text{person}(x), \text{time}(t))$ equivalent to (\ref{1}) ?

(1) can be paraphrased as "There exists an x, and for any t if x is a person and t is a time, then x can be fooled at time t" (I would use fool-able instead of can-fool, as it is closer to the intended meaning).
(2) would be "There exists an x, and for any t, you can fool x is a person and t is a time."
They are not equivalent: person(x)  and time(t) are boolean predicates, which return a truth value: they are true if x is a person, and t is a time, respectively. So in (1) they act as a constraint on the values that x and t can take. If x was a saucepan, then person(x) would be false, and thus you wouldn't be able to claim that you can fool a saucepan.
So can-fool takes two arguments: one for which person is true, and one for which time is true. But in (2), the arguments are actually the boolean truth values: if x was "Falstaff" and t was "yesterday", then in (1) the premise would be true, as person("Falstaff") and time("yesterday") are true, and so you conclude can-fool("Falstaff", "yesterday").
In (2) that becomes can-fool(person("Falstaff"), time("yesterday")), which evaluates to can-fool(true, true), and that won't work.


In DQN, when do the parameters in the Neural Network update based on the reward received? I'm aware that we back-propagate after computing the loss between:
The Neural Network Q values and the Target Network Q values
However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.
Suppose one epoch is run and the reward is +10, surely we need to update the parameters using this too to tell the Network to push the probability of these actions, given these parameters up.
How does the algorithm know +10 is good? Suppose the reward range is -10 for loss and +10 for win.

However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.

Yes. That is all it needs to do because we have defined the policy around the Q values like so:
$$\pi(s) = \text{argmax}_a \hat{q}(s,a,\theta)$$
Where $\theta$ is the neural network weights.
Therefore, if the estimates of Q are approximately the same as the action value of the optimal policy, the policy in DQN is approximately the optimal policy.

How does the algorithm know +10 is good?

It does not, at least not directly. The algorithm knows, approximately, what the action values are if it acts consistently with their current estimates by always choosing the maximising action at each step.
The learning process will learn that +10 is relatively good in your scenario because it never finds anything better when exploring.


What is the difference between goal-based and utility-based agents? What is the difference between goal-based and utility-based agents? Please, provide a real-world example.

Utility is a fundamental to Artificial Intelligence because it is the means by which we evaluate an agent's performance in relation to a problem. To distinguish between the concept of economic utility and utility-based computing functions, the term "performance measure" is utilized.

The simplest way to distinguish between a goal-based agent and a utility-based agent is that a goal is specifically defined, where maximization of utility is general.  (Maximizing utility is itself a form of goal, but generalized as opposed to specific.)

A goal-based navigation agent is tasked with getting from point A to point B.  If the agent succeeds, the goal has been satisfied.
A utility-based navigation agent could seek to get from point A to point B in the shortest amount of time, with the minimum expenditure of fuel, or both.

In the above example, the utility agent is also goal based, but where the performance measure for the goal agent is a binary [succeed/fail], the utility agent can use real numbers and measure performance by degree. The utility agent allows more granularity in evaluation.

For an example of a non-goal based utility agent consider a form of a partisan sudoku in which players compete to control regions on the gameboard by placement of weighted integers.

In a game with 9 regions, the goal based agent seeks to control a specific number of regions at the end of play.  If the agent is conservative, the goal might be 5 regions. If the agent is hyper-aggressive, the goal might be 9 regions.  When evaluating the environment (gameboard), if the agent dominates the desired number of regions, it could choose to consolidate (reinforce); if the agent does not dominate the desired number of regions, it could choose to expand (attack).

The above strategy can be effective, but is limited by the specificity of the goal.  A hyper-aggressive goal would work well against a weak opponent, but against a strong opponent it might prove disastrous.  If the agent is sophisticated, where performance has been poor, it might alter it's goal by switching to a "turtling strategy" and seek to control fewer regions, but, because the new goal is still specific, the agent may miss opportunities to improve it's final status beyond the adjusted goal.

The utility-based agent can approach the game with no specific goal beyond improving it's status. Rather than seeking to control a set number of regions, the utility-agent evaluates whether a given choice improves or worsens it's status. ("Do I dominate more or less regions if I take this position?") The utility agent can distinguish between sets of beneficial choices ("which choice maximizes my expected benefit?") and, where no benefit can be obtained, distinguish among the set of choices with the least downside ("among the set of bad choices, which is the least bad choice?")

In this example, the utility-agent doesn't even need to understand the victory condition (controlling the more regions than the opponent at the end of play.)  Instead, the utility-agent merely seeks to maximize the number of controlled regions over the course of play, which will result in victory if the agent makes more optimal choices than the opponent.


What are some books/paper that deal with fundamental and philosophical issues of ML and relate it to the global discourse of AIs? In my experience, most of the time, when people talk about AI nowadays they mostly mean machine learning. Despite this, ML is usually seen as a mere technique to build high-performance software; rarely I see people discuss the foundational questions of it, such as, from which "philosophy" of AI did ML emerge? Why is ML compelling in AI research, if not by its performance? What are the fundamental differences between statistical/probabilistic AI and logical AI? For reference, this hasn't even been mentioned on my master-level course on machine learning. Even myself I used to have a distaste for ML because I thought it was just mindless data-crunching.

But lately I've been reading through "Probability Theory: The Logic Of Science" and I'm starting to appreciate the theoretical side of ML, for instance, how bayesian probability can be seen as a model of plausible reasoning in humans, and how probability theory extends logic (motivating, maybe, why probabilistic AIs were the next logical [no pun intended] step after logical AI). I would like now to delve deeper in the topic.

What are some books/paper that deal with fundamental and philosophical issues of ML and relate it to the global discourse of AIs?

I'll recommend two sources:

The venerable Russell & Norvig book, which is a common text in AI courses. Russell & Norvig end each chapter with a summary of the history of the developments of the techniques they have just discussed. These sections are often skipped by novice readers, but are almost exactly what you are looking for. The ones in the back half of the book should, together, give you a good sense of the order in which developments occurred, why techniques were developed, and what advances techniques enabled.
R&N does a good job of covering what happened, but not always why. For that, you want a philosophy of AI book. I recommend Mind Design II as a starting place. This book is a chronologically organized collection of papers and academic essays by the big thinkers in philosophy of mind and in AI research. Often, the papers are responses to one another. With some side reading about the history of each author, you can begin to get a good sense of the big philosophical movements in the field over the last 70 years, and why things have changed.

If you don't want to read the book, I can give you a summary (spoilers ahead!):

1920's: behaviorists, and others, propose that Mind = Brain, and specifically focus on intelligent behavior.
1950: Turing proposes that a computer could be programmed to exhibit intelligent behavior.
1960's: Cognativists in AI, Psych, and Linguistics argue that behavior is not enough. Minds think, and thought takes the form of reasoned algorithms. The lynchpin of their argument is domains like language understanding, which they claim cannot be modeled without logic. Their work produces search algorithms, and early rule-based planning and language systems.
1970's Searle argues that computers can't think because of Phenomenology. Most AI folk ignore him, and go on working, but Philosophy & Psych take greater notice.
1970's Dryfus (and others) argue compelling that logic and reason cannot explain human thought, by dissecting the rule based systems of the day through the Frame Problem. AI researchers take some notice.
The Connectionists, especially Hinton (AI) & Churchland (Philosophy) propose the first post-Cognativist theories of mind. These focus on the idea that Mind is in the Connections (specifically, the firing patterns) of neurons, not in the brain itself. This view of minds spurs (re)-development of Neural Networks. In the 1980's, this is mostly ignored.
During the 1990's, Connectionists, and others working on probabilistic methods in AI demonstrate systems for language that are substantially better than rule based methods. Cognativists begin to decline, because the domains they claimed needed reasoned algorithms are actually better handled by statistical algorithms. Connectionist views gain support in AI and elsewhere.
Today, with rising computational power, statistical techniques come to the forefront of many AI domains. Simultainiously, the Churchlands, Rodney Brooks, and others propose further Post-Cognativist schools of thought based around dynamical systems theory and embodied cognition, which were somewhat influential in robotics. Cognativism continues to enjoy some support within the AI, Psych, and Linguistics communities, but this is much diminished. Some hybrid systems use a mix of statistical and rule-based techniques.


What is the purpose of "reshaping it into the shape the network expects and scaling it so that all values are in the [0, 1] interval."? I am a deep learning beginner recently reading this book "Deep learning with Python", the example explains the process of implementing a greyscale image classification using MNIST in keras, in the compilation step, it said,

  Before training, we’ll preprocess the data by reshaping it into the shape the network expects and scaling it so that all values are in the [0, 1] interval. Previously, our training images, for instance, were stored in an array of shape (60000, 28, 28) of type uint8 with values in the [0, 255] interval. We transform it into a float32 array of shape (60000, 28 * 28) with values between 0 and 1.

Images stored in an array of shape (60000, 28, 28) of type uint8 with values in the [0, 255] interval. For my understanding, the values are between 0-255 of each px and storied as 3D matrix. Can someone explain why needs to "transform" it into the network expects by scaling it and make "all values are in the [0, 1]interval."?

Please also make suggestions if I didn't explain some parts correctly.

The purpose of rescaling  gradient descent problems is to reframe the problem for quicker convergence / calculation of linear coefficient parameters.  in the Stanford video series, Andrew Ng provides a intuitive explanation and enables one to hone an intuitive understanding.

Multivariate input regression gradient descent converges faster when the inputs are in the same order of magnitude.  For example, if predicting house prices based on X1= the number of rooms and X2= area of the home in square feet.  X1 is on scale of 0-6 bedrooms and and X2 is typically 1000-3000 square feet.  Given the diffence in magnitude, this problem is a good candidate for feature scaling.


Traffic signs dataset I'm looking for annotated dataset of traffic signs. I was able to find Belgium, German and many more traffic signs datasets. The only problem is these datasets contain only cropped images, like this:

While i need (for YOLO -- You Only Look Once network architecture) not-cropped images.

I've been looking for hours but didn't find dataset like this. Does anybody know about this kind of annotated dataset ?

EDIT:

I prefer European datasets.

Look at Google's Open Image Dataset @ https://storage.googleapis.com/openimages/web/index.html

They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships.

Here is the link for the traffic signs dataset.


Would the people of the 19th Century call our conventional software today artificial intelligence? It is possible that the view of what is impressive enough in computer behavior to be called intelligence changes with each decade as we adjust to what capabilities are made available in products and services.

They would probably have followed the same sequence we do:

be amazed at the capabilities,
ask how it is done,
wonder whether this is really intelligence and (or) point out our narrow the performance was,
require more next time to be impressed again.


What are the segment embeddings and position embeddings in BERT? They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.

ELMo paper - https://arxiv.org/pdf/1802.05365.pdf

BERT paper - https://arxiv.org/pdf/1810.04805.pdf

These embeddings are nothing more than token embeddings.

You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.


Can mutation enable a disabled connection? In the add node mutation, the connection between two chosen nodes (e.g. A and B) is first disabled and then a new node is created between A and B with their respective two connections.

I guess that the former A-B connection can be re-enabled via crossover (is it right?).

Can the former A-B connection also be re-enabled via mutation (e.g. "add connection")?

Yes, the original gene is disabled, but is left in the genome. This can be seen on page 10, figure 3 of the paper linked (taken from the original paper NEAT Paper) where gene 3 is disabled, but not removed from the genome. This gene can be re-enabled by receiving the gene with the identical innovation number from a mating partner with the gene enabled during crossover.

The original paper does not mention a mutation to re-enable genes, but various other publications and implementations after the original paper do. This is desirable for a number of reasons. Re-enable mutate allows for dropout to used in the implementation. It is also possible that certain genomes are disabling genes too quickly and this can help to correct for that.


Why does Q-learning converge under 100% exploration rate? I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the following picture.

Here are the values of my parameters.

Learning rate = 0.1
Discount factor = 0.95
Default reward = 0

Reaching the trophy is the final reward, no negative reward is given for bumping into walls or for taking a step.
After 500 episodes, the arrows have converged. As shown in the figure, some states have longer arrows than others (i.e., larger Q-values). Why is this so? I don't understand how the agent learns and finds the optimal actions and states when the exploration rate is 100% (each action: N-S-E-W has 25% chance to be selected)

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random policy (i.e. 100% exploration) is guaranteeing this and the other conditions are met (which they probably are) then Q-learning will converge.
The reason that different state-action pairs have longer arrows, i.e. higher Q-values, is simply because the value of being in that state-action pair is higher. An example would be the arrow pointing down right above the trophy -- obviously this has the highest Q-value as the return is 1. For all other states it will be $\gamma^k$ for some $k$ -- to see this remember that a Q-value is defined as
$$Q(s, a) = \mathbb{E}_\pi \left[\sum_{j=0}^\infty \gamma^j R_{t+j+1} |S_t = s, A_t = a \right]\;;$$
so for any state-action pair that is not the block above the trophy with the down arrow $\sum_{j=0}^\infty \gamma^j R_{t+j+1}$ will be a sum of $0$'s plus $\gamma^T$ where $T$ is the time that you finally reach the trophy (assuming you give a reward of 1 for reaching the trophy).


Understanding notation of Goodfellow's GAN objective function What is the meaning of $V(D,G)$? How do we get these expectation parts?
I was trying to understand it following this article: Understanding Generative Adversarial Networks (D.Seita), but, after many tries, I still can't understand how he got from $\sum_{n=1}^{N} \log D(x)$ to $\mathbb{E}(\log(D(x))$.

To understand this equation first you need to understand the context in which it is first introduced. We have two neural networks (i.e. $D$ and $G$) that are playing a minimax game. This means that they have competing goals. Let's look at each one separately:

Generator

Before we start, you should note that throughout the whole paper the notion of the data-generating distribution is used; in short the authors will refer to the samples through their underlying distributions, i.e. if a sample $a$ is drawn from a distribution $p_a$, we'll denote this as $a \sim p_a$. Another way to look at this is that $a$ follows distribution $p_a$.

The generator ($G$) is a neural network that produces samples from a distribution $p_g$. It is trained so that it can bring $p_g$ as close to $p_{data}$ as possible so that samples from $p_g$ become indistinguishable to samples from $p_{data}$. The catch is that it never gets to actually see $p_{data}$. Its inputs are samples $z$ from a noise distribution $p_z$.

Discriminator

The discriminator ($D$) is a simple binary classifier that tries to identify which class a sample $x$ belongs to. There are two possible classes, which we'll refer to as the fake and the real. Their respective distributions are $p_{data}$ for the real samples and $p_g$ for the fake ones (note that $p_g$ is actually the distribution of the outputs of the generator, but we'll get back to this later).

Since it is a simple binary classification task, the discriminator is trained on a binary cross-entropy error:

$$
J^{(D)} = H(y, \hat y) = H(y, D(x))
$$

where $H$ is the cross-entropy $x$ is sampled either from $p_{data}$ or from $p_g$ with a probability of $50\%$. More formally:

$$
x \sim
\begin{cases}
p_{data} \rightarrow & y = 1, & \text{with prob 0.5}\\
p_g \;\;\;\,\rightarrow & y = 0, & \text{otherwise}
\end{cases}
$$

We consider $y$ to be $1$ if $x$ is sampled from the real distribution and $0$ if it is sampled from the fake one. Finally, $D(x)$ represents the probability with which $D$ thinks that $x$ belongs to $p_{data}$. By writing the cross-entropy formula we get:

$$
H(y, D(x)) = \mathbb{E}_y[-log \; D(x)] = \frac{1}{N} \sum_{i=1}^{N}{ \; y_i \; log(D(x_i))}
$$

where $N$ is the size of the dataset. Since each class has $N/2$ samples we can split this sum into two parts:
$$
= - \left[ \frac{1}{N} \sum_{i=1}^{N/2}{ \; y_i \; log(D(x_i))} +  \frac{1}{N} \sum_{i=N/2}^{N} \; (1 - y_i) \; log((1 - D(x_i))) \right]
$$

The first of the two terms represents the the samples from the $p_{data}$ distribution, while the second one the samples from the $p_g$ distribution. Since all $y_i$ are equally likely to occur, we can convert the sums into expectations:

$$
= - \left[ \frac{1}{2} \; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \frac{1}{2} \; \mathbb{E}_{x \sim p_{g}}[log \; (1 - D(x))] \right]
$$

At this point, we'll ignore $2$ from the equations since it's constant and thus irrelevant when optimizing this equation. Now, remember that samples that were drawn from $p_g$ were actually outputs from the generator (obviously this affects only the second term). If we substitute $D(x), x \sim p_g$ with $D(G(z)), z \sim p_z$ we'll get:

$$
L_D = - \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

This is the final form of the discriminator loss.

Zero-sum game setting

The discriminator's goal, through training, is to minimize its loss $L_D$. Equivalently, we can think of it as trying to maximize the opposite of the loss:

$$
\max_D{[-J^{(D)}]} = \max_D \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

The generator however, wants to maximize the discriminator's uncertainty (i.e. $J^{(D)}$), or equivalently minimize $-J^{(D)}$.

$$
J^{(G)} = - J^{(D)}
$$

Because the two are tied, we can summarize the whole game through a value function $V(D, G) = -J^{(D)}$. At this point I like to think of it like we are seeing the whole game through the eyes of the generator. Knowing that $D$ tries to maximize the aforementioned quantity, the goal of $G$ is:

$$
\min_G\max_D{V(D, G)} = \min_G\max_D \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

Disclaimer:

This whole endeavor (on both my part and the authors' part) was to provide a mathematical formulation to training GANs. In practice there are many tricks that are invoked to effectively train a GAN, that are not depicted in the above equations.


Will/can AI be the worst enemy to humanity? This is the question which always is being discussed. Will artificial intelligence be the undefeatable enemy for human beings?

Like the Slaughter Bots (Killer Microdrones technology) can kill a human or an living thing in just some seconds. And it neither could be deceived by its target nor it could be stopped by any source. Once it's set to kill the target, it can never be stopped and it identifies its target. It is also the innovation of AI.

See Microdrones technology introduction.

I highly suggest you read Hannu Rajaneimi's Quantum Thief trilogy, William Gibson's Neuromancer trilogy, and Phillip K. Dick's Do Android's Dream of Electric Sheep, Isaac Asimov's I, Robot.

It's a nuanced subject, and no one knows.

The most credible scenarios I've found for the destruction of humanity by automata involves not high-intelligence, but very low intelligence.  (The Grey Goo scenario.)  Dick, a gifted philosopher in his own right, proposes that empathy is a natural function of high intelligence, which provides a credible, increasingly validated ray of light.

But Hawking and Musk are credible imo per their comfort with very large numbers (complexity) and proven world-class abstract thinking capabilities.  It's possibly that strong "hyperpartizan" AI could run amok.  We've already seen issues with high-frequency trading algorithms, and automation is proceeding in weapons tech.

One might say "we are cursed to live in such interesting times."


What is the effect of picking action deterministicly at inference with Policy Gradient Methods? In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution.

When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?

When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?

Depends what you mean by "evaluating the policy". Unlike in value-based methods, such as Q learning, the policy in gradient methods is not implied by anything else, it is described directly by the probability density function that is being optimised.

Taking the maximum probability item will technically change the policy (unless you are already using deterministic policy gradient), and you would be evaluating a different but related policy to that found by your policy gradient.

However, in a standard MDP environment, and after at least some training, this should be a reasonable process that would give some indication of how well the agent is performing.

In some cases, the nature of the environment means the agent is relying on a stochastic policy. In some partially-observable scenarios it may be better to decide randomly - a simple example is a corridor that needs to be traversed, but where the state features don't give enough information to determine the true direction. A deterministic policy will not be able to traverse the corridor in both directions, but a stochastic policy will get through it both ways, eventually. Another example is in adversarial situations where another agent can learn your agent's policy (the classic version of that being Scissor/Paper/Stone where two ideal opposed agents would learn probability $\frac{1}{3}$ for each action according to Nash equilibrium)

If you don't think you have these special cases, then it should be OK to derive a deterministic policy from your policy gradient agent, and assess that. That's not quite the same as assessing the "learned policy", but is quite sensible to do once you think the agent has converged anyway, since it may still be selecting non-optimal actions at some low probability, and you could get closer to optimal behaviour by removing that.


Training an artificial neural network using PSO Hi guys I've been studying such combination where the idea is to replace the classic descendant gradient for an algorithm that is less sensitive to local optimum, so the PSO tries to select the best weight and bias for the model.

But I've seen everywhere that only one hidden layer is used (no one explains why just one layer is being used) and all those codes break when I try to use more than one hidden layer, so the questions are:

Can't PSO be used to optimize an Artificial Neural Network with more than one hidden layer?
In that case, why is that?

Particle Swarm Optimization can be used to optimize a neural network with more than one hidden layer. Instead of optimizing a single weight matrix, and two bias vectors, you are just optimizing more of them. However, PSO is not often used for larger neural networks, because, particle swarm optimization is not all that efficient at working with a large amount of data, which you need to train larger neural networks. Particle Swarm Optimization involves taking many particles, and evaluating the fitness in all of them. If you have let’s say 1,000,000 training examples, you are doing a lot more error calculations, even if you used a small batch size, than if you used another technique lake Backproppogation, or a related technique. Backproppogation and related techniques are used much more in larger neural networks because they are more efficient at working with larger sets of data. As for why the code that you have been using breaks with networks with more than one hidden layer, I cannot explain, but it is fairly easy to write your own basic implementation of PSO to train multi layer neural networks.


On which data evaluate an object detection model ? (similar or real life data ?) I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.

Even I have more than 2k different poses, but the background does not change much (I have only 5 different points of view).

I trained my model on these images (70% for the training and 30% for validation).

Now, I want to evaluate the model on an unbiased dataset.

Should I keep images of my dataset for this purpose or should I use a real life dataset ?

(A good solution would be to have a real life training set, but I don't have)

I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.

Bonus question: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

Precision-Recall ? (but I have to choose a pose classification threshold...)

If you want to evaluate on real thermal image dataset, you can use this one.

Thermal Image dataset

is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

  Mean Average Precision is the relevant metric but it's more technical. You can start explaining with False Positives and False
  Negatives in the predictions. In turn, it leads to Precision and Recall. It mostly depends upon your use case because there will always be a trade-off between them.


Can the IQ of an AI program be measured? Can an AI program have an IQ?

In other words, can the IQ of an AI program be measured?

Like how humans can do an IQ test.

Short answer: No.

Longer answer: It depends on what IQ exactly is, and when the question is asked compared to ongoing development. The topic you're referring to is actually more commonly described as AGI, or Artificial General Intelligence, as opposed to AI, which could be any narrow problem solving capability represented in software/hardware.

Intelligence quotient is a rough estimate of how well humans are able to generally answer questions they have not previously encountered, but as a predictor it is somewhat flawed, and has many criticisms and detractors.

Currently (2016), no known programs have the ability to generalize, or apply learning from one domain to solving problems in an arbitrarily different domain through an abstract understanding. (However there are programs which can effectively analyze, or break down some information domains into simpler representations.) This seems likely to change as time goes on and both hardware and software techniques are developed toward this goal. Experts widely disagree as to the likely timing and approach of these developments, as well as to the most probable outcomes.

It's also worth noting that there seems to be a large deficit of understanding as to what exactly consciousness is, and disagreement over whether there is ever likely to be anything in the field of artificial intelligence that compares to it.


Q-learning in gridworld with random board I'm trying to use Q-learning in order to solve Wumpus world environment.
Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense that he is on the gold field), then he has to pick it up and leave cave in the entry position. Some fields are safe, other contain pit or wumpus (monster). If agent move to the pit or wumpus field he dies. The fields next to wumpus or pit (not diagonally!) have properties that agent can sense - stench (wumpus), breeze (pit). Agent achieve positive reward if he leaves cave with gold and negative if he dies. Action space: turn right/left, move forward, shoot arrow (if shot in good direction it can kill wumpus, only 1 available), pick up gold, leave cave. There: https://www.javatpoint.com/the-wumpus-world-in-artificial-intelligence you can find more detailed description of environment.
It is easy to solve this problem if the gridworld is constant. I have a huge problem to even start thinking about it if gridworld is random in every learning episode (random fields and random size) and also random in testing. How should I define it (especially state space)? How should I create q table? I'm using Python.
Thank you in advance for any help.

From your linked description of the game, we can see it has a key property when used normally in AI teaching:

Partially observable: The Wumpus world is partially observable because the agent can only perceive the close environment such as an adjacent room.

This makes sense, the problem of avoiding hazards would be trivial if the full map was revealed to the agent. The problem is also not about learning a specific map.
You could use the agent's perception to construct a simple state table based on current observations:

Current location
Current facing
Boolean flags for stench, breeze, glitter, bump, scream

This would be relatively easy to build into a Q table, and might have some success. However, it would likely perform a lot worse than the propositional logic and planning suggested suggested at https://www.javatpoint.com/the-wumpus-world-in-artificial-intelligence for two reasons:

Agent will not take account of knowledge specific to the current map that it has gathered. This is critical, because the stench and breeze sensors only tell you that at least one of the adjacent rooms has a hazard. In theory you know that this will not be any of the rooms that the agent has already visited, but a simple state representation based on current observations will not capture this.

Agent will not plan using the deterministic rules of the game that you know and could code for.

There are a few different approaches you could take to improve on these issues. Sticking with Q learning and trying to solve this by improving the state representation, you could look at the suggested knowledge-building structure from the java T point site, and replicate something like it as input features:

Current facing
Whether or not the last move forward resulted in a bump
Whether or not there has been a scream
For each allowed room, a binary flag set to true if the room has been visited by the agent
For each allowed room, a set of binary variables covering the three observations that are about the room: stench, breeze, glitter
Either the current location separately, or another binary variable added to each grid square set to true for the one room that the agent is in

When multiplied by 16 to cover each room, you will end up with ~84 binary flags. Although many combinations will not be possible, this is still going to be far too large a state space to use with a Q table. You would probably use a simple neural network and DQN agent to solve this problem using Q learning.


What dataset might Elon Musk's Dall-E have used? Dall-E, it can generate many imaginative images from the description, even some peculiar images, how did they actually create this kind of dataset to train this AI , because there is not much of that kind of data which include weird images and descriptive text, how did they create this massive dataset. Does anyone have any idea?
If you have no idea what I am talking about, please refer to this link: https://openai.com/blog/dall-e/.

DALL·E is a 12-billion parameter version of GPT-3 trained to generate images from text descriptions

should be the same data they used to train the GPT-3


Understanding how to calculate $P(x|c_k)$ for the Bernoulli naïve Bayes classifier I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes, however when looking at how $P(x|c_k)$ is calculated I don't understand it. The Wikipedia page says its calculated as follows $P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)} $.

They mention that $p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, does that mean $P(x|c_k)$? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is $p_{ki}$?

And in the first part, after the product symbol, are they raising this probability to the power pf $x_i$ or does that again just mean 'probability of class $c_k$ generating the term $x_i$'?

I also don't understand the intuition behind why or how this calculates $P(x|c_i)$.

Thanks for your help everyone.

Bernoulli naïve Bayes

$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$

Let's examine the example of document classification.
Let K different text classes and n different terms that our vocabulary contains. $x_i$  are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document x. x is a vector of dimension n.

$P(x|c_k)$ is the probability that given the class k, document x to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x_i = 1$, then $1 - x_i = 0$ and inversely. In other words, for each term it takes the probability that the document does contain this term or it does not.

$p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, that is it could be the prior probability a document that belongs to k class contains this term of the vocabulary.


How do I choose the search algorithm for a particular task? How do I choose the search algorithm for a particular task? Which criteria should I take into account?

The choice of the most appropriate search algorithm for a particular task is often based (but not exclusively) on its time complexity, space complexity, termination (if the algorithm always halts), optimality guarantees (if the algorithm is guaranteed to find the optimal solution), available implementations (as software libraries) and (if known) the actual performance (for such particular task). There are algorithms that have the same time or space complexities, but that, in practice, have different performance, which also depends on the problem being solved and implementations (which might or not be optimised).

For example, consider a search tree where you have a finite number of nodes and paths. Suppose that solutions lie at the leaves of this tree and that you are interested in one solution, but not necessarily the optimal one. In that case, in practice, DFS might find solutions faster than BFS, because of its nature and the task being solved. However, in the case you have a tree with infinite-length paths, then DFS might not terminate. In that case, proceeding layer by layer (BFS) might be a more appropriate strategy.


How machine learning can help with sustainable development and biological conservation? Recently White House published the article: Preparing for the Future of Artificial Intelligence which says that the government is working to leverage AI for the public good and toward a more effective government.
I'm especially interested in how AI can help with computational sustainability, environmental management and Earth's ecosystem, such as biological conservation?

There are a variety of aspects where AI can help for the public good. Future studies of computational methods can contribute to a sustainable management ecosystem by its data acquisition, interpretation, integration and model fitting.
Prof. Tom Dietterich is a leader in combining computer science and ecological sciences to build the new discipline of Ecosystem Informatics which studies methods for collecting, analyzing and visualizing data on the structure and function of ecosystems.
His group is involved in many aspects of the ecosystem, such as:

Models that can predict species distribution and their presence/absence elsewhere in order to create species distribution and migration/dispersal maps (such as DataONE Datanet, eBird project, BirdCast) .
Bio-economic models require solving large Spatio-temporal optimization problems under uncertainty.
Ecosystem prediction problems require integrating heterogeneous data sources.
Algorithms for deployment (sensor placement), cleaning and analysis of sensor network data of resulting data to increase agricultural productivity (Project TAHMO), like deployment of 20,000 hydro-meteorological stations in Africa (e.g. computational problem where to place it).
Systems for capturing, imaging, and sorting bugs combined with general image processing/machine learning/pattern recognition tools for counting and classifying them (BugID project). The goal is to develop algorithms for automating biodiversity based on visual pattern recognition by using the computer vision method.

For further information about this work, check the following resources, see:

(video) "Challenges for Machine Learning in Computational Sustainability" (CRCS) (2013)
(slides) Tom Dietterich, Understanding and Managing Ecosystems through AI (2013)
(study) Adapting environmental management to uncertain but inevitable change (2015)


How will we recognize a conscious machine? How will we recognize a conscious machine (or AI)? Is there any consciousness test? For example, if a machine is aware of its previous experiences, can it be considered conscious?

There are two main subjects you need to look at to understand the problem:

The Turing Test

  The Turing test, developed by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. wiki

See also: Turing Test (Stanford Philosophical Dictionary)

There is a linguistic element, which is that intelligence can be interpreted in many legitimate ways.  Some maintain that Artificial Intelligence has not yet been achieved; other feel a simple automated switch constitutes the most basic form of AI.

Bear in mind that "intelligence" is distinct from consciousness in the sense of "self awareness", but the generalized Turing Test can also be understood as a gauge of the appearance of consciousness.

This leads to Searle's Chinese Room Argument.  I highly recommend reading the Stanford Philosophy link, but the wiki gives a simpler synopsis:

  The Chinese room argument holds that a program cannot give a computer a "mind", "understanding" or "consciousness", regardless of how intelligently or human-like the program may make the computer behave.

The real problem may be, how does one know an algorithm is truly conscious and not merely simulating consciousness?

Philip K. Dick approaches from the opposite direction in Electric Sheep, where one of the conclusions is that "life is life" whether organic or artificial.  This might be said to lead to the "Duck Test" for consciousness: "If it looks like a duck, and quacks like a duck, then it's probably a duck."  (Dick's philosophy was heavily influenced by Christianity, and his view on artificial life may be though of as radically humanist.)

(A parallel philosophical argument might be "how is the appearance of free will distinct from the actuality of free will?"  Free will has been fruitlessly argued about for millennia, but whether the universe is actually deterministic or not, we perceive ourselves to have free will.  If the universe turns out not to be strictly deterministic, it wouldn't functionally change anything.  To make matters more fuzzy, true randomness is nature is only found at the quantum level and then only within certain models. Quantum uncertainty has been proposed as a basis for free will, but we don't know.  Possibly uncertainty is a condition of individuality, which must be subjective in relation to a system or other subjectivities.  Nevertheless, how do we know we are not "robots" acting out a pre-determined sequence of actions, which we perceive as decisions?)

I personally very much like the idea of recursion as function of self awareness.

  At the phenomenal level, consciousness can be described as a singular, unified field of recursive self-awareness, consistently coherent in a particular way; that of a subject located both spatially and temporally in an egocentrically-extended domain, such that conscious self-awareness is explicitly characterized by I-ness, now-ness and here-ness. SOURCE: Peters, Frederic Consciousness as Recursive, Spatiotemporal Self-Location

I'd also posit that being able to read and interpret your own code is a form of self-awareness.

This brings me to the idea that understanding (Chinese Room) is a red herring, and what we're really talking about is interpretation.

We are trapped in subjectivity, humans and automata, and perfect certainty is only achievable in a very limited set of circumstances, such as solved games. (There is also the idea of completeness of a system vs. consistency.)

When we talk about understanding in the sense of anything abstract, by which I means semantics as opposed to syntactics, meaning vs. form, one can say that understanding is a function of interpretation, regardless of the accuracy of the interpretation.

--------------------------------------

Fun Speculation:

I've learned never to underestimate the insight of artists, and the best speculative fiction authors are narrative philosophers in the tradition of Plato.  Dick related memory to identity, regardless of whether the memories are real or artificial.  This might be thought of as the narrative conception of the self--I am a product of my experience.  The real "me" is not merely my body, but "the story of me"--the subjective history that led to this moment of me-ness.

I think it is a not unreasonable assumption that artificial consciousness may arise out of understanding of narrative.  Words are just symbols, and there are all kinds of semantic issues, but actions have a concrete aspect.  Game theory studies actions in the form of choices, not only in the sense of equilibria, but also as a form of communication.   (See: iterated dilemma)  The choices agents make in iterated dilemmas constitute a narrative history than can be analyzed and understood mathematically, and these analyses are used for decision making.

It seems to me that the idea of consciousness can be related to decision making.  If you're just a transistor, that consciousness is quite limited, more akin to a cell than a complex organism. It may come down to whether one considers intelligence and consciousness to be spectrums as opposed to thresholds.

If you believe consciousness to be a spectrum, then limited artificial consciousness has already been achieved. Consciousness analogous to human-level self-awareness is still in the future.

See Also:
Definitions of the Self (wiki); Self Knowledge (wiki)


How do I prove that the MSE is zero when all predictions are equal to the corresponding labels? In the back-propogation algorithm, the error term is:
$$
E=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2,
$$
where $\hat{y}_k$ is a vector of outputs from the network, $y_k$ is the vector of correct labels (and we work out the error by calculating predicted-observed, squaring the answer, and then summing the answers for each $k$ (and dividing by 2).
How do you prove that if this answer is $0$ (i.e., if $E=0$), then $\hat{y}_k=y_k$ for all $k$?

This is very easy to prove.
Let's first prove that, if $\hat{y}_k = y_k$, then the $E = 0$. I will leave all steps, so that it's super clear.
\begin{align}
E
&=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 \\
&=\frac{1}{2}\sum_k(y_k - y_k)^2\\
&=\frac{1}{2}\sum_k(0)^2\\
&=\frac{1}{2}\sum_k 0\\
&=\frac{1}{2} 0\\
&=0\\
\end{align}
To prove the other way around, i.e. if $E = 0$, then $\hat{y}_k = y_k$, you can do as follows
\begin{align}
\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2
&=E\\
&=0
\end{align}
Recall now that any number squared is non-negative (i.e. positive or zero). Given that $(\hat{y}_k - y_k)^2 $ is non-negative, then $\sum_k(\hat{y}_k - y_k)^2$ is a sum of non-negative numbers. The only way that a sum of non-negative numbers is equal to zero is if all numbers are zero, so we must have $\hat{y}_k = y_k$ (because any non-zero number squared is non-zero).
(Note that $E$ is the mean squared error, i.e. a loss function, and it's not the back-propagation algorithm, which is just the algorithm that you use to compute partial derivatives of $E$ with respect to the parameters of the model, which are not even visible in the way you wrote $E$).


What’s the difference between LSTM and RNN? What's the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?

You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more recurrent (or cyclic) connections. Or you could say that layer $l$ of neural network $N$ is a recurrent layer, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).
In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name). On the other hand, an LSTM can refer to an LSTM unit (or neuron), an LSTM layer (many LSTM units), or an LSTM neural network (an NN with LSTM units or layers), depending on the context.
An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so an LSTM network is a recurrent network. The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is more sophisticated. More precisely, it is composed of the so-called gates that supposedly regulate better the flow of information through the unit.
Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called peephole connection).

This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices). You can easily see from this diagram that an LSTM unit (or layer) is composed of gates and recurrent connections. It's also composed of a cell. To understand the details (i.e. the purpose of all these components, such as the gates), you should e.g. read the paper that originally proposed the LSTM by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers.
Given the presence of cyclic connections, any recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard RNN or an LSTM network (or maybe a variant of it, e.g. the GRU).

RNNs are particularly suited for tasks that involve sequences (because of the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it is more computationally effective. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the vanishing gradient problem.


Is it possible to use an internal layer's outputs in a loss function? For a network of the form:
Input(10)
Dense(200)
Dense(100+10)
Dense(20)
Output()

Those +10 outputs are what I want to add to the standard 20 outputs, for my loss function.
Is this possible - in theory or even with some pre-existing library?

Yes, you can do that, and it is a standard practice. One famous example is the "Inception" network architecture. To keep inner subnets from "dying out",  several outputs from inner layers are extracted and passed into FC->Softmax. Then all the outputs are averaged in the loss function.
From practical point of view, you won't be able to implement such things with basic Sequential model construction. Most libraries allow you to move beyond it, though, and this distinction is usually well-documented. For example in the tensorflow guide on functional API it states right in the introduction:

The functional API is a way to create models that are more flexible than the tf.keras.Sequential API. The functional API can handle models with non-linear topology, shared layers, and even multiple inputs or outputs.

Hope that'll help you to get started.


Why do language models place less importance on punctuation? I have very outdated idea about how NLP tasks are carried out by normal RNN's, LSTM's/GRU's, word2vec, etc to basically generate some hidden form of the sentence understood by the machine.

One of the things I have noticed is that in general researchers are interested in generating the context of the sentence, but oftentimes ignore punctuation marks which is on of the most important aspects for generating context. For example:

  “Most of the time, travellers worry about their luggage.”

  “Most of the time travellers worry about their luggage”

Source

Like this there exists probably 4 important punctuation marks .,? and !. Yet, I have not seen any significant tutorials/blogs on them. It is also interesting to note that punctuations don't have a meaning (quite important, since most language models try to map word to a numerical value/meaning), they are more of a 'delimiter'. So what is the current theory or perspective on this? And why is it ignored?

Language models almost always map every word to an embedding. There are many embedding algorithms with most of them having interpolation properties i.e. If $E(word)$ represents embedding of a word then $E(king)-E(male)+E(female) \sim E(queen)$. The smoother the interpolation properties the better the model understands the word, these properties don't exactly make much sense when it comes to delimiters.

Yet, there are instances where a delimiter embedding is learned ( always has an embedding ). While using these first all punctuation in the text is converted to one specific word, say 'dlmt', the embedding algorithm learns an embedding for this word treating it as if it would have any word. This maintains interpolation properties where the delimiter is understood to be a word that is used to split context.

I have observed that delimiters such as question mark or exclamation marks at the end of sentence are also understood to be breaks in context, in these cases the model learns if the statement is a question or so just by the context given by the words and stops in the sentence


Which courses in computer science and logic are relevant to Machine Learning? Although I have a decent background in math, I'm trying to understand which courses from CS and logic to look into. My aim is to get into a Machine Learning PhD program.

I worked as a professor for a time, and often advised students on this. For a PhD in machine learning, I think the ideal background is:

Core CS Courses

Programming (typically 3-4 courses). Language choice is not highly important, but Python, C++, Java, and perhaps JavaScript, are reasonable picks, if only because of their prevalence.
Core topics: data structures, algorithms, operating systems, databases
numerical linear algebra or numerical methods
advanced algorithm design and analysis

Together these will allow you to read and write the code that even highly optimized versions of ML algorithms are written in, and to understand what might be going wrong within them.

2. AI & ML courses, usually offered through a CS department

a broad survey course in AI (like AI:AMA by Russel & Norvig), usually offered to senior undergraduates.
A course applied machine learning or data mining.

You may also take other AI courses, but they are not as common to see offered to undergraduates, so many students wait until graduate school:

reinforcement learning
soft computing
computational learning theory
Bayesian Methods
Deep Learning
Multiagent Systems
Information Retrieval
Natural Language Processing
Computer Vision
Robotics

Together, these will give you the broadest possible background in AI & ML. These can allow you to find new applications of ML, or to pull AI techniques from one area into another as you need.

3. Statistics courses

a 1 or 2 term course in probability theory, ideally a version that requires and uses calculus.
at minimum a course in statistical hypothesis testing.

Much stronger would be to also take courses in:

regression
generalized linear models
experiment design
causal inference
Bayesian methods

These courses allow you to reason formally and comfortably about uncertainty. They also give you the correct framework for answering questions about whether your ML algorithm is working, and what patterns an ML algorithm uncovers mean.

4. Mathematics courses

3 semesters of calculus, going at least as far as multi-variate/vector calculus.
optionally, a more advanced course that builds on calculus, like real analysis, but only to reinforce calculus concepts.
at least 1, and preferably 2, courses in linear algebra
at least 1, and preferably 2, courses in discrete mathematics.
ideally, something like Knuth et al.'s Concrete Mathematics
ideally a course in advanced optimization techniques
optionally, courses in logic, but be aware that this is almost a fringe area in AI now, and essentially irrelevant to a PhD in machine learning. The parts you need are usually covered in a broad survey AI course.

These courses give you the basic mathematical fluency to understand most machine learning algorithms well.


How is $v_*(s) = \max_{\pi} v_\pi(s)$ also applicable in the case of stochastic policies? I am reading Sutton & Bartos's Book "Introduction to reinforcement learning". In this book, the defined the optimal value function as:
$$v_*(s) = \max_{\pi} v_\pi(s),$$ for all $s \in \mathcal{S}$.
Do we take the max over all deterministic policies, or do we also look at stochastic policies (is there an example where a stochastic policy always performs better than a deterministic one?)
My intuition is that the value function of a stochastic policy is more or less a linear combination of the deterministic policies it tries to model, however, there are some self-references, so it is not mathematically true).
If we do look over all stochastic policies, shouldn't we take the supremum? Or do we know, that the supremum is achieved, and therefore it is truly a maximum?

The value function is defined as $v_\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ where $G_t$ are the (discounted) returns from time step $t$. The expectation is taken with respect to the policy $\pi$ and the transition dynamics of the MDP.
Now, as you pointed out the optimal value function is defined as $v_*(s) = \max_\pi v_\pi(s)\; ; \;\forall s \in \mathcal{S}$. All we are doing here is choosing a policy $\pi$ that maximises the value function; this can be a deterministic or a stochastic policy, though intuitively it is likely to be deterministic unless for some states that are two (or more) actions with the same expected value, in which case you can take any of said actions with equal probability, thus making the policy stochastic.
For a finite MDP (which is what I assumed above too), we know that an optimal value function exists (this is mentioned in the book) so taking the maximum is fine here.


Why does self-playing TicTacToe not become perfect? I trained a DQN that learns TicTacToe by playing against itself with a reward of -1/0/+1 for a loss/draw/win. Every 500 episodes I test the progress by letting it play some episodes (also 500) against a random player.

As shown in the picture the net learns quickly to get an average reward of 0.8-0.9 against the random player. But after 6000 episodes performance seems to deteriorate. If I play manually against the net after 10000 episodes it plays okay, but by no means perfect.

Assuming that there is no hidden programming bug, is there anything that might explain such a behavior? Is there anything special about self-play in contrast to training a net against a fixed environment ?

Here further details: The net has two layers with 100 and 50 nodes (and a linear output layer with 9 nodes), uses DQN and a replay buffer with 4000 state transitions. The shown epsilon values are only used during self-play, during evaluation against the random player exploration is switched off. Self-play actually works by training two separate nets of identical architecture. For simplicity one net is always player1 and the other always player2 (so they learn slightly different things). Evaluation is then done using the player1 net vs. a random player which generates moves for player2.

Thanx

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have given in the question and comments, I can make a few observations and guesses about where to look:

Your metric of average reward against a random player is sensible. In this case, you could also use a perfect player (that ideally randomised choice of any optimal move), where you would see a maximum averaged return of zero - this would be helpful to know if your agent had learned a fully optimal behaviour, because it would consistently score zero. In general for more complex games a perfect player is not available to test with, but as you are learning here it might help you.
Your DQNs might be unable to fit the value function. You can test that in this case by getting the value function from an optimal self-play player (all the values will be -1, 0, or 1) and using a supervised learning approach, separately from your agent. You should be able to get a loss very close to zero - if you cannot do that, then something could be wrong with your network architecture.
Whilst you are training, even though you are using a variation of Q-learning (which learns an optimal policy even whilst exploring other actions), your DQNs are not learning optimal play. That is because you have used two agents. In DQN, the algorithm is not aware that there are other learning agents, and it will treat any other agents as if they were part of the environment. Which means that the agents will spend some effort trying to set the game up for each other to make an exploration mistake. That could lead to non-optimal choices and a little bit of instability. Your decay of epsilon should help with that, although you are caught between a rock and a hard place here. You want to learn off-policy and explore, but are forced to reduce exploration. There are a couple of ways to resolve that, I will explain a bit further down . . .
10,000 games may not be enough. In the experiments I have done with TicTacToe agents, it seems between 20,000 and 50,000 games are required for a naive learner. More may be required if you have done something that makes learning inefficient. In addition, I found when adding more sophisticated learning approaches (in my case using eligibility traces) the agents appear to become close to optimal very quickly, but actually have flaws which take a long time to shake out, just as long as running a more naive algorithm. When the flaws got found and fixed, it upset the value function for a while and I saw fluctuations in my metrics similar to yours.
Q-learning with NNs is inherently unstable. DQN implements some ideas to fix that, but it is not perfect. It is not uncommon to need to adjust the batch size and/or time steps between taking frozen copy of network for the TD target calculation. The initial stability followed by poor performance looks a lot like that instability too.

Regarding your use of two opposing agents, I can see two possible improvements:

Alternately train one or other agent in each game, don't train both at once. That will mean each agent is learning to play against the other agent playing its best without exploratory moves.
Combine networks into single agent description. As this is a zero-sum game, you can take player A's network for calculating values, and just have player B try to minimise the action value on its turn. That means use min and argmin functions for steps that represent player B's turn wherever player A would use min or argmin, including in the Q-value updates - this is typically easy to add to the inner loop of Q-learning, and should improve learning efficiency (essentially you are hard-coding knowledge that this is a zero-sum game and taking advantage of that symmetry).

Both of these ideas will free you up from caring about the value of epsilon, or decaying it - you can probably just leave it fixed at e.g. 0.1

Finally, as a test of whether your agent can cope with learning optimal play in general, you could have it learn against an already optimal agent. That is obviously not something you can do for more complex games, but might help you debug agent code and hyper-parameters of the network - it divides your problem up into "can it learn this at all" and "can it learn through self-play".