What are multi-hop relational paths? What are multi-hop relational paths in the context of knowledge graphs (KGs)?

I tried looking it up online, but didn't find a simple explanation.

Before trying to explain this term in your context, let me briefly describe the term in other contexts.

In computer networking, the term "hop" refers to a node (e.g. a router) that a packet goes through before reaching its destination from its source. In a multi-hop situation, you have several nodes involved in the process of sending the packet from the source to the destination.

A knowledge graph is a graph that accumulates and conveys knowledge of the real world, where nodes represent entities of interest and edges relations between those entities.

So, multi-hop relational paths are probably relational paths involving more than one node or edge in the knowledge graph.

But what do we mean by "relational"?

If you are familiar with the basics of databases, the word "relational" shouldn't be so unfamiliar. In fact, there the so-called relational databases, relational models and relational algebra. Intuitively, the word "relational" is used to denote what you think it denotes, i.e. relations. See also What a relational database is by Oracle.

And what is a path?

In section 2.2.3 of the tutorial Knowledge Graphs, Aidan Hogan et al. provide a description of a path (expressions) in the context of knowledge graphs

  Navigational graph patterns. A key feature that distinguishes graph query languages is the ability to include path expressions in queries. A path expression $r$ is a regular expression that allows matching arbitrary-length paths between two nodes, which is expressed as a regular path query $(x,r,y)$, where $x$ and $y$ can be variables or constants (or even the same term).


Are there any projects that can measure the concept similarity between texts? Are there any algorithms or software libraries that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.?
For example, these phrases:

Outside, it is warm.
Outside, it is hot.
Outside, it is not cold.
It is not cold outside.

Should be similar to this phrase:

It is warm outside.

Ideally, the algorithm or software would be capable of generating a score from 0 to 1, based on the concept similarity. The goal is to use this algorithm or software to map a large number of statements to a single, similar original statement. It is for this mapping of a given statement to the original statement that the aforementioned similarity score would be generated.
Does such an algorithm of software already exist?

Doc2Vec

Doc2Vec comes to mind, here's the original publication. The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet.

There are a number of implementations of Doc2Vec. If you're using Python, one to look at is gensim.

Word2Vec

Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the original publication, and another publication that does a nice job explaining it further.

Tensorflow has a tutorial for setting up a Word2Vec model. Gensim also has a Word2Vec implementation.


How can I model this problem of delivering assets by choosing a route with reinforcement learning? I would like to build a model based on reinforcement learning (RL) for the following scenario

Recommend the best route (of cities listed for a given country) that satisfies the required criteria (museum, beaches, food, etc) for a total budget of $2000.

Based on the recommendation, the user will provide its feedback (as a reward), so the recommendations can be fine-tuned (by reinforcement learning) the next time. I modeled the system this way:

States = (c,cr), where $c$ is the city and $cr$ is the criteria (history, beach, food, etc)

Actions = (p) is the price of visiting the city

Reward: acceptance of the cities selected by end user as a route (1 or 0)

The objective is to decide which list of cities together satisfy the
given budget.
Is this MDP model right and how can I implement this? May be the only option is using Monte Carlo methods and linear/dynamic programming.. Is there any other way?

I do not see how you came to choose prices as actions. Normally, actions are something like go left, go right, jump, stay etc.
Analogously, I would say that in your case the actions are visiting a certain location, whereas locations are what you referred to as states. I'd go for something like that:
locations = {location1=(c1,cr1), location2=(c1,cr2), ...}
Actions = {Visit location1, Visit location2, ...}
States = {--set of all the possible paths the model can generate until the budget is possibly exhausted--}

The reward function could then be a combination of both the acceptance (vs. rejection) of a route/path by the user and the inverse of the cost associated with the suggested path (because you want the model to favor cheap paths in order to keep your own business costs low).
How you balance these two terms is up to exploration.
For rapid prototyping, check out stable baselines, which offers a bunch of highly optimized RL algorithms.


How to manage high numbers of input layer data points Not sure if this is the correct forum, but I have been working with a large (non-image) dataset that will eventually be used to train a neural network.  I have been puzzling over how to manage wide data sets.  For this application "wide" is maybe 10,000 or 20,000 points wide.  It is not really possible to store this as a row in a conventional RDMS (which are usually limited to several hundred columns).  Is it better to use a huge CSV file or maybe a no-sql technology like Cassandra (the data is originally in JSON format)?

Training with images makes use of pictures that are very large, like 7360×4912 (36 mega pixels). For image recognition at any angle and size, an image is rotated (about 60 times) and resized (depends on the original size, possibly 20 times) many times. This means you end up with a ton of data. In my example here, you would get some 60 × 36 × 2 = 4Gb of data for that one image. Although you really only save the data of interest (noise in the light signal) so in the end the AI does not use as much, but it would still be a quite large data set... especially when you know that a well trained image recognition system uses a good 2,000 images.

So 10 to 20 thousands data points isn't much at all. I guess it depends how many of those you have (one per second would be 86,400 a day...)

I would consider using a system like Cassandra because the database can grow as big as you'd like, if that is one of the problem you're facing. Also you can save the data in binary and the number of columns can vary on each row. However, Cassandra expects a key for each row. This is an important point as it makes use of that key to distribute the data over multiple computers (for faster access later and distribution of the data set over all computers.) It also includes replication "for free" so if a computer fails, you can just replace it (no need to restore it).

What is not a very good idea with Cassandra is creating many indexes. Those use rows (i.e. one index = one row) and thus the few computers handling that one row will work extra-hard on queries on that index... But I would imagine that for AI you would not really need such.

A really large CSV will be difficult to work with because you'll be responsible for accessing the rows (unless you have to access all the rows each time anyway?) Also the file format is Text and the file size will be limited by your OS (which should be really large, but be careful as many I/O calls in higher level libraries miss the fact that the size could be 64 bits.)

If you are looking at a longer/more permanent solution, I would definitely look into Cassandra, especially if you're using Java as a language (also I use it with C++ and it works just fine with C++ too.) If you're just writing a throw away app. anyway, then CSV is probably much easier than handling a whole Cassandra cluster! That's a learning curve...


Can I use augmented data in the validation set? I am trying to predict nursing activity using mobile accelerometer data. My dataset is a CSV file containing x, y, z component of acceleration. Each frame contains 20-second data. The dataset is highly imbalance, so I perform data augmentation and balance the data. In the data augmentation technique, I only use scaling and my assumption is, if I scale down or up a signal the activity remains the same. Using this assumption I augmented the data and my validation set not only contain the original signals but also the augmented (scaling) signals. Using this process, I am getting quite a good accuracy that I never being expected using only data augmentation. So, I am thinking that I performed a terrible mistake somewhere. I check the code, everything is right. So now I think, since my validation set has augmented data, that's the reason of getting this high accuracy (maybe the augmented data is really easy to classify).

You should not use augmented data in the validation nor in the test set.
Validation and test set are purely used for hyperparameter tuning and estimating the final performance, i.e. estimating the generalization error. These two data sets should be as close as possible to other data, which you could have acquired, but you actually haven not, i.e. your true data distribution.
It's fine to augment training data, since it mimics other samples from the true distribution by applying transformations, noise, etc. and thus helps to increase generalization performance (assuming your augmentation assumptions are right). But evaluation should always be performed on original data.


How can AI techniques be used in software testing? How can Artificial Intelligence be applied to software testing?

In large software with many actions and possible flows like web applications, enterprise software, etc, it is really hard and time taking to test out all possible scenarios via traditional approach. So, building a machine learning model is an interesting approach to solving this. A reinforcement learning system with an end goal to crash or make the software unresponsive can be tried.

There is research being done on this idea. You can take read this research paper which explores Reinforcement Learning as an approach to automated GUI robustness testing. Also, some companies like Appdiff trying to incorporate AI in software testing with mobile apps in context, but similar thinking can be reasonably extended to web apps.


How can I create an artificially intelligent aimbot for a game like CS:GO? How can I create an artificially intelligent aimbot for a game like Counter-Strike Global Offensive (CS:GO)?

I have an initial solution (or approach) in mind. We can train an image recognition model that will recognize the head of the enemy (in the visible area of the player, so excluding the invisible area behind the player, to avoid being easily detected by VAC) and move the cursor to the position of the enemy's head and fire.

It would be much more preferable to train the recognition model in real-time than using demos. Most of the available demos you might have might be 32 tick, but while playing the game, it works at 64 tick.

It is a very fresh idea in my mind, so I didn't actually think a lot about it. Ignoring facts like detection by VAC for a few moments.

Is there any research work on the topic? What are the common machine learning approaches to tackle such a problem?

Later on, this idea can be expanded to a completely autonomous bot that can play the game by itself, but that is a bit too much initially.

Thare are many ways to approache this. One approach to start would be to try to describe the problem using a formalism closer to reinforcement learning.

Output:

Aiming in any shooter type game, as I recall, involves moving the mouse. So the output of your aimbot has two dimentions. Depending on the required accuracy, you can consider these two dimentions continous, with a limited range, or if you consider each pixel as a integer, you might be able to discretize your action space. (I assume mouse XY coordinates should be input to the game, not increments)

Input:

You  definetly need screen information. You can take the whole screen as input to a CNN, similarly to DeepQLearning for Atari.

Reward function

This might be tricky since your rewards need to be as dense as possible, however, the only feedback you get from the game is that someone was shot.
It might be ebough but this will definetly increase your training time.

Training data / Environemnt:

Your environment for training is the game itseld. Using a curriculum learning approach would probably make the training process more efficient.

You can also try an imitation learning approach, since I assume you are happy to provide expert training examples (in this case probably headshots in the game environment).

You can read more about how to apply reinforcement learning for games here. The Unity ML-Agents Library also includes sample tracking problems and their solutions.


Is the Cognitive Approach (SOAR) equivalent to the Chinese Room argument? Soar is a cognitive architecture.

https://en.wikipedia.org/wiki/Soar_(cognitive_architecture)
https://soar.eecs.umich.edu/

There is something called "the Chinese box" or "Chinese room" argument:

https://en.wikipedia.org/wiki/Chinese_room
https://plato.stanford.edu/entries/chinese-room/

The "Chinese room" seems to be begging its question, but that is not what I am asking.  I am asking if there is any literal difference between a tool like "SOAR" and the formalism of the "Chinese box".  Is SOAR identical or equivalent to a "Chinese Box"?

Searle's Chinese room is analogical and is intended to present an easy-to-understand picture of the essential elements and processes of the digital computer. In the room the man (CPU) has a book of intructions (program) for responding to Chinese input questions. That is just one program of many possible programs the room could run. Each different program would be a different instruction book. SOAR would be just one of those books.


Unable to understand V* at infinite time horizon using Bellman equation for solving MDP I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct.

My reasoning was

$V^*(a)$ = 10 fixed, because the optimal action is to terminate and receive the reward 10.

$V^*(b) = 10 \times 0.2 = 2$ using Bellman optimality eqn $V^*(s) = R(s)+ \gamma \ \rm{max}_{a} \sum_{s'} P(s'|s,a) V^*(s')$, where the optimal actionfrom b is to left.

Similarly, I get $V^*(c) = 10 \times (0.2)^2 = 0.4$

For the state $d$, it is optimal to move to the right and exit at $e$ to receive 1, therefore $V^*(d) = 1 \times 0.2 = 0.2$.

And $V^*(e) = 1$ fixed.

However, there autograder says it's incorrect and doesn't show explanation. Can anyone explain what the right approach or answer is?

Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).


How do I model the blocked N queens problem as a search problem? The blocked N-queens is a variant of the N-queens problem. In the blocked N-queens problem, we also have a NxN chess board and N queens. Each square can hold at most one queen. Some squares on the board are blocked and can not hold any queens. Conditionality is that queens do not dare to attack each other. At the entrance to this problem are the queues and blocked areas.

How do I model the blocked N queens problem as a search problem, so that I can apply search algorithms like BFS?

In general, the process of modelling a problem as a search problem consists in creating a graph which contains nodes, which represent the possible states in your problem, and edges, which represent the relations between these states (that is, you will have an edge between nodes $A$ and $B$ if it is possible to go from state $A$ to state $B$, and vice-versa, in your problem).

In the case of the 8 (or, in general, N) queens problem (or blocked 8 queens problem), the states are all possible configurations of the board (that is, all possible combinations of the positions of all the 8 queens). In the graph that represents this problem, you will have a node for all possible valid configurations, where a valid configuration is one that is allowed by the rules of the game. You will also have an edge between configurations (or states) $A$ and $B$ if you can go from state $A$ to state $B$. For example, configurations where you have two queens on the same square or where you have a queen in a blocked area will not be represented in your graph as a node (or state).

In the case of the 8 queens problem, you actually have more than one solution, so you will have more than one goal state (or configuration of the board).

When using the BFS, the goal will then be to find a path from the initial configuration of the queens (that is, the initial state) to one of these 18 (or 12) goal states.

In the case of the 8 queens problem, there are 4,426,165,368 possible configurations (so you will likely not be able to draw this graph).


Are training sequences for LMs sampled in an IID fashion? If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 chunks of 10 tokens each (token 1-11, 2-12, and so on).
However these training sequences are not iid, right? If so, are there any papers that try and deal with this?

Language models explicitly assume that word sequences are not independent and identically distributed (iid). A word-based model that assumed iid within each sequence could only predict probabilities of words according to some other context than surrounding words, which would not be very useful for a language model.
The training processes for statistical models often require iid input/output pairs when sampling e.g. minibatches. Neural networks definitely train better when datasets are shuffled to remove any chance of correlation between examples as they are used to update parameters.
How to resolve these two different needs? When training a sequence-based model on many sequences, it is the distribution of sequences that needs to be iid and representative of the overall population. The distribution of items within the sequence is what is being learned, so should not be obscured or removed.
As an analogy, you do not usually want to shuffle the rows of pixels in an image when training image classifiers. The spatial pattern in the image needs to be preserved in the same way that the sequential pattern in a sentence needs to be preserved, because the pattern is part of the data being modelled. With image data, you accept non-iid relationships between the pixels that are next to each other within a single image, then apply shuffling and stratifying algorithms at the level of individual images.

If so, are there any papers that try and deal with this?

There may be some original papers from 50 years ago which compare iid with non-iid data when training sequences fed into RNNs, but it has been a standard part of engineering practice for decades now to shuffle datasets, and the separate sequences used to train RNNs are no different.
From your comments:

However the sequences that are sampled for training are not iid if they are multiple sequences generated sequentially from the same document, which from what I understand happens often?

You are correct, the raw set of sequences is not iid if they are collected in that fashion. However, the dataset is always shuffled or resampled for training purposes, it is not fed into training routines in that raw state. The shuffling (of selected sequences which are kept intact internally) happens in-between the raw data collection and the training.

There are some simple statistical models that do not require iid data to train on. This occurs for example in tabular reinforcement learning, which can learn online from a single continuous sequence of states, actions and rewards. An equivalent language model would be word- or letter-based ngrams.


RL: how can the cart-pole problem be a continuing task? In Introduction to Reinforcement Learning (2ed), Sutton and Barto, there is an example of Pole-Balancing problem (Example 3.4).

In this example, it said, this problem can be treated with 'episodic task' and 'continuing task'.

I think that it can only be treated as episodic task because it has an end of playing, which is falling the rod.

I have no idea how this can be treated as continuing task.... Even in OpenAI Gym cartpole env, there is an only episodic mode.

The key is that reinforcement learning through something like, say, SARSA, works by splitting up the state space into discrete points, and then trying to learn the best action at every point.

To do this, it tries to pick actions that maximize the reward signal, possibly subject to some kind of exploration policy like epsilon-greedy.

In cart-pole, two common reward signals are:

Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
Receive a reward that linearly increases with the distance the pole is off the ground.

In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.

However, an offline algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An online algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.


Why is a Nvidia Single Board computer less than ideal for AI model training? Conventional NVIDIA GPUs, such as the Titan and or GT1080, are used to train AI models.  Why would a Jetson Nano board be less than ideal as a substitute for a conventional GPU?

CONTEXT

I would like to run simple concept demonstrator training exercises with and without a GPU: that being said, I am not interested in buying a desktop and outfitting it with a $1000+ GPU for said exercise.

Jetson Nano specs Price:$99:

  128-core Maxwell GPU

  Quad-core ARM A57 @ 1.43 GHz

  472 GFLOPS max
   compute

Titan V(for example) Price:$2500 :

  640 tensor cores

  5120 cuda cores

  110000 GFLOPS max compute

Some simple maths gets us to a speed of 11,800 GFLOPS for 25 of these perfectly synced in parallel(not likely or gonna be fun). Vs the 110000 it just isn't even a question.


Do genetic algorithms also evolve? After witnessing the rise of deep learning as automatic feature/pattern recognition over classic machine learning techniques, I had an insight that the more you automate at each level, the better the results, and I, therefore, turned my focus to neuroevolution.
I have been reading neuroevolution publications with the same desire to automate at every level.
Do genetic algorithms also evolve themselves? Do they get better at searching through the solution space for each generation over time?

In principle, yes, you can also evolve the genetic algorithm (or, in general, evolutionary algorithm), i.e. you can evolve its operations (such as the mutation and cross-over) and hyper-parameters (such as the size of the population or mutation rate). For example, you could use genetic programming to evolve the cross-over operation of a genetic algorithm. However, these genetic operators and hyper-parameters are usually designed and determined by a human and do not change during the evolution process. Nevertheless, there is a framework for evolutionary computation in which reproduction and mutation can also evolve, known as auto-constructive evolution, and there are other examples in the literature of meta-optimization methods applied to evolutionary algorithms, such as

Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook (1987, Schmidhuber's thesis)

Parallel optimization of evolutionary algorithms (1994, by T. Bäck)

Adaptive and Self-adaptive Evolutionary Computations (1995, by Peter Angeline)

Meta-Genetic Programming: Co-evolving the Operators of Variation (1998, by B. Edmonds)

More generally, the optimization of an optimization algorithm is known as meta-optimization and/or hyper-parameter optimization/tuning, so the idea of meta-optimization is not just restricted to evolutionary algorithms, but can be applied also to e.g. deep learning.


A way to leverage machine learning to reduce DFS/BFS search time on a tree graph? I'm not very knowledgeable in this field but I'm wondering if any research or information already exists for the following situation:
I have some data that may or may not look similar to each other. Each represents a node that represents a vector of size 128.

And they are inserted into a tree graph according to similarity.

and a new node is created with an edge connecting the most similar vertex node found in the entire tree graph.
Except I'm wasting a lot of time searching through the entire graph to insert each new node when I could narrow down my search according to previous information. Imagine a trunk node saying "Oh, I saw a node like you before, it went down this branch. And don't bother going down that other branch." I could reduce the cost of searching the entire tree structure if there was a clever way to remember if a similar node went down a certain path.
I've thought about some ways to use caching or creating a look-up table but these are very memory intensive methods and will become slower the longer the program runs on. I have some other ideas I am playing around with but I was hoping someone could point me in the right direction before I started trying out weird ideas.
Edit: added a better (more realistic) graph picture

It sounds like you may be looking for the A* search algorithm. It is a search algorithm, like DFS and BFS, but it will explore only the most promising branches based on a heuristic function you supply. The difficult part of implementing this is deciding on a low-cost, admissible heuristic.
Excited to reflect nbro's suggestion from comments.


What is the definition of each of these neural network cell types? I found this nice-ish-looking diagram, but it has a wholly inadequate descriptions for each of the cell types, aside from including names.

What is the definition/description of each of these cell types?

Input Cell
Backfed Input Cell
Noisy Input Cell
Hidden Cell
Probablistic Hidden Cell
Spiking Hidden Cell
Output Cell
Match Input Output Cell
Recurrent Cell
Memory Cell
Different Memory Cell
Kernel
Convolution or Pool

The diagram is nice-ish-looking, and it does lack a convenient list of node type descriptions.  There is some description in their Zoo Flashcards page. Although the zoo is not a clean presentation in every respect, it is quite instructive in several ways. These are of particular interest.

The symbolic shapes and colors visually display the relationship between cell types and the types of networks that employ them, which is quite instructive as an overview.
The use of the term cell, as opposed to neuron, acknowledges the fundamental difference between current artificial network cells and the amazingly intricate and self-modifying nature of biological neurons.

What is still missing is a concise list of each cell type and its purpose for existing in the contexts of its use.  This may be useful.

Input Cell — Provides a point of input and often uses an identity activation function (the output = the input)
Back-fed Input Cell — Provides a point where feedback is deliberately introduced
Noisy Input Cell — Provides a point where chaotic (pseudo-random), thermal, or quantum noise can be introduced
Hidden Cell  — A cell that attenuates (applies a parameterized weight to) the input from all or a select range of activation function outputs from the cells of the immediately prior layer
Probabilistic Hidden Cell — Applies a radial basis function to the difference between the test case and the cell's mean1
Spiking Hidden Cell — Employs a spiking model of activation rather than a perceptron model, working toward a more accurate simulation of biological neurons at the cell level
Output Cell — Provides a point of output and applies the range (min and max) or ordinality (binary, category index)
Match Input Output Cell — Used in auto-encoders to converge on the identity function from network input to network output, usually through a narrowed data path through which features are extracted after convergence
Recurrent Cell — Used in RNN designs such as recurrent artificial networks as a method of applying historic signal state to improve convergence where the time domain is indicated as an important dimension of learning 2
Memory Cell — Used in LSTM (long short term memory) designs as a method of applying historic signal state to improve convergence where the time domain is indicated as an important dimension of learning 2
Different Memory Cell — Used in GRU (gated recurrent unit) designs as a method of applying historic signal state to improve convergence where the time domain is indicated as an important dimension of learning 2
Convolution Cell — Passes the result of a convolution kernel multiplied to a relative range of the cells of the immediately previous layer to the next layer
Pool Cell — Aggregates (usually arithmatically) the results of the cells of a convolution layer

A table of cell types with the features for each might be a next step in presenting the cell types in a comparative summary.  The table would particularly helpful if it had sort up and down gadgets in the header row for each column.

Footnotes

[1] Typically $\varphi(\mathbf{x}) = \sum_{i=1}^I a_i e^{\left[ -\beta \left \Vert \mathbf{x} - \mathbf{c}_i  \right \Vert ^2 \right]}$ where c is the center value of the cell

[2] These cell types integrate historical influence into the signal path at the cellular level, introducing statefulness beyond the learning parameters and network hyper-parameters, providing the potential of Turing completeness to the network after training.


How to deal with very, very small time-series? I have an ensemble of 231 time series, the largest among them being 14 lines long. The task at hand is to try to predict these time-series. But I'm finding this difficult due to the very small size of the data. Any suggestions about what algorithm to use? I'm thinking about going for a hidden markov model, but I don't know if that's a wise choice.

Of course depends on your type of data, but Holt-Winter models can have different degree of complexity and use moving average, trend, and seasonality. This is most useful if the data is not hierarchical, meaning that the time-series are independent from each other.
If time-series are relatives of each other then you can also try aggregating them, predict at aggregate level and then disaggregate. The following can be a good resource:
https://otexts.com/fpp2/


Metrics for evaluating models that output probabilities I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to one of two classes. What metrics can/should be used to evaluate these types of models?

I'm currently using mean squared error, but I would like to know if there are other metrics, and what the advantages/disadvantages of those metrics are.

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even though the latter value is closer and would score better under MSE). It also applies that you care that 0.9 is by the same logic "closer" to 0.8 than it is to 0.97. With cross-entropy loss, extreme confidence (predicting close to $0$ or close to $1$) is penalised more heavily when it is wrong.

For completeness, the loss function (per data point) is:

$$\mathcal{L}(\hat{y},y) = -(y\text{log}(\hat{y})+ (1-y)\text{log}(1-\hat{y}))$$

This is likely to be the same loss function as you are using for your objective (or at least it should be), so for tests, simply also use it as your metric*.

$\hat{y}$ is your predicted probability of being in class A, and can be in range $[0,1]$. Ideally you have ground truth probabilities and $y$ is also in that range. In which case the only problem is that the "perfect" score is no longer $0$ but some positive number. If that bothers you, then you could offset by the perfect score, pre-calculating it on each data set (just set $\hat{y} = y$ for each item and you will find the minimum possible score).

If you don't have ground truth probabilities, but you do have classes, then $y$ will either be 0 or 1, and the metric still works. To get an accurate metric, you will need enough samples that the relative frequencies of each class depending on input has a significant effect. That is, you need more data, both training and test, in order to train for accurate probabilities instead of targeting simpler classification accuracy metrics.

Similar logic also works for multi-class probabilities. However, many off-the-shelf libraries use an optimisation in the loss function - assuming only one true class - which makes using probabilities as ground truth impossible. You might therefore need to write your own loss function and gradient functions based on multi-class cross-entropy loss in that case.

* I am making the assumption here that you use standard conventions for noting loss functions (typically per item), cost functions (typically aggregated across a data set and possibly multiple loss functions) and metric functions which don't have to be differentiable or usable as either of the former. The cost function is usually also fed into optimisers as the objective function - i.e. it has the important job of driving parameters to reach a maximum or minimum value. For gradient based solvers, that means it must be differentiable.


One vs multiple output neurons Consider an MLP that outputs an integer 'rating' of 0 to 4. Would it be correct to say this could be modeled in either of the following ways:

map each rating in the dataset to a 'normalized set' between 0 and 1 (i.e. 0, 0.25, 0.5, 0.75, 1), have a single neuron with sigmoid activation at output provide a single decimal value and then take as the rating whatever is closest to that value in the 'normalized set'
have 5 output neurons with a softmax activation function output 5 values, each representing a probability of one of the 5 ratings as the outcome, and then take as the rating whichever neuron gives the highest probability?

If this is indeed the case, how does one typically decide 'which way to go'? Approach 1 certainly appears to yield a simpler model. What are the considerations, pros/cons of each approach? Perhaps a couple of concrete examples to illustrate?

This depends on whether the output is a continuous or discrete variable. If the output variable is discrete (there are a finite number of possibilities that it can be), as in a classification task (such as this one, where you are trying to place the input into one of 5 categories), you want to use one output neuron for each class. If the variable is continuous, however, you should only use one output neuron.

This is because of how the training process works. When training your network successively makes adjustments to try and reduce the errors. These adjustments are made in the direction of the error – so if the network predicts a value which is too high then the network's weights are adjusted to make the output value lower. On the other hand if the network's predicts a value which is too low the network's weights are adjusted to make the output bigger.

If you have output neurons labeled 0 to 4 and a training sample with some input value and a target prediction of 2 then the neural network will make its prediction. Once the prediction has made each neuron is adjusted individually – in this case neuron 2 will be adjusted in the direction of the correct probability and all the other neurons will be adjusted in the direction of the incorrect probability. In this way you have one prediction for each class.

Backpropagation is a about error attribution, and using multiple neurons allows the error of the neural network to be better attributed as the neural network can adjust each neuron individually, and thus adjust the required probabilities for each class.

Using a single neuron with a sigmoid activation function would be less good as the sigmoid function saturates values close to 0 and 1 so there would be an unnatural bias towards category 0 and category 4 over the other categories. The neural network could learn to overcome this, but it would take more time.


What is a recurrent neural network? Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions.

So, what is a recurrent neural network, and what are their advantages over regular (or feed-forward) neural networks?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.

An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.

There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.


How could decision tree learning algorithms cope with imbalanced classes? Decision trees and random forests may or not be more suited to solve supervised learning problems with imbalanced labels (or classes) in datasets. For example, see the article Using Random Forest to Learn Imbalanced Data, this Stats SE question and this Medium post. The information across these sources does not seem to be consistent.

How could decision tree learning algorithms cope with imbalanced classes?

Decision Tree learners, on their own, are not a good way to deal with imbalanced data. The most commonly used algorithms, by default, make no attempt to address this problem.

If you look carefully at the three sources you post, you will find that they actually all agree on this point.

Two of the sources actually propose methods of addressing this shortcoming, by making adjustments to the decision tree learning algorithms. The proposed adjustments are essentially standard solutions to these problems, being applied to decision trees.

An example technique, discussed in the first paper you reference, is changing the weightings of the classes. An inefficient/approximate way to do this is to increase the number of examples from the minority class. For example, if you had an 80/20 split, you could add 3 new copies of each minority class example to move to an 80/80 = 50/50 split. Of course, if you add new data points, your algorithm may take longer to run. Instead, you can just modify the weightings of the classes in your optimization function. This approach is algorithm-specific, and will depend on your loss function, but achieves the same effect, just without needing to increase the number of points you use.


MCTS for non-deterministic games with very high branching factor for chance nodes I'm trying to use a Monte Carlo Tree Search for a non-deterministic game. Apparently, one of the standard approaches is to model non-determinism using chance nodes. The problem for this game is that it has a very high min-entropy for the random events (imagine the shuffle of a deck of cards), and consequently a very large branching factor ($\approx 2^{32}$) if I were to model this as a chance node.

Despite this issue, there are a few things that likely make the search more tractable:

Chance nodes only occur a few times per game, not after every move.
The chance events do not depend on player actions.
Even if two random outcomes are distinct, they might be "similar to each other", and that would lead to game outcomes that are also similar.

So far all approaches that I've found to MCTS for non-deterministic games use UCT-like policies (e.g. chapter 4 of A Monte-Carlo AIXI Approximation) to select chance nodes, which weight unexplored nodes maximally. In my case, I think this will lead to fully random playouts since any chance node won't ever be repeated in the selection phase.

What is the best way to approach this problem? Has research been done on this? Naively, I was thinking of a policy that favors repeating chance nodes more over always exploring new ones.

You can try using an "Open-Loop" MCTS approach, instead of the standard "closed-loop" one, and eliminate chance nodes altogether. See, for example, Open Loop Search for General Video Game Playing.

In a "standard" (closed-loop) implementation, you would store a game state in every normal (non-chance) node. Whenever there is a chance event, you would stochastically traverse to one of its children, and then have a normal node with a "deterministic" game state again.

In an open-loop approach, you do not store game states in any node (except possibly the root nodes), because nodes no longer deterministically correspond to specific game states. Every node in an open-loop MCTS approach only corresponds to the sequence of actions that leads to it from the root node. This completely eliminates the need for chance nodes, and results in a significantly smaller tree because you only need a single path in your tree for every possible unique sequence of actions. A single sequence of actions may, depending on stochastic events, lead to a distribution over possible game states.

In every separate MCTS iteration, you would re-generate game states again by applying moves "along the edges" as you traverse through the tree. You also "roll the dice" again for any stochastic events. If your MCTS iteration traverses a certain path of the tree often enough, it will still be able to observe all the possible stochastic events through sampling.

Note that, given an infinite amount of time, the closed-loop approach with explicit chance nodes will likely perform much better. But when you have a small amount of time (as is the case in the real-time video game setting considered in the paper I linked above), an open-loop approach without explicit chance nodes may perform better.

Alternatively, if you prefer the closed-loop approach with explicit chance nodes, you could try some mix of:

Allowing MCTS to prioritise promising parts of the search tree over parts that have not been visited at all (i.e. do not automatically prioritise nodes with $0$ visits). For example, instead of giving unvisited node a value estimate of $\infty$ (this is how you could interpret the automatic selection of them), you could give them a value estimate equal to the value estimate of the parent node, and just apply the UCB1 equation directly.
Use AMAF value estimates / RAVE / GRAVE in your selection phase. This allows you to very quickly learn some crude value estimates for moves that you have never selected in the Selection phase yet, by generalising from observations of playing them in the Play-out phase. I have noticed that the "standard" implementation of RAVE / GRAVE, without an explicit UCB-like exploration term, does not mix well with my previous suggestion of using a non-infinite value estimate for unvisited children. It may be good to consider a UCB-like variant with an explicit exploration term instead.


Are Graph Neural Networks generalizations of Convolutional Neural Networks? In lecture 4 of this course, the instructor argues that GNNs are generalizations of CNNs, and that one can recover CNNs from GNNs.
He presents the following diagram (on the right) and mentions that it represents both a CNN and a GNN. In particular, he mentions that if we particularize the graph shift operator (i.e the matrix S, which in the case of a GNN could represent the adjacency matrix or the Laplacian) to represent a directed line graph, then we obtain a time convolutional filter (which I hadn't heard of before watching this, but now I know that all it does is shift the graph signal in the direction of the arrows at each time step).

That part I understand. What I don’t understand is how we can obtain 2D CNNs (the ones that we would for example apply to images) from GNNs. I was wondering if someone could explain.
EDIT: I found part of the answer here. However, it seems that the image convolution as defined is a bit different from what I’m used to. It seems like the convolution considers pixels only to the left and above of the “current” pixel whereas I’m used to convolutions considering both left, right, above, and below

Excuse my lack of rigor. Although I believe this could be rigorously proven for certain definitions of GNN, the term is still too loose for me to honestly claim one way or another on this. Hopefully the following thoughts will be helpful anyway.
I prefer the term Message Passing Networks as a generalization of many things people like to call GNN. In a generic Message Passing Network, every node has an associated vector.
Each node's vector, $x_i$, is then updated as a function of itself and its neighbors. (I've left off time indices for readability...)
For instance:
$$ x_i := Update(x_i, \sum_{x_j\in N(x_i)} Message(x_i,x_j, e_{ij})) $$
where $N(x_i)$ is the set of vectors associated with nodes neighboring $x_i$ and $Update$ and $Message$ are arbitrary parameterized functions.
In the case where $Message(x_i, x_j, e_{ij}) = e_{ij} x_j$ and $e_{ij}$ is a scalar, and $Update(a,b) = b$, this becomes the following:
$$ x_{i} := \sum_{x_j\in N(x_i)} e_{ij} x_j $$
This means that each node becomes a weighted sum of its neighbors and this is precisely convolution (given that you have a stride of 1 ... and that you have padding ... and that the graph is a grid where all upward edges are equal, all leftward edges are equal, all rightward edges are equal and all downward edges are equal.)


Is it a good idea to use different width and height of the kernel in a CNN? I always see that the width and height of the kernel are the same. But is it a good idea to use different numbers?
Recently I tried to use GoogLeNet (which expects images to be 224x224) on my images (500x150) and I got an error:

Negative dimension size caused by subtracting 7 from 5 for 'average_pooling2d_5/AvgPool'...

I know that this error is because the height of my image is too small. If I use the height of about 200, then everything is ok. So, maybe, in this situation, I could just use a smaller height and bigger width in the kernel. For example (5, 3).
Is it a good idea in this case? Or in general? How can it affect the accuracy of the network and the ability to extract different features?

It depends on your application. In case of text recognition, non-uniform kernels are used since the information about text is less on the horizontal axis and more on the vertical axis.
If in your case it is applicable then, it will be good idea. But, if it is not you are better off using a smaller uniform kernel (2x2, maybe). You can also zero-pad your image to make it uniform before putting it through convolutions. Also, check if you are doing 'valid' or 'same' padding in your convolutions since 'valid' convolutions chip away at your image dimensions.


What are the current trends/open questions in logics for knowledge representation? What are the future prospects in near future from a theoretical investigation of description logics, and modal logics in the context of artificial intelligence research?

Some of the work on descriptive logics and modal logics was done within the context of artificial intelligence from a research funding perspective.  Some was part of the normal academic apparatus of mathematics departments.  Furtherance of these fields in the AI context has been hindered by historically low return on investment.

Although first order logic, propositional logic, and fuzzy logic have been investigated conceptually and developed considerably, there are no theorems of note that have been proven without human assistance.  Currently, mathematics is furthered solely by willful acts of humans, although that fact is not immutable.

There have been no proofs offered that artificial machinery cannot ever apply logic to prove theorems, develop disciplines, or assemble new logical systems.  There are also no existing proofs to the contrary.

It is an open question in academia and with governments and corporations investigating the potentials of AI.  To characterize any of the work in progress as a trend would be to exaggerate the effort level along one of many theoretical paths.

A search of academic articles using either descriptive logics and modal logics as search terms, with date constrained to 2008-2018, returns a trend of variety.  Another search of the same decade for articles that reference the twentieth century work in those disciplines produces the same wide distribution.

There is no media hype or academic, corporate, or government claims of any research results that can be translated into useful application.  All of these statistics are research inhibitory.  Investors and educational institutions like paths with clear advantages.

Changes to this condition may come from work being done on the ontological and epistomological side in neural research as it applies to human linguistic networks in the brain.  Although lower animals can appear to draw conclusions, they are largely learning patterns of behavior based on past experience without applying logic.

Dolphins, elephants, dogs, and primates can communicate and exhibit forms of intelligence and personality traits, but existing research indicates that they are unable to learn dialog or develop the ability to form new ideas.

Those investigating semantic approaches in AI seem to exhibit a single orientation, and that may be the one identifiable trend.  They suspect that there is one single causal origin responsible for three things.

The ability to develop complex languages
The ability to construct new schools of thought
The ability to develop a logical argument for or against a proposition or a theory or consider its likelihood logically

There may be something to the idea that natural language rather than formal grammars are the key to higher thinking.


What types of machine learning model would fit? I am working on a supervised machine learning problem where I have more than 10 probably 50 or 100 predicting label categories. Which type of model can be used to work on this type of problem in anaconda python.

Welcome to AI.SE @Par!

What you have might be either a multi-label or a multi-class classification problem. If the classes are disjoint (each example belongs to just 1 of the 50 classes), it's a multi-class problem. If not (so each example can belong to several classes at once), it's a multilabel problem.

Multi-label classification is usually handled by training a separate model for each of the labels. If you want to label a new point, you then ask each model what it thinks, and assign the union of the labels that the models suggest together. An alternative approach is to use something like a neural network, which can have many outputs. You can then have one output neuron for each possible label.

Multi-class classification can be addressed by using the multilabel techniques, but this is usually not a good idea. The three main approaches that are used are "one-v-one", "one-v-all", and "many-v-many".

In 1-v-1, we train one model to discriminate each pair of classes (n(n+1)/2 models in total for n classes). To classify a new point, we ask each model which class it belongs to and assign a "vote" to the class the model returns. The class with the most votes overall is selected as the label for the new point. In case of a tie, we can report several possible answers to the user.
In 1-v-all, we train one model to discriminate each class from all the other classes (n models in total for n classes). To classify a new point, we ask each model whether it belongs to the model's primary class or not. Ideally, just one model claims the new point. If more than one does, we can report a tie, or use some notion of classifier confidence to select a winner.
In many-v-many, we train k models each of which is tasked with discriminating some subset of the classes from all the rest. To classify a new point, we ask all of these models to label the point. We then pick the class that that is most consistent with the results of all the models. This can also be done by deliberately constructing the models to form an error correcting code.

So which approach should you use? Well, Rifkin & Klautau's 2004 JMLR paper argues convincingly that the answer is to use one-versus-all classification. This is also pretty easy to do in most packages. For example, in Python's ScikitLearn, you can do it with OneVsRest. If you're not sure what to try, that's probably a safe bet.


What are the connections between ethics and artificial intelligence? What are the connections between ethics and artificial intelligence?

What are the issues that have arisen, especially in the business context? What are the issues that may arise?

This is a good related read from Nature: There is a Blind Spot in AI Research

  Fears about the future impacts of Artificial Intelligence are distracting researchers from the real risks of deployed systems


Is it standard to say that an evaluation function estimates the “goodness” of a position? Wikipedia states that:

An evaluation function, also known as a heuristic evaluation function or static evaluation function, is a function used by game-playing computer programs to estimate the value or goodness of a position (usually at a leaf or terminal node) in a game tree.

Is "goodness" an actual term in use in this context, or should it more properly be something like "perceived optimality"?
I ask because, for instance, in Combinatorial Game Theory, a lighthearted term such as "loopy" is preferred by some mathematicians (Demaine) over the more serious term "cyclic" (Fraenkel).
On a related note, is the use of "position" instead of "node" preferred here as an acknowledgment of the heuristic nature of evaluation functions?  (My understanding is that "position", "node" and "game" may all be interchangeable in certain contexts.)

Yes, "goodness" is a common description of the value generated by an evaluation function.
For example,
"Artificial Intelligence" p. 77;
"Knowledge-Free and Learning-Based Methods in Intelligent Game Playing" p. 15;
"Tenth Scandinavian Conference on Artificial Intelligence" p. 125; and
"Algorithms and Networking for Computer Games" p. 80.

The "position" is to the complete state of the game at some instant. A single position includes the locations of every piece on a game board at that instant, and also includes a few other bits of information -- who's turn it is; whether castling is still a legal option; etc.

I suspect that "the position" is a synecdoche, referring to the total state by a small part, the physical location (physical position) of some piece on the game board, based on earlier usage in military tactics of "positioning" troops relative to natural cover or relative to enemy troops.


What would an implementation of this Neural Network look like? I'm relatively new to neural networks and was wondering what an implementation of this paper would look like. More specifically, how are the correct values of Kp, Ki, and Kd determined at run time so it can be back propagated?

In the paper there is a diagram of the network showing how there are 3 outputs which share features but have their own set of associated weights. Back propagation works by only looking at one output at a time.


How can I find a specific word in an audio file? I'm trying to train and use a neural network to detect a specific word in an audio file. The input of the neural network is an audio of 2-3 seconds duration, and the neural network must determine whether the input audio (the voice of a person) contains the word "hello"  or not.
I do not know what kind of network to use. I used the SOM network, but I did not get the desired result. My training data contains a large number of voices that contain the word "hello".
Is there any python code for dis problem?

After some research on the internet, I realized that using VOSK toolkit in python, it can be found (detect) any particular word in audio file or real time audio streaming.
https://alphacephei.com/vosk/


Where can I find the original paper that introduced RNNs? I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?

The two tech reports below both call RNNs explicitly "recurrent net(work)s".

Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.

Jordan, Michael I. (May 1986). Serial order: a parallel distributed processing approach. Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, University of California.

Jordan was a student of Rumelhart, so I would lean on identifying 1 as the paper introducing RNNs, with the caveat that the first sentence in the section "Recurrent Nets" of 1 reads:

We have thus far restricted ourselves to feedforward nets. This may
seem like a substantial restriction, but as Minsky and Papert point
out, there is, for every recurrent network, a feedforward network with
identical behavior (over a finite period of time).

This is interesting for two reasons:

After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.
The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.

Unfortunately, I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.


How do we choose the kernel size depending on the problem? Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of the filter (or kernel)!

In most cases, intuition should be to go for small filters for detecting high-frequency features and large kernels for low-frequency features, right? For example, $3 \times 3$ kernel filters for edge detection, color contrast, etc., and maybe $11 \times 11$ for detecting full objects, when the objects occupy an area of roughly $11 \times 11$ pixels.

Is this "intuition" more or less generally true? How can we decide which kernel's sizes should be chosen for a specific problem - or even for one specific convolutional layer?

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.


Is there any practical application of knowing whether a concept class is PAC-learnable? A concept class $C$ is PAC-learnable if there exists an algorithm that can output a hypothesis with probability at least $(1-\delta)$ (the "probably" part), and an error that is less than $\epsilon$ (the "approximately" part), in time that is polynomial in $1/\epsilon$, $1/\delta$, $n$ and $|C|$.

Tom Mitchell defines an upper bound for the sample complexity, $m >= 1/\epsilon (ln(|H|) + ln(1/\delta))$ for the finite hypotheses. Based on this bound, he classifies whether target concepts are PAC-learnable or not. For example, $n$ conjunction boolean literal concept class.

It seems to me that PAC-learnability seeks to act more like a classification of certain concept classes.

Are there any practical purposes for knowing whether a concept class is PAC-learnable?

Is there any practical application of knowing whether a concept class is PAC-learnable?

If you know that a concept class is PAC-learnable (i.e. its VC dimension is finite), then there's a possibility that you can design an algorithm that can find a function (or concept) that is arbitrarily close to your target (or desired) function.

This is not really an application, but a consequence, which can lead to applications.

However, note that asking if PAC learning is useful in practice is like asking if special relativity is useful in practice. Yes, they are useful, but in the sense that they can be used to predict the outcomes of an experiment or explain the rules in their specific context. In the case of machine learning, PAC learning can be used to explain e.g. the probably required number of data points needed to learn a concept (a target function) approximately.

See also Are PAC learning and VC dimension relevant to machine learning in practice? for more concrete "applications" of PAC-learning and the VC dimension.


How can I determine whether a car in a video is moving or not? How can I classify a given sequence of images (video) as either moving or staying still from the perspective of the person inside the car?
Below is an example of the sequence of 12 images animated.

Moving from the point of the person inside the car.

Staying still from the point of the person inside the car.

Methods I tried to achieve this:

A simple CNN (with 2d convolutions) with those 12 images (greyscaled) stacked in the channels dimension (like Deepmind's DQN). The input to the CNN is (batch_size, 200, 200, 12).

A CNN with 3d convolutions. The input to the CNN is (batch_size, 12, 200, 200, 1).

A CNN+LSTM (time-distributed with 2d convolutions). The input to the neural network is (batch_size, 12, 200, 200, 1).

The late fusion method, which is taking 2 frames from the sequence that are some time steps apart and passing them into 2 CNNs (with same weights) separately and concatenating them in a dense layer As mentioned in this paper. This is also like CNN+LSTM without the LSTM part. The input to this net is (batch_size, 2, 200, 200, 1) -> the 2 images are first and last frames in the sequence

All the methods I tried failed to achieve my objective. I tried tuning various hyperparameters, like the learning rate, the number of filters in CNN layers, etc., but nothing worked.
All the methods had a batch_size of 8 (due to memory constraint) and all images are greyscaled. I used ReLUs for activations and softmax in the last layer. No pooling layer was used.
Any help on why my methods are failing or any pointers to a related work

CNNs are translation invariant.

You are over complicating the problem.  The easiest thing you can do is define a region of interest (ROI) of the hood.  In the first case where the car is moving and the reflections are dynamic.  In the second case they are static.  Just do frame-to-frame image subtraction of the hood.  If the vehicle is moving you will have lots of 'edge energy'.  If it is not moving it will be just noise.

You can apply the same method to the whole image too.  In the static case the image subtraction method may become messy as the clouds are moving along with vehicles and pedestrians.  For this case use the image subtractions as input to your method.

Another approach is to run an image stabilization algorithm.  OpenCV has one.  Look at the transformation outputs (translation, rotation, scale, rigid, similarity, affine, etc.).  If you can't make a simple filter on them to determine the two cases, train a classifier.


Does this "flavor" of AI have a name? In order to build a Scientific Reference Parser, I am contemplating a kind of "AI" system, and would like to know if something similar is already an established "design pattern" in AI research.

The input for the system would be Scientific References with structures like the following:
"Co-authors, title, Journal, volume, issue, begin page, year"
Of course, many other variations are possible, and I want to build a system that can make "best guesses" in case of unfamiliar patterns.

At the moment this is done by manually chaining the results of different methods together, ranging from Regex patterns to more complex algorithms like N-Grams, LSH and random forests. I contemplate a AI system that automatically "chains" all these methods together in the most optimal way. The way I imagine this to work is by means of what I call "a bag of functions". So, how would this work?

For each of the methods I use at the moment, I would specify their input requirements, and specify what they provide as output. (e.g.: input = reference, output = title). These could also be parameters like: input = year, output = is valid year? . Note that if a function outputs "title" that is an attempt at providing the title, but this is not necessarily correct (if for example, the regex pattern grabbed a wrong portion of text).
For each of these functions, I would build a training set, and log their execution time (cost) and their probability of providing a correct result.
Then, I would build a system that chains these functions together to get from a certain input, to a certain output. eg: from -a function that takes a reference, and outputs the list of co authors- to -a function that takes a list of co authors, and breaks it apart into separate authors- to -a function that takes an author, and tries to break it apart into last name and initial-.
Once a "chain" of functions is found, this chain can in turn be stored as a "function" and can be reused later on by the algorithm. For each function, the success rate and run time is stored, so the algorithm can choose to go for the fastest known route, or experiment with new functions.
In the settings you could specify the max run time (cost) or the minimum success rate. This way you could push the system to experiment with new combinations of functions.

I'm not sure I explained the intent clearly, and I'm not sure the design would hold once I try to implement this in reality. Just wanted to throw this out here to see if anyone recognizes the design. This feels like a combination of a shortest path algorithm (to connect the functions) with normal statistical probability (to determine the success rate of a function) with a "self learning" system (because combinations of functions can be "remembered" and reused).

The added advantage would be that I don't need to manually guess what parsing method I should give a higher or lower likelihood of being correct in what specific scenario. It would allow me to just "throw" a new function into the bag, and let the system test it in all kinds of configurations, learning when best to use it, and when to avoid it.

Any feedback would be greatly appreciated! :)

This sounds a little bit like a Blackboard Architecture approach.  One of the biggest challenges in these is figuring out how to handle coordination between the various "experts" (ie, algorithms).  Some people talk about an "executive" which manages the coordination, and that leaves you needing to figure out how to train the "executive".  Alternatively, you may be able to do this without a master "executive", but nobody knows exactly what the best way to do this is.

You may also find some inspiration / useful ideas in the realm of Multi-agent Systems if you model each component of your systems as a discrete "agent".


When will an artificial intelligence replace a software developer or tester? Artificial intelligence techniques have been successfully applied in many scenarios and to automate several jobs.
When will an artificial intelligence system do the job of a software developer or tester? Is this possible (for example, in the next 10 years)?

The ultimate goal of machine learning is to bypass the developer...
When we will have a "master algorithm" that can learn how to generalize any function or algorithm from examples, it can essentially replace any developer, skip the 'development" stage, going from problem directly to algorithm.

We can't know when this will happen, but as we surrounded with multiple creatures (humans and animals) which can demonstrate learning algorithms and predictive model learning without any "developer" - we can assume that such an algorithm is possible.

If I would have to guess, I would say that we are probably very near the point where "developers" and "testers" will be replaced by learning algorithms. We could be a decade or two away from the point where people will not write any code or any testing at all. Programs and automation will be derived directly from describing the problems themselves in a natural language, visualizations or data collections. However, we still need some breakthroughs in combining feature learning with active memory, unsupervised learning, and artificial common sense.


Is Mask R-CNN suited to solve a multi-class classification problem where the classes are related? I want to create a model to solve a multi-class classification problem.
Here are more details about my problem.

Every picture contains only one object

The background is very simple

All objects belong to the same family of objects (for example, all objects are knives), but there are different specific subtypes

the model will learn and predict the name of the object (example: the model learn all types of knives, and when it get an image it will tell us the name of the knife)

To be clear, let's say I have 50 types of knives, and the output of the model has to recognize the correct name of the knife. Knife name could be:

Chef's Knife,
Heavy Duty Utility Knife,
Boning Knife, etc.

To solve this problem, I have started to use annotated, segmented (masked) images (COCO-like dataset) and the Mask R-CNN model.
As a first step, I got a prediction, but I really don't know if I'm on the right way.
For this problem, Mask R-CNN could be the solution, or it is impossible to recognize a tiny difference between two objects from the same class (for example Chef's Knife, Heavy Duty Utility Knife)?

Mask RCNN can be a very heavy function for a simple class classification. It is designed to handle multiple object in a single image. So I would suggest you could use much simpler models like VGGnet or Resnet which are backbones of the MaskRCNN.
The biggest hurdle you might face is the dataset. If you are trying to capture even small difference between knives and classify them you will need atleast 2000 images per knife type. which might be difficult to get. You may have to do a data augmentation.


How to design fitness function for multiple objectives? I am currently building a neural network with genetic algorithms that learns to fly a 2D drone to a target. My goal is that it achieves all tasks as fast as possible, but I want the drone to also fly stable and upright. The way I tried to calculate the fitness was to create a function that has the greatest value when the drone does everything I want right.
fitness += 1/distToTarget + cos(drone_angle)

My current inputs are:
difference_target_X
difference_target_Y
velocity_X
velocity_Y
angular_velocity (degree per second)
drone_angle     | = 0;     |_ = 90     _| = -90

The output (I don't think it is important but)
left_thruster_angle
left_thruster_boost
right_thruster_angle
right_thruster_boost

The NN is programmed in unity and the drone uses a 2D rigid body and the NN adds a force to the thruster at the right angle.
How do I get the drone to set the best weights to fulfill all tasks: fly stable, fly fast, fly to the target?

Your fitness function has two objectives that are added together, but they are not necessarily on the same scale. The component cos(drone_angle) must have a value from 0..1. The component 1/distToTarget will have a range that depends on how you measure distToTarget; e.g. if distToTarget has a range 0..1000, then this part of the fitness function will always be small far from the target (e.g. distance of 500) and massive when it gets very close (e.g 0.1 distance from the target). So the contribution of both components may not always be equal. Another potential complication is that the cosine function is nonlinear and make a very rapid transition between 0 and 1, as opposed to a smooth transition.
I recommend reworking the fitness function to make the two components more equal, e.g. something like
fitness(angle, distance)=
    w1 * |desired_angle - angle| / maximum_angle_error
  + w2 * distance / maximum_distance

In this function that should be minimised to zero, both components contribute linearly to the final fitness, both components have a range of 0..1, and you can tune w1 and w2 to give different importance to different components depending on your preference (e.g. you may prefer w1>w2 if staying upright is the most important).


solving xor function using a neural network with no hidden layers xor is a non-linear dataset. It cannot be solved with any number of perceptron based neural network but when the perceptions are applied the sigmoid activation function, we can solve the xor dataset.

But I came across a source where the following statement is stated as False

A two layer (one input layer; one output layer; no hidden layer) neural network can represent the XOR function.

However, I have trained a model with no hidden layers, gives the following result:

[INFO] data=[0 0],ground-truth=0, pred=0.5161, step=1
[INFO] data=[0 1],ground-truth=1, pred=0.5000, step=1
[INFO] data=[1 0],ground-truth=1, pred=0.4839, step=0
[INFO] data=[1 1],ground-truth=0, pred=0.4678, step=0

So, in if I apply a softmax classifier, I can separate the xor dataset with a nn without any hidden layer. This makes the statement incorrect.

Is it true that we cannot separate a non linear dataset without any hidden layers in a neural network? If yes, where am I wrong in my reasoning from the training of the nn I have done above

Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, exp(xi)/sum(exp(xj) for j in all x), where xi is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)).

Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space.
, [src: https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]

Now, whatever you do, you cannot separate the points with a single wx+b straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped wx+b.

Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be (w2(w1*x + b1)+b2, which again is a linear equation not able to separate the classes 0 and 1.

So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get w2*(1/+exp(w2*x+b1))+b2 which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.

Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?


Why does the value of state change depending on the policy used to get to that state? From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state.
If I have understood these concepts correctly, why does the value of a state change with the policy with which an agent gets there?
I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there (different policies may have different ways, and hence different values, for getting to a particular state).
If there can be a concrete example (perhaps on a grid world or on a chessboard), that might make it clear why that might be the case.

I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there

It doesn't.

The value of a state changes depending on what the agent will do next. That is where the dependency on the policy comes in, not in past behaviour, but expectations of future behaviour. The future behaviour depends on the state transitions and rewards presented by the environment, plus it depends on the distribution of actions chosen by the policy.

More formally, the value function of a state is not just a relative and arbitrary scoring system, but equals the expected (discounted) sum of rewards, assuming the MDP follows the given dynamics, including action selection:

$$v_{\pi}(s) = \mathbb{E}_{A \sim \pi}[\sum_{k=0}^\infty \gamma^k R_{t+k+1} | S_t = s]$$

Without identifying a policy, it is not possible to assess a value function. In value-based control methods, the policy to evaluate can be implied, somewhat self-referentially, as the policy that acts greedily (or maybe $\epsilon$-greedily) according to the current estimates of the value function.

  If there can be a concrete example (perhaps on a GridWorld or on a chess board), that might make it clear why that might be the case

A very simple deterministic MDP with a start state and two terminal states illustrates this:

Start in state B. Taking the left action is followed by a transition (with $p=1$) to terminal state A, and a reward of $0$. Taking the right action is followed by a transition (with $p=1$) to terminal state C, and a reward of $3$.

What is the value of state B? It depends on what the policy chooses. A deterministic left policy $\pi_1$ has $v_{\pi_1}(B) = 0$, a random policy $\pi_2$ choosing left and right with $p=0.5$ has $v_{\pi_2}(B) = 1.5$. The optimal policy chooses action right always and has $v_{\pi_3}(B) = v^*(B) = 3.0$


What would be the importance sampling ratio for off-policy TD learning control using Q values? The off-policy TD learning control using state value function from page 34 of David Silver's RL lecture is:
$$ V(S_t) \leftarrow V(S_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \gamma V(S_{t+1})) - V(S_t) \right). $$
I'd like to change this update rule to action value function Q, something like:
$$ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \gamma Q(S_{t+1},A_{t+1})) - Q(S_t,A_t) \right). $$
Then what is the corresponding importance sampling ratio?
Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not?

Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not?

You could assign values of 1 to each to get the right answer, but the situation is different. You can see that more clearly in the definition of action value, $q(s,a)$:
$$q_{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1}|S_t=s, A_t=a]$$
The condition on the expectation $|S_t=s, A_t=a$ means that the value of $A_t$ is already assumed. There is no need to know or use the associated probability, and no need to adjust between probabilities of different policies when estimating returns for the first action choice.
In addition, the last action choice used to bootstrap in Q learning is always the maximising action over current Q values. That is an on-policy choice with respect to the target policy $\pi$, so does not need to be adjusted for.
For single-step Q learning, there is no need to use importance sampling to adjust for policy differences between behaviour and target policies. The correct update equation rule is:
$$ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left( R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t,A_t) \right). $$
This does change for n-step updates, where you do need to take account of differences in action choice between target and behaviour policies for $A_{t+1}, A_{t+2}$ up to $A_{t+n-1}$. However, you should bear in mind that $\pi(a|s)$ is zero for any non-maximising action - if you use weighted importance sampling then any trajectory with an exploring action before calculating the TD update contributes zero update.
So, if you are calculating over longer trajectories, in e.g. Q($\lambda$), it is common to see this simplified to a logical test and some kind of shortcut update. You rarely see explicit use of importance sampling - with probability calculations and ratios - in Q learning.


How do biological neurons weights get initialized? When trying to map artificial neuronal models to biological facts it was not possible to find an answer regarding the biological justification of randomly initializing the weights.

Perhaps this is not yet known from our current understanding of biological neurons?

In short

I mentioned in another post, how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of central nervous system development and synaptogenesis.

A bit more detail (with the most relevant parts italicized below)

The neocortex (one of its columns, more specifically) is a region of the brain that somewhat resembles an ANN. It has a laminar structure with layers that receive and send axons from other brain regions. Those layers can be viewed as "input" and "output" layers of an ANN (axons "send" signals, dendrites "receive"). Other layers are intermediate-processing layers and can be viewed as the ANN "hidden" layers.

When building an ANN, the programmer can set the number of layers and the number of units in each layer. In the neocortex, the number of layers and layer cell counts are determined mostly by genes (however, see: Human echolocation for an example of post-birth brain plasticity). Chemical cues guide the positions of the cell bodies and create the laminar structure. They also seem to guide long term axonal connections between distant brain regions. The cells then sprout dendrites in certain characteristic "tree-like" patterns (see: NeuroMorpho.org for examples). The dendrites will then form synapses with axons or other cell bodies they encounter along the way, generally based on the encountered cell type.

This last phase is probably the most analogous to the idea of random weight initialization in ANNs. Based on where the cell is positioned and its type, the encountered other neurons will be somewhat random and so will the connections to them. These connections are probably not going to be very strong initially but will have room to get stronger during learning (probably analogous to initial random weights between 0 and ~0.1, with 1 being the strongest possible connection). Furthermore, most cells are either inhibitory or excitatory (analogous to negative and positive weights).

Keep in mind this randomization process has a heavy spatial component in real brains. The neurons are small and so they will make these connections to nearby neurons that are 10-200 microns away. The long-distance connections between brain regions are mostly "programmed-in" via genes. In most ANNs, there is generally no distance-based aspect to the initialization of connection weights (although convolutional ANNs implicitly perform something like distance-based wiring by using the sliding window).

There is also the synaptic pruning phenomenon, which might be analogous to creating many low weight connections in an ANN initially (birth), training it for some number of epochs (adolescence), and then removing most low-weight connections (consolidation in adulthood).


Viola-Jones algorithm: Haar-like features, how are the features extracted? If I have an image like this
1 2 3 4 5 6 7 8
a b c d e f g h
...

And I apply a Haar-like feature with a template
1 1 1 1
-1 -1 -1 -1

Then in the first position we get X1 = 1+2+3+4+a+b+c+d. If we slide one side to the right, we again get X2 = 2+3+4+5+b+c+d+e.
This way we will get X1 and X2 and X3 and so on. Now, how are these values combined to get the feature? Because when we say a feature we are not just running that template in one place, rather we will run it over multiple places in the image. It gives lots of values like X1,X2 and X3 and so on. Now, how are those combined to get the final feature which will be passed to Adaboost?

I would look at table 1 of the original paper. While you're reading the alogorithm, try to really focus on Step 2 when you get to it.

In summary, each feature is used to train it's own classifier. So in your example, the calculated features X1, X2, ... Xn you describe coorespond to apply some set of feature transforms f_1, f_2, ... f_n to a single image. This is a bit backwards from what actually happens. What the method really does is train a classifier for each feature. So if you had n features, you would have n classifiers. Then in an adaboost fashion, you upweight the classifier that performed the best. I.e, you are upweighting the classifier based soley on th best performing feature. You then repeat and re-weight all the classifiers until you reach convergence.


What is the difference between a Hopfield Network and a Neural Network or a Perceptron? Neural Net can be feed-forward or recurrent. Perceptron is only feed forward.

So, what is Hopfield Network then?

Hopfield stores some predefined patterns (lower energy states) and when an non seen pattern is fed to the Hopfield net, it tries to find the closest match among the stored patterns. Hence the output of a Hopfield is always one of the predefined patterns which matches closely to the unseen input pattern.

This is not the case with Feed Forward Neural Nets (where no such predefined patterns are stored) and every input generates a corresponding output. Here (in Feedforward) the output is generated by a predefined function (which is self-adjusted during training session) where as in Hopfield predefined patterns are stored and outputted (no such functions exist).


How do we make our outputs to have the same size as the true mask? When we are doing multi-label segmentation tasks, our y_true (the mask) will be (w, h, 3), but, in our model, at the last layer, we will be getting (w, h, number of classes) as output.
How do we make our outputs to have the same size as the true mask so that to apply the loss function, given that, currently, the shapes are not equal? Also, if we are done with applying the loss function and trained the model, how do I make results in the shape of (w, h, 3) from (w, h, number of classes)?

You can create a mapping from classes to colors for a simple one is:
y # y.shape = (W, H, n_classes)
_, y_color = y.max(dim=-1, keepdim=True) / n_classes # y_color.shape = (W, H, 1)
y_color = torch.cat([y_color] * 3, dim=-1) # y_color.shape = (W, H, 3)

(using pytorch like code)
This mapping is visualizable, of course, you may get nicer visualizations if your class to color mapping is more sophisticated.


Should we focus more on societal or technical issues with AI risk I have trouble finding material (blog, papers) about this issue, so I'm posting here.

Taking a recent well known example: Musk has tweeted and warned about the potential dangers of AI, saying it is "potentially more dangerous than nukes", referring the issue of creating a superintelligence whose goals are not aligned with ours. This is often illustrated with the paperclip maximiser though experiment. Let's call this first concern "AI alignment".

By contrast, in a recent podcast, his concerns seemed more related to getting politicians and decision makers to acknowledge and cooperate on the issue, to avoid potentially dangerous scenarios like an AI arms race. In a paper co-authored by Nick Bostrom:
Racing to the Precipice: a Model of Artificial Intelligence Development, the authors argue that developing AGI in a competitive situation incentivises us to skim on safety precautions, so it is dangerous. Let's call this second concern "AI governance".

My question is about the relative importance between these two issues: AI alignment and AI governance.

It seems that most institutions trying to prevent such risks (MIRI, FHI, FLI, OpenAI, DeepMind and others) just state their mission without trying to argue about why one approach should be more pressing than the other.

How to assess the relative importance of those two issues? And can you point me any literature about this?

Neither AI alignment nor AI governance are important yet. We are so far away from AGI that we don't even know what is missing.

We don't set up safety instructions for interstellar travel, so why should we do it for AGI? I can also come up with a lot of dangers of that...

There are real dangers of AI, though. Including societal issues:

Blind trust:  Trusting a machine, although it is not perfect. Just because it is good in most cases and seems to be rational/objective. Example: Machine Bias
Unemployment: AI has the potential to replace a lot of low-skill jobs with very few super high skilled jobs. For example, self-driving cars (< 100 people for one manufacturer, I guess) could replace all jobs in trasportation (1,076,200 of 31,373,700 in Germany (source) - that is 3.4%!). See CG gray video.
Weapon Systems: The thread of leathal autonomous weapons is real. There is the danger of efficient slaughterbots and the danger of errors - similar to the Wech Baghtu wedding party airstrike or that one

There are a couple of other problems, but I think they are less severe:

Better fakes: lyrebird.ai gives a good demo
Spam: I think it's easier to improve the spam filters, but humans might have a harder time
Data is dominance: Companies like Amazon will have a dominant position on the market as they were the first to aquire important customer information
Bubbles / false hopes: AI (or let's better say Machine learning)  is often seen as a golden hammer. It is not.


Why does (not) the distribution of states depend on the policy parameters that induce it? I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question -

While transitioning from the first line to the second, the gradient with respect to policy parameters $\theta$ was pushed into the summation. What bothers me is how it skipped over $\mu (s)$, the distribution of states - which (the way I understand it), is induced by the policy $\pi_\theta$ itself! Why then does it not depend on $\theta$?
Let me know what's going wrong! Thank you!

The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that $\mu(s)$ depends on the policy $\pi$ which in turn depends on the policy parameters $\theta$, and so there should be a derivative of $\mu$ wrt $\theta$, however the Policy Gradient Theorem doesn't require you to take this derivative.
In fact, the great thing about the Policy Gradient Theorem is that the final result does not require you to take a derivative of the state distribution with respect to the policy parameters. I would encourage you to read and go through the derivation of the Policy Gradient Theorem from e.g. Sutton and Barto to see why you don't need to take the derivative.

Above is an image of the Policy Gradient Theorem proof from the Sutton and Barto book. If you carefully go through this line by line you will see that you are not required to take a derivative of the state distribution anywhere in the proof.


Is the choice of the optimiser relevant when doing object detection? Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with their annotations. We want to train a model using YOLOv3.
Does the choice of optimizer really matter in terms of training the model? Would the Adam optimizer be better than the Adadelta optimizer? Or would they all basically be the same?
Would some optimizers be better because they allow most of the weights to achieve their "global" minima?

I have experimented with this to a small degree and have not noticed that much of an impact.
To date, Adam appears to give the best results on a variety of image data sets. I have found that "adjusting" the learning rate during training is an effective means of improving model performance and has more impact than the selection of the optimizer.
Keras has two callbacks that are useful for this purpose. Documentation is at https://keras.io/callbacks/. The ModelCheckpoint callback enables you to save the full model or just the model weights based on monitoring a metric. Typically, you monitor validation loss and set the parameter save_best_only=True to save the results for the lowest validation loss. The other useful callback is ReduceLROnPlateau, which allows you to adjust the learning rate based on monitoring a metric. Again, the metric usually monitored is the validation loss. If the loss fails to reduce after a user-set number of epochs (parameter patience), the learning rate will be adjusted by a user-set factor (parameter factor). You can think of the training process as traveling down a valley. As you near the bottom of the valley, it becomes more and more narrow. If your learning rate does not adjust to the "narrowness" there is no way you will get to the bottom of the valley.
You can also write a custom callback to adjust the learning rate. I have done this and created one which first adjusts the learning rate based on monitoring the training loss until the training accuracy reaches 95%. Then it switches to adjust the learning rate based on monitoring the validation loss. It saves the model weights for the lowest validation loss and loads the model with these weights to make predictions. I have found this approach leads to faster training and higher accuracy.
The fact is you can't tell if your model has converged on a global minimum or a local minimum. This is evidenced by the fact that, unless you take special efforts to inhibit randomization, you can get different results each time you run your model. The loss can be envisioned as a surface in $N$ space, where $N$ is the number of trainable parameters. Lord knows what that surface is like and where your initial parameter weights put you on that surface, plus how other random processes cause you to traverse that surface.
As an example, I ran a model at least 20 times and got resultant losses that were very close to each there. Then I ran it again and got far better results for exactly the same data.


Is the number of feature maps equal to the number of kernels in the LeNet 5 architecture? In LeNet 5's first layer, the number of feature maps is equal to the number of kernels. However, the second convolutional layer has a depth different from the 3rd layer. Does the filter size dictate the number of feature maps?

The # of kernels will be the channel length, looking at the image you posted in your comment from post, I do not understand where you see the inconsistency.


You need to include optimizer you used to make sure it is correct.
By the way, your drop-out layers are not going to do anything, so you should take them away.

You likely don’t have test and train data in time-series because all data points are connected. It just has prediction value and ground truth of each period.

I recommend you use the whole dataset and rotate changing hyper-parameters of LSTM to find the best model.


Why are the rewards of my RL agent for the Atari Breakout game decreasing after a certain number of episodes? The agent is trying to master the Atari Breakout game.

Here is my code

Is that normal that reward_100 decreased that much after it hits 4.5? Is there a way to avoid that behavior?

Be aware that reward_100 is simply mean_reward = np.mean(self.total_rewards[-100:]). In other words, it is the mean over the last 100 rewards. On the graph, reward_100 represents de y-axis and th number of episodes the x-axis.

It seems that decaying the learning rate solved my problem. I changed learning_rate from 0.001 to 0.0001