Is it possible to generate "Karel the robot" programs with genetic programming? Karel the robot is an education software comparable to turtle graphics to teach programming for beginners. It's a virtual stack-based interpreter to run a domain-specific language for moving a robot in a maze. In its vanilla version, the user authors the script manually. That means, he writes down a computer program like

move forward
if reached obstacle == true then stop
move left.

This program is then executed in the virtual machine.
In contrast, genetic programming has the aim to produce computer code without human intervention. So-called permutations are tested if they are fulfilling the constraints and, after a while, the source code is generated. In most publications, the concept is explained on a machine level. That means assembly instructions are generated with the aim to replace normal computer code.
In "Karel the robot" a high-level language for controlling a robot is presented, which has a stack, but has a higher abstraction. The advantage is, that the state space is smaller.
My question is: is it possible to generate "Karel the robot" programs with genetic programming?

A genetic algorithm is used to find an optimal solution to a problem. The parameters of the problem are encoded in the population, where each individual contains a solution to the problem in itself. Each individual is evaluated using a fitness function, and then the highest scoring individuals are used to create a new set of individuals by mixing up their parameters, and this new set replaces the worst scoring individuals from the previous generation. Over time, the average fitness of the population will (hopefully) increase, and a good solution be found. Essentially, the search space is explored in parallel through the many individuals in the population, which means that local maxima can probably be avoided.

The question now is: What is the problem you want to optimise?

The Karel language is fairly simple, and has few tokens. A program should be easily encodable in the right format, though genetic algorithms usually require individuals to have the same size (though shorter programs could be padded with no-ops). Recombining the best programs could mean swapping tokens. Maybe syntactic constraints could be factored into this to avoid crass syntactic errors. That should not be an insurmountable problem.

However, that still leaves the question of the fitness function. A basic criterion would be syntactic: if the program doesn't parse, it's not 'fit'. This could potentially wipe out a lot of offspring, and doesn't really help you much: it would produce more or less random programs which are syntactically well-formed.

What you really want is to evaluate the outcome of the programs. Like, which program can best navigate a maze. So you need to execute them and see what they produce. Then you run straight away into the halting problem: your little program might contain an endless loop and will never terminate. So you need to add a kind of time-out, or a limit on the steps the program can execute. This should be easy to do in a virtual machine.

So, to answer your question: Yes, it is possible to turn creating Karel programs into a problem that a genetic algorithm can be applied to. However, you need to be clear about what problem you want to solve. I don't quite understand what you mean by "learning from demonstration" or how you can achieve the aim of replicating human actions, but if you can somehow encode that in a fitness function, GAs are a tool you can use.


How is the depth of a CNN layer determined? I am looking at a diagram of ZFNet in an attempt to understand how CNNs are designed effectively. I'm working with the CIFAR10 set in pytorch.

In the first layer, I understand the depth of 3 (224x224x3) is the number of color channels in the image.

In the second layer I understand the 110x110 is is (224 - ( 7 * 2)) / 2

I also understand how pooling works to create a size reduction.

But where does the depth of 96 come from in the second layer? Is this the new "batch size"? Is it totally arbitrary? Bonus points if someone can direct me to a reference that can help me understand how all these dimensions relate to each other.

The 96 is the amount of filter maps (also: filter kernels). It is a fundamental of convolutional neural networks. The exact number is not arbritary, although there is no equation or exact rule of restricting the number.

If you have a CNN one single convolution operation would be pointless: since it used for the whole image information it can generalize, but only to specific (meaning: finite amount of) features. Easy example: if a 7x7 filter in the first layer concentrates on round shapes, it can not generalize on let's say red cubes at the same time.

Therefore you convolutional layers have several filter kernels, i.e. several weights where each is used for a convolution. The result of each of these convolutions is one filter map, i.e. the image information convolved by a kernel.

Typically you look at your filter kernels and your problem's domain to figure out what an appropriate number of filter kernels could be. You also have to keep in mind that too few kernels could possibly lose information and overfit to specific patterns, while too many kernels could possibly underfit. Especially when you have far more parameters (primarily the weights of your network) than training data your network will normally perform bad and you have to reduce its size.

The filter kernels should not be confused with batch size. The batch size is the amount of samples (here: images) you train in parallel. Each training step of your network does not only consist of feeding a single image, but feeding batch size number of images, usually combined with batch normalization steps between the layers. Hence, this has nothing to do the amount of filter kernels / convolutions of your network.


Price Movement Forecasting Issue I am working on a project for price movement forecasting and I am stuck with poor quality predictions.
At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the sequence of the last 45-60 observations. I tested several different ideas, but they all seems to give similar results. The model is trained to minimize MSE.
For each idea I tried a model predicting 1 step at a time where each prediction is fed back as an input for the next prediction, and a model directly predicting the next 10 steps(multiple outputs). For each idea I also tried using as input just the moving average of the previous prices, and extending the input to input the order book at those time-steps.
Each time-step corresponds to a second.
These are the results so far:
1- The first attempt was using as input the moving average of the last N steps, and predict the moving average of the next 10.
At time t, I use the ground truth value of the price and use the model to predict t+1....t+10
This is the result:
Predicting moving average:

On closer inspection we can see what's going wrong:
Prediction seems to be a flat line. Does not care much about the input data:

The second attempt was trying to predict differences, instead of simply the price movement. The input this time instead of simply being X[t] (where X is my input matrix) would be X[t]-X[t-1].
This did not really help.
The plot this time looks like this:

Predicting differences:

But on close inspection, when plotting the differences, the predictions are always basically 0.

At this point, I am stuck here and running our of ideas to try. I was hoping someone with more experience in this type of data could point me in the right direction.
Am I using the right objective to train the model? Are there any details when dealing with this type of data that I am missing?
Are there any "tricks" to prevent your model from always predicting similar values to what it last saw? (They do incur in low error, but they become meaningless at that point).
At least just a hint on where to dig for further info would be highly appreciated.
UPDATE
Here is my config
{
    "data": {
        "sequence_length":30,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 51,
                "input_timesteps": 30,
                "input_dim": 101,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 51,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "linear"
            }
        ]
    }
}

Notice the last layer with 101 neurons. It is not an error. We just want to predict the features as well as the price. In other words, we want to predict the price for time t+1 and use the features predicted to predict the price and new features at time t+2, ...

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation.  This is based on training on a history of 45 input time steps and forecasting out 10 steps in the future.  I have assumed that your 'num_steps' is a stride through the training data.  Note that I have not tested this and you will most likely need to tweak a parameter or two:

{
    "data": {
        "sequence_length": 55,
        "train_test_split": 0.85,
        "normalise": true,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 100,
                "input_timesteps": 45,
                "input_dim": 2,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "lstm",
                "neurons": 100,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

Try this out and plot your model's loss and accuracy by epoch.  See Display Deep Learning Model Training History in Keras for some information and code in this area.  Let me know if you are not using Keras and I will point you to another article/code solution.


Would Google's self-driving-car stop when it sees somebody with a T-shirt with a stop sign printed on it? In Hidden Obstacles for Google’s Self-Driving Cars article we can read that:

Google’s cars can detect and respond to stop signs that aren’t on its map, a feature that was introduced to deal with temporary signs used at construction sites.
Google says that its cars can identify almost all unmapped stop signs, and would remain safe if they miss a sign because the vehicles are always looking out for traffic, pedestrians and other obstacles.

What would happen if a car spotted somebody in front of it (but not on the collision path) wearing a T-shirt that has a stop sign printed on it. Would it react and stop the car?

Google’s self-driving car most likely uses mapping of traffic signs using google street view images for roadway inventory management. If traffic signs are not in its database, it can still “see” and detect moving objects which can be distinguished from the presence of certain stationary objects, like traffic lights. So its software can classify objects based on the size, shape and movement patterns. Therefore it is highly unlikely that a person would be mistaken for a traffic sign. See: How does Google's self-driving car identify pedestrians?

Image: Technology Review

To support such a claim, Illah Nourbakhsh, a professor of robotics at Carnegie Mellon University, gave an interview to the New York Times magazine cover story on autonomous driving cars, and includes this hypothetical scenario, saying:

  If they’re outside walking, and the sun is at just the right glare level, and there’s a mirrored truck stopped next to you, and the sun bounces off that truck and hits the guy so that you can’t see his face anymore — well, now your car just sees a stop sign. The chances of all that happening are diminishingly small — it’s very, very unlikely — but the problem is we will have millions of these cars. The very unlikely will happen all the time.

Even so, the risk would be minimal, since the car is always looking out for traffic, pedestrians and other obstacles.

Sources:

How Google's self-driving cars see the world
The Dream Life of Driverless Cars at The New York Times


Is it difficult to learn the rotated bounding box for a (rotated) object? I have checked out many methods and papers, like YOLO, SSD, etc., with good results in detecting a rectangular box around an object, However, I could not find any paper that shows a method that learns a rotated bounding box.
Is it difficult to learn the rotated bounding box for a (rotated) object?
Here's a diagram that illustrates the problem.

For example, for this object (see this), its bounding box should be of the same shape (the rotated rectangle is shown in the 2nd right image), but the prediction result for the YOLO will be Ist right.
Is there any research paper that tackles this problem?

Cartesian Bias and Pipeline Efficiency

You are experiencing a techno-cultural artifact of Cartesian-centric imaging running all the way back to the dawn of coordinate systems. It is the momentum of practice as a consequence of applying Cartesian 2D coordinates to rasterize images appearing at the focal planes of lenses from the dawn of television and the earliest standards of raster based capture and display.

Although some work was done toward adding tilt to bounding rectangles in the late 1990s and since, from a time and computing resource conservation perspective, it is computationally and programmatically less costly to include the four useless triangles of pixels and keep the bounding box orthogonal with the pixel grid.

Adding a tilt angle to the bounding boxes is marginally competitive when detecting ships from a satellite only because two conditions offset the inefficiencies in that narrow domain. The ship appears as an oblong rectangle with rounded corners from a satellite positioned in geosynchronous orbit. In the general case, adding a tilt angle can slow recognition significantly.

Biology Less Biased

An interesting side note is that the neural networks of animal and human vision systems do not have that Cartesian-centricity, but that doesn't help this question's solution, since non-orthogonal hardware and software is virtually nonexistent.

Early Non-Cartesian Research and Today's Rasterization

Gerber Scientific Techonology research and development in the 1980s (South Windsor, Connecticut, U.S.) investigated vector capture, storage, and display, but the R&D was not financially sustainable for a mid-side technology corporation for the reasons above.

What remains, because it is economically viable and necessary from an animation point of view, is rasterization on the end of the system that converts vector models into frames of pixels. We see this in on the rendering  SVG, VRML, and the original intent of CUDA cores and other hardware rendering acceleration strategies and architectures.

On the object and action recognition side, the support of vector models directly from imaging is much less developed. This has not been a major stumbling block for computer vision because the wasted pixels at one tilt angle may be of central importance at another tilt angle, so there are no actual wasted input pixels if the centering of key scene elements is widely distributed in translation and tilt, which is often the case in real life (although not so much in hygienically pre-processed datasets).

Conventions Around Object Minus Camera Tilt and Skew from Parallax

Once edge detection, interior-versus-exterior, and 3D solid recognition come into play, the design of CNN pipelines and the way kernels can do radial transformation without actually requiring $\; \sin, \, \cos, \, \text{and} \, \arctan \;$ functions evaporate the computational burden of the Cartesian nature of pixel tensors. The end result is that the bounding box being orthogonal to the image frame is not as problematic as it initially appears. Efforts to conserve the four triangles of pixels and pre-process orientation is often a wasted effort by a gross margin.

Summary

The bottom line is that efforts to produce vector recognition from roster inputs have been significantly inferior in terms of resource and wait time burden, with the exception of insignificant gains in the narrow domain of naval reconnaissance satellite images. Trigonometry is expensive, but convolution kernels, especially now that they are moving from software into hardware accelerated computing paths in VLSI, is computable at lower costs.

Past and Current Work

Below is some work that deals with tilting with regard to objects and the effects of parallax in relation to the Cartesian coordinate system of the raster representation. Most of the work has to do with recognizing 3D objects in a 3D coordinate system to project trajectories and pilot or drive vehicles rationally on the basis of Newtonian mechanics.

Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs, James T. Klosowski, Martin Held, Joseph S.B. Mitchell, Henry Sowizral, and Karel Zikan, 1998

Sliding Shapes for 3D Object Detection in Depth Images, Shuran Song and Jianxiong Xiao, 2014

Amodal Completion and Size Constancy in Natural Scenes, Abhishek Kar, Shubham Tulsiani, Joao Carreira and Jitendra Malik, 2015

HMD Vision-based Teleoperating UGV and UAV for Hostile
Environment using Deep Learning, Abhishek Sawarkar1, Vishal Chaudhari, Rahul Chavan, Varun Zope, Akshay Budale and Faruk Kazi, 2016

Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds, Z Liu, H Wang, L Weng, Y Yang, 2016

Amodal Detection of 3D Objects:
Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images, Zhuo Deng, 2017

3D Pose Regression using Convolutional Neural Networks, Siddharth Mahendran, 2017

Aerial Target Tracking Algorithm Based on Faster R-CNN Combined with Frame Differencing, Yurong Yang, Huajun Gong, Xinhua Wang and Peng Sun, 2017

A Semi-Automatic 2D solution for Vehicle Speed Estimation from Monocular Videos, Amit Kumar, Pirazh Khorramshahi, Wei-An Lin, Prithviraj Dhar, Jun-Cheng Chen, Rama Chellappa, 2018


How to select number of hidden layers and number of memory cells in an LSTM? I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN.

Is there an article where this problem is being investigated, i.e., how many memory cells should one use? I assume it totaly depends on the application and in which context the model is being used, but what does the research say?

Your question is quite broad, but here are some tips.

Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns?

The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and error.

If you are also interested in feedforward networks, see the question How to choose the number of hidden layers and nodes in a feedforward neural network? at Stats SE. Specifically, this answer was helpful.

  There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below:

  $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$

  $N_i$ = number of input neurons.
  $N_o$ = number of output neurons.
  $N_s$ = number of samples in training data set.
  $\alpha$ = an arbitrary scaling factor usually 2-10.

  Others recommend setting $alpha$ to a value between 5 and 10, but I find a value of 2 will often work without overfitting. You can think of alpha as the effective branching factor or number of nonzero weights for each neuron. Dropout layers will bring the "effective" branching factor way down from the actual mean branching factor for your network.

  As explained by this excellent NN Design text, you want to limit the number of free parameters in your model (i.e. its degree or the number of nonzero weights) to a small portion of the degrees of freedom in your data. The degrees of freedom in your data is the number samples * degrees of freedom (dimensions) in each sample or $N_s * (N_i + N_o)$ (assuming they're all independent). So $\alpha$ is a way to indicate how general you want your model to be, or how much you want to prevent overfitting.

  For an automated procedure you'd start with an alpha of 2 (twice as many degrees of freedom in your training data as your model) and work your way up to 10 if the error (loss) for your training dataset is significantly smaller than for your test dataset.


Determining minimal state representation for maze game I came across this question set. It asks following question:

Let’s revisit our bug friends from assignment 2. To recap, you control one or more insects in a rectangular maze-like environment with dimensions M × N , as shown in the figures below. At each time step, an insect can move North, East, South, or West (but not diagonally) into an adjacent square if that square is currently free, or the insect may stay in its current location. Squares may be blocked by walls (as denoted by the black squares), but the map is known.
For the following questions, you should answer for a general instance of the problem, not simply for the example maps shown.
(a) You now control a single flea as shown in the maze above, which must reach a designated target location X. However, in addition to moving along the maze as usual, your flea can jump on top of the walls. When on a wall, the flea can walk along the top of the wall as it would when in the maze. It can also jump off of the wall, back into the maze. Jumping onto the wall has a cost of 2, while all other actions (including jumping back into the maze) have a cost of 1. Note that the flea can only jump onto walls that are in adjacent squares (either north, south, west, or east of the flea).

i. Give a minimal state representation for the above search problem.
Sol. The location of the flea as an (x, y) coordinate.
ii. Give the size of the state space for this search problem.
Sol. M ∗ N
(b) You now control a pair of long lost bug friends. You know the maze, but you do not have any information about which square each bug starts in. You want to help the bugs reunite. You must pose a search problem whose solution is an all-purpose sequence of actions such that, after executing those actions, both bugs will be on the same square, regardless of their initial positions. Any square will do, as the bugs have no goal in mind other than to see each other once again. Both bugs execute the actions mindlessly and do not know whether their moves succeed; if they use an action which would move them in a blocked direction, they will stay where they are. Unlike the flea in the previous question, bugs cannot jump onto walls. Both bugs can move in each time step. Every time step that passes has a cost of one.

i. Give a minimal state representation for the above search problem.
Sol. A list of boolean variables, one for each position in the maze, indicating whether the position could contain a bug. You don’t keep track of each bug separately because you don’t know where each one
starts; therefore, you need the same set of actions for each bug to ensure that they meet.
ii. Give the size of the state space for this search problem.
Sol. $2^{MN}$

I don't get why the (a).i. uses $(x,y)$ coordinates whereas (b).i. uses boolean list. I guess they can be used interchangeablly right? And correspondingly the answers to ii will change.
Update
I now understand following:

For single flea maze, the representation $(x,y)$ will have $M\times N$ state space, whereas boolean list will have $2^{M\times N}$ state space. For two bug maze, the representation $(x_1,y_1,x_2,y_2)$ will have $(M\times N)^2$ state space, whereas boolean list will have $2^{M\times N}$ state space. I am able to understand, we prefer $(x,y)$ representation for single flea maze since $M\times N < 2^{M\times N}$. But for two bug maze, I am not able to understand why we prefer boolean list representation (and not $(x_1,y_1,x_2,y_2)$ representation), since $(M\times N)^2<2^{M\times N}$.

I don't get why the (a).i. uses (,) coordinates whereas (b).i. uses boolean list. I guess they can be used interchangeablly right?

No these cannot be used interchangeably. The situations really are different enough that completely different state representations are necessary.
In the case of (a) you know the current bug's location, and nothing else about the problem changes over time. So any state representation that captures a single bug's location is valid. The (x,y) coordinates are a natural choice given the framing of the question, but also you could enumerate all the possible grid positions for the bug, giving the same size of state space.
In case of (b) you do not know where either bug is, but you don't need to care about the difference between each bug to track it separately (this is becasue your goal state does not rely on the identity of either bug). Therefore the state representation has to capture where any bug could possibly be.
The proposed solution is simple, but you could also do it by tracking possible locations of each bug. That could work with $17^2$ $(x_1,y_1,x_2,y_2)$ tuples initialised to all possible position combinations that the two bugs start in. The state space in that case is large (it is not just $17^2$, but includes all valid reachable combinations of the combined $17^2$ $(x_1,y_1,x_2,y_2)$ tuples) and difficult to assess because many state combinations are not reachable. Also note this alternative proposal does differentiate between the bugs - you could potentially improve upon that by removing start states that simply swap between bug locations, although this is still not as efficient as the binary representation.
Technically the proposed solution for (b) also works the same for any number of bugs that are spread out in the maze and want to all be in the same location at the end.


Understanding the loss function in deep Q-learning I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight of the network $\theta$.
The loss function to minimise is $||\delta_{t+1}||^2$ where $\delta_{t+1}$ is shown below. The loss function is from the website talking about function approximation. Even though it is not explicitly deep Q learning, the loss function to minimise is similar.
$$\delta_{\mathrm{t}+1}=\mathrm{R}_{\mathrm{t}+1}+\max _{\mathrm{a}\in\mathrm{A}} \boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{t+1}, \mathrm{a}\right)-\boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{\mathrm{t}}, \mathrm{a}\right)$$
Source: https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566.
Intuitively, I am not able to understand why the loss function is defined as such. Once the network converges to a $\theta$ using gradient descent, does that mean that the $Q_{max}(s,a)$ is found?
In essence, I am not able to grasp intuitively how the neural network is able to generalise the learning to unseen states.
The algorithm I am looking at to help me understand the deep Q networks is below.

Source: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, look up TD-learning to have a better grasp of the concept.


What kind of problem is "email text extraction"? I need to retrieve just the text from emails. The emails can be in HTML format, and can contain huge signatures, disclaimer legalese, and broken HTML from dozens of forwards and replies. But, I only want the actual email message and not any other cruft such as the whole quotation block, signatures, etc.

This isn't really a problem that could be solved with regex because HTML mail can get very, VERY messy.

Could a neural network perform this task? What kind of problem is this? Classification? Feature selection?

It's certainly possible to treat this as a natural language processing problem, basically you're looking to assign "salience" scores to the text.

Really, though, that's overkill for this kind of problem. Writing a regex or a CFG parser (or better: finding an existing parser) is likely to be easier and more reliable.


How to classify human actions? I'm quite new to machine learning (I followed the Coursera course of Andrew Ng and now starting deeplearning.ai courses).

I want to classify human actions real-time like:

Left-arm bended
Arm above shoulder
...

I first did some research for pre-trained models, but I didn't find any.
Because I'm still quite new, I want to have advice about how I should solve this.

I thought maybe I need to create for every action enough pictures and from there on I can do image classification.
Or I use PoseNet from TensorFlow so that I have the pose estimation points. And from there on I create videos of a couple of seconds with every pose I want to track and I save the estimation points. From there on, I use a classification algorithm (neural network) to classify those points.

What is the most efficient option or are they both bad and is there a better way to do this?

My suggestion is to go with 1st option. reason is you will get to know much about data and initial stage will find some challenges in developing the model, over a period of time you will get to better results after hypertunning. Please go through article , ignore you have already read this article


What are the various methods for speeding up neural network for inference? One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?

These are some ways to speed up inference:

Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8.
Reference paper: https://arxiv.org/pdf/1502.02551
Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.
Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, but, the speed is on average 7x faster. Reference paper: https://pjreddie.com/media/files/papers/xnor.pdf

https://software.intel.com/en-us/articles/binary-neural-networks

Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.


What does the "number of channels" correspond to in U-Net? I'm studying the U-Net CNN architecture. I'm new to CNNs and am confused regarding the "number of channels".
Referring to the U-Net diagram, the input image is convolved with a 3x3 mask which generates a 570x570 output. This output image is then convolved again by a 3x3 mask to produce a 568 x 568 signal. However, what do the 64's correspond to?
The U-net says something about a multi-channel feature map. But how does convolving an image by a 3x3 mask results in a "64".

In this example you have a gray scale image of size 572x572 and 1 (gray) channel. The first convolution operation consists of 64 filters of size 3x3 and 1 channel per filter. The channel of the filters always fits the channel size of the previous layer (here: the Input).
In the second convolution step of this explicit architecture, you again use 64 filters of size 3x3. In this case, each of these filters consists of 64 channels according to the previous output (64 feature maps/channels). The output of the second convolution consists of 64 feature maps according to the amount of 64 filters in the second convolution. This video from Andrew Ng visualizes it perfectly.


Can I do deep learning with the 1060 or the 1070 ti? Before I start, I want to let you know that I am completely new to the field of deep learning! Since I need a new graphics card either way (gaming you know) I am thinking about buying the GTX 1060 with 6GB or the 1070 ti with 8GB. Because I am not rich, basically I am a pretty poor student ;), I don't want to waste my money. I don't need deep learning for my studies I just want to dive into this topic because of personal interest. What I want to say is that I can wait a little bit longer and don't need the results as quickly as possible.

Can I do deep learning with the 1060 (6GB seem to be very limiting, according to some websites) or the 1070 ti? Is the 1070 ti overkill for a personal hobby deep learner?

Or should I wait for the new generation Nvidia graphics card?

Regarding specific choices I can't recommend, but if you are completely new, you should probably learn/code some more until you get a GPU. There is a lot to learn in machine learning before GPU speedups make a significant difference, and until then doing the computations on any old CPU would be just fine, especially if you are just starting since you won't be doing anything too complex. You will know when computational resources are your main bottleneck, and until then it shouldn't really matter too much.

Or, you could also rent computing power from say, AWS or Google


What are the current big challenges in natural language processing and understanding? I'm doing a paper for a class on the topic of big problems that are still prevalent in AI, specifically in the area of natural language processing and understanding. From what I understand, the areas:

Text classification
Entity recognition
Translation
POS tagging

are for the most part solved or perform at a high level currently, but areas such as:

Text summarization
Conversational systems
Contextual systems (relying on the previous context that will impact current prediction)

are still relatively unsolved or are a big area of research (although this could very well change soon with the releases of big transformer models from what I've read).

For people who have experience in the field, what are areas that are still big challenges in NLP and NLU? Why are these areas (doesn't have to be ones I've listed) so tough to figure out?

According to a nice article by Sebastian Ruder https://ruder.io/4-biggest-open-problems-in-nlp/ based on answers from top NLP researchers https://docs.google.com/document/d/18NoNdArdzDLJFQGBMVMsQ-iLOowP1XXDaSVRmYN0IyM/edit

Natural language understanding
NLP for low-resource scenarios
Reasoning about large or multiple documents
Datasets, problems, and evaluation

I recommend having a look at the article. More details in the slides https://drive.google.com/file/d/15ehMIJ7wY9A7RSmyJPNmrBMuC7se0PMP/view


How does a PDDL solver find a solution for a given problem? As far as I know, in PDDL, an environment is designed as well as the initial state described. When we describe the target state, the solver creates some sort of a graph. How is the graph built and what are the keys (keywords) in PDDL referring to?
I know that there are many flavours of PDDL, but let's go with the standard or the most common version of PDDL.

The question doesn't really make sense: PDDL is a description language that is used to formulate a problem. This description then is the input to a planner; how the planner arrives at the intended solution is not related to the PDDL description.

There are a number of planning algorithms, and you can implement any of them to make use of a PDDL description. The output of the solver is a plan, which is usually an ordered sequence of actions, and a tree or graph structure might be a good way of capturing this.


What approach to use for selecting one of the category according to short category text? I need some tool to classify articles based on short category text which consists of two or three words separated by '-'. The RSS/XML tag content is for example:

Foreign - News

Football - Foreign

I created my own categories and now I need to classify categories from parsed RSS of this news source, so it fits news categories defined by me.
I would for example need all articles containing category "football" to be identified as a category Sport but sometimes those categories XML tags contains exact match like Foreign - News should belong in the DB to category defined by me as Foreign.
I can of course also use longer description text if that would be needed but I think for this simple problem that would not be even necessary.
Since I used only trained decision trees frameworks so far for another project, I would like to hear advice about approach, AI technique or particular framework I can use to solve this problem. I don't want to get into a dead-end street by my own poor in this field not experienced decision.

For this application, you can frame it as text classification. Look at SpaCy. You just need to create embeddings for your text and put a Softmax in the end. You can get those embeddings from BERT or anything else out there. You can in fact just use GLOVE vectors and others like it, concatenate them and then train a classifier.


Do you need to store prevous values of weights and layers on recurrent layer while BPTT? The Back propagation through time on recurrent layer is defined similar to  normal one, means somethin like

self.deltas[x] =  self.deltas[x+1].dot(self.weights[x].T) * self.layers[x] * (1- self.layers[x]) where

self.deltas[x+1] is error from prevous layer, self.weights[x] is weights map and self.layers[x](1- self.layers[x]) is bakwards activation of sigmoid function where self.layers[x] is vector of sigmoid. But while normal backpropagation the values are there, while BPTT i can not take the current self.layers[x] : i need the previous ones, right ?

So unlike normal BP, do i need extra store old weights and layers, for example in circular queue, and then apply the formula where self.deltas[x+1] is layer from next time ?

Not realy implementation, just basic understanding in order to can implement it.

Lets see the picture:

Here are : self.layers[0] = $x_{t+1}$, self.layers[1] = $h_{t+1}$ , self.layers[2] = $o_{t+1}$, in order to perform backprop $h_{t+1}$ -> $h_{t}$ -> $h_{t-1}$...  I DO NEED to have layers $h_t$ ,$h_{t-1}$... and weights $v_{t+1}$, $v_t$...  EXTRA stored in additional to the network $x_{t+1}$ -> $h_{t+1}$ -> $o_{t+1}$, right?
Thats all the question.

And i do not need to store previous outputs $o[t, o_{t-1}, etc..]$, because backprop from them ot->ht, etc was already calculated.

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

Notation of my explanation is different from notation in the figure of question.
My notation is as below:

V - Hidden Layer - Output Layer (gradients of V are independent of
previous time steps)
U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed slightly different notation than in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/enter link description here

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.


What kind of experience do I need to get into an AI master's programme? I'm about to go into my third and final year of my mathematics degree and I recently discovered some master's programmes in AI that really interest me. I have very little experience in programming. I've done some very basic programming in Python when I was doing my A-levels and some of my Uni modules so far included the use of MATLAB, Maple and R. I'm also going to be doing a module called Formal Languages and Theory of Computation this year, but I feel like this is not going to be enough to show my interest in the subject when I apply.

Can anyone advise on what sort of things I should be doing over next year in order to prepare myself for a course in AI and how do I go about doing them?

Thanks!

Well, mathematics is a very strong field in AI as that is what most AI algorithms revolve around. You only need to learn how to write code (Mostly python or c++, python is easier to learn in my opinion) and how to write mathematical equations in code. Once you have that down it should be enough but I don't how courses work at yours as I know our University require a degree in computer science.

If you want to impress somebody then I can pass you an old University assignment I had about backpropagation in neural networks. It does not involve any code but to determine the partial derivative of an equation that is used to update weights in a neural network. If you do the assignment correctly you should know the basics of how a fully connected neural network works, which is the cornerstone of most neural networks, and also how to train it through backpropagation using only maths. Not only would you know how neural networks work but you would have something to show for it. Witch hopefully could contribute to applying for your Masters.

Lastly, I know of people that wrote scientific papers in the realm of computer science, to prove they would survive in masters, and then applied for a Master's degree in Computer Science. If you are really serious about it you can do this but just double-check with someone as you might do all the work for naught.


How to add weights to one specific input feature to ensure fair training in the network? I am trying to create a multiclass product-rating network based on product reviews and other input features. Two of the other input features are "product category" and "gender". However, I want to avoid unfair bias in the classification task between male/female. Since some product categories are more likely to be reviewed by males or females (hence, not balanced), I am seeking for an approach to solve this "imbalance"-like issue.

The options and things that I consider at the moment are:

Downsample the training examples in each product category to balance for gender
Add weights to the training examples for gender, or
Add weights to the loss function (either log-likelihood or cross-entropy)

Even though downsampling might be the easiest option, I would like to explore the options of adding weights in the network in some way. However, most literature are only discussing adding weights to the loss function in order to solve for imbalanced data related to the target value (which is not the issue that I am addressing).

Can someone help me or point me in the right direction to solve this challenge?

I think your approach to tackle this as an imbalanced problem is correct. The easiest thing you could do is to add weights to the samples, during training, so that the model "pays more attention" to the under-represented class.

There are also a couple of other ways for this to be done: oversampling and undersampling, but initially, I'd focus on adding weights, since its easier to implement.


How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network? Consider the following loss function

$$
L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
$$

where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are represented as neural networks, where $w^-$ and $w$ are the corresponding weights.

But how do you calculate $max_{a'} Q(s', a', \mathbf{w^-})$? Do you really need to hold always an older version of the network? If yes, why and how old should it be?

You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.

The details depend a little on your neural network architecture:

If you have a network that takes the state vector as input and outputs all possible action values $(\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2) ...)$, then you can run it once forward with the next_state as input to getthe array $(\hat{q}(s',a_0), \hat{q}(s',a_1), \hat{q}(s',a_2) ...)$, and take the maximum element (you don't need to care which action $a'$ caused it). You will then have the problem that you now have a loss for $Q(s, a, \mathbf{w})$ for the single action $a$ just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate $Q(s, a, \mathbf{w})$ then substitute in this new estimated value against the action $a$ and train using this modified vector.
If you have a network that takes the state vector and action combined as input and outputs a single estimate $\hat{q}(s,a)$, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination.

Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.

  do you really need to hold always an older version of the network. If yes why and how old should it be?

You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.

How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.

One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. $\mathbf{w}^{\_} = (1 - \beta)\mathbf{w}^{\_} + \beta \mathbf{w}$ where $\beta$ might be $0.001$

In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.


What is the weight matrix in self-attention? I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.
From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,
The query is the dot product of the query weight matrix and the word vector,
ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?

The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.
This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.


What programmable devices can be used to teach artificial intelligence? Could you give examples of affordable programmable devices that could be used in university classes to teach students about A.I. and demonstrate it?

The devices are expected to do some form of self-learning, pattern recognition, or any other features of A.I., and to be programmable or customizable.

LEGO Mindstorms is widely used to demonstrate AI in schools and universities [1, 2]. With LEGO as basis, you are very flexible. You can build what you want very easily. The AI programs can be written in different languages from very easy graphical once to Lisp and C++. The newest version has an SD Card drive, USB interface and a powerful ARM processor. You can use four motors and four sensors directly. There exists touch, sound, sonar, gyro, infrared and color sensors. There is also a big community which provides you with a lot of ideas, hardware and programs [3].


How can I implement an AI system that learns to recognise a context-free grammar? Suppose I have context-free grammar (CFG) $L$ that pumps out words in that language. I want a machine learning system that can detect if a word $w$ came from $L$ or not. It has access to a stream from $L$ that is constantly producing words in $L$ at random. In this case, the system can only be trained with positive examples. The system also has access to a verifier for $L$. The system (if it chooses to) can generate strings and verify that the string is in $L$ or not. However, the system could just cheat and just use the verifier, so during the test phase, it gets disabled (during the learning phase, it uses the verifier as much as it wants). Moreover, $L$ can be augmented and changed, and the system should adapt to that change.

For instance, let's consider this regex (a regex is a CFG): $$(aa)*$$

This language only produces a string of $aa$'s of even length. Suppose I modified the regex to $(aa|bb)*$. This system should adapt to recognize this new language.

What kind of methods/approaches should I consider in my design?

The process of automatically learning a grammar from examples of a language is called grammar induction. Since you mention that L can be "augmented and changed", it might be feasible to solve this problem using an adaptive parser.


Number of classes vs number of parameters/layers? How to estimate the number of parameters in CNN for object detection?
I know that there are some well-known architectures that was trained on a lot of data (AlexNet, ResNet, VGG, GoogleLeNet). But they were trained for example for classifying 1000 classes. Or they were used as backbones in the algorithms like YOLO to localize 80 classes of objects.
Now let's say that I want to classify only 5 classes. Or I want to perform object detection and I am interested only in cars and people. I want to detect/classify this small number of objects. So the network must learn only the features of cars and people (instead of learning the features of hundreds of objects).
So my intuition is that I can use smaller network with fewer number of parameters. Correct me if I am wrong. And my second intuition is that the number of layers should not have a big impact. I mean, you shouldn't decrease the number of layers only because you have less classes. Because the network learns more and more sophisticated features in deeper layers. And it wouldn't be able to detect advanced features of cars (or other objects) if you don't have enough layers.
Recently I tried to use CenterNet https://arxiv.org/abs/1904.07850 to detect digits on 64x64 grayscale images and I achieved success having quite simple 900k convnet. Then I tried to use slightly modified GoogLeNet to detect cars using 224x224, 448x448, and 512x512 images. I trained it on 450 images. After a lot of trials and errors I still cannot train a good model. GoogLeNet is quite small network in compare to other well-known architectures, but I heard that it's very good. It was carefully designed to be very powerful despite being small (7M parameters).
So to be clear. My question is about the dependencies between the number of classes and the number of layers and parameters.

You are correct about your first assumption, but not about your second assumption.
More layers does not always mean better pattern detection. The analogy that in the deeper layers the network learns more complex features is an oversimplified one. It is true to some extent, although it is not enough to explain very complex architectures like GoogLeNet.
Moreover, using more layers than neccessary increases the risk of observing vanishing gradients. Because the network is too deep i.e. has too many layers, the gradient will be vanishingly small when backpropagating so many layers in the beginning will not get any significant update during training.


How can I remove the noise from an EEG signal? I am working on a project that takes signals from the brain, preprocesses them, and then makes the machine learn about what human is thinking about. I am struck on preprocessing the signal (incoming from the EEG). I am having a problem when I attempt to remove noise. I used SVM but to no avail. I need some other suggestions from experts who have worked on a project similar to this. What can I do to preprocess the signal?

This might be more of a signal-processing question, rather than a artificial intelligence question, but I will try my best to be of help.

Do you know what the noise you are trying to remove is? How it behaves/where it stems from?
Or, do you know how your output signal should look, post processing?

If you know these things and you are familiar with MATLAB or any other matrix multiplication software, they come with great prebuilt toolboxes for traditional approaches to remove noise from signals.

If you are not exactly sure what patterns you are looking for, I suggest perhaps looking into Autoencoders to discover the hidden patterns. Though it is important to note that the origin of the noise may greatly effect its abilities.
If you plan on using such a technique it is important that you have a sufficiently large dataset of the signals available.

Without the clarifications to these questions, along with @nbro's questions, it is hard to be more specific.


Autoencoder network for feature selection not converging I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For optimization, I am using the ADAM optimizer. Moreover, to improve convergence, I have also introduced learning rate decay. The model shows good convergence initially, but later starts to generate losses (12 digits) in the same range of values for several epochs, and is not converging. How to solve this issue?

The trick was to normalize the input dataset values with the respective mean and standard deviation in each column. This reduced the loss drastically, and my network is training more efficiently now. Moreover, normalizing the data also helps you calculate the weights associated with each input node more easily, especially when trying to find out variable importance.


Convolutional Neural Network: does each filter in each convolution layer create a new image? Say I have a CNN with this structure:

input = 1 image (say, 30x30 RGB pixels)
first convolution layer = 10 5x5 convolution filters
second convolution layer = 5 3x3 convolution filters
one dense layer with 1 output

So a graph of the network will look like this:

Am I correct in thinking that the first convolution layer will create 10 new images, i.e. each filter creates a new intermediary 30x30 image (or 26x26 if I crop the border pixels that cannot be fully convoluted).

Then the second convolution layer, is that supposed to apply the 5 filters on all of the 10 images from the previous layer? So that would result in a total of 50 images after the second convolution layer.

And then finally the last FC layer will take all data from these 50 images and somehow combine it into one output value (e.g. the probability that the original input image was a cat).

Or am I mistaken in how convolution layers are supposed to operate?

Also, how to deal with channels, in this case RGB? Can I consider this entire operation to be separate for all red, green and blue data? I.e. for one full RGB image, I essentially run the entire network three times, once for each color channel? Which would mean I'm also getting 3 output values.

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.

About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.


What is the computational complexity of the forward pass of a convolutional neural network? How do I determine the computational complexity (big-O notation) of the forward pass of a convolutional neural network?
Let's assume for simplicity that we use zero-padding such that the input size and the output size are the same.

What is the time complexity?
The time complexity of an algorithm is the number of basic operations, such as multiplications and summations, that the algorithm performs. The time complexity is usually expressed as a function of the input's size $n$ (but this does not always have to be the case: for instance, you can express the time complexity as a function of the output's size).
Example
Rather than giving you a full answer to your question, I will try to help you by explaining, with the simplest example, how you should calculate the time complexity.
For simplicity, let's assume that we have a kernel $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ and input image $\mathbf{I} \in \mathbb{R}^{3 \times 3}$ (i.e. the kernel has the same dimensions as the input), we use a stride of $1$ and no padding. If we convolve $\mathbf{I}$ with $\mathbf{H}$, how many operations will we perform? The convolution is defined as a scalar product, so it is composed of multiplications and summations, so we need to count both of them. We have $9$ multiplications and $8$ summations, for a total of $17$ operations.
\begin{align}
\mathbf{I} \circledast \mathbf{H}
&=
\begin{bmatrix}
i_{11} & i_{12} & i_{13} \\
i_{21} & i_{22} & i_{23} \\
i_{31} & i_{32} & i_{33}
\end{bmatrix}
\odot
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & h_{33}
\end{bmatrix}\\
&=
\sum_{ij}
\begin{bmatrix}
i_{11} h_{11} & i_{12} h_{12} & i_{13} h_{13} \\
i_{21} h_{21} & i_{22}  h_{22} & i_{23} h_{23} \\
i_{31} h_{31} & i_{32} h_{32} & i_{33} h_{33}
\end{bmatrix}\\
&=
i_{11} h_{11} + i_{12} h_{12} + i_{13} h_{13} +
i_{21} h_{21} + i_{22}  h_{22} + i_{23} h_{23} +
i_{31} h_{31} + i_{32} h_{32} + i_{33} h_{33}
\end{align}
Time complexity
What is the time complexity of this convolution? To answer this question, you first need to know the input's size, $n$. The input contains $9$ elements, so its size is $n = 9$. How many operations did we perform with respect to the input's size? We performed $17$ operations, so the time complexity $\mathcal{O}(2*n) = \mathcal{O}(n)$, i.e. this operation is linear. If you are not familiar with the big-O notation, I suggest that you get familiar with it, otherwise, you will not understand anything about computational complexity.
To calculate the time complexity in the case the input's dimensions are different than the kernel's dimensions, you will need to calculate the number of times you slide the kernel over the input. You can't ignore this (as I ignored the constant $2$ above) because the number of times you slide the kernel over the input depends on the input's size, so it's a function of the input. Anyway, the paper A guide to convolution arithmetic for deep learning contains a lot of information about convolution arithmetic, so it will be helpful.
Non-linearities, pooling and fully connected layers
Note that, in the above example, I ignored the non-linearities and pooling layer. You can easily extend my reasoning to include these operations too. I also ignored the operations in the final fully connected layers. You can find how to calculate the number of operations in an MLP in this answer. If you also perform other operations or have other layers other than convolutional, pooling and fully connected, of course, you will also need to take them into account.
Forward pass
Moreover, the time complexity of the forward pass of a CNN depends on all these operations in these different layers, so you need to compute the number of operations in each layer first. However, once you know how to compute the number of operations for one convolutional layer, one pooling layer, and one fully connected layer, you can easily compute the number of operations for the other convolutional, pooling, and fully connected layers. Then you just need to sum all these operations and express your time complexity as a function of the input (and probably number of layers).
Space complexity
If you also want to compute the space complexity, you just need to do the same thing, but as a function of the space that you use, i.e. how many variables you use to perform the convolution.


Is there any computer vision technology that can detect any type of object? Is there any computer vision technology that can detect any type of object? For example, there is a camera fixed, looking in one direction always looking at a similar background. If there is an object, no matter what the object is (person, bag, car, bike, cup, cat) the CV algorithm would notice if there is an object in the frame. It wouldn't know what type of object it is, just that there is an object in the frame.

Something similar to motion detector but that would work on a flat conveyor belt. Even though the conveyor belt moves will look similar between frames. Would something like this be possible? Possibly something to do with extracting differences from the background, with the goal being to not have to train the network with data for every possible object that may pass by the camera.

TL;DR

This is possible. You need a correctly labeled dataset. Your dataset has two labels:

$y\in \{\text{background},\text{object in frame}\}$ or simply $y\in \{0,1\}$

This labelling avoids needing to know what object is in frame only that there is an object in frame.

Examples With Similar Objectives

Here as a paper (link) that was seeking to classify animals in a frame. The first part of their pipeline needed to know whether there was an animal in frame or not. Excerpt:

  Task I: Detecting Images That Contain Animals. For this task, our models take an image as input and output two probabilities describing whether the image has an animal or not (i.e. binary classification)

Here is another paper (link) regarding detecting animals in images. Excerpt:

  Wildlife detection, which is actually a binary classiﬁer capable of clasifying input images into two classes: “animal” or “no animal”

The methods used in these papers could be adapted to fit the conveyor belt use case.

Here is a system (link) that detects moving objects using background extraction. The method they use to check whether or not the image is a background image or not could be adapted to your use case.

Ideas for Implementation

In the conveyor belt use case, a very simple model might work. Thus, it is recommended that you try a simple model first. For example, you could flatten your inputs to a vector and feed these into a shallow feed forward network. A simple logistic regression classifier might even work on flattened inputs for this use case.

If your desired performance is not reached with a simple model you can analyze their performance using methods like learning curves to decide how to proceed.

If you find that your model has high bias:

You can either use a larger/deeper model like a custom tuned CNN or you can use a pre-trained network like VGG16 and  use transfer learning to achieve your goal.

Here is a post that describes using a CNN to classify cats vs dogs (link). Simply replace the cat/dog dataset with your object/no-object dataset. The post even describes using transfer learning with VGG16.

If you find that your model has high variance

More data can help. Furthermore, in the conveyor belt example, hand labeling data would be easy and quick. Thus, it would not take very much time to get many more examples.

I hope this helps.


Understanding the reconstruction loss in the paper "Anomaly Detection using Deep Learning based Image Completion" I would like to implement the approach represented in this paper. Here they used following reconstruction loss:

$$
L(X)= \frac{\lambda \cdot || M \odot (X - F(\overline{M} \odot X)) ||_{1} + (1 - \lambda) \cdot || \overline{M} \odot (X - F(\overline{M} \odot X)) ||_{1}}{N}
$$

Unfortunately, the author does not explain the function $F$.
Does someone know a similar function or could understand the function's purpose from the context?

$F$ in this context is the output of the Convolutional Neural Network that's being trained, which is of the same size as $X$.


How can I train a neural network for another input set, without losing the learning of the previous input set? I read this tutorial about backpropagation.

So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as output. And the neural network is adjusting its weight values accordingly. So once after the neural network is trained this way, suppose we are giving another input set, say [6,8], also then will the neural network update its weight values (overwriting previous values), right? This will result in losing the previous learning, right?

Yes, this is actually a limitation known as catastrophic forgetting.
A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on the weights important for those tasks". See Overcoming catastrophic forgetting in neural networks for details. Another approach is Learning without forgetting.

If the tasks are different, the approach you are talking about is called transfer learning. You might want to have a look at multi-task learning as well

If the tasks are the same, you could try creating a join of both datasets and training on that.


Can YOLO detect large objects? I have a rather basic question about YOLO for bounding box detection.
My understanding is that it effectively associates each anchor box to an 8-dimension output.
During testing, does YOLO take each anchor box and classify on it alone? What happens if the object is big and spans over several anchor boxes (e.g., covering 70% of the image)? How can YOLO classify and detect objects spanning over many anchor boxes?

The example given based on the yolov1 paper:

The last layer has a tensor of the dimension 7x7x30. but the dimension of the last tensor is not in every case 7x7x30.

let be:
S: the number of grid cells in X and Y direction
C: the number classes to train
B: the number of bounding boxes in every grid cell

The dimension of the output tensor is calculated with this formula: SxSx(5*B+C). The given example in their paper has the following values:
S:7
B:2
C:20
(7X7X(5*2+20) = 7x7x30)

With this configuration you can detect at most 96 object (SSB -> 7*7*2) and 2 objects per grid cell. (e.g. it's not possible to detect many small object in a few grid cells)

now lets consider what are each feature map of the last tensor used for:

The data of every bounding box is stored in 5 feature maps:
- 1 Feature map for the bounding box center in x direction
- 1 Feature map for the bounding box center in y direction
- 1 Feature map for the height of the bounding box
- 1 Feature map for the width of the bounding box
- 1 Feature map for the "confidence score" of the feature map

(confidence score = P(Object)*IoU(BoundingBox,Object))

P(Object): the probability that in this cell is an object
IoU: Intersection over Union. A ration between the overlapping area and the union area:

additional there are a feature map for every class. In every cell the following probability is calculated: P(Cass|Object)

Long story short:
If the bounding box is bigger than one grid cell, then some neurons share the same values (they share the bounding box center, the bounding box size and the confidence score, because they refer to the same bounding box).

Source:
https://pjreddie.com/media/files/papers/yolo_1.pdf

edit no Bounding boxes on the image

if there are no bounding boxes on the image, then the confidence score would be zero, because there are no overlapping area. you can configure the threshold of the confidence score. look at this sample picture with threshold 0:

additionally the feature maps of the classes would be zero because every neuron represents this probability P(Class|Object) because P(Class|Object) = P(Class And Object) / p(Object); (P(Class) = 0 -> P(Class and Object) = 0)

Is adding images without bounding boxes to the training set useful?
I think it helps to lower the number of false positive matches. But that is just a guess. I have trained yolo to detect this plant https://en.wikipedia.org/wiki/Rumex_obtusifolius and I added a lot of picture without bouding boxes, becuase a low number of false positive matches was important. The result: the specificity was >99%.

Hope this helps


How to use pretrained checkpoints of BERT model on semantic text similarity task? I am unaware to use the derived checkpoints from pre-trained BERT model for the task of semantic text similarity.

!python create_pretraining_data.py \
          --input_file=/input_path/input_file.txt \
          --output_file=/tf_path/tf_examples.tfrecord \
          --vocab_file=/vocab_path/uncased_L-12_H-768_A-12/vocab.txt \
          --do_lower_case=True \
          --max_seq_length=128 \
          --max_predictions_per_seq=20 \
          --masked_lm_prob=0.15 \
          --random_seed=12345 \
          --dupe_factor=5

!python run_pretraining.py \
      --input_file=/tf_path/tf_examples.tfrecord \
      --output_dir=pretraining_output \
      --do_train=True \
      --do_eval=True \
      --bert_config_file=/bert_path/uncased_L-12_H-768_A-12/bert_config.json \
      --init_checkpoint=/bert_path/uncased_L-12_H-768_A-12/bert_model.ckpt\
      --train_batch_size=32 \
      --max_seq_length=128 \
      --max_predictions_per_seq=20 \
      --num_train_steps=20 \
      --num_warmup_steps=10 \
      --learning_rate=2e-5

I have run a pre-trained BERT model with some domain of corpora from scratch. I have got the checkpoints and graph.pbtxt file from the code above. But I am unaware on how to use those files for evaluating semantic text similarity test file.

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d

You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one


About the definition of "soft label" and "hard label" In semi-supervised learning, there are hard labels and soft labels. Could someone show me what's exactly the meaning of the two things?

According to Galstyan and Cohen (2007), a hard label is a label assigned to a member of a class where membership is binary: either the element in question is a member of the class (has the label), or it is not.

A soft label is one which has a score (probability or likelihood) attached to it. So the element is a member of the class in question with probability/likelihood score of eg 0.7; this implies that an element can be a member of multiple classes (presumably with different membership scores), which is usually not possible with hard labels.

One use of soft labels in semi-supervised learning could be that the training set consists of hard labels; a classifier is trained on that using supervised learning. The classifier is then run on unlabelled data, and adds soft labels to the elements. This enlarged data set is then used for further training, where the algorithm can treat hard and soft labels differently.


How to add more features to the input of a machine learning algorithm? I am trying to perform a binary classification of tweets using machine learning.

The usual way of doing this seems to be putting a hand-classified tweet's words into a big vector, then use that vector as input to an algorithm, which then predicts new tweets based on this data.

Is there a standard method, or algorithm, that can include other input, such as the location of a tweet, in this process?

I could just add tweet location at the end of the vector, I suppose, but that would give it a very small weighting.

Any pointers are much appreciated.

Data pre-processing and feature extraction are by far the most important part of any machine learning algorithm. It's even more important that the model you choose to do the classification.

Unfortunately, pre-processing and feature extraction are completely different for each type of data. You need to play around with the data yourself to find out what works best with the nature of your data. With experience you start noticing some patterns with different data types. For example, as you are doing, building a word vector is an effective means of feature extraction with text based data.

"I could just add tweet location at the end of the vector I suppose, but that would give it a very small weighting."

This is entirely untrue for any machine learning algorithm I would choose. Your model should not associate weights to the inputs based on their array location. They should be associated based on the explained variance (information gain) it provides.

After you do your pre-processing and feature extraction you can then further refine your feature set with some common methods which can be found in libraries such as: principle component analysis (PCA) and linear discriminant analysis (LDA).


Too small gradient on large neural network When training on large neural network, how to deal with the case that the gradients are too small to have any impact?

FYI, I have an RNN, which has multiple LSTM cells and each cell has hundreds of neurons. Each training data has thousands of steps, so the RNN would unroll thousands of times. When I print out all gradients, they are very small, like e-20 of the variable values. Therefore the training does not change the variable values at all.

BTW, I think this is not an issue of vanishing gradients. Note that the gradients are uniformly small from the beginning to the end.

Any suggestion to overcome this issue?

Thank!

I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.


Is a multi-layer Kohonen network possible? The Kohonen network is one fully connected layer, which clusters the input into classes by a given metric. However, the one layer does not allow to operate with complex relations, that's why deep learning is usually used.

Is it possible then to make multi-layered Kohonen network?

AFAIK, the output of the first layer is already cluster flags, so the activation function on the non-last layers must be different from the original Kohonnen definition?

Kohonen networks by definition are single layer FCNN's, but what differentiates them from others is their unsupervised training procedure.

This procedure is a function of the input, the weights and some hyperparameters. This means if you have a multilayer network you could train only the final layers weights using this procedure. Think of it this way, let $f$ be the final layer, and $g$ be the composition of all other layers, such that the whole network $N(x) = f \circ g(x)$. Using kohonens training procedure, you could learn $f$'s weights where the input is $g(x)$ rather than x. But then how would you learn $g$'s weights?

So you could do this by iterative learning. let $N$ be a 2 layer network: $N = f_2 \circ f_1$. first learn $f_1$ using the single layer procedure, and now you could learn $f_2$ in the same manner, except the input features its trying to cluster would be $f_1(x)$ rather than $x$.

There does exist cons to this procedure though: Using a kohonen network at each layer will try to cluster it to the best of its abilities, not so that its final composition will be best (a common problem in many optimization procedures that arent end-to-end), but that its current representation is. This may lead to non-optimal results, not achieving that deep representation that you're looking for that you could achieve in autoencoders or other deeper unsupervised models.


Are there any ongoing projects which use the Stack Exchange for machine learning? Are there any ongoing AI projects which use the Stack Exchange for machine learning?

There certainly appear to have been research projects involving some form of text mining / information retrieval /etc. and StackExchange sites.

Some examples I was able to find through google/google scholar (unlikely to be anywhere near an exhaustive list):

TACIT: An open-source text analysis, crawling, and interpretation tool describes numerous text-crawlers for a variety of sites (including Stack Exchange sites, but also Twitter, Reddit, etc.). At first glance, this appears to be primarily about crawling, not about doing anything else with the data afterwards. Searching for other papers that cite this one on Google Scholar may yield interesting results though, it may lead to papers that used this for crawling and did more with the data afterwards.
Chaff from the Wheat : Characterization and Modeling of Deleted Questions on Stack Overflow describes research into the quality of Stack Overflow questions in some sense (specifically, predicting whether questions will get deleted for example). I'm not 100% sure if this is also the kind of stuff you're interested in; it is Stack Exchange + Machine Learning as implied by the title of your question, but not necessarily about retaining information from answers as implied by the text in your question.
Text mining stackoverflow: An insight into challenges and subject-related difficulties faced by computer science learners also describes text mining in StackOverflow questions and answers, though at a very quick glance it appears to be primarily about topic detection etc., not necessarily about automated question answering for example.
Different Facets of Text Based Automated Question Answering System appears to be a relatively recent survey on the topic of Automated Question Answering research. Stack Exchange is mentioned a few times as an example of a source of data for such systems, but doesn't appear to be used otherwise.
Extending PythonQA with Knowledge from StackOverflow is specifically about incorporating Questions and Answers from StackOverflow in an automated Question and Answering system for questions about the Python programming language. The paper provides a link to more details (http://pythonqas2.epl.di.uminho.pt), but that link appears to be down. I suppose you could always try contacting the authors directly if you're interested in more information on this.

More generally, Automated Question Answering systems appears to be a rather active area of research still, not a trivial / "solved" problem. StackExchange can be one source of data for such systems, but there are plenty of other sources of data too (Wikipedia, Quora, etc.).


How does Google's self-driving car identify pedestrians? Based on the article Google's self-driving cars can now spot cyclists: Sensors can read hand signals and predict rider's behaviour, Google's self-driving cars can spot cyclists, cars, road signs, markings, traffic lights, and pedestrians.

How exactly does it identify pedestrians? Is it based on face recognition, shape, size, distance, infrared signature?

The AI of the car uses sensor data to process all the data and classifies objects based on the size, shape and movement patterns. It can recognize surroundings from a 360 degree perspective by making predictions about vehicles, people and objects around it will move.

It can detect pedestrians, but as moving, column-shaped blurs of pixels, so it really cannot tell whether it's a rock or a crumpled piece of paper.

However it is programmed to determine certain patterns when a police officer has halted traffic or the car is being signaled to move forward.

It also recognizes cyclists as objects outlined in red and can slow down to let the cyclist enter into a lane.

Above images are provided by Chris Urmson who heads up Google's driverless car program.

Sources:

How Google's self-driving cars see the world
Hidden Obstacles for Google’s Self-Driving Cars
(video) Chris Urmson: How a driverless car sees the road


What is the use of the $\epsilon$ term in this back-propagation equation? I am currently looking at different documents to understand back-propagation, mainly at this document. Now, at page 3, there is the $\epsilon$ symbol involved:

While I understand the main part of the equation, I don't understand the $\epsilon$ factor. Searching for the meaning of the $\epsilon$ in math, it means (for example) a error value to be minimized, but why should I multiply with the error (it is denoted as E anyways).

Shouldn't the $\epsilon$  be the learning rate in this equation? I think that would be what makes sense, because we want to calculate by how much we want to adjust the weight, and since we calculate the gradient, I think the only thing that's missing is the multiplication with the learning rate. The thing is, isn't the learning rate usually denoted with the $\alpha$?

The change in nomenclature from what you expected is Lisa Meeden's choice, for unknown reasons. Those with whom she published in the past used $\epsilon$ to represent error, the result of a loss function. Why she did not use $\alpha$ may be because the letter $a$ was used elsewhere in the formula, but, if that was the reasons, that wasn't a great one.

Changes in nomenclature between articles is common, for instance between Carnegie Mellon literature and Cal Tech. Robotics control systems people use different nomenclature from those who write game competition systems. The online mappings between choice of Greek letters and their meaning cannot be relied upon. This is not particular to AI. I've seen it in science and engineering literature, even between text books in courses that are part of the same university curriculum.


How does an episode end in OpenAI Gym's "MountainCar-v0" environment? I am working on OpenAI's "MountainCar-v0" environment.  In this environment, each step that an agent takes returns (among other values) the variable named done of type boolean. The variable gets a True value when the episode ends. However, I am not sure how each episode ends. My initial understanding was that an episode should end when the car reaches the flagpost. However, that is not the case.
What are the states/actions under which the episode terminates in this environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.


How many trees should be generated in a random forest? What are ways of determining the number of trees to be generated in a random forest algorithm?

The number of estimators in Random Forest is a hyper-parameter. If you are using SKLearn's Random Classifier you can use one of the following techniques to find a (near) optimal hyperparameter settings (Note:You can tweak other hyperparameters like min_leaf_size etc as well with this approach);

GridSearchCV You can specify a grid of all the hyperparameters and a scoring criteria. This function will then evaluate all combinations of these parameters for you and return the setting which performed best on the validation set.

RandomSearchCV You can specify a grid of all the hyperparameters and a scoring criteria. This function will then evaluate n randomly choosen combinations of these parameters for you and return the setting which performed best on the validation set.

Bayesian Optimization You can treat the hyperparameter settings and corresponding score as a black box function and use exploitation-exploration paradigm using bayesian optimization.


Open-source tools or libraries to experiment with neural networks and evolutionary algorithms I'd like to do some experimenting with neural net evolution (NEAT). I wrote some GA and neural net code in C++ back in the 90s just to play around with, but the DIY approach proved to be labor-intensive enough that I eventually dropped it.
Things have changed a lot since then, and there are lots of very nice open-source libraries and tools around for just about any interest one might have. I've Googled different open-source libraries (e.g. DEAP), but I could use some help choosing one that would be a good fit.

I spent much of my time writing code to visualize what was going on (neural net state, population fitness) or final results (graphs, etc).
Maybe this would have to be fulfilled by a separate open-source library, but visualization support would be something that would allow me to spend more time on the problem/solution and less on implementation details.

I know C/C++, Java, C#, Python, Javascript and a few others. Something that's a nice trade-off between a higher-level language and good performance on home hardware would be a good choice.

Can someone with experience suggest a good open-source library or set of tools?

https://github.com/wagenaartje/neataptic

As this is written in Javascript and does not (yet) offer GPU support, it is quite slow. However, it is very nice to fiddle around with flexible network architectures. The only visualisation that it offers right now is a map of network architecture, but graphs could easily be implemented.


What model structure I should use to train on low res and blurry images? I am looking for advice or suggestion.
I have photos like these: photo_1 and photo_2 and many more similar to that. The average shape of these photos is about 160 x 100. What we are doing is we are trying to find wheather or not person in a photo is wearing safety vest and helmet (if person is wearing both it is 1, if something is missing or both are missing it is 0). Training data consists of about 5k almost equally distributed image sets. I have tried to use augmentation techniques (flipping, adding noise, brighness correction) but results didn't improove.
I tried to train on many pretrained popular models: resnet101, mobilenet_v2, efficientneyb3, efficientneyb0, DenseNet121, InceptionResNetV2, InceptionV3, ResNet152V2, ResNet50V2, but results are not eyepleasing. I have tried different input sizes ranging from 224x224 to 112x112 but result didn't improve as much as I would have liked it to be. And the weird thing is that the image shape does not correlate to wheather or not there are more wrong predictions using bigger or smaller images.
As a side not I would lik to ask couple questions:

Should I use my own written small net?
Are the models that I use too big for this problem?

Any advice will be appreciated.

Yes, you may use the small model you have but you need to tune it.
The models you have used are not too big if any overfitting happens you can just use dropout to alleviate that.
You need to improve the quality of your images. One way is to reconstruct them using autoencoders or GANs (Super-resolution GANs). Then, you can classify them.


Factors that causing totally different outcomes from an exactly same model and datasets Here is a model that trains time series data in (batch, step, features) way.
I have kept the random state for train test split function the same. Every parameter below the same, running the model training yields different outcomes every time and the outcomes are drastically different.
What may be the factors that led to this? Regularization?

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=666)
def attention_model(X_train, y_train, X_test, y_test,num_classes,dropout=0.2, batch_size=68, learning_rate=0.0001,epochs=20,optimizer='Adam'):

    Dense_unit = 12
    LSTM_unit = 12

    attention_param = LSTM_unit*2
    attention_init_value = 1.0/attention_param

    u_train = np.full((X_train.shape[0], attention_param),
                      attention_init_value, dtype=np.float32)
    u_test = np.full((X_test.shape[0],attention_param),
                     attention_init_value, dtype=np.float32)

    with keras.backend.name_scope('BLSTMLayer'):
        # Bi-directional Long Short-Term Memory for learning the temporal aggregation
        input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
        x = Masking(mask_value=0)(input_feature)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)

        y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.000029),kernel_regularizer=l2(0.027),recurrent_regularizer=l2(0.025),return_sequences=True, dropout=dropout))(x)
#         y = Bidirectional(LSTM(LSTM_unit, kernel_regularizer=l2(0.01),recurrent_regularizer=l2(0.01), return_sequences=True, dropout=dropout))(y)

    with keras.backend.name_scope('AttentionLayer'):
        # Logistic regression for learning the attention parameters with a standalone feature as input
        input_attention = Input(shape=(LSTM_unit * 2,))
        u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)

        # To compute the final weights for the frames which sum to unity
        alpha = dot([u, y], axes=-1)  # inner prod.
        alpha = Activation('softmax')(alpha)

    with keras.backend.name_scope('WeightedPooling'):
        # Weighted pooling to get the utterance-level representation
        z = dot([alpha, y], axes=1)

    # Get posterior probability for each emotional class
    output = Dense(num_classes, activation='softmax')(z)

    model = Model(inputs=[input_attention, input_feature], outputs=output)

    optimizer = opt_select(optimizer,learning_rate)
    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)

    hist = model.fit([u_train, X_train],
                     y_train,
                     batch_size=batch_size,
                     epochs=epochs,
                     verbose=2,
                     validation_data=([u_test, X_test], y_test))

#kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),
    return hist

batch_size= 150
#217
epochs = 1000
learning_rate = 0.00081
optimizer = 'RMS'
num_classes = y_train.shape[1]
dropout=0.22

tf.keras.backend.clear_session()

history = attention_model(X_train, y_train, X_test, y_test, num_classes,dropout = dropout,batch_size=batch_size, learning_rate=learning_rate,epochs=epochs,optimizer=optimizer
)

By default, Keras sets shuffle argument True, so you should set the numpy seed before importing Keras.
CPU
from numpy.random import seed
seed(25)
from keras.models import Sequential

GPU
tf.random.set_seed(seed)


Why is the state value function sufficient to determine the policy if a model is available? In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated:

  If a model is not available, then it is particularly useful to estimate action values (the values of state– action pairs) rather than state values. With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

However, I don't see how this is true in practice. I can see how it'd work trivially for discrete state and action spaces with deterministic environment dynamics, because we could compute $\pi(s) = \underset{a}{\text{argmax}}\ V(\text{step}(s, a))$ by just looking at all possible actions and choosing the best one. As soon as I think about continuous state and action spaces with stochastic environment dynamics, computing the $\text{argmax}$ seems to be become very complicated and impractical. For the particular case of continuous states and discrete actions, I think estimating an action value might be more practical to do even if a forward model of the environment dynamics is available, because the $\text{argmax}$ becomes easier (I'm especially thinking of the approach taken in deep Q learning).

Am I correct in thinking this way or is it true that if a model is available it's not useful to estimate action values if state values are already available?

With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

As soon as I think about continuous state and action spaces with stochastic environment dynamics, computing the $\text{argmax}$ seems to be become very complicated and impractical.

For stochastic dynamics the calculations would be more complex, but will often be quite tractable. It depends on size of the distribution, and ease of calculating probabilities to make the correct weighted sums. Instead of $|\mathcal{A}(s)|$ calls to $Q(s,*)$ you will need to make roughly $|\mathcal{S'}(s)| \times |\mathcal{A}(s)|$ calls to $V(s')$ where $\mathcal{S'}(s) \in \mathcal{S}$ are all possible states that might result from the starting state. In the worst case this is $|\mathcal{S}| \times |\mathcal{A}|$
Despite the additional work here, you have been more efficient earlier. By only evaluating $V(s)$, you have removed the action dimension, which otherwise splits up your estimates. Your value function estimates will, all else being equal, converge faster because of this. So this is sometimes a compromise you might be willing to make.
For continuous states, this might still be practical, if both action choices and possible transitions are still discrete.
Once either action space is continuous, or transitions are continuous over some probability density function, then finding the maximising action via the model becomes impractical.