What is the relation between Monte Carlo and model-free algorithms? Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D points inside the square and count the number of points inside and outside the circle.

In reinforcement learning, an MC method is a method that "samples" experience (in the form of "returns") from the environment, in order to approximate e.g. a value function.

A temporal-difference algorithm, like $Q$-learning, also performs some form of sampling: it chooses an action using a stochastic policy (e.g. $\epsilon$-greedy) and observes a reward and the next state.  So, couldn't $Q$-learning also be considered a MC method? Can an MC method be model-based?

In Reinforcement Learning (RL), the use of the term Monte Carlo has been slightly adjusted by convention to refer to only a few specific things.

The more general use of "Monte Carlo" is for simulation methods that use random numbers to sample - often as a replacement for an otherwise difficult analysis or exhaustive search.

In RL, Monte Carlo methods are generally taken to be non-bootstrapping sample-based approaches to estimating returns. This is a labelling convention within RL - probably because someone called an initial model-free learner a "Monte Carlo method", and the name stuck whilst many refinements and new ideas have since been published under different names.

The historical use of the term is important, since if you mention you are using "Monte Carlo Control", it usually means a very specific subset of methods within RL, to most readers.

  So, couldn't -learning also be considered a MC method?

In the general sense perhaps. The argument is perhaps stronger for it if the environment is being simulated on a computer for the agent to learn from.

However, if you start to unilaterally call Q-learning a MC method, you are probably going to just confuse people who have learned the conventions in RL.

  Can an MC method be model-based?

In general, yes, because the model can be sampled for planning, and a policy can also be separately sampled - so it is possible to run an MC method with or without a model - depending on whether you sample from the model taking "virtual actions" (for e.g. planning or refining your agent) or from the environment taking real actions. Many RL techniques blur the line between online learning and planning. For instance, using a simulated environment or historical data can be framed as planning for the real environment.

Monte Carlo Tree Search is an example of a model-based technique using the term "Monte Carlo" within RL frameworks. It is used famously within DeepMind's AlphaZero in order to refine a policy and value estimates during self-play.

  What is the relation between Monte Carlo and [other] model-free algorithms?

In the context of RL, Monte Carlo is presented as one way to estimate expected utility (or return) - by sampling from the environment and policy until the a complete trajectory is available:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{T-t} \gamma^k R_{t+1+k} | S_t = s ]$$

MC is contrasted with Temporal Difference (TD) approaches, such as Q-learning, which sample bootstrap estimates using the Bellman Equation:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1}) | S_t = s ]$$

The two approaches can be combined in various ways, including TD($\lambda$) methods. With  TD($\lambda$), if you set $\lambda = 0$ then the algorithm is identical to single-step TD learning, and if you set $\lambda = 1$ then it is very similar to a Monte Carlo method. Often setting it to some intermediate value is more efficient than either extreme.


Matrix Dimension for Linear regression coefficients While reading about least squares implementation for machine learning I came across this passage in the following two photos:

Perhaps I’m misinterpreting the meaning of beta  but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y}  would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted.

Can someone provide clarification?

Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:

Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?

Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, & Friedman.

I have used W as beta and Y_pred as Y_hat.

Apparently, as far as conventions go both feature vector X and weight vector W are assumed as column vectors. Although this is not important in your case but when we use Neural Nets, this is particularly important, and a weight vector for a layer is given by p*N_n where p is the number of features being input from the previous layer and N_n is the number of nodes in the layer. Check a NN structure and you will understand what I am saying.

As far as the special case in your current question, the book is correct, although comments say Y_predicted should be vector, it is not, since we are doing dot product between X and W (although it is represented in a vectorised form).

Also  how can W  have dimensions p*K (assuming it is not a NN), it means multiple solutions to the same problem? Instead it is X which will have dimensions p*K and W will have p*1. It means you have K training examples with p features.

If its a NN with K nodes then dimensions for X like p*N and W as p*K is totally possible. The only downside of this notation is that you have to again take a transpose of the result.

Here is the explanation of the notation used:

Notation for NN


How are the Bellman optimality equations and minimax related? Is the philosophy between Bellman equations and minimax the same?

Both the algorithms look at the full horizon and take into account potential gains (Bellman) and potential losses (minimax).

However, do the two differ besides the obvious on the fact that Bellman equations use discounted potential rewards, while minimax deals with potential losses without the discount? Are these enough to say they are similar in philosophy or is are they dissimilar? If so, then in what sense?

They have similar philosophies, in the sense that minimax and algorithms based on the Bellman optimality equations are used to solve optimization problems, but they are also different because they solve different problems.

Minimax (at least, the minimax version that I am aware of) is typically used to solve two-player games (e.g. chess, tic-tac-toe, etc.), while Bellman optimality equations (I assume you are referring to the Bellman equations that algorithms such as policy iteration are based on) do not assume the existence of two players (unless you consider the environment a player).


What are the approaches to teach AI to how to render html page based on its source code? I'm wondering, instead of implementing new web browsers over and over again with millions line of code which is very difficult to manage, would it be possible to use ANN or GA algorithm to teach it about the rendering process (how the page should look like)?

So as an input I would imaging the html source code, output is the rendered page (maybe in some interactive image like SVG, some library or something, I'm not sure).

The training data can be dataset of websites providing input source code and their rendered representation by using other browsers for the guidance as expected output.

Which approach would you take and what are the most challenging things you can think of?

The rendering process for browsers is very well defined, and has a very rigid definite ruleset where (virtually) every accountability is noted and handled. This is not optimal for Machine Learning, which works when we have a large pool of examples, and we don't know the ruleset; it will figure it out. Even if you were to train an Neural Network to process that input, there are several things you must account for:

1. Variance in data.

Not all webpages are equal in length or complexity, and making a neural network to generate output from HTML would produce garbage most of the time.

2. Training time.

The time it would take for a neural network to understand HTML tags, attributes, the DOM Tree, and each and every element, including new ones being added every few years, and how each one renders and behaves, would take an extremely long time, most likely several years on a fast computer, if it even were possible

3. Interactivity.

Web pages aren't just static, they change according HTML, CSS and JavaScript. Not only would you have to design your system to account for the rendering step, you would also make it have to understand the Turing Complete scripting language JavaScript, as well as the less complicated, but inherently intertwined with HTML, CSS stylesheet language. If you thought the rendering process was easy, try training a neural network to handle complicated scripting patterns.

4. New Standards

Not all HTML is equal, because of different standards. WHATWG began working on HTML5 in 2004, and browsers started to implement not long after. In 2004, there were very few examples of HTML5 sites to train your network to begin with. Sure, now it's standardized and every website uses it, but what about HTML6? When the first specification is released (probably 2017-2025), virtually no websites will use it, because no one will support it. Only when it finally becomes standard, probably in the late 2020s or early 2030s, will you have enough data to train your monstrous system of neural networks

As for AI in general, one could argue that browsers already use A.I. in their rendering process. They intelligently decide what to render (taking CSS into account), when in order to get the most efficient render time, they selectively use different JavaScript parsers on different sections of the code to optimize the speed, the whole system has been optimized on another ruleset to make rendering and interacting with a webpage as seamless and easy-to-use as possible. Your system will never be as good as what hundreds of humans have optimized over 20 years.

Trying to solve HTML rendering with Neural Networks is akin to trying to nail a nail with a screwdriver. It's just not going to work

Hope this was helpful!


Where can I find short videos of examples of RL being used? I would like to add a short ~1-3 minute video to a presentation, to demonstrate how Reinforcement Learning is used to solve problems. I am thinking something like a short gif of an agent playing an Atari game, but for my audience it would probably be better to have something more manufacturing/industry based.
Does anyone know any good sources where I could find some stuff like this?

You can take some ideas from this YouTube video .
In addition, you should consider that page which is about Deep Reinforcement Learning used in a game (Pong from Pixels) .


Game AI design for a multiplayer random board game? I'm writing an AI for a board game, and previously I would just create a value maximizing state machine and tune the factors one at a time.

However, the issues with this is getting apparent. My last AI did not look into the future, hurting it's long term chances, and manual tuning of weights is proving to be a chore.

I've looked into minimax algorithms, but with a non perfect information game and elements of random chance I'm not too confident it will be effective.

I've also looked into traditional neural networks, but evaluating board states is tricky and the game does not split into moves well.

The game I'm writing the AI for is Ticket To Ride, but I would appreciate tips for any board game with similar mechanics.

Many successful game-playing engines use some form of search to look ahead and plan the next move. This is possible even in stochastic environments by building a tree with probabilities assigned, or by simulating the randomness during planning and relying on lots of samples to get reasonable estimates of value.

There are many approaches possible. However, as Ticket to Ride is a relatively simple game that you could automate to play to the end very quickly (1000s of times per second), then you could look into Monte Carlo Tree Search (MCTS) as a mechanism for the game-playing agents to look ahead and adjust their play to current odds.

The basics of Monte Carlo Tree Search:

On its turn, the agent builds up an evaluation tree local to its current state - each node of the tree is a game state, augmented with statistics from MCTS (how often node was chosen, how many times agent managed to win eventually from that node) and each link is a choice that a player makes. This is built up slowly, typically just adding one node per full simulated game.
For look-ahead search "inside" the tree, the agent explores options according to their relative promise, selecting moves to simulate according to its best estimates of value at the time. It needs to explore alternatives to make sure it doesn't miss anything, and there are a few variations of doing this. One successful approach to action selection inside the tree is Upper Confidence Bound applied to Trees (or UCT for short).
When play gets to a leaf node of the tree, the tree might be expanded slightly, and then afterwards the agent simulates plays of the game either to the end or to a robust evaluation point. This later play might be entirely random - or might still be guided probabilistically by some heuristic - and is called a rollout. The important thing here is to play quickly to get samples just to roughly assess the position at the leaf of the tree.
The evaluation resulting from rollout is fed back into the tree, which builds up sampled statistics of the most promising variations of play.
After a certain number of repeats of the whole algorithm (local tree search, expanding the tree, rollout and update), the agent picks the most promising next step from the tree. At this point it may just discard the whole tree - in a game with lots of randomness, this is probably what you would do now, as things you were considering as just probabilities before will have become historical fact, and the chances are that re-running the tree build from the next start position will be more accurate.

There are a few variants possible, including combinations with neural networks as used in AlphaZero.

The basic principle is to sample from many choices, focusing over time on what looks "best" to each player based on statistics of those choices.

As Ticket To Ride is a card game where cards can be drawn from an unseen deck, and it would be considered cheating to simulate and rollout with the actual deck in play, you would need to have realistic shuffles of the remaining unknown cards used during rollout. It may still work quite well without the realistic part and for speed just assume a random unlimited deck of cards (because re-shuffling on each imagined rollout would be costly).


What is the difference between a normal processor and a processor designed for AI? What is the difference between a normal processor and a processor designed for AI?

Parallel processing is very important when computing ANNs. Though they are not designed for AIs, GPUs are widely used in machine learning because of their higher capability of paralel processing and it is due to their higher number of cores comparing CPUs.

All in all, the dedicated hardware should have GPU-like architecture.


How could AI be used to augment human biological intelligence? One of the most compelling applications for AI would be in augmenting human biological intelligence. What are some of the currently proposed methods for doing this aside from vague notions such as "nanobots swimming around our brains and bodies" or "electrodes connected to our skulls"?

'Direct augmentation' of human intelligence, of the sort that you would see in science fiction, looks to be very hard. Most of our promising approaches deal with avoiding damage rather than adding capabilities--there's no drug that you can take now that will make you smarter to the degree that missing a night of sleep can make you dumber.

The most informative area of current practice is probably game-playing, where 'centaurs,' or humans working with computers, outcompete human players or computer players.

But a centaur player doesn't have a wire jutting out of their skull to jack into the computer; they're looking at a laptop screen. One of the reasons to be pessimistic about cyborg augmentation is because current I/O technology is already so good. Why install a new wire to put information into your visual cortex, when you come already equipped with two?

If you could think code directly onto the screen, how much better would that be than typing code through a keyboard? Probably some, but I find it difficult to imagine that it'll be more than twice as good. So most human-computer intelligence augmentation will look like people using software, and software using human inputs, rather than humans and computers evolving together.

Transcranial Direct Current Stimulation (TDCS) and similar approaches cause temporary changes in mental abilities by raising or lowering the activation potentials of neurons in particular regions of the brain. (I've done it myself, a few years ago, and what weak effects I noticed were probably negative. Not too much surprise for a DIY setup!)

It looks like it has a number of useful implications. One article about TDCS that I found particularly striking was the journalist who tried it gushing about how their anxiety disappeared for a few days, presumably because the part of their brain behind the anxiety was dampened. One could imagine it being useful for the treatment of many different mental disorders.

That said, I'm pessimistic that it will translate into superior peak performance, and I think that's the sort of thing that's more relevant for discussions of augmentation. (Is there TDCS that we could do that would make Terrence Tao better at doing mathematics?)

Where improved AI methods will come into play is by improving our models of the brain, allowing us to better target interventions, much in the way that AI methods are improving our treatment of cancer (through superior diagnosis and targeting of radiotherapy, as two easy examples). These effects will all be indirect--for example, AI empowering an app or gadget that helps you sleep better won't directly augment your intelligence, but will cause population-level increases in effective intelligence through reducing sleep deprivation.

I haven't talked yet about nootropics, chemicals that increase intelligence, but it's reasonable to expect that AI will improve drug discovery there like it improves drug discovery for anything else. But the same caveats apply--the effect of nootropics seem to be negatively correlated with intelligence (that is, the smarter someone already is, the harder it is to increase their intelligence further).


What is "Computational Linguistics"? It's not clear to me whether or not someone whose work aims to improve an NLP system may be called a "Computational Linguist" even when she/he doesn't modify the algorithm directly by coding.

Let's consider the following activities:

Annotation for Machine Learning:
analysis of Morphology, Syntax, POS tagging
Annotation, analysis, and annotation of entities (NER) and collocations; supporting content categorization; chunking; word sense disambiguation.
Recording of technical issues of the annotation tool to improve its reliability.
Recording of linguistic and logical particular rules adopted by the research team who develops the NLP algorithm to improve consistency between annotation and criteria previously adopted to train the NLP.

May be these activities considered "Computational Linguistics"? If not, which is their professional category and how should they be included in the resume in a word which synthesizes them?

Yes. A computational linguist is someone who (among other things) uses computers to process/model/analyse/... natural language. Coding might be one aspect of it, but is about the least important: you can always get a non-linguist programmer to do coding for you.

I studied "Computational Linguistics" at university, and while programming was taught as part of the course, coding was only a minor aspect in the actual subject matter. The senior professor (and head of the department) wasn't able to do any coding himself; he came from the linguistics side of it.

Being able to program is useful, as it speeds things up and makes you more independent, but it is by no means an important part of being a computational linguist.

UPDATE: I have been accused of misrepresenting the field of CL. However, it is a broad, interdisciplinary field, and comprises many elements. Sure, on the academic/research side you might do more programming than in the applied/commercial side, but I maintain that you can easily work as a computational linguist without actually doing any programming. For most tasks, readily available software exists by now, so you don't actually need to program anything new.


What are the real-life applications of transfer learning? What are the real-life applications of transfer learning in machine learning? I am particularly interested in industrial applications of the concept.

One application I know of being used in industry is of image classification, by only training the last layer of one of the inception models released by Google,  with the desired number of classes. I can't provide specific details.

Transfer learning is useful when:

You do not have the resources (time, processing power, etc.) to train a DL  model from scratch.
You can compromise a bit on accuracy.


How to know what kind of memory is stored in the connection weights? So, I have seen few pictures re-created by a Neural Network or some other Machine Learning algorithm after it has been trained over a data set.

How, exactly is this done? How are the weights converted back into a picture or a memory which a Neural Net is holding?

A real life example would be when we close our eyes we can easily visualize things we have seen. Based on that we can classify things we see. Now in a Neural Net classification part is easily done, but what about the visualization part? What does the Neural Net see when it closes its eyes? And how to represent it for human understanding?

For example a deep net generated this picture:

SOURCE: Deep nets generating stuff

There can be many other things generated. But the question is how exactly is this done?

You should use google for this question, it is extremely vague. Some technical documents will give very clear insight into what's happening here.

https://blog.openai.com/generative-models/

http://proceedings.mlr.press/v37/gregor15.pdf

If you really want to understand "what the AI is thinking", well, you may never know. The idea of AI is that it can handle complex data (high degree of dimensionality) and is too complex for Humans to comprehend.


How fast is TensorFlow compared to self written neural nets? I made my first neural net in C++ without any libraries. It was a net to recognize numbers from the MNIST dataset. In a 784 - 784 - 10 net with sigmoid function and 5 epochs with every 60000 samples, it took about 2 hours to train. It was probably slow anyways, because I trained it on a laptop and I used classes for Neurons and Layers.
To be honest, I've never used TensorFlow, so I wanted to know how the performance of my net would be compared to the same in TensorFlow. Not too specific but just a rough approximation.

I wanted to know how the performance of my net would be compared to the same in Tensor Flow. Not to specific but just a rough aproximation.

This is very hard to answer in specific terms because benchmarking is very hard and is often wrong.

The main point of TensorFlow as I see it is to make it easier for you to use a GPU and further allows you to use a large supply of programs written in Python/JavaScript that still give C++ level performance.

  How fast is TensorFlow compared to self written neural nets?

This is answering the general question of using TensorFlow/PyTorch vs a custom solution, rather than your specific question of how much of a speed up you'd get.

There was a relatively recent MIT paper, Differentiable Programming for Image Processing and Deep Learning in Halide trying to discuss the performance vs flexibility vs time spent coding a solution of 3 different languages.

Specifically they compared a solution in their language Halide vs PyTorch vs CUDA.

  Consider the following example. A recent neural network-based
  image processing approximation algorithm was built around a new
  “bilateral slicing” layer based on the bilateral grid [Chen et al. 2007; Gharbi et al. 2017]. At the time it was published, neither PyTorch
  nor TensorFlow was even capable of practically expressing this
  computation. As a result, the authors had to define an entirely
  new operator, written by hand in about 100 lines of CUDA for the
  forward pass and 200 lines more for its manually-derived gradient
  (Fig. 2, right). This was a sizeable programming task which took
  significant time and expertise. While new operations—added in just
  the last six months before the submission of this paper—now make
  it possible to implement this operation in 42 lines of PyTorch, this
  yields less than 1/3rd the performance on small inputs and runs
  out of memory on realistically-sized images (Fig. 2, middle). The
  challenge of efficiently deriving and computing gradients for custom
  nodes remains a serious obstacle to deep learning.

So in general you'll probably get faster performance with TensorFlow/PyTorch than a custom C++ implementation, but for specific cases if you have CUDA knowledge on top of C++ then you will be able to write more performant programs.


Using CNN to identify buildings from aerial images I want to train a CNN (Vggnet) to identify different types of buildings from aerial images.

However seeing that a CNN "ignores" size, e.g. the same type of dog in one image can be large and small in another image but will still be classified as a dog.

My issue is that non-residential buildings are mostly larger than residential houses, now I want to use this property to distinguish between residential and non residential. Is this even possible?

Thanks

You need to know the basic mechanics of CNN , you can't simply finetune a pretrained net (VGGNET) everytime. VGGnet is trained on imagenet dataset so most of the features in it are not so relevant for your task (recognising buildings from aerial view). so it is much better to train a cnn from scratch. You can't simply say that it fails to classify residential , non-residential . It is best to find what could be done by trying,

My issue is that non-residential buildings are mostly larger than residential houses, now I want to use this property to distinguish between residential and non residential.

Most of the times it is better to observe the features recognised by the net and then come with an approach , rather than enforcing your views.

You may find this helpful https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Thesis.pdf


Can I apply DQN or policy gradient algorithms in the contextual bandit setting? I have a problem which I believe can be described as a contextual bandit.

More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, based on the context, I have to choose one of the ten available actions. The actions do not influence the next context.

Based on the above I have the following questions:

Is this a contextual bandit or an MDP with a discount equal to zero (one step RL)? I have read that, in contextual bandits, we receive a different context for each action and I am a little bit confused.
Can I use the DQN algorithm with TD Target only the observed reward instead of the reward plus the predicted value of the next state?
Can I use a policy gradient algorithm, like REINFORCE or A2C? If yes, should I use a baseline and what this baseline should be?
I have seen in the literature that there are some algorithms for contextual bandits such as LinUCB, LinRel, NeuralBandit, etc. And I am wondering why the DQN, A2C and REINFORCE algorithms, which seem to work well in MDP setting, are not used in contextual bandits, given the fact that this problem can be described as an MDP with a discount equal to zero?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.

Therefore, methods used in RL to solve MDPs will work to solve contextual bandits. You can either treat a contextual bandit as a series of 1-step episodes (with start state chosen randomly), or as a continuous problem with discount factor zero.

  Can I use the DQN algorithm with TD Target only the observed reward instead of the reward plus the predicted value of the next state?

Yes. That's identical mathematically to having a discount of zero, or having 1-step episodes.

  Can I use a policy gradient algorithm, like REINFORCE or A2C? If yes, should I use a baseline and what this baseline should be?

Yes. Once converted to an MDP, you can use the same baselines in these algorithms as normal (A2C's use of advantage instead of action value is already a baseline). Generally adding a baseline can help reduce variance, so it may still help when applying RL to contextual bandit problems.

  I have seen in the literature that there are some algorithms for contextual bandits such as LinUCB, LinRel, NeuralBandit, etc. And I am wondering why the DQN, A2C and REINFORCE algorithms, which seem to work well in MDP setting, are not used in contextual bandits

There are a couple of reasons that contextual bandit problems are not solved using RL techniques more often:

The goal in contextual bandits is commonly focused on creating a highly efficient online learner that minimises regret. Regret is the long term difference in total reward between always exploiting the best action choice compared to the exploration necessary to find it. Some RL solvers - e.g. DQN - are poor by this metric.
The lack of timesteps and state transitions can be used in algorithm design to be more efficient.
Improvements to RL methods designed to help with sparse rewards and the assignment problem in MDPs are pointless for contextual bandits and may be wasteful or even counter-productive.

Some RL algorithms do resolve to be nearly identical to their contextual bandit counterparts, and have the same performance characteristics e.g. REINFORCE with baseline for 1-step episodes is essentially the Contextual Gradient Bandit algorithm.

It is also worth noting than many problem domains where contextual bandit algorithms do well - e.g. website recommendations and advertising - have research showing that that a more sophisticated MDP model and RL-like approach can do even better. Although that is not quite the same as your question, it typically means extending the model so that timesteps and state transitions are meaningful.


How to calculate the confidence of a classifier's output? I'm training a classifier and I want to collect incorrect outputs for human to double check.

the output of the classifier is a vector of probabilities for corresponding classes. for example, [0.9,0.05,0.05]

This means the probability for the current object being class A is 0.9, whereas for it being the class B is only 0.05 and 0.05 for C too.

In this situation, I think the result has a high confidence. As A's probability dominants B's and C's.

In another case, [0.4,0.45,0.15], the confidence should be low, as A and B are close.

What's the best formula to use to calculate this confidence?

Obvious answer for a binary (2 classes)classification is .5. Beyond that the earlier comment is correct. One of the things I have seen done is to run your model on the test set and save the prediction probability results.  Then create a threshold variable call it thresh. Then increment thresh from 0 to 1 in a loop. On each iteration compare thresh with the highest predicted probability prediction call it P. If  P>thresh declare that as the selected prediction, then compare that with the true prediction. Keep track of the errors for each value of thresh. At the end select the value of thresh that has the least errors.
There are also some  more sophisticated methods for example "top 2 accuracy" where thresh is selected based on  having the true class within either the prediction with the highest probability or the second highest probability . You can construct a weighted error function and select the value of thresh that has the net lowest error over the test set.
For example an error function might be as follows. If neither P(highest) or P(second highest) = True class, error=1. If P(second highest) = true class, error=.5. If p(highest)=true class error=0. I have never tried this myself so I am not sure how well this works. When I get some time will try it on a model with 100 classes and see how well it does. I know in the Imagenet competition they evaluate not just the top accuracy but also the "Top 3" and "Top 5' accuracy. In that competition there are 1000 classes.
I never thought of this before but I assume you could train your model specifically to optimize say the Top 2 accuracy by constructing a loss function used during training that forces the network to minimize this loss.


What does an oscillating validation error curve represent? I have been training my CNN for a bit now and, while both the training loss and the training error curves are going down during training, both my validation loss and my validation error curves are kind of zig-zagging and oscillating along the epochs. What does this represent?

Without more information the best diagnostic is:
You potentially have a bug in your code.
Justification
Neural network systems are capable of failing silently. The system can still appear to "learn" even in the presence of said bug - a while back I had a very similar issue with a toy CNN project. My network could get 99% accuracy on the training set but always achieved 8-13% on the validation set. This looked like overfit but none of the methods solved the issue. Finally, I found that I wasn't feeding the data correctly into the network during train time but I was feeding the data correctly during validation time.
Conclusion
Provide the following for better diagnostic:

Is this a custom CNN you hardcoded?
are you using python?
what's the objective function?
can you provide the loss curve for the training set?
show us some code too (this will be very helpful for working toward a solution)

I hope this helps, and best of luck!


L2 normalization in Facenet paper acting as a Constraint? In the Facenet paper, Under section 3.2 The authors mention that:

  The embedding is represented by f(x) ∈ Rd . It embeds an image x into
  a d-dimensional Euclidean space. Additionally, we constrain this
  embedding to live on the d-dimensional hypersphere, i.e. ||f(x)||2 = 1.

I don't quite understand how the above equation holds! As far as I understand L2 Norm is same as euclidean distance but I don't quite understand how this imposes
||f(x)||2 = 1 criteria.

The constraint is enforced with bespoke code. If FaceNet was implemented in NumPy, and the embedding layer vector (pre-constraint) was in the NumPy array h, then the code might look like:

e = h / np.linalg.norm(h)

The variable e would then contain the desired embedding with L2 distance of 1. In practice, even with NumPy, the code might be more complex due to handling mini-batches. More likely this will not be implemented in NumPy.

In this Keras/TensorFlow-based FaceNet implementation you can see how it may be done in practice:

# L2 normalization
X = Lambda(lambda  x: K.l2_normalize(x,axis=1))(X)

This scaling transformation is considered part of the neural network code (it is part of the Keras model building routine in the above snippet), so there needs to be corresponding support for back propagation through the embedding. When using an automatic differentiation framework such as Theano or TensorFlow, then no extra code is required.


How do map providers like Google calculate the distance between two coordinates and find turn by turn directions? I have searched on how Google or any map provider calculates distance between two coordinates. The closest I could find is Haversine formula.

If I draw a straight line between two points, then Haversine formula can be helpful. But since no one will travel straight and typically move through the streets, I want to know if there are any methods to calculate turn by turn points and see how to find multiple ways to travel to the destination from the source.

Right now my idea is

Have the two coordinates within a map window.
Make an algorithm detect the white lines (path) in the window.
Make it understand how they are connected.
Feed it to an algorithm to solve the Travelling Salesman Problem to find the best path between them.

But these things see very memory and process intensive. Even with the knowledge that Google has the powerhouse to process, to serve so many directions and distance matrix in fractions of seconds in amazing. I want to know if there are different approaches to this?

Roads maps are well defined and you can access them online. For example you can visit OpenStreetMap to download road network of a given region.

Such a road network definition contains nodes (junctions) with lat/lon coordinates and edges between nodes (roads). An edge is defined by a connection of two nodes. Edges can represent one-way roads: you need to define double ways as two edges going in both ways. Haversine formula gives you the straight distance between two nodes: the length of the edge.

This definition of a road network gives you a graph (in the mathematical point of view) and is largely covered by the literature. You can search the graph and compute many of its parameters.

From a road network definition you can run shortest path algorithm to find the closest way between two nodes. From a given coordinate you can either pick the closest node or get the closest edge (and compute distance to the edge and the coordinates of the perpendicular projection point).

Algorithms like A* and Dijkstra are not so time consuming and you can find shortest path efficiently.


Is randomness anti-logical? I came across a comment recently "reads like sentences strung together with no logic."  But is this even possible?

Sentences can be strung together randomly if the selection process is random. (Random sentences in a random sequence.) Stochasticity does not seem logical—it's a probability distribution, not based on sequence or causality.

but

That stochastic process is part of an algorithm, which is a set of instructions that must be valid for the program to compute.

So which is it?

Is randomness anti-logical?

Some definitions of computational logic:

The arrangement of circuit elements (as in a computer) needed for computation
also: the circuits themselves Merriam Websters  A system or set of principles underlying the arrangements of elements in a computer or electronic device so as to perform a specified task. Logical operations collectively. Google Dictionary
The system or principles underlying the representation of logical operations. Logical operations collectively, as performed by electronic or other devices.
Oxford English Dictionary

Some definitions of randomness

Being or relating to a set or to an element of a set each of whose elements has equal probability of occurrence. Lacking a definite plan, purpose, or pattern.
Merriam Websters. Made, done, happening, or chosen without method or conscious decision. Google Dictionary Having no definite aim or purpose; not sent or guided in a particular direction; made, done, occurring, etc., without method.  Seeming to be without purpose or direct relationship to a stimulus. Oxford English Dictionary

I think the answer here lies in that the dictionary definition of randomness you have is not the one used in statistics, ML, or mathematics. We define randomness to mean there exists a distribution with generally greater than 0 uncertainty.

Depending on who you talk to, we live in a random universe (the way we define quantum mechanics depends on a wave function (essentially a probability distribution)

So why if a sequence is drawn from a distribution is it illogical? First, even as humans we can make a strong argument that what we say is random. I mean we speak to convey some form of message or context, but there exists multiple ways to deliver this, but we choose a single one. Our brains inherently model $p(\vec w|c)$ where $\vec w$ is the sequence and $c$ is our context or message we want to convey.

Takeaway: Generating a sequence in an ergodic or uniform manner would be illogical, but that is not what is being modeled or done in practice. Normally its drawn from some complex distribution.

Sidenote: My above claim could make it seem that being uniformly random implicates something illogical, and I want to emphasize that is not the case. It is domain to domain, sometimes that is the most logical solution, just in the case of sentence generation it normally isnt. I would define a logical algorithm as one that given the information at hand acts in a sensible manner towards achieving some goal, and so if something purely random does that, I don't see the problem.


What is the derivative function used in backpropagration? I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function?

These terms are confusing: derivative of act. function, partial derivative wrt. loss function??

I'm still not getting it correct.

Overview

The derivatives of functions are used to determine what changes to input parameters correspond to what desired change in output for any given point in the forward propagation and cost, loss, or error evaluation &mdash whatever it is conceptually the learning process is attempting to minimize. This is the conceptual and algebraic inverse of maximizing valuation, yield, or accuracy.

Back-propagation estimates the next best step toward the objective quantified in the cost function in a search. The result of the search is a set of parameter matrices, each element of which represents what is sometimes called a connection weight. The improvement of the values of the elements in the pursuit of minimal cost is artificial networking's basic approach to learning.

Each step is an estimation because the cost function is a finite difference, where as the partial derivatives express the slope of a hyper-plane normal to surfaces that represent functions that comprise forward propagation. The goal is to set up circumstances so that successive approximations approach the ideal represented by minimization of the cost function.

Back-propagation Theory

Back-propagation is a scheme for distribution of a correction signal arising from cost evaluation after each sample or mini-batch of them. With a form of Einsteinian notation, the current convention for distributive, incremental parameter improvement can be expressed concisely.

$$ \Delta P = \dfrac {c(\vec{o}, \vec{\ell}) \; \alpha} {\big[ \prod^+ \! P \big] \; \big[ \prod^+ \!a'(\vec{s} \, P + \vec{z}) \big] \; \big[ c'(\vec{o}, \vec{\ell}) \big]} $$

The plus sign in $\prod^+\!$ designates that the factors multiplied must be downstream in the forward signal flow from the parameter matrix being updated.

In sentence form, $\Delta P$ at any layer shall be the quotient of cost function $c$ (given label vector $\vec{\ell}$ and network output signal $\vec{o}$), attenuated by learning rate $\alpha$, over the product of all the derivatives leading up to the cost evaluation. The multiplication of these derivatives arise through the recursive application of the chain rule.

It is because the chain rule is a core method for feedback signal evaluation that partial derivatives must be used. All variables must be bound except for one dependent and one independent variable for the chain rule to apply.

The derivatives include three types.

All layer input factors, the weights in the parameter matrix used to attenuate the signal during forward propagation, which are equal to the derivatives of those signal paths
All the derivatives of activation functions $a$ evaluated at the sum of the matrix vector product of the parameters and signal at that layer plus the bias vector
The derivative of the cost function $c$ evaluated at the current output value $\vec{o}$ with the label $\vec{\ell}$

Answer to the Question

Note that, as a consequence of the above, the derivatives of both the cost (or loss or error) function and any activation functions are necessary.

Redundant Operation Removal for an Efficient Algorithm Design

Actual back propagation algorithms save computing resources and time using three techniques.

Temporary storage of the value used for evaluation of the derivative (since it was already calculated during forward propagation)
Temporary storage of products to avoid redundant multiplication operations (a form of reverse mode automatic differentiation)
Use of reciprocals because division is more costly than multiplication at a hardware level

In addition to these practical principles of algorithm design, other algorithm features arise from extensions of basic back-propagation. Mini-batch SGD (stochastic gradient descent) applies averaging to improve convergence reliability and accuracy in most cases, provided hyper-parameters and initial parameter states are well chosen. Gradual reduction of learning rates, momentum, and various other techniques are often used to further improve outcomes in deeper artificial networks.


How to architect a network to find bounding boxes in simple images? I have an application where I want to find the locations of objects on a simple, relatively constant background (fixed camera angle, etc). For investigative purposes, I've created a test dataset that displays many characteristics of the actual problem.
Here's a sample from my test dataset.

Our problem description is to find the bounding box of the single circle in the image. If there is more than one circle or no circles, we don't care about the bounding box (but we at least need to know that there is no valid single bounding box).
For my attempt to solve this, I built a CNN that would regress (min_x, min_y, max_y, max_y), as well as one more value that could indicate how many circles were in the image.
I played with different architecture variations, but, in general, the architecture a was very standard CNN (3-4 ReLU convolutional layers with max-pooling in between, followed by a dense layer and an output layer with linear activation for the bounding box outputs, set to minimise the mean squared error between the outputs and the ground truth bounding boxes).
Regardless of the architecture, hyperparameters, optimizers, etc, the result was always the same - the CNN could not even get close to building a model that was able to regress an accurate bounding box, even with over 50000 training examples to work with.
What gives? Do I need to look at using another type of network as CNNs are more suited to classification rather than localisation tasks?
Obviously, there are computer vision techniques that could solve this easily, but due to the fact that the actual application is more involved, I want to know strictly about NN/AI approaches to this problem.

There are some special architectures of CNNs which are designed exactly for the task you mention. The Detector library includes a collection of these architectures, this paper describes the Mask R-CNN network in detail, which is designed for image segmentation tasks.


Why isn't the ElliotSig activation function widely used? The Softsign (a.k.a. ElliotSig) activation function is really simple:

$$ f(x) = \frac{x}{1+|x|} $$

It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally extremely simple (easy for, e.g., a GPU).

Why it is not widely used in neural networks? Is it because it is not infinitely derivable?

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.

ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions,

  the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.

Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.

Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.

As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.


Asus zenbook with deep learning Is the laptop Asus ZenBook Pro 90NX0152-M02980 enough to do deep learning model?

Specs:

Processeur Intel Core i7-7500U (Dual-Core 2.7 GHz / 3.5 GHz Turbo -
cache 4 MB)
8 GB Memory
SSD M.2 SATA de 256GB

Here is a little bit of aglimpse

Asus Zenbook pro,is quite good for technical nor software development tasks,despite having ordered one recently.

That said,you need to that each and every computer/machine for effective and efficient computing purpose,the first priority is to analyse critically the processor chip,which generational architecture/design does it have inline with GPU/Graphics ,Resolution,motherboard,hard-drive...e.t.c..

Therefore,basing on such technical specs,what matters most,is not the brand nor logo..! Go for any brand,keeping in mind your objective "DeepLearning"


With gradient descent w/MSE on a regression, must/should every Epoch use the exact same training samples? Let's say I've got a training sample set of 1 million records, which I pull batches of 100 from to train a basic regression model using  gradient descent and MSE as a loss function.  Assume test and cross validation samples have already been withheld from the training set, so we have 1 million entries to train with.

Consider following cases:

Run 2 epochs (I'm guessing this one is potentially bad as it's basically 2 separate training sets)

In the first Epoch train over records 1-500K
In the second epoch train over the 500K-1M

Run 4 epochs

In the first and third Epoch train over records 1-500K
In the second and fourth epoch train over the 500K-1M

Run X epochs, but each epoch has a random 250K samples from the training set to choose from

Should every epoch have the exact samples?  Is there any benefit/negative to doing so? My intuition is any deviation in samples changes the 'topography' of the surface you're descending, but I'm not sure if the samples are from the same population if it matters.

This relates to a SO question: https://stackoverflow.com/questions/39001104/in-keras-if-samples-per-epoch-is-less-than-the-end-of-the-generator-when-it

Your goal in regression should be to obtain the factors which result in the best fit model without over-fitting.  The more data you have in the training set, the better your regression will be.  Thus you would want to train on the most data, but you also want to have some data held out to validate that your model is not over-fit.  So this is where you should have your data split into say a 80/20 training and validation set.  And if data is scarce or you want that 20% to contribute to the model then you could do a 5 fold cross validation.

In the spirit of research perhaps you should try both of these routes, and report your findings.


Extracting multiple softmax values from image I am currently in the pre-process of starting an image classification and extraction project which needs to output multiple softmax and absolute values from a single image like such:

{
 time: "20:20",
 teams: [
   {
     red: { goals: 2},
     blue: { goals: 1},
   },
   {
     scored_by : [{
      john: 80%,
      kyle: 51%,
      darren: 20%
     }
   ]}
 ]
}

I can create multiple models which are responsible for different task such as reading the time from the image as well as the score and eventually combine both. I would however like to make sure I maximize on efficiency to make sure the process is as fast as possible.

Any pointers in the right direction would be greatly appreciated.

With kind regards,
Dennis

It is not unheard of to share network weights with multiple output layers. I have seen it on DeepMind's Asynchronous Deep Learning paper, and I have also used it here.

The idea is to share all the layers and just have multiple outputs. However, this might decrease the accuracy of your networks, as is the usual performance VS accuracy trade-off.

To optimize this, just calculate the loss of both outputs and sum them when feeding the optimizer, like this

            self.target_policy_fast_t = tf.placeholder('float32', [None, a_size], name='target_policy_fast_t')
            self.loss_policy_fast = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_fast_before_softmax, labels=self.loss_policy_fast_t))

            self.target_policy_slow_t = tf.placeholder('float32', [None, a_size], name='target_policy_slow_t')
            self.loss_policy_slow = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_slow_before_softmax, labels=self.target_policy_slow_t))

            self.loss = tf.reduce_mean(self.loss_policy_fast + self.loss_policy_slow, name='loss')


What is the dimensionality of the output map, given the dimensionality of the input map, number of filters, stride and padding? I am trying to understand the dimensionality of the outputs of convolution operations. Suppose a convolutional layer with the following characteristics:

Input map $\textbf{x} \in R^{H\times W\times D}$
A set of $F$ filters, each of dimension $\textbf{f} \in R^{H'\times W'\times D}$
A stride of $<s_x, s_y>$ for the corresponding $x$ and $y$ dimensions of the input map
Either valid or same padding (explain for both if possible)

What should be the expected dimensionality of the output map expressed in terms of $H, W, D, F, H', W', s_x, s_y$?

where, $p_{x}$ and $p_{y}$ are padding values. (equal on both sides).
You can have different padding on left and right side. (similarly top and bottom). If same padding, equation will have $2*p_{x}$ or $2*p_{y}$, else you can just add values of both padding and replace in the equations. (For example $p_{xLeft} + p_{xRight}$)

Padding is optional. Generally thumb rule is add padding such that the equations above produce integers and also add padding such that if stride is 1, output is same as input and if stride is 2, output is half of input.


Two DQNs in two different time scales I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries.
In episode $e$, the agent must first select an adversary. Then for each step $t$ in the episode $e$, it plays the game against the chosen adversary. Every step $t$, it receives a reward following the chosen action in step $t$ (for the chosen adversary). How to maximize the expected rewards using DQN? It is clear that choosing the "wrong" (the strongest) adversary won't be a good choice for the agent. Thus, to maximize the accumulated rewards, the agent must take two actions at two different timescales.
I started solving it using two DQNs, one to decide the adversary to play against and one to play the game against the chosen adversary. I have two duplicate hyperparameters (batch_size, target_update_freq, etc), one for each DQN. Have you ever seen two DQNs like this? Should I train the DQNs simultaneously?
The results that I am getting is not that good. The accumulated reward is decreasing, the loss isn't always decreasing...

From comments, you say there is no "outer" goal for picking an adversary other than  scoring highly in an individual episode.
You could potentially model the initial adversary choice as a partially separate Markov Decision Process (MDP), where choosing the opponent is a single-step episode with return equal to whatever reward the secondary MDP - which played the game - obtains. However, this "outer" MDP is not much of an MDP at all, it is more like a contextual bandit. In addition, the performance of the inner game-playing agent will vary both with the choice of opponent, and over time as it learns to play better against each opponent. This makes the outer MDP non-stationary. It also requires the inner MDP to know what opponent it is facing in order to correctly predict correct choices and/or future rewards.
That last part - the need for any "inner" agent to be aware of the opponent it is playing against - is likely to be necessary whatever structure you chooose. That choice of opponent needs to be part of the state for this inner agent, because it will have an impact on likely future rewards. A characterisation of the opponents also needs to be part of whatever predictive models you could use for the outer agent.
A more natural, and probably more useful, MDP model for your problem is to have a single MDP where the first action $a_0$ is to select the opponent. This matches the language you use to describe the problem, and resolves your issue about trying to run a hierarchy of agents. Hierarchical reinforcement learning is a real thing, and very interesting for solving problems which can be broken down into meaningful sub-goals that an agent could discover autonomously, but it does not appear to apply for your problem.
This leaves you with a practical problem of creating a model that can switch between choosing between two sets of radically different actions. The select an opponent action only occurs at the first state of the game, and the two sets of actions do not overlap at all. However, in terms of the theoretical MDP model this is not an issue at all. It is only a practical issue of how you get to fit your Q function approximator to the two radically different action types. There a a few ways around that. Here are a couple that might work for you:
One shared network
Always predict for all kinds of action choice, so the agent still makes predictions for switching opponents all the way to the end of the game. Then filter the action choices down to only those available at any time step. When $t=0$ only use the predictions for actions for selecting an opponent, for $t \ge 1$ only use predictions relating to moves in the game.
Two separate approximators
Have two function approximators in your agent, use one for predicting reward at $t=0$ that covers different opponent choices, and use the other for the rest of the game. If $n$ is small and there is no generalisation between opponents (i.e. no opponent "stats" that give some kind of clue towards the end results), then for the first approximator, you could even use a Q table.
For update steps you need to know whether any particular action value was modelled in one or other of the Q functions - and this will naturally lead you to bootstrap
$$\hat{q}_{o}(s_0, a_0, \theta_0) \leftarrow r_1 + \gamma \text{max}_{a'}\hat{q}_{p}(s_1, a', \theta_1)$$
where $\hat{q}_{o}$ is your approximate model for action values of selecting opponents (and $a_0$ must be an opponent choice) at the start of the game, and $\hat{q}_{p}$ is the nodel you use for the rest of it (and $a'$ must be a position play in the game). I've misused $\leftarrow$ here to stand in for whatever process used to update the action value towards the new estimate - a tabular method that would be a rolling average with current estimate, in neural networks of course that is gradient descent using backpropagation.


Can the hidden layer prior to the ouput layer have less hidden units than the output layer? I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.
I remember that the Professor saying that there is no rule for choosing the number of hidden units and that having many of them along with many hidden layers can cause the network to overfit the data and under learn.
However, I still have this question where assuming that we have a network with an input layer of n  input nodes, a first hidden layer of 4 hidden units, a second layer of X hidden units and an output layer of 5 units. Now if I follow the Professor's saying, it would mean that I am allowed to have X = 3 or X = 4 in layer 2.
Is that actually allowed? Won't we have some sort of information gain passing from 4 (or 3) nodes to 5?  The example is illustrated below.

A layer with bigger number of nodes than previous one is something very common. Some examples are:

strategies encoder-decoder (autoencoders) where the encoder typically  has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes.

bidirectional recurrent networks where in the forward direction number nodes decreases and in the backward increases.

generators, that from a random vector generates, by example, a full image.

As general rule: decrease number of nodes forces the net to filter/resume/abstract/summarize the internal signal information (discarding useless information or noise) while increase number of nodes means apply current information to generate an answer value for a specific question/target.
Allow me a strongly simplified example: assume you want a system that, from a photo of an animal, answers the questions: number of legs? has beak ? flies ? . Net inputs are images of birds and dogs.
The net architecture can have layers of decreasing size until a single node that will decide "is bird or dog ?". From this single item of information (the only one need to answer all the questions) the output layer will have 3 nodes, each one answering one of the specific target questions: number or legs ? 4 if dog, 2 if bird, etc .


How do I segment each part of a DICOM image? As I'm beginner in image processing, I am having difficulty in segmenting all the parts in DICOM image.

Currently, I'm applying watershed algorithm, but it segments only that part that has tumour.

I have to segment all parts in the image. Which algorithm will be helpful to perform this task?

The image below contains the tumour.

This image is the actual DICOM image

I looked at some picture from DICOM, and it seems that these images usually don't have many local minima, so I guess that's why watershed isn't optimal.
However, it seems that in DICOM the color gradients are often strong, so I suggest to use techniques that works well with gradients:

Direct Gradient computations, either convolutional gradients or morphological gradients (selects shape of your different objects)
Normalized Cut on Similarity Graphs

If you want to know more about these you can read this post.


How to solve the "dangerous feedback loops" in machine learning? From the article Dangerous Feedback Loops in ML

Let’s say our model has leads from Facebook, Google, and Bing. If our first model decides that the probability of conversion is 3%, 5%, and 1% from these given sources, and we have finite amount of callbacks we can make, we will only callback the 5% probability. Now fast forward two months. The second model finds these probabilities are now: 0.5%, 8.5%, and 0%. What happened?

Because we started only calling Google leads back, we increased our chances of converting these leads, and likewise, because we stopped calling Facebook and Bing leads, these leads never converted because we never called them. This is an example of a real world feedback loop

How can we solve this problem?

I read the article you linked, and what you are missing are that the given conversion probabilities are assessed pre-callback - i.e. they include an assessment of whether you will even call them back or not. So of course the probabilities change if you change your behaviour. The writer of the article has created a bit of a straw man argument by defining a model and decision process that don't go well together. They should have used a model that predicted conversion rates after callback, then it could be used as they wanted.
So the sales example is pretty easy to solve. Make the model accept the lead source as a feature, and predict the conversion rate after callback. That will give you the "action value" of choosing to callback, which is much more useful value to have if you are deciding whether or not to callback.
To cover the possibility that probabilities change over time, you have to be willing to test that hypothesis by calling back at least some leads that the model predicts have a low probability of success, in order to update the model. This is related to exploration vs exploitation trade off in bandit problems or in reinforcement learning. The callback optimisation problem looks a lot like a bandit problem in fact.
Many problems of choosing how to act optimally, based on an updating statistical model of results, can be re-framed successfully as bandit problems or reinforcement learning problems. This is one way to try and address the feedback loops issue, because these include theory around decision making. It is not a magic fix for all these problems, but contextual bandits are very strong in advertising strategies that are very similar to the example you give.
To be able to address feedback loops like this well does require that source data is unbiased, so that adding conditional actions to your model helps obtain better ground truth. So the other examples that the linked article gives - recidivism and recruitment - are much harder to address, because biased data is inserted as ground truth even if you add the conditions (e.g. for exploring and tracking results in a statistically neutral way). For some, adding these conditions may be enough to help, it will make the model accurate enough that it starts to address bias. For others, there is no really good solution other than to be aware of weakness of the model.
You can attempt to de-bias the model by using some abstract ideal. A common technique used in recruiting for instance is to remove features that might be causing bias: Literally remove race, sex and related determiners from the CV on the grounds that these should not be part of any fair model, regardless of whether or not a ML model could make of use them to predict results. This might accidentally throw out data that would improve accuracy too, but can be used to achieve a decision process that observers agree should be free of contentious soruces of bias.
Which features to remove or adjust is an open and subjective question. Unfortunately ML models can often determine protected characteristics from other data, and with "black box" ones such as neural networks it is not clear whether or not they are doing so. It's still an open question, and the main defence against biased models is basically awareness that it is a problem and that feeding all your data into a computer to get a statistical model does not in any way "purify" it or achieve some higher level of objective truth.


Why are deep neural networks and deep learning insufficient to achieve general intelligence? Everything related to Deep Learning (DL) and deep(er) networks seems "successful", at least progressing very fast, and cultivating the belief that AGI is at reach. This is popular imagination. DL is a tremendous tool to tackle so many problems, including the creation of AGIs. It is not enough, though. A tool is a necessary ingredient, but often insufficient.

Leading figures in the domain are looking elsewhere to make progress. This report/claim gathers links to statements by Yoshua Bengio, Yann LeCun and Geoff Hinton. The report also explains:

  The main weaknesses of DL (as I see them) are: reliance on the simplest possible model neurons (“cartoonish” as LeCun calls them); use of ideas from 19th century Statistical Mechanics and Statistics, which are the basis of energy functions and log-likelihood methods; and the combination of these in techniques like backprop and stochastic gradient descent, leading to a very limited regime of application (offline, mostly batched, supervised learning), requiring highly-talented practitioners (aka “Stochastic Graduate Descent”), large amounts of expensive labelled training data and computational power. While great for huge companies who can lure or buy the talent and deploy unlimited resources to gather data and crunch it, DL is simply neither accessible nor useful to the majority of us.

Although interesting and relevant, such kind of explanation does not really address the gist of the problem: What is lacking?

The question seems broad, but it may be by lack of a simple answer. Is there a way to pin-point what DL is lacking for an AGI ?

Everyone dealing with neural networks misses an important point when comparing systems with human like intelligence. A human takes many months to do anything intelligible, let alone being able to solve problems where adult humans can barely manage. That and the size of human brain is enormous compared to our neural networks. Direction might be right, but the scale is way off. Number of neurons in human brain can be matched memory-wise but the amount of parallelism to simulate it real-time cannot yet be achieved (at least for a random researcher). While a little old this might give you an idea of how much we lack the processing power.


What are the differences between 1-step SARSA and SARSA? SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?

N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

Off-policy n-step SARSA: Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$

where

$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$

You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.

Off-policy one-step SARSA: You can think of Q learning as one-step off policy SARSA.


How can I solve part b of exercise 3.6 from the book "Artificial Intelligence: A Modern Approach"? I am trying to solve part b of the exercise 3.6 (page 113) from the book Artificial Intelligence: A Modern Approach.
More specifically, I need to give a complete problem formulation (that is precise enough to be implemented) for the following problem.

A 3-foot-tall monkey is in a room where some bananas are suspended from the 8-foot ceiling. He would like to get the bananas. The room contains two stackable, movable, climbable 3-foot-high crates.
Give the initial state, goal test, successor function, and cost function for each of the following. Choose a formulation that is precise enough to be implemented.

Initial state: initial position of the monkey.
Possible actions

climb on the crate,
get down the crate,
move the crate from one spot to another,
stack one crate on another,
walk from one spot to another,
grab bananas (if standing on the crate)

Goal test: did the monkey get the bananas?
Cost function: the number of actions completed


How does a robot protect its own existence What are the many ways that artificial intelligence robots protect their existence?

Isaac Asimov's "Three Laws of Robotics"

A robot may not injure a human being or, through inaction, allow a human being to come to harm.

A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Right now, the scenarios where automata protect their own existence is limited.

In the case of autonomous vehicles, primary goals would certainly be collision avoidance, and other hazards (deep water, cliff faces, etc.)

It's possible that arial drones could have certain automated defense mechanisms specific to airborne threats such a missiles.

Bots might be said to protect their existence by copying themselves onto new systems.


How likely is it that Watson-like AI will replace Wikipedia-like encyclopedias? Is there any risk in the near future of replacing all encyclopedias with Watson-like AI where knowledge is accessible by everybody through API?

Something similar happened in the future in The Time Machine movie from 2002.

Obviously maintaining 40 million articles and keeping it up-to-date and consistent could be beyond brain power of few thousands of active editors. Not to mention thousands of other encyclopedias including paperback version or large number of books used by universities which needs to be updated every year by a huge number of people.

What are the pros and cons of such a change?

I get the impression that (perhaps even more than Bluemix) this is what the Wolfram Language is looking to offer in the longer term.

Seems to me that the main pros and cons are two sides of the same coin:

With Wikipedia, there's no 'search filter' between you and the text. Adding an algorithmic level of indirection between the user and the knowledge that they're looking for is subject to hidden biases.

If those biases are intended in your best interests, and the search is context-sensitive enough to present you with information in the form that is most useful and digestible to you, then this is a good thing. Otherwise, not. Like many topics in AI, problems arise because we're simply not that good at modelling human context yet.

Of course, we're already subject to this filter bubble effect via search engines and social media. The current consensus seems to be that even more of this would not be a good thing for society.


What is layer freezing in transfer learning? Transfer learning consists of taking features learned on one problem and leveraging them on a new, similar problem.
In the Transfer Learning, we take layers from a previously trained model and freeze them.
Why is this layer freezing required and what are the effects of layer freezing?

Why is this layer freezing required?
It's not.
What are the effects of layer freezing?
The consequences are:
(1) Should be faster to train (the gradient will have far less components)
(2) Should require less data to train on
If you do unfreeze the weights, I'd think your performance would be better because you are adjusting (i.e., fine-tuning) the parameters to your specific problem at hand. I am not sure what the marginal improvements are in practice, as I have not experiemented much with fine-tuning (like are the improvements typically a 0.01% reduction in error rate? Not sure.)


What is Cognitive Intelligence? Similarly to the question, What is artificial intelligence?

Cognitive Intelligence, as well as being a part of Artificial Intelligence, is an area that mainly covers the technology and tools that allow our apps, websites, and bots to see, hear, speak and understand the needs of the user through natural language

What is the definition of Cognitive Intelligence?

The term it comes from cognitive science - depending on the paradigm, it can have many meanings correlated with neuroscience as well as the brain.
Activities based on this sphere is :

language
learning
thinking
perception
awareness
decision making
intelligence

The mentioned cognitive processes - activate the appropriate regions in the brain narrow cognitive intelligence - it's an effective response to in particular in unforeseen and uncertain situations.
However, it requires the ability to set goals, learn, plan and think about your own thinking.


What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"? In this paper Fairness Through Awareness, the notation $\mathbb{E}_{x \sim V} \mathbb{E}_{a \sim \mu_x} L(x,a)$ is being used (page 5 top line), where $V$ denotes the set of individuals (so I guess set of feature vectors?) and the meaning of the other variables can be found in the paragraph above the mentioned notation. What does the $\sim$ in the expectation stand for?
Another notation that I do not know is $\Delta (A) $, where $A$ is the set of outcomes, for instance, $A = \{ 0,1\}$. What does it stand for?

The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write two explicit expectations here because $a$ is a random variable with distribution $\mu_x$ but $X$ is also a random variable with distribution $V$. I believe you are right that $V$ would be analogous to a set of features. So we are saying that $X$ is a random variable over the distribution over the features, or individuals in this context.
As for the $\Delta(A)$, I have never seen this notation used before -- I am not sure if it is standard notation. I Googled to see if it was something that I just hadn't seen before, but there was no such answer. However, from the context of the paper they define $M: V \rightarrow \Delta (A)$ to be mappings from individuals to probability distributions over $A$, so I guess that $\Delta (A) $ is probability distributions over $A$.


Train and Test Accuracy of GRU network not increasing after 2nd epoch So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel.
My problem is about the classification of 8 hour long timeseries with 11 different events with 1 second timesteps or to be more specific: sleep recordings.
Since the GRU can´t handle the whole timeseries at once, I split it in 500 timepoint pieces with 50 seconds (10%) overlap.  Also since I don´t have much data, after the training/test split I´m oversampling the train data by duplicating the underrepresented classes up to 10 times. So at the end I get a set of ca. 14.000 training snippets and ca 1.800 to test the model. That to the data.

Next thing is the model, represented with the Keras Code:

verbose, epochs, batch_size = 2, 50, 1200  # 1200 batch_size was the maximum the GPU can handle
    n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], trainy.shape[1]

    model = Sequential()
    model.add(GRU(128, input_shape=(n_timesteps, n_features), return_sequences=True, dropout=0.7,
                  kernel_regularizer=regularizers.l2(0.01), activation="relu"))
    model.add(
        GRU(64, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
            activation="relu"))
    model.add(
        GRU(32, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
            activation="relu"))
    model.add(
        GRU(16, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
            activation="relu"))
    model.add(TimeDistributed(Dense(units=16, activation='relu')))
    model.add(Flatten())
    model.add(Dense(n_outputs, activation='softmax'))

    # define custom optimizer with configurable learning rate
    sgd = optimizers.SGD(lr=0.1, momentum=0.9, decay=0.0, nesterov=False)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

So as a short summary:
I am using 4 GRU layers with a "pyramid" shape, so the network has to be more specific towards the end. Also one fully connected layer as some kind of "adapter" and one output layer with the size of my features. I am using SGD as an optimizer.

The results are always pretty bad, here are the loss and accuracy plots of an example run featuring 25 epochs:

Despite the huge dropout each stage, it still seems to be overfitting. Also as you can see, the accuracy of test and train is not changing after the 2nd epoch. The result is this sad looking confusion-matrix, showing only one class beeing detected by the network, which is not even the one with the highest amount of data in the dataset:

[[231   0   0   0   0   0   0   0]
 [ 55   0   0   0   0   0   0   0]
 [647   0   0   0   0   0   0   0]
 [141   0   0   0   0   0   0   0]
 [444   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0]
 [118   0   0   0   0   0   0   0]
 [ 74   0   0   0   0   0   0   0]]

So what are possible approaches in order to:
a) fight the overfitting,
b) just one class beeing detected and
c) fix the stationary accuracy to be growing again?

Since I´m pretty new to neural networks I am thankful for your time and effort!

Couple reccomendations:
1) I dont think your overfitting, your test loss is not ever increasing and is staying reasonbly proportional to train loss -- This may indicate that whatever loss your using is not a good indicator of the metric of interest (in this case, it seems you want that to be accuracy, but data is imbalnced so maybe look at avg precision?)
2) Use a lr scheduler, or something like a ReduceLROnPlateau callback to reduce the lr once the loss converges
3) Adding more sources of the underrepresented class is valid but i reccomend just using class weighting. Effectively its the same (as an expectation) and will save you train time.

Good Luck!


Are successive actions independent? The proof of the consistency of the per-decision importance sampling estimator assumes the independence of
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|S_k)}.$$

See the proof of Theorem 1 in "Eligibility Traces for Off-Policy Policy Evaluation".
The result is also stated in Equation (5.14) of Sutton and Barto's RL book.

I'm guessing that this is itself a consequence of an assumption of independence between
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}\quad\text{ and }\quad \frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}.$$

I don't understand how this assumption can be justified. Consider the extreme case of a nearly deterministic policy $\pi$ and deterministic MDP dynamics. It would seem to me that the two values above are then surely not independent.

Am I missing something?

This is a consequence of the Markovian assumption, which underpins all of RL.

The Markovian assumption says that it doesn't matter how we reached a given state, only that we reached it, when deciding how likely it is that we move to subsequent states. This naturally implies that our choice of actions must also depend only on the current state.

You are correct that the assumption is somewhat unrealistic. However, it usually can be made to yield a reasonable approximation of the true dynamics without involving too many state variables. By solving this approximation, we hope to find a policy that also works well in the real problem.

Here's an example. In robot navigation, the real dynamics of the robot do depend on where the robot has been in the past, because as the robot's battery drains, the voltage levels it outputs will change slightly, and its wheels may become more prone to slippage. Thus, logically, our choice of action should change based on both the current state and our choice of actions in previous states (which drained the battery to a greater or lesser degree). However, if we try to incorporate this into the model, we'll end up having a combinatorial explosion in size of the dynamics function $P(S_{t+1} | S_t, a_{t}, S_{t-1}, a_{t-1} ... S_0, a_0)$ that actually captures this process (in particular, it will now be a function of 2t inputs). This, in turn, will necessitate a combinatorial blowup in the complexity of our policy (it will increase in complexity by an exponent of 2t). To keep things tractable, we accept that the dynamics will become lossy, or we can add some extra detail to the local state (e.g. a battery level) to capture the more complex dynamics in a Markovian way. Either way though, we'll be back to a world where future actions won't depend on past ones.

To be clearer, if we assume that $P(S_{t+1} | S_t) = P(S_{t+1} | S_{t}, S_{t-i})$ for any $i$, then the other relationships you mention should not seem surprising. That assumption is the Markovian assumption. It converts the state transition function into a Matrix that represents a Markov Chain. If we don't make that assumption, then most RL algorithms do not apply to our problem.


How do I increase the size of an (almost) balanced dataset? I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for imbalanced data (as they balance the dataset). How can I do this and is it even worth it?
P.S. I know copying the same data points and appending them at the end doesn't add much value, but can we do it, and can it help to increase the prediction accuracy?

Random over sampling creates duplicates of existing examples, so applying this to your training data would be the same as increasing the weight of the oversampled examples. If it's done to all of the examples uniformly then the effects will probably cancel out. SMOTE, on the other hand, creates synthetic examples that are linear combinations of existing examples. Thus it can be thought of as a type of data augmentation, and in some situations this might improve your model's predictions.


A model for each sub-problem vs one model for the whole problem Let's say one wants to use a neural net to learn some function $g(x)$. Let's say that we know that $g$ is a combination of two functions (or two sub-problems), $g(x)=f_2(f_1(x))$, and that we have two datasets

composed of $x$ samples and their corresponding $g(x)$ labels, and
composed of $x$ samples and their corresponding $f_1(x)$ labels.

Should we use two nets, one to learn the mapping from $x$ samples to $f_1(x)$ using dataset 1 and another net to learn the mapping from $f_1(x)$ to $g(x)$ (note that we can build a dataset composed of $f_1(x)$ samples and $g(x)$ labels with the trained net), or just one net to learn mappings from $x$ to $g(x)$ using dataset 1?

Intuitively, the first option seems to be better since we take advantage of our knowledge that $f_1$ is a "sub-problem" of $g$.

The tendency in literature in the last years (at least for computer vision problems) seems to point towards the single model option (I'll try to remember to come back and add some links to papers mentioning this when I find them), although this IMO is really data- and problem-dependent.

In your case, I would set up a network for the mapping $x$ to $g(x)$, with a training-only auxiliary loss calculated on the mapping $x$ to $f1(x)$ and compare this with a model trained only on "$x$ to $g(x)$".


Why everyone is using CNN for image segmentation? I'm a newbie in artificial intelligence.

I have started to research how to do image segmentation and all the papers that I have found are about CNN. Most of them use the same network, U-net, but with little variations: with more or fewer layers, different parameter values, etc.; but with not very different results.

It seems that CNNs are in fashion and everyone uses them. Or there are other reasons that I don't know.

If everyone is getting not very different results, why are they using the same approach instead of trying different ones?

CNN is used since it is effectively an optimized use case for dealing with image data.

CNN effectively automatically extracts features from images. Other techniques are more likely to not take full advantage of the data. CNN is able to make full use of the data by also including information from adjacent pixels and downsample through layers.

Here is a paper on the performance of CNN on image data
Here is a paper comparing CNN's to other methods


Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?
Here's the update formula.

Removing the learning rate will likely yield poor convergence to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.
Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and gradually until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $\alpha = 0.1$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their book, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.


Berkeley AI Course Question on Nearly Zero Sum Games I am drawing this question from Berkeley's AI course (also not sure if it is the correct place to ask, so I apologize ahead of time)
https://inst.eecs.berkeley.edu/~cs188/pacman/course_schedule.html

Currently, I am working on section 3's Homework.

My question is: the question (Part 1, question 6). Why is it that we can only guarantee that if the Min agent acts suboptimally, the best we can hope for is the following

It seems that we can put any arbitrary value for the second node e.g. whey does it have to be -Episolon. It could be any range of values, e.g. Epsilon, in which case we would have optimised the Player A

(using X for epsilon because keyboard)

This is just a hypothesis, but if you have a maximising agent and a minimising agent, then the optimal outcome for A (maximising) is to sweep the board (X,0), while the optimal outcome for B (minimising) is (-X,0) because B is minimising A's score, not their own.

A's optimal outcome is then complicated by the factor for sub-optimality, which we then imagine approaches zero.

There seem to be a bunch of assumptions that are not articulated, though, if this hypothesis is true.


In Monte Carlo learning, what do you do when an end state is reached, after having recorded the previously visited states and taken actions? When you train a model using Monte Carlo-based learning the state and action taken at each step is recorded, and then at some point an end state is reached and the agent receives some reward - what do you do at that point?

Let's say there were 100 steps taken to reach this final reward state, would you update the full rollout of those 100 state/action/rewards and then begin the next episode, or do you then 'bubble up' that final reward to the previous states and update on those as well?

E.g.

Process an update for the full 100 experiences. Can either stop here, or...
Bubble up the final reward to the 99th step and process an update for the 99 state/action/reward.
Bubble up the final reward to the 98th step and process an update for the 98 state/action/reward.
and so on right the way to the first step...

Or, do you just process an update for the full 100-step roll-out and that's it?

Or perhaps these are two different approaches?  Is there a situation where you'd one rather than the other?

I am assuming you are asking about Monte Carlo simulation for value estimates, perhaps as part of a Monte Carlo control learning agent.
The basic approach of all value-based methods is to estimate an expected return, often the action value $Q(s,a)$ which is a sum of expected future reward from taking action $a$ in state $s$. Monte Carlo methods take a direct and simple approach to this, which is to run the environment to the end of an episode and measure the return. This return is a sample out of all possible returns, so it can just be averaged with other observed returns to obtain an estimate. A minor complication is that the return depends on the current policy, and in control scenarios that will change, so the average needs to be recency-weighted for control e.g. using a fixed learning rate $\alpha$ in an update like $Q(s,a) \leftarrow Q(s,a) + \alpha(G - Q(s,a))$
Given this, you can run pretty much any approach that calculates the returns from observed state/action pairs. You will find that the "bubble up" approach is used commonly - the process usually termed backing up - working backwards from the end of the episode.
If you have an episode from $t=0$ to $t=T$ and records of states, actions, rewards $s_0, a_0, r_1, s_1, a_1, r_2, s_2, a_2, r_3 . . . s_{T-1}, a_{T-1}, r_T, s_T$ (note indexing, reward follows state/action, there is no $r_0$ and no $a_T$), then the following algorithm could be used to calculate individual returns $g_t$:

$g \leftarrow 0$
for $t = T-1$ down to $0$:
$\qquad g \leftarrow r_{t+1} + \gamma g$
$\qquad Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha(g - Q(s_t,a_t))$

This working backwards is an efficient way to process rewards and assign them with discounting to action values for all state, action pairs observed in the episode.

Or perhaps these are two different approaches?

It would be valid to calculate only the return for the first state/action, and randomly select state/actions to start from (called exploring starts). Or in fact take any arbitrary set of estimates generated this way. You don't have to use all return estimates, but you do need to have an algorithm that is guaranteed to update values of all state/action pairs in the long term.

Is there a situation where you'd one rather than the other?

Most usually you will see the backed up return estimates to all observed state/action pairs, as this is more sample efficient, and Monte Carlo is already a high variance method that requires lots of samples to get good estimates (especially for early state/action pairs at the start of long episodes).
However, if you work with function approximation such as neural networks, you need to avoid feeding in correlated data to learn from. A sequence of state/action pairs from a single episode are going to be correlated. There are a few ways to avoid that, but one simple approach could be to take just one sample from each rollout. You might do this if rollouts could be run very fast, possibly simulated on computer. But other alternatives may be better - for instance put all the state, action, return values into a data set, shuffle it after N episodes and learn from everything.


What activation functions are currently popular? I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? e.g. ReLU, ELUs, etc. I'd appreciate any statistic on this or insight.

Currently both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.
Also these AFs, more specifically ReLU, are very fast learning AF which make them even more useful in research.
However, depending on type of NN you working on, it's always a good practice to pay some attentions to new studies.


Converting age and sex variables to a 64-unit dense layer I am studying a preprint for my own learning (https://www.medrxiv.org/content/medrxiv/early/2020/04/27/2020.04.23.20067967.full.pdf) and I am befuddled by the following detail of the neural network architecture:

This is in accord with the paper's description of the architecture (p. 5):

Age and sex were input into a 64-unit hidden layer and was concatenated with the other branches.

How can the two scalars of age and sex be implemented as a 64-unit dense layer?

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.


How can I access the weights at each training iteration of an MLP with scikit-learn? I'm building an MLP with scikit-learn. Is there a way I can access the weights and biases of the output layer per iteration? There is an option mlp.coefs_, but it outputs the trained weights when the model is compiled. I need weights per iterations how can I get weights and biases per iteration?

You can use the MLP function partial_fit to perform a single training iteration at a time. If you do retrieve the weights between calls to this function, you can see what they look like after each iteration.


Why do activation functions need to be differentiable in the context of neural networks? Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?

It is almost mandatory to have a differentiable activation function unless, of course, you have an alternative to training the network by back-propagating the error.