Is private research with low funds still feasible in the field of AI? My question assumes that a private researcher doesn't have access to anything stronger than a modern PC with a high end GPU to implement his projects. He can also use cloud computing but with limited funds as well.

Is it still feasible to do research with those restrictions? AlphaGo used 1,202 CPUs and 176 GPUs to beat Lee Sedol. Is this enormous power only required to achieve the final optimizations or have we already reached a state where high end research can only be done with larger funding?

Please include in your answer examples of recent research results and the required infrastructure that was used to create them.

Yes, you can do research in Artificial Intelligence with low funds (but you need a lot of time!). Notably, because AI is not the same as applied machine learning (indeed running ML programs on big data requires a lot of computer power). For example, knowledge representation and reasoning or natural language processing (both are subfields of AI) generally don't require a lot of computer power. Even when interested in machine learning (which is not the same as AI, just a subfield of it among others), you can use a powerful PC or laptop for that.
A lot of recent papers (probably most of them) in journals like  Artificial Intelligence, or in conferences like IJCAI (see their past proceedings) are somehow theoretical, and when something is implemented, it runs on a laptop or desktop. Notice that both AI journal and IJCAI conferences are peer-reviewed with a very selective process.
Actually it is difficult and rare to find a research paper in AI mentioning that costly equipment was needed to do the research. Costly supercomputers used for research are generally not used by AI researchers (but by researchers in physics or bioinformatics), and AI researchers often don't even have access to such facilities.
For examples, recent IJCAI2016 papers such as Coco: Runtime Reasoning About Conflicting Commitments, Interdependent Scheduling Games, Control of Fair Division, Verifying Pushdown Multi-Agent Systems against Strategy Logics, etc. don't mention any costly computation.
Actually, it is likely that most recent papers don't use and don't need large scale costly cloud computing. And some of that research might not have been implemented (perhaps by some intern) in anything more than a toy prototype.

Please include in your answer examples of recent research results and the required infrastructure that was used to create them.

Here are some recent research publications, they all explicitly mention the needed computer equipment, follow the link to read them:
Event recent IJCAI experimental papers like A Multicore Tool for Constraint Solving, Compiling Constraint Networks
into Multivalued Decomposable Decision Graphs, External Memory Bidirectional Search, Multiple Constraint Acquisition, Completion of Disjunctive Logic Programs, Eliminating Disjunctions in Answer Set Programming by Restricted Unfolding and On the Empirical Time Complexity of Random 3-SAT at the Phase Transition mention at most a multi-core workstation (e.g. at most a dual Xeon socket workstation,  often a cheaper laptop or desktop). I guess that most authors need their GPU only as a graphics card, to use their display screen. BTW, papers in other journals, such as JAIR (like Improving the Efficiency of Dynamic Programming on Tree
Decompositions via Machine Learning), when they mention computer equipment, gives similar kind of machines (laptop or at most high-end desktop).
(Actually, I don't remember having read an AI paper mentioning costly computing equipment; and I believe the reason for that is that in the academic community access to supercomputers is nearly reserved to research in other domains: physics, bioinformatics, etc. For an AI researcher gaining such access is difficult and uncommon; BTW, in H2020 European research grants computing cost above 15% of the labor cost needs to be dully justified, so is exceptional)
However, you'll better publish your software as free software or open-source, and you need a lot of time (preferably full-time, or at least half-time) to do the research work, publish it, and follow outside progress in your area. BTW, contacting a nearby university could be helpful (you could attend some seminars, etc.)
You can find interesting blogs (e.g. the one of J.Pitrat) of AI researchers working on just a laptop or a desktop.
So, most research in AI (even in Machine Learning, read papers from JMLR) are done by researchers working on a laptop or desktop. I am not even sure you need a powerful GPU to do the research. You certainly can do interesting research with a desktop computer (probably running Linux, with 32Gb RAM, some AMD Ryzen or Intel i5 or i7) costing in 2017 less than 1500€ or maybe 2500€ (and sometimes a low end laptop is enough). In rare cases (a small minority of papers), you might need a dual-socket workstation (something costing perhaps 5000€).
Notice that Big Data is not considered as a field of Artificial Intelligence. I guess (but don't really know) that research in Big Data requires access to more computer power than a PC. BTW, I am not sure that AlphaGo is really a research project, it is more an industrial demo.
PS. See also http://norvig.com/21-days.html (notice who is the author!) and this Bismon draft report (developing programming techniques, in a GPLv3+ alpha-stage software prototype, notably frame-based or semantic-network related ones, with reflection and introspection, reusable and relevant to AI), and the RefPerSys project.


How do I identify the number and type of objects in the same picture? I need to identify the number and type of all objects in a picture, so there can be multiple objects of the same type.

For example, I have a picture with $10$ animals, and I want my program to tell me that, on the picture, I have $3$ elephants, $2$ cats and $5$ dogs. However, I do not need the detection of the location of the objects. All I need is the information on the number of objects of each class, without their possible locations.

I wanted to ask you guys for help in defining the type of problem I am dealing with and maybe some suggestions about where to start looking for a solution. It would be nice if you could point out some directions, algorithms or network architectures to solve the problem described below.

Your problem might involve a combination of object recognition (or object classification), which is the problem of determining which objects are in an image, and object detection (which could also be called object localization), which is the task of locating a specific object in the image.

A naive approach to solve your problem would then be to first locate the position of all possible objects in the image (object detection), then, for each these positions, perform object recognition (or classification). Then you would count the number of detected and classified objects for each possible class.

There are several models that can be used to solve your problem.

The YOLO (You Only Look Once) model (introduced in the paper You Only Look Once: Unified, Real-Time Object Detection) (2016) can be used to detect multiple objects in an image. See the official YOLO implementation, which is part of the Darknet framework. Have also a look at this article, which explains how to use the official implementation of YOLO.

You could also use the SSD model (introduced in the paper Single Shot MultiBox Detector). There is also R-CNN (Regions with CNN features), Fast R-CNN or Faster R-CNN, but these should perform worse than both YOLO and SSD.

So, even if you do not care about the locations, YOLO (and other mentioned methods) can give them to you, in addition to the number and type of objects in the image.


How can these 7 AI problem characteristics help me decide on an approach to a problem? If this list1 can be used to classify problems in AI ...

  Decomposable to smaller or easier problems
  Solution steps can be ignored or undone
  Predictable problem universe
  Good solutions are obvious
  Uses internally consistent knowledge base
  Requires lots of knowledge or uses knowledge to constrain solutions
  Requires periodic interaction between human and computer

... is there a generally accepted relationship between placement of a problem along these dimensions and suitable algorithms/approaches to its solution?

References

[1] https://images.slideplayer.com/23/6911262/slides/slide_4.jpg

The List

This list originates from Bruce Maxim, Professor of Engineering, Computer and Information Science at the University of Michigan.  In his lecture Spring 1998 notes for CIS 4791, the following list was called,

  "Good Problems For Artificial Intelligence."

  Decomposable to easier problems
  Solution steps can be ignored or undone
  Predictable Problem Universe
  Good Solutions are obvious
  Internally consistent knowledge base (KB)
  Requires lots of knowledge or uses knowledge to constrain solutions
  Interactive

It has since evolved into this.

  Decomposable to smaller or easier problems
  Solution steps can be ignored or undone
  Predictable problem universe
  Good solutions are obvious
  Uses internally consistent knowledge base
  Requires lots of knowledge or uses knowledge to constrain solutions
  Requires periodic interaction between human and computer

What it is

His list was never intended to be a list of AI problem categories as an initial branch point for solution approaches or a, "heuristic technique designed to speed up the process of finding a satisfactory solution."

Maxim never added this list into any of his academic publications, and there are reasons why.

The list is heterogeneous.  It contains methods, global characteristics, challenges, and conceptual approaches mixed into one list as if they were like elements.  This is not a shortcoming for a list of, "Good problems for AI," but as a formal statement of AI problem characteristics or categories, it lacks the necessary rigor.  Maxim certainly did not represent it as a, "7 AI problem characteristics," list.

  It is certainly not a, "7 AI problem characteristics," list.

Are There Any Category or Characteristics Lists?

There is no good category list for AI problems because if one created one, it would be easy to think of one of the millions of problems that human brains have solved that don't fit into any of the categories or sit on the boundaries of two or more categories.

It is conceivable to develop a problem characteristics list, and it may be inspired by Maxim's Good Problems for AI list.  It is also conceivable to develop an initial approaches list.  Then one might draw arrows from the characteristics in the first list to the best prospects for approaches in the second list.  That would make for a good article for publication if dealt with comprehensively and rigorously.

An Initial High Level Characteristics to Approaches List

Here is a list of questions that an experienced AI architect may ask to elucidate high level system requirements prior to selecting an approaches.

Is the task essentially static in that once it operates it is likely to require no significant adjustments?  If this is the case, then AI may be most useful in the design, fabrication, and configuration of the system (potentially including the training of its parameters).
If not, is the task essentially variable in a way that control theory developed in the early 20th century can adapt to the variance?  If so, then AI may also be similarly useful in procurement.
If not, then the system may possess sufficient nonlinear and temporal complexity that intelligence may be required.  Then the question becomes whether the phenomenon is controllable at all.  If so, then AI techniques must be employed in real time after deployment.

Effective Approach to Architecture

If one frames the design, fabrication, and configuration steps in isolation, the same process can be followed to determine what role AI might play, and this can be done recursively as one decomposes the overall productization of ideas down to things like the design of an A-to-D converter, or the convolution kernel size to use in a particular stage of computer vision.

As with other control system design, with AI, determine your available inputs and your desired output and apply basic engineering concepts.  Thinking that engineering discipline has changed because of expert systems or artificial nets is a mistake, at least for now.

Nothing has significantly changed in control system engineering because AI and control system engineering share a common origin.  We just have additional components from which we can select and additional theory to employ in design, construction, and quality control.

Rank, Dimensionality, and Topology

Regarding the rank and dimensions of signals, tensors, and messages within an AI systems, Cartesian dimensionality is not always the correct concept to characterize the discrete qualities of internals as we approach simulations of various mental qualities of the human brain.  Topology is often the key area of mathematics that most correctly models the kinds of variety we see in human intelligence we wish to develop artificially in systems.

More interestingly, topology may be the key to developing new types of intelligence for which neither computers nor human brains are well equipt.

References

http://groups.umd.umich.edu/cis/course.des/cis479/lectures/htm.zip


How to change the backward pass for an LSTM layer that outputs to another LSTM layer? I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far.

On pg. 21 of the pdf (pg. 13 of the paper), he derives the backward pass equations for output gates. He writes

$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} \delta_{k}(t)$$.

If we replaced $\delta_{k}(t)$, the expression becomes

$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} f'(net_k(t)) e_k(t)$$.

He states that the result of the partial derivative $\frac{\partial y^k(t)}{\partial y_{out_{j}}}$ comes from differentiating the forward pass equations for the output units.

From that and from the inclusion of $e_k(t)$, the paper implies that there is only one hidden LSTM layer. If there are multiple hidden LSTM layers, it wouldn't make sense.

   Because if $k$ is the index of LSTM cells that the current cell is outputting to, then $e_k(t)$ would not exist since the cell output isn't compared with the target output of the network. And if $k$ is the index of output neurons, then $w_{k c_{j}^{v}}$ would not exist since the memory cells are not directly connected to output neurons. And $k$ cannot mean different things since both components are placed under a sum over $k$. Therefore, it only makes sense if the paper assumes a single LSTM layer.

So, how would one modify the backward pass derivation steps for an LSTM layer that outputs to another LSTM layer?

My understanding is that you have posted the last equation on p. 19 in the dissertation (please correct me if I'm wrong).

The derivation is indeed for a single LSTM layer as ek(t) is the error at the output (the assumed network topology is also mentioned just below Eq. (2.8) in the dissertation):

  Finally, assuming a layered network topology with a standard input layer, a hidden layer consisting of memory blocks, and a standard output layer[...]

Let's assume now that there are two stacked LSTM layers. In this case, the derivation in the dissertation applies to the last hidden LSTM layer (the one below the output layer), where Eqs. (3.10) -- (3.12) give you the partial derivatives for the weights at each gate for a cell in that layer. To derive the deltas for the hidden LSTM layer below, you have to compute the partial derivatives with respect to the portions of netcvj(t), netinj(t) and netfj(t) terms corresponding to the outputs of the preceding hidden LSTM layer, and then use those in the same way you used ek(t) for the current LSTM layer. Underneath all that unappealing notation is just the usual multi-layer backprop rule (well, with the truncated RTRL twist). If you stack more LSTM layers, just keep propagating the errors further down through the respective gates until you reach the input layer.

For a slightly more intuitive explanation, if you look at Fig. 2.1 in the dissertation, you can assume that in a multi-layered LSTM the IN in fact includes the OUT of the preceding LSTM layer.

Edit

There is a nice diagram of the flow of partial derivatives here (also see subsequent slides).

In this example, all xt terms represent the external input to that layer at time step t. In a multi-layer LSTM, this includes input from the LSTM layer below the current one. To propagate the error to the layer below, take the derivatives with respect to all gate weights corresponding to xt and apply the chain rule.

The reason why the derivation for the output gate weights is different is that the gating is applied after the cell state is updated, whereas the cell, input and forget gates are applied before that. This matters when computing the gradients.


Are AI algorithms capable of self-repair? Do AI algorithms exist which are capable of healing themselves or regenerating a hurt area when they detect so?
For example: In humans if a certain part of brain gets hurt or removed, neighbouring parts take up the job. This happens probably because we are biologically unable to grow nerve cells. Whereas some other body parts (liver, skin) will regenerate most kinds of damage.
Now my question is does AI algorithms exist which take care of this i.e. regenerating a damaged area? From my understanding this can be achieved in a NN using dropout (probably). Is it correct? Do additional algorithms (for both AI/NN) or measures exist to make sure healing happens if there is some damage to the algorithm itself?
This can be particularly useful in cases where say there is a burnout in a processor cell  processing some information about the environment. The other processing nodes have to take care to compensate or fully take-over the functions of the damaged cell.
(Intuitionally this can mean 2 things:

We were not using the system of processors to its full capability.
The performance of the system will take a hit due to other nodes taking over functionality of the damaged node)

Does this  happen in the case of brain damage also? Or is my inferences wrong? (Kindly throw some light).
NOTE : I am not looking for hardware compensations like re- routing, I am asking for non-tangible healing. Adjusting the behavior or some parameters of the algorithm.

Yes, this was an active area of research in a number of different AI fields.

Probably the most directly related work is Bongard, Zykov & Lipson's self-repairing robots from the early 2000's.

There's some more recent work from Mark Yim that you can see here too.

There are lots of different ways to do this, but Bongard et al's approach was probably the most elegant. The basic idea was to frame it as a learning problem: the robot is able to learn the shape of its body by performing controlled experiments. When the body is damaged, the robot can detect that it's body has changed shape (sensors don't report the expected values when it tries to move), perform new experiments to determine the extent of the damage, and then generate new movements that work around the damaged area. Lipson covers the basics of this system very briefly in this video.

The more modern system uses a similar approach, but tries to repair its body, rather than working around the damage. It's got an internal model of what it's body should look like, and then a set of cameras that help it locate the various pieces and move them to reassemble.

Dropout is sort of a similar idea, but dropout is usually done to encourage redundancy during training, which can help a model avoid overfitting. It's usually not done explicitly to heal a damaged system, although it would make a system more resistant to damage in the first place.


What are the available selection methods in genetic algorithms? In a genetic algorithm, there are different steps. One of those steps is the selection of chromosomes for reproduction. What are the available selection strategies in genetic algorithms?

Overview
There are many selection operators (or methods) for evolutionary algorithms. These selection methods differ in the way they evaluate the individuals given their fitness or how they compute their fitness in the first place. Some selection operators select individuals deterministically, while others select them stochastically, which can prevent premature convergence, given that, for instance, in fitness proportionate selection (aka roulette wheel selection), a well-known selection operator, all individuals can be selected, although the fitter individuals have a higher probability of being selected. Each selection operator has different advantages and disadvantages compared to other selection operators.
All these details are covered from chapter 22 (p. 166 and 203 of the linked pdf) onwards of the well-written book Evolutionary Computation 1 Basic Algorithms and Operators by Thomas Bäck et al. These concepts are also described in Computational Intelligence: An Introduction (section 8.5, p. 135), in case you want to have an alternative reading.
For completeness, I will enumerate and briefly describe some of the well-known selection operators.
Fitness proportionate selection (FPS)
This selection method is also known as roulette wheel selection (RWS). In FPS, the probability of an individual being selected is proportional to its fitness. More precisely, let $f_i$ be the fitness of individual $i$, then the probability of that individual being selected is
$$p_{i}={\frac {f_{i}}{\Sigma _{j=1}^{N}f_{j}}}$$
FPS does not handle minimization problems directly and it requires the fitnesses to be non-negative (given that probabilities are in the range $[0, 1]$), but the minimization problem can be transformed into a maximization one. The solutions are thus selected probabilistically in this case. There are other problems/issues with this selection method, which are described in the cited book (chapter 23, p. 172).
Tournament selection (TS)
In TS, at each generation, a set/group $Q = \{ a_1, \dots, a_q \}$ of $|Q| = q$ individuals is sampled (with or without replacement) at random from the current population of all individuals $P$. Successively, this group $Q$ takes part in a so-called tournament, i.e. a loop of $\lambda$ iterations, where, for each iteration $j$, the individual from the group $Q$ with the highest fitness $a_1'$ is selected (deterministically or stochastically) and inserted in a new set $S$, which will contain the individuals that you will mutate and recombine. After this first iteration $j$, $S$ contains only one individual, $S = \{a_1' \}$. In the second iteration, we perform the same thing, and obtain $S = \{a_1', a_2' \}$, and so on. So, after $\lambda$ iterations, $S$ contains $\lambda$ individuals, i.e. $|S| = \lambda$. Note that $a_1'$ is not necessarily equal to $a_1 \in Q$. The individuals in $S$ are the ones that undergo mutation and cross-over in this generation. In the next generation, you repeat this tournament process for selecting the best $q$ individuals. Typically, you need to have at least $q = 2$ (i.e. binary tournament selection), so that you have the opportunity to choose 2 different individuals for the cross-over. If $q=1$, individuals are picked randomly from the population (given that you do not perform any tournament at all).
Rank-based selection (RS)
In RS, only the rank ordering of the individuals within the current population determines the probability of being selected. So, in RS, an individual has a probability of being selected not proportional to its fitness (as in FPS), but it is based on its rank (order) with respect to the other individuals. So, if the individual $a$ has is ranked $1$st and individual $b$ is ranked $2$nd, even if $a$ had a fitness a lot higher than individual $b$, then its probability of being selected would not be a lot higher than the probability of selecting $b$, while, in FPS, this would be the case. There are different ways of computing the rank, such as linear or non-linear. The details of how the rank can be computed are described in chapter 25 of the cited book.
Other selection methods

Boltzmann selection (chapter 26)
Soft brood selection (chapter 27)
Disruptive selection (chapter 27)
Competitive selection (chapter 27)
Truncation selection (chapter 29)

Other related concepts
There's also the related concept of elitism, where the best individuals of the current population are carried over to the next population/generation, and hall of fame, which is a data structure that stores the best individual for each generation.


Building AI from chess - data shape from simulation Problem

My problem is the following: Given 1000 wins, losses, and ties from a chess simulation I am using, what shape should each game be (I.e., sequence of moves leading to win/loss/tie) in order to build an deep neural network on it?

Literature Review

Kind of got me in the right direction.
Great theoretical way of thinking about it, but left me wanting detail.
Gave a good high level tutorial, but not enough detail.
Alpha beta pruning python example, but no neural network.
Neural network using reinforcement learning, but I cant figure out what the (X,y) shapes are and their meaning from just digging into the source code.

Current Situation

I currently have a sample of 1000 wins, 1000 losses, and 1000 ties from the python-chess api, so if i is the index of a game, then this is the structure of the current dataset I am working with:

game_i -> (num_moves_i,8,8,16)

So each game_i where i in {1..3000} and num_moves_i is variable depending on the game (E.g., 14 for a good winning game, or 765 moves for a tie game). The 16 represents a one_hot_encoding for one of the 16 unique board pieces. The data set is also an alternating board state, so:

game_i[0] == board state of whites first move
game_i[1] == baord state of blacks first move

Furthermore, I also have alpha-beta pruning and maximin working, so for each move I have an intrinsic value associated with it using recursion three levels deep. Leading my current approach to a regression of a given move, essentially leading me to believe the AI would simply learn the heuristic and predict a value the heuristic would give.

Summary

Clearly my proof of concept 1000 winning games is not enough to make a meaningful AI, but that isn't my goal. I want to learn the techniques, not produce an enterprise scale chess AI.

Does the tensor shape make sense?
Is this a reinforcement learning problem? If so, how can I shape my current framing into that type of thinking? Theory in this area would be greatly appreciated as I am less familiar with it.
Is this a RNN/LSTM problem? (E.g., predict the next board state).
Is this a regression problem?
Is this a sequence mining problem?

What is the standard approach to framing this problem, once you have data falling through the pipeline.

Your support is more than appreciated.

* UPDATE (STILL IN RESEARCH) *

With further research, a candidate label for the training data is the tensor and the item from the state-space that was selected with that board state. Then only keep games containing sequences of moves which obtain a cumulative value >= epsilon. This would require a one_hot_encoding of all moves played in all games we wish to train on, as labels. E.g., (game_i,board_ij,e2e6)

Not an expert here by any means. I think the simplest way to approach the chess problem is to have three components a) a game logic component, that can generate all the possible legal moves from a given board position b) an evaluation function that can score a given game position and c) your minimax/alphabeta algorithm that searches through the tree of possible moves generated by a) looking for the branch that returns the best score using b). Here, the machine learning part would be the evaluation function. You would just transform each board position in your data into a feature vector, and train some kind of classifier to return the probability of win/lose at that given board position. I think it's highly doubtful that this would work with such a small data set though.


How to know if a real-time classifier is achivable in a low-power emdedded system? Say I have an Machine/Deep learning algorithm I developed on a desktop pc to achieve a real-time classification of time series events from a sensor. Once the algorithm is trained and performs good, I want to implement it on an low power embbeded system, with the same sensor, to classify events in real-time:

How can I know if the low power embedded system is fast enough to allow real-time classification regarding the algorithm (knowing it in advance would avoid to implement and try multiple architectures) ?
Machine/Deep learning algorithm are usually developed in python. Is there easy ways to transfer the code from python to a more embeddable langage ?

Regarding your first point, it depends on what neural network you would like to use, the sensor temporal resolution, and the capabilities of the embedded system. You can figure out the number of operations required for a forward pass of your network, then when combined with the internal clock of the embedded system, you can calculate the approximate time it would take for one classification event in real time.
A good explanation is given here
What is the computational complexity of the forward pass of a convolutional neural network?
If you have the computational complexity of your network, and the number of CPU cycles per seconds, you can roughly approximate the time.


When could a linear discriminant give excellent or possibly even the optimal classification accuracy? I am actually reading the linear classification. There is a question in the question set behind the chapter in the book as follows:

Sketch two multimodal distributions for which a linear discriminant could give excellent or possibly even the optimal classification accuracy.

I have no idea about how to get the optimal solution on linear classification, any ideas?

A linear model has a linear decision boundary. So in the case of the question, you need to draw two multimodal distributions whose domains do not overlap at all and then you can just say the linear model would should all to the right of some number would be class 1 and all to the left would be class 2


Difference in UCB performance when scaling the rewards I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I make the reward continuous (and bounded [0, 1]) then $\epsilon$-greedy remains good but UCB1 performance plummets. As an experiment, I just scaled the reward of 1 by a factor of 1/10 which negatively influences the performance.
I have plotted the reward values estimated by the algorithms and see that (due to the confidence interval term) UCB1 largely overestimates the rewards.
Is there a practical trick to fix that? My guess is that the scaling coefficient $c$ in front of the upper confidence bound is meant just for this case. Nevertheless, the difference in performance is staggering to me. How do I know when and what scaling coefficient will be appropriate?
===
update 1 The reward distribution is very simple. There are 17 arms, for arm 3 and 4 the learning algorithm gets a reward of 1, the other arms return reward of 0. No stochasticity, the algorithm has 1000 iterations.
If I scale the reward by a factor 1/10, for instance, then UCB1 takes a whole lot of time to start catching up with $\epsilon$-greedy

Epsilon greedy is unaffected by scaling of rewards, it always selects a random action with a probability of epsilon.
On the other hand, if we look at the formulation of UCB (Section 2.7 of Reinforcement Learning, Sutton and Barto):
$$A_t \doteq  \underset{a}{\operatorname{argmax}} [\mathcal{Q}_t(a) + c \sqrt{\frac{\ln t}{N_t(a)}}]$$
Where $Q_t(a)= \frac{R_1 + R_2 + \dots + R_{n-1}}{n-1}$ is the average of rewards seen so far for that action. By scaling the rewards, for example by 1/10, you are essentially scaling $Q_t(a)$ by 1/10 which will change UCBs behaviour to favour the exploratory right hand term $c\sqrt{\frac{\ln t}{N_t(a)}}$. If you want to offset your scaling of the rewards you should scale c by the same amount. Theoretically, I believe this should result in performance the same as that achieved pre-scaling.
In fact, another way of looking at it is that when using UCB, you could dispense with the $c$ constant altogether and simply scale the rewards to tune the exploitation/exploration trade-off. Smaller rewards favour exploration, larger rewards favour exploitation.


How does the optimization process in hindsight experience replay exactly work? I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the infamous sparse reward problem. It is based on the intuition that human beings constantly try and learn something useful even from their past failed experiences.
I have almost completely understood the concept. But in the algorithm posited in the paper, I don't really understand how the optimization works. Once the fictitious trajectories are added, we have a state-goal-action dependency. This means our DQN should predict Q-Values based on an input state and the goal we're pursuing (The paper mentions how HER is extremely useful for multi-RL as well).
Does this mean I need to add another input feature (goal) to my DQN? An input state and an input goal, as two input features to my DQN, which is basically a CNN?
Because in the optimization step they have mentioned that we need to randomly sample trajectories from the replay buffer and use those for computing the gradients. It wouldn't make sense to compute the Q-Values without the goal now, because then we'd wind up with duplicate values.
Could someone help me understand how exactly does the optimization take place here?
I am training Atari's "Montezuma's Revenge" using a double DQN with Hindsight Experience Replay (HER).

Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments


How to compute the action probabilities with Thompson sampling in deep Q-learning? In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy $\mu(a)$ (e.g., if we want to use importance sampling).
In my case, I am using Deep Q-Learning and selecting actions using Thompson Sampling. I implemented this following the approach in "What My Deep Model Doesn't Know...": I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.
So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.
I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.
Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.


How does the neural-network know how to tweak weights for a specific neuron? I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside each neuron in the hidden layers that defines how this is an important neuron for the correct result in some way? How does the network know how to tweak those weights for that specific neuron?

tl;dr
The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.
A deeper look...
Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a loss $J$. The loss (or cost) function tells us how far off we are from the target. This is what we want to reduce, so that the predictions fall closer to the target.
By computing the partial derivative of the loss function $J$ w.r.t a parameter $w_i$ (so this is just one partial derivative and not the full gradient vector)
$$
\frac{\partial J}{\partial w_i}
$$
the NN uncovers two pieces of information:

The slope of $J$ w.r.t $w_i$, which tells the NN how much $w_i$ affects $J$.
Its sign, which tells the NN which way to tweak $w_i$ to decrease (or increase) the value of $J$.

By making the parameter updates depend on the derivative
$$
w_i^{new} \leftarrow w_i^{old} - \lambda \frac{\partial J}{\partial w_i}
$$
the NN causes parameters that affect the loss the most, to be updated the most.
You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.


Is Lisp still being used to tackle AI problems? I know that language of Lisp was used early on when working on artificial intelligence problems. Is it still being used today for significant work? If not, is there a new language that has taken its place as the most common one being used for work in AI today?

Overall, the answer is no, but the current paradigms owe a lot to LISP. The language most commonly used today is python.

Relevant answers:

Stack Overflow thread explaining why LISP was thought of as the AI language: Why is Lisp used for AI
Quora answer by Peter Norvig, who wrote a popular textbook on the subject and is currently Director of Research at Google: Is it true that Lisp is highly used programming language in AI?

LISP pioneered many important concepts in what we now call functional programming, with a key attraction being how close the programs were to math. Many of these features have since been incorporated into modern languages (see the Wikipedia page). LISP is very expressive: it has very little syntax (just lists and some elementary operations on them) but you can write short succinct programs that represent complex ideas. This amazes newcomers and has sold it as the language for AI. However, this is a property of programs in general. Short programs can represent complex concepts. And while you can write powerful code in LISP, any beginner will tell you that it is also very hard to read anyone else's LISP code or to debug your own LISP code. Initially, there were also performance considerations with functional programming and it fell out of favor to be replaced by low level imperative languages like C. (For example, functional programming requires that no object ever be changed ("mutated"), so every operation requires a new object to be created. Without good garbage collection, this can get unwieldy). Today, we've learned that a mix of functional and imperative programming is needed to write good code and modern languages like python, ruby and scala support both. At this point, and this is just my opinion, there is no reason to prefer LISP over python.

The paradigm for AI that currently receives the most attention is Machine Learning, where we learn from data, as opposed to previous approaches like Expert Systems (in the 80s) where experts wrote rules for the AI to follow. Python is currently the most widely used language for machine learning and has many libraries, e.g. Tensorflow and Pytorch, and an active community. To process the massive amounts of data, we need systems like Hadoop, Hive or Spark. Code for these is written in python, java or scala. Often, the core time-intensive subroutines are written in C.

The AI Winter of the 80s was not because we did not have the right language, but because we did not have the right algorithms, enough computational power and enough data. If you're trying to learn AI, spend your time studying algorithms and not languages.


Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct? I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is.
Assuming that it has only 1 hidden layer, it is a classification problem with 1 output node (so it's a binary classification task), has N input nodes for N features in the data set, and every node is connected to every node in the next layer.
I'm thinking that to ensure that the network is able to extract all of the useful relations between the data, then every piece of data must be linked to every other piece of data, like in a complete graph. So, if you have 6 inputs, there must therefore be 15 edges to make it complete. Any more and it will be recomputing previously computed information and any less will be not computing every possible relation.
So, if a network has 6 input nodes, 1 hidden node, 1 output node. There will be 6 + 1 connections. With 6 input nodes, 2 hidden nodes, and 1 output node, there will be 12 + 2 connections. With 3 hidden nodes there will be 21 connections. Therefore, the hidden layer should have 3 hidden nodes to ensure all possibilities are covered.
This answer discusses another method. For the sake of argument, I've tried to keep both examples using the same data. If this idea is computed with 6 input features, 1 output node, $\alpha = 2$, and 60 samples in the training set, this would result in a maximum of 4 hidden neurons. As 60 samples is very small, increasing this to 600 would result in a maximum of 42 hidden neurons.
Based on my idea, I think there should be at most 3 hidden nodes and I can't imagine anymore being useful, but would there be any reason to go beyond 3 and up to 42, like in the second example?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.

It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network).

Even with one input neuron it is not possible. That is because even with one input, the output can be an arbitrarily complex mapping to classes. A good example with two inputs that would require an infinite number of hidden neurons to supply a simple classifier would be classifying x,y points as being in the Mandelbrot set.

In some, more constrained, examples, with well-defined functions, you can construct a minimal neural network that solves the problem perfectly. For instance a neural network model of XOR can be made with two hidden neurons (and six links). However, this kind of analysis is limited to simple problems. You might be able to come up with some variation of your idea if all inputs were boolean, and the neural network limited to some combined bitwise logic on all the inputs.

Your idea of matching number of edges to number of possible interactions between inputs does not work because you are only considering the most basic kind of interaction between two variables, whilst variables can in practice combine in all sorts of ways to form a function.

In addition, each neuron in a hidden layer works with a linear weighted sum, plus a fixed transformation function. This is in no way guaranteed to match the function shape that you are trying to approximate with the neural network. An analogy that you might be aware of is discrete Fourier transforms - it is possible to model any part of a function by combine sine and cosine waves of different frequencies, but some functions will require many such waves in order to be represented accurately.

Your link to the answer in Cross Validated Stack Exchange gives you a rule of thumb that the writers find often works with the kinds of data that they work with. This is useful experience. You can use such rules as the starting point for searching for architecture that works on your problem. This will likely be more useful than your idea based on counting the possible variable interactions. However, in both cases, the most important step is to perform a test with some unseen examples, and to search for the best neural network architecture for your problem.

There are things you can do with variable interactions though. For instance, try looking for linear correlations between simple polynomial combinations of variables and your target variable, e.g. plot $x_1 x_2$ vs $y$ or $x_3^2 x_4$ vs $y$ . . . you may find some combinations have a clear signal implying a relationship. Take care if you do this sort of thing though, if you test very many of these, you will find a linear relationship purely by chance that looks good initially but turns out to be a dud when testing (it's a form of overfitting). So you should generally test a lot less than the size of your dataset, and limit yourself to some modest maximum total power.


Do the rows of the design matrix refer to the observations or predictors? I attempt to understand the formulation of dictionary learning for this paper:

Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution
Multimodal Task-Driven Dictionary Learning for Image Classification

Both papers used the exact formulation in two different domains.
Based on my understanding, in common machine learning, we formulate our matrices, from vectors, as rows to be observations, columns to be predictors.
Given a matrix, $A$:
\begin{array}{lcccccc}
& p_1 & p_2 &  p_3 &  p_4 &  p_5 & \text { label } \\
o_1 & 1 & 2 & 3 & 4 & 1 & 1 \\
o_2 & 2 & 3 & 4 & 5 & 2 & 1 \\
o_3 & 3 & 4 & 5 & 6 & 2 & 0 \\
o_4 & 4 & 5 & 6 & 7 & 3 & 0
\end{array}
So, using a math notation and excluding the label, I can define this matrix, $A = [o_1, o_2, o_3, o_4]  ∈ R^{4×5}$, as $A = [{(1, 2, 3, 4, 1), (2, 3, 4, 5, 2), (3, 4, 5, 6, 2), (4, 5, 6, 7, 3)}]$, and in numpy:
import numpy as np

A = np.array([[1, 2, 3, 4, 1],
              [2, 3, 4, 5, 2],
              [3, 4, 5, 6, 2],
              [4, 5, 6, 7, 3]])

A.shape
# (4, 5)

Am I right?

Based on my understanding, in common machine learning, we formulate our matrices, from vectors, as rows to be observations, columns to be predictors.

The rows (or, in general, the first dimension of your tensor) are typically the observations. For example, in TensorFlow, the first dimension of the input tensor typically refers to the batch size, i.e. the number of observations. If you are using Pandas (a Python library to manipulate data), the rows are typically the observations and the columns are the predictors.
However, in general, it does not really matter which convention you use, provided that you use one of the conventions consistently in your implementation (i.e. you choose one of the conventions and you stick with it throughout all your code, to avoid complexity), and make it clear in the documentation. So, you can have a matrix where either the rows or columns are observations and, consequently, the columns or, respectively, rows are the features (aka predictors or independent variables).
Anyway, it is probably a good idea to be consistent with existing literature and implementations/libraries, so you should probably use the rows for the observations.


What is the intuition behind TD($\lambda$)? I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD($\lambda$) as a type of "truncated" Monte Carlo learning?

TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.
More precisely, TD($\lambda$) is temporal-difference learning with a $\lambda$-return, which is defined as an average of all $n$-step returns, for all $n$, where an $n$-step return is the target used to update the estimate of the value function that contains $n$ future rewards (plus an estimate of the value function of the state $n$ steps in the future). For example, TD(0) (e.g. Q-learning is usually presented as a TD(0) method) uses a $1$-step return, that is, it uses one future reward (plus an estimate of the value of the next state) to compute the target. The letter $\lambda$ actually refers to a parameter used in this context to weigh the combination of TD and MC methods. There are actually two different perspectives of TD($\lambda$), the forward view and the backward view (eligibility traces).
The blog post Reinforcement Learning: Eligibility Traces and TD(lambda) gives a quite intuitive overview of TD($\lambda$), and, for more details, read the related chapter of the book Reinforcement Learning: An Introduction.


Should the agent play the game until the end or until the winner is found? I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. For example, the game might last 100 turns, but it's possible to know that one of the players won at move 80, because some winning condition was satisfied. The last 20 moves don't change the outcome of the game. If people were playing this game, they, would play it to the very end, but the agent doesn't have to.
The agent will be using memory replay to learn from the experience. I wonder, is it helpful for the agent to have the experiences after the winning condition was satisfied for a more complete picture? Or is it better to terminate the game immediately, and why? How would this affect agent's learning?

You should probably grant reward at the point that the game is logically won. This will help the agent learn more efficiently, by reducing the number of timesteps over which return values need to be backed up.
Stopping the episode at that point should also be fine, and may add some efficiency too, in that there will be more focused relevant data in the experience replay. It seems like on the surface that there is no benefit to exploring or discovering any policy after the game is won, and from the comments no expectation from you as agent developer that the agent has any kind of behaviour - random actions would be fine.
It is still possible that the agent could learn more from play after a winning state. It would require certain things to be true about the environment and additional work from you as developer.
For example, if the game has an end phase where a certain kind of action is more common and it gains something within the game ("victory points", "gold" or some other numbered token that is part of the game mechanics and could be measured), then additional play where this happened could be of interest. Especially if the moves that gained this measure could also be part of winning moves in the earlier game. To allow the agent to learn this though, it would have to be something that it predicted in addition to winning or losing.
One way to achieve this is to have a secondary learning system as part of the agent, that learns to predict gains (or totals) of this resource. Such a prediction could either be learned separately (but very similarly to the action value) and fed into the q function as an input, or it could be a neural network that shares early layers with the q function (or policy function) but with a different head. Adding this kind of secondary function to the neural network can also have a regularising effect on the network, because the interim features have to be good for two types of prediction.
You definitley do not need to consider such an addition. It could be a lot more work. However, for some games it is possible that it helps. Knowing the game, and understanding whether there is any learning experience to be had as a human player to play on beyond winning or losing, might help you decide whether looking into trying to replicate this additional experience for a bot. Even if it works, the effect may be minimal and not worth the difference it makes. For instance running a more basic learning agent for more episodes may still result in a very good agent for the end game. That only costs you more run time for training, not coding effort.


How does a single hidden layer affect output? I'm learning about multilayer perceptrons, and I have a quick theory question in regards to hidden layer neurons.

I know we can use two hidden layers to solve a non-linearly separable problem by allowing for a representation with two linear separators. However, we can solve a non-linearly separable problem using only one hidden layer.

This seems fine, but what kind of representation does one hidden layer add? And how is the output of the network affected?

I've drawn a diagram of a multilayer perceptron with one hidden layer neuron. I used this same layout to solve a non-linearly separable problem. The single hidden layer node is inside the red square. Forgive my poor MS-Paint skills.

When we add a single layer with a non-linear activation function, right after the application of the activation function, a new basis function is found (for that neuron), which is some combination of the weights and biases, which acts as a new way to view or analyze the feature sets.

With an increasingly deeper network, we keep finding representations, which are new basis vectors of the combination of the previous layer's weights and biases, that is, higher-level representations.

If they're error-free, they'll give better performances, but if small errors creep in earlier basis vectors, the error increases through depth.

A nice analogy is the Taylor series, where $1$, $x$, $x^2$, and so on, are the basis vectors for estimating the function in 1D.


What does "statistical efficiency" mean in this context? Consider the following statement(s) from Deep Learning book (p. 333, chapter 9: Convolutional Networks)

Convolution is thus dramatically more efficient than dense matrix
multiplication in terms of the memory requirements and statistical
eﬃciency.

Book is saying that statistical efficiency is due to the decrease in the number of parameters due to convolution (using kernel) compared to fully connected feed forward neural networks.
What is meant by statistical efficiency in this context? And how does decrease in the number of parameters increase statistical efficiency?

Statistical efficiency in this context essentially means that a CNN would require fewer training examples than a fully connected network to learn. Intuitively this seems reasonable: more parameters to learn should mean more samples needed. Of course it is always desirable to minimise the number of training samples needed, so that's a definite advantage of CNNs.
There is a paper on the efficiency of CNNs which attempts to make that statement more precise. They examine the case of a convolutional network using a linear activation function.


Variational Autoencoder task for better feature extraction I have a CNN with the regression task of a single scalar.
I was wondering if an additional task of reconstructing the image (used for learning visual concepts), seen in a DeepMind presentation with the loss and re-parametrization trick of Variational Autoencoder, might help the principal task of regression.

So you can imagine some convolutions with the role of feature extraction with some output X (let's say a vector of 256 values), that X goes into the VAE which computes Z and then the reconstructed image. And then the original regression task would take either X or Z in order to compute that scalar value.

Has anyone tried such an approach, is it worth the work?
Thank you

I have not worked on this but I think I can give you a theoretical perspective of using VAE's. Regression is a Supervised Learning task and is basically a mapping from Input to Output where the Neural Net will approximate the function $f(input) = output$.

VAE's on the other hand are good for finding how a latent variable affects the output. For example, if you have a task of training on a persons facial emotions, and if your latent space contains 2 variables $z_1$ and $z_2$ then you might find varying $z_1$ varies the amount of smile on the face, while varying $z_2$ might give the amount of drooping of eyes. I suggest you check this video from Stanford at ~44:00 to see this actually happens or check this blog. So VAE's might have been useful if your output contained more features which would vary according to variations in latent variables, but a single scalar output can only tell you about the rate of effect on varying a latent variable.

But if your job is only for better regression, Auto-Encoders are the better alternative, since it has an inherent de-noising ability and sufficient training might help in de-noising the input, and thus provide better results if classified on the basis of latent variables.

An approach, which I think is kind of similar (to your thinking) has been proposed by Kingma, et al. for Semi Supervised learning in this paper. The paper has very poor description of the method so I would suggest you check out this blog. They have used an additional classifier for reconstruction of the original input and trained the classifier when labels are present.


Is the cube root function suitable as a n activation function? I am trying to design a neural network on Python.

Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph:

Is this suitable?

There are a few traits that you want the activation function to have, and cube roots rate as OK-ish:

Nonlinear – check.
Continuously differentiable – no. There is a problem at $x=0$. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach $x=0$, because $\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$
Range considerations – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.
Monotonic – check.
Monotonic derivative - no.
Approximates identity near the origin – no, the approximation is bad near the origin.

If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.

I would worry about the high gradients near $x=0$, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than $1$ (or less than $-1$) then treat the gradient as if it were $1$ (or $-1$) for the rest of backpropagation.

The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.


What should I do when I have a variable-length sequence when instantiating an LSTM in Keras? In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time. What if my sequence is larger than 128? How to interpret this? Can anyone please explain me? Thank in advance.

In Keras, what you specify is the hidden layer size. So :

LSTM(128)

gives you a Keras layer representing a LSTM with a hidden layer size of 128.

As you said :

  From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time

So if you picture your RNN for one time step, it will look like this :

And if you unfold it in time, it look like this :

You are not limited in your sequence size, this is one of the feature of RNN : since you input your sequence element by element, the size of the sequence can be variable.

That number, 128, represent just the size of the hidden layer of your LSTM. You can see the hidden layer of the LSTM as the memory of the RNN.

Of course the goal is not for the LSTM to remember everything of the sequence, just link between elements. That's why the size of the hidden layer can be smaller than the size of your sequence.

Sources :

Keras documentation
This blog

Edit

From this blog :

  The larger the network, the more powerful, but it’s also easier to overfit. Don’t want to try to learn a million parameters from 10,000 examples – parameters > examples = trouble.

So the consequence of reducing the size of hidden state of LSTM is that it will be simpler. Might not be able to get the links between the element of the sequence. But if you put a too big size, your network will overfit ! And you absolutely don't want that.

Another really good blog on LSTM : this link


When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true? When discussing why the policy improvement theorem is true, when we do Monte Carlo control by updating greedily, it says on page 98 of Sutton and Barto's book (2nd edition) that:
$$
\begin{aligned}
q_{\pi_{k}}\left(s, \pi_{k+1}(s)\right)
&=
q_{\pi_{k}}\left(s, \underset{a}{\arg \max } q_{\pi_{k}}(s, a)\right)
\\
&=
\max _{a} q_{\pi_{k}}(s, a)
\\
&
\geq
q_{\pi_{k}}\left(s, \pi_{k}(s)\right)
\\
&
\geq v_{\pi_{k}}(s) \end{aligned}
$$
I don't understand why the last inequality is not an equality.
The policy improvement theorem was derived on page 78 for deterministic policies.
So, the $\pi_{k}(s)$ we see in $q_{\pi_k}(s, \pi_{k}(s))$ is a fixed action, call it $a'$. Then, in this case, since $v_{\pi_k}(s)= \sum_a \pi_k(a|s) q_{\pi_k}(s, a) =   1 * q_{\pi_k}(s, a') = q_{\pi_k}(s, a')$ (where the second equality is because the probability of all other actions is 0), shouldn't the last inequality be an equality? When is it possible that we have a greater than relation?

You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$.
The $\geq$ inequality is also correct, of course, and it could be that the authors' intention was to show that $\pi_{k+1}$ and $\pi_k$ satisfy the condition for the policy improvement theorem:

Let $\pi$ and $\pi'$ be any pair of deterministic policies such that, for all $s\in\mathcal{S}$,
$$q_\pi(s,\pi'(s))\geq v_\pi(s)\tag{4.7}$$

So they kinda relaxed the equality to show that the condition is satisfied "word-for-word" for $\pi'=\pi_{k+1}$ and $\pi=\pi_k$.
Although, I must say that they don't seem to be consistent in this - for example, they keep the strict equality at the derivation at the bottom of the page 78. So it might as well be that you've discovered a typo in Sutton and Barto.


Will attention based networks prevail over RNN and LSTM? There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish Vaswani et all, 2017 caught my attention earlier.  Are their claims that attention networks will supercede RNNs and LSTMs credible?

I think there are two parts to answering this question.

First, about the specific paper that has been mentioned.

The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more modest claim that the new attention-based architecture can outperform existing modes on a machine translation task (which is supported with experiments), and that, maybe, in the future, attention-based models could work well in other domains, or provide advantages in interpretability.

However, there's also a second way to interpret this question. Attention-based networks are getting a lot of buzz in the AI community this year and to some extent last year. It seems likely that there are a lot of research efforts underway to apply them to other kinds of problems. If these efforts pan out, then they could indeed become the tool of choice.

At this point however, the answer to whether they will has to be "Maybe". There is no good underlying theory for why some NN architectures work better than others on a given kind of problem. We have post-hoc explanations for some (like, for instance, CNNs), but in the absence of a general theory, we're better off waiting for the empirical results before making a call. In particular, the No Free Lunch Theorem tells us that if Attention-based networks work very well on one kind of problem, we should actually expect them to work poorly on others! I suspect we'll have a better answer to this question in 1-2 more years however.


What is $ \nabla_{\theta_{k-1}} \theta_{k}$ in the context of MAML? I am attempting to fully understand the explicit derivation and computation of the Hessian and how it is used in MAML.  I came across this blog: https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html.
Specifically, could someone help to clarify this for me: is this term in the red box literally interpreted as the gradient at $\theta_{k-1}$ multiplied by the $\theta_k$?

$\nabla_{\theta_{k-1}} \theta_k$ is gradient of $\theta_k$ with respect to $\theta_{k-1}$, it follows chain rule as noted in the side comment in the image. $\nabla_{\theta} \mathcal L(\theta_k)$ is also not a Hessian but a gradient vector.


What is the pros and cons of increasing and decreasing the number of worker processes in A3C? In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I understand it correctly.
But I wonder how I should decide the number of the child processes to implement. I think the more child processes are, the better it is to disentangle the correlation between the samples is, but I'm not sure what is the cons of setting a large number of child processes.
Maybe the more child processes are, the larger the variance of the gradient is, leading to the instability of the learning? Or is there any other reason?
And finally, how should I decide the number of the child processes?

The correct number of child processes will depend on the hardware available to you.

Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running.

If your problem fits nicely in your computers' memory, then processes will spend almost all of their time running. If it's too big for memory, they will periodically need to wait for disk.

You should use approximately 1 Child process per CPU core. If you are training on a GPU, then it depends whether the process can make use of the entire GPU at once (in which case, use just 1), or whether a "process" is really more like a CUDA thread here (in which case you'd want one per CUDA core).

If you think your processes will wait for disk, use more than one per core. About 50% more is a good starting point. You can use a program like top to monitor CPU usage and adjust the number of processes accordingly.

To answer your question more explicitly:

Having more child processes (up to a point, discussed above), will increase hardware utilization, and make your training run faster. With a Core i7 CPU for instance, you might be able to run 8 or 16 child processes at a time, so you'd train 8-16x times faster.
Having more child processes than processing units (CPU cores, CUDA cores), will begin to cause frequent context switching, where the processing units have to pause to change between different jobs. Changing jobs is extremely expensive, and ultimately, your program cannot train faster than it would by using all the available hardware. If you have more processes than processing units, reducing the number should make your program train faster.


Has "deep vs. wide" been resolved? All else being equal, including total neuron count, I give the following definitions:

wide is a parallel ensemble, where good chunks of the neurons have the same inputs because the inputs are shared and they have different outputs.
deep is a series ensemble, where for the most part neurons have as input the output of other neurons and few inputs are shared.

For CART ensembles the parallel (wide) ensemble is a random forest while the series (deep) ensemble is a gradient boosted machine.  For several years the GBM was the "winningest" on kaggle.
Is there a parallel of that applied to Neural networks?  Is there some reasonable measure that indicates whether deep outperforms wide when it comes to neural networks?  If I had the same count of weights to throw at a tough problem, all else being equal should they be applied more strongly in parallel or in series?

I am not sure what you are really looking for but I leave here this paper here, where some intuition into that direction is provided. This paper compares the performance of a deep learning model scaling in 3 dimensions: resolution, width and depth. As depicted in their definition:

If you go to section 3.2 you will see how scaling the different dimensions independently (resolution, width and depth) affects performance and how by performing a compound scaling the performance of the model is maximized (so there is a close relation). It is a very thorough ablation study. For me this was the paper where I understood how these Width, Depth and Resolution parameters come together.
Rethinking Model Scaling for Convolutional Neural Network: https://arxiv.org/abs/1905.11946


How do I classify whether a document is legal or not given a set of keywords that appear only in legal documents? Let's say that I want to classify whether a document is a legal document or not. I have a list of keywords that will be presented only in legal documents.
What is the proper way or algorithm to calculate probability based on this list?

Maybe this is what you are looking for:
https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm
Basically you would

build and store a finite-state machine that resembles a trie with additional links between the various internal nodes using the given keywords.
for the candidate document, go through it with the above finite-state machine.

Then based on the result

If there is any match, then the document is valid.
Otherwise it is possible that the document is invalid. However, in terms of the probability, I do not have any good idea yet.


What are proxy reward functions? The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.

... the observed proxy reward function is the approximate solution to a reward design problem

(source: Inverse Reward Design)
But I have trouble getting how they fit the overall reward objective and got confused by some examples of them.
I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from this post, made me question that.

Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...

What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?

In the examples above, how would high discount rates form a proxy reward?

I'm also curious about how they are used as a source of multiple rewards

In the paper that you cite, Inverse Reward Design (2017), the authors actually define what they mean by "proxy reward function".

We formalize this in a probabilistic model that relates the proxy (designed) reward to the true reward

So, the proxy reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word "proxy" is thus consistent with the general usage of the word in computer science, i.e. a "proxy reward function" is a reward function that is used instead of the intended (optimal) reward function.


How many types of variational auto-encoders are there? I have been studying about auto-encoders and variational auto-encoders. I would like to know how many variants of VAEs are there today.
If there are many variants, can they be used for feature extraction for complex reinforcement learning tasks like self-driving cars?

There are many variations of the original VAE (proposed in the 2013 paper Auto-Encoding Variational Bayes), with different purposes (such as the generation of discrete data or graphs). Of course, I cannot enumerate all of them, so here I will list only the ones I am currently aware of.

Conditional variational auto-encoder (CVAE) (2015, NeurIPS)
Adversarial autoencoders (AAE) (2015, ICLR)
Ladder variational autoencoder (LVAE) (2016,  NeurIPS)
VAE/GAN (2016, PMLR)
Vector quantised variational autoencoder (VQ-VAE) (2017, NeurIPS)
Denoising variational autoencoder (DVAE) (2017, AAAI)
Grammar variational autoencoder (GVAE) (2017, ICML)
Syntax-directed variational autoencoder (SD-VAE) (2017, ICLR)
PixelVAE (2017)
GraphVAE (2018, ICANN)
Constrained graph variational autoencoder (CGVAE) (2018, NeurIPS)
Character VAE (CVAE) (2018, ACS central science)
Regularized VAE (RVAE) (2018, NeurIPS)
Junction tree variational autoencoder (JT-VAE) (2018, PMLR)
InfoVAE/MMD-VAE (2019, AAAI)
Molecular hypergraph grammar variational autoencoder (MHG-VAE) (2019, ICML)
DAG variational autoencoder (D-VAE) (2019, NeurIPS)
Denoising adversarial autoencoders (iDAAE and DAAE) (2019, IEEE Transactions on Neural Networks and Learning Systems)
MACRo-mIcro Disentangled Variational Auto-Encoder (MacridVAE) (2019, NeurIPS)

VAEs have been used for drug design (see e.g. JT-VAE) and in reinforcement learning (see e.g. world models). I don't know if they have been used for self-driving cars, but it's possible.


Planning a Neural Network I'm trying to understand how to effectively plan and write a Neural Network but running into problems with understanding how they should be written. I'm working with classification with writing a backpropagation Neural Network however any other types/examples would be greatly appreciated.

One of my problems is understanding the possibilities of them, for example if you have a single neuron I assume you can have 2^1 possibilities, like in my example I can't imagine how I could predict something more than above or below a straight line. Similarly I assume with 3 neurons therefore 2^3 possibilities you could predict if something is one of the 4 quadrants of a graph and then predict if it's in the upper part or lower part of the quadrant so 8 classifications. Is this true or is a network capable of predicting more/less than that?

On to my specific problem.

I've written a single Neuron/Perceptron that can predict whether something is above or below a straight line graph given the correct training data and using a sign activation function.

Following from this I'm trying to write a Neural Network that can predict whether something is in the 1st, 2nd, 3rd or 4th quadrant of a graph.

One idea I have had is to have 2 input perceptrons, the first taking the x value, the 2nd taking the y value, these then try and predict individually whether the answer is on the right or left of the centre, and then above or below respectively. These then pass their outputs to the 3rd and final output neuron. The 3rd Neuron uses the inputs to try and predict which quadrant the coordinates are in. The first two inputs use the sign function.

The problems I'm having with this is to do with the activation function of the final neuron, one idea was to have a function that somehow scaled the output into a integer between 0 and 1, so 0 to 0.25 would be quadrant 1, and so on up to 1. Another idea would be to convert it to a value using sin and representing it as a sine wave as this could potentially represent all 4 quadrants.

Another idea would be to have a single neuron taking the input of the x and y value and predicting whether something was above or below a graph (like my perceptron example), then having two output neurons, which the 1st output neuron would be fired if it was above the line and then passed in the original x coordinate to that output neuron. The 2nd output neuron would be fired if it was below then pass in the original x value as well to determine if it was left or right.

Are these ideas adequate examples of writing a network?

Also my final question would be if you wanted your network to have 8 possible outputs would you need 8 output neurons or could you represent the 8 values with a single more advanced activation function?

Thanks!

First of all what you are trying to do can be achieved by simple logical programming. Secondly you are making things overly complex.

1 node in a Neural Net can predict as many values as you would like it to, as it outputs a real number no matter the activation function. It is us who round of the number to 0 or 1 or maybe 0, 0.5, 1 depending on the classification job at hand.

2 Neurons can predict 4 classes 00, 01, 10, 11 depending on how you use it. Now in your specific of quadrant detection 2 Neurons are sufficient, since not only can you decompose the problem into two separate problems, both the problems are linearly separable too. This is not always the case or it maybe too complex to find linearly separable classes, so in general one output node is reserved for 1 class only. If it is that class only that output node is activated while the rest is 0. Memory and process consuming, but easier.

Coming to your first approach. I don't exactly how hope to achieve 4 class classification with a single output node. Minimum of 2 nodes is required. The problem with your approach is that if you think critically your hidden nodes are not doing anything other than scaling the inputs, so almost the same type input is propagated and then you are asking a node to perform a a 4 task classification which is not linearly separable as per me. sin will not be able to separate it if you think critically.

The second idea is better, just remove the hidden node and you will be able to classify it based on 00, 01, 10, 11 outputs. Although this problem can be solved by simple logical computation and your Neural Net is basically doing nothing.

Please point out to me if you find any mathematical mistakes. Hope this helps!


How does adding a small change to an neuron's weighted input affect the overall cost? I was reading the following book: http://neuralnetworksanddeeplearning.com/chap2.html

and towards the end of equation 29, there is a paragraph that explains this:

However I am unsure how the equation below is derived:

I believe he's just saying that:

$$
\frac{\partial C}{\partial z_j^l} \Delta z_j^l \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C
$$

so that the change in cost function can be arrived at simply for a small enough perturbation $\Delta z_j^l$.

Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just:
$$
\partial C \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx  \frac{\partial C}{\partial z_j^l} \Delta z_j^l
$$


Wouldn't convolutional neural network models work better without flattening the input in any stages? The above model is what really helped me understand the implementation of convolutional neural networks, so based on that, I've got a tricky hypothesis that I want to find more about, since actually testing it would involve developing an entirely new training model if the concept hasn't already been tried elsewhere.

I've been building a machine learning project for image recognition and thought about how at certain stages we flatten the input after convoluting and max pooling, but it occurred to me that by flattening the data, we're fundamentally losing positional information. If you think about how real neurons process information based on clusters, it seems obvious that proximity of the biological neurons is of great significance rather than thinking of them as flat layers, by designing a neural network training model that takes neuron proximity into account in deciding the structure by which to form connections between neurons, so that positional information can be utilized and kept relevant, it seems that it would improve network effectiveness.

Edit, for clarification, I made an image representing the concept I'm asking about:

Basically: Pixels 1 and 4 are related to each other and that's very important information. Yes we can train our neural network to know those relationships, but that's 12 unique relationships in just a 3x3 pixel grid that our training process needs to successfully teach the network to value, whereas a model that takes proximity of neurons into consideration, like the real world brain would maintain the importance of those relationships since neurons connect more readily to others in proximity.

My question is: Does anyone know of white papers / experiments closely related to the concept I'm hypothesizing? Why would or would that not be a fundamentally better model?

I have had similar thoughts about neural networks before. Convolution layers are layers of two dimensional nodes effectively passing the spacial data so why don't we use two dimensional hidden layers to receive information out of them.

I'm sure someone has used this type of implementation before. I believe the papers bellow are using this. Part of the point of neural networks is that the weights are trained in order to optimize finding the best solution so regardless of the spacial information it learns to 'focus'/increase weight on locations that are associated with deciding the solution.

Think of the problem where your neural network examines an image and gives true or false. Training images are True if the center is red and one of the corners is blue or if the center is blue and one of the corners are red. Flattening the layers or not should have basically no effect on this model. In other circumstances like object detection or labeling outlines yes I believe not flattening will benefit the model. With that said flattening the data does not erase spacial relationship each layer will still be trained to detect the spacial information that gives a correct answer the flattened layers just won't have the benefit of neighbors when the layers are one dimensional instead of two.

In a a CNN with multi class detecting as the task you could allow each class to have its own CNN like hidden layers that narrow to a decision node and decide if they match that class or not. Imagine a palm tree shape where the palm trunk is the image convolutions and each leaf on the top are the two dimensional hidden layers that narrow to an output layer.

 Multi-dimensional NN    and
 Three dimensional Neural Network

I know I spoke in a lot of abstraction so if any part doesn't make sense, I'll make an edit to clarify.


What is the correct notation for an operation that applies to each element of an array independently? I am looking for the standard notation to define element-wise / Hadamard-style functions, if there is one.
That is to say, if the operator I am looking for were represented by a hexagon ⬡, I could use it as such:
$$A(x) = \underset{i}{\Large{⬡} } f(x_i)$$
$$A : \mathbb{R}^n \rightarrow \mathbb{R}^n$$
$$f : \mathbb{R} \rightarrow \mathbb{R}$$
It is very convenient to define such functions explicitly because I want to manipulate them: $B \circ A$ . It seems to me that the following notation is correct: $A_i(x) = f(x_i)$ but I worry it is nonstandard and confusing.
My functions are non-linear so I cannot simply apply them directly to the array as a vector.
Related:

https://math.stackexchange.com/questions/159005/notations-for-array-operations
https://math.stackexchange.com/questions/20412/element-wise-or-pointwise-operations-notation

The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:
$$ \mathcal{L} = - \frac{1}{4} F_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}_i y_{ij} \psi_j \phi + h.c.) + |D_\mu \phi|^2 - V(\phi) $$
But if you try to expand all the indices in all the objects above - then it barely fits on a page.
On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example $f(x_i)$ can be read as:
$$f(x_0, x_1, \dots, x_N)\quad \text{or as}\quad f(x_0),f(x_1), \dots, f(x_N)$$
One way to implicitly resolve this ambiguity is to show that the index $i$ "escapes" the argument brackets:
$$a_i = f(x_i)\quad \text{or e.g.} \quad \sum_if(x_i)$$
This can only be interpreted as $f(x_i)$ being element-wise. Also, at least in my opinion, using $x_i$ with index and $x$ without index in the same expression is extremely confusing.
And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets $f[x_i]$ or capital letters $F(x_i)$ the vector-argument functions.


Intuitively, how does it make sense to take an action $A'$ when the environment already ended? The update equation for SARSA is $Q(S,A) = R + \gamma Q(S',A')$. Consider this: I take an action $A$ that leads to the terminal state. Now my $S'$ would be one of the terminal states. So...

Intuitively, how does it make sense to take an action $A'$ when the environment already ended? Or is this something you just do anyway?

Once a terminal state-action pair is reached, you update the previous state-action pair and then start the game loop all over again. But this means that the terminal state-action pair ($Q(S',A')$ in my example) is never updated. So, if your initial estimate of $Q(S',A')$ was wrong, you would never be able to fix it which would be very problematic. (And you can't set all the terminal values to zero because you are using function approximators)

So, how do I resolve these issues?

Intuitively, how does it make sense to take an action A' when the environment already ended?

It doesn't make sense, in that nothing can happen once the agent reaches a terminal state. However, it is often modelled as an "absorbing state" where the action is unimportant (either null or value ignored) with value by definition of $0$.

And you can't set all the terminal values to zero because you are using function approximators

The value is zero by definition. There is no need to approximate it. So don't use function approximators for action values in terminal states.  When $S'$ is terminal, the update becomes:
$Q(S,A) \leftarrow Q(S,A) + \alpha(R - Q(S,A))$
Look at any implementation of Q learning and you will see a conditional calculation for the update value, that uses some variant of the above logic when $S'$ is terminal. For OpenAI Gym environments for instance, it will use the done flag.


Given a dataset with no noisy examples, is the training error for the ID3 algorithm always 0? Given a dataset with no noisy examples (i.e., it is never the case that for 2 examples, the attribute values match but the class value does not), is the training error for the ID3 algorithm is always equal to 0?

Yes, if you can assume that your data is separable on the features given then ID3 will find a decision tree for it (Note: this will not necessarily be an optimal tree, or even a good tree). To understand why let's look at a proof.

Assume we have one feature left and some number of examples in a leaf that does not have separated data points then, either:

This feature perfectly splits the data into their desired categories. In this case ID3 will split over this feature and be done.
This feature does not split the data into their desired categories. In this case, we have at least 2 examples with the same feature value, but they do not share the same class. This means either these two examples have the same feature value for all features (a contradiction to our assumptions) or we have created a leaf with non-separable data points (which we prove is impossible later on).

Now for the inductive step.

Assume that ID3 used some number of features to somewhat separate our examples already. Thus, we have some number of features not used and some number of examples not separated on any current leaf, and one of the following is true:

All our leafs contain data points of only one class. In which case, ID3 is done and returns.
All leafs contain data points that are separable by the remaining features. Then ID3 splits over the local optimum and continues.
At least one leaf contains data points that are not separable by the remaining features of each own's branch. Then there exists two examples p and q such that p and q share all the same remaining feature values, but do not share the same class. Thus, one of the following is true:

There is a feature that we already used that differs between p and q.
This is a contradiction as we said ID3 already separated over this feature, and thus p and q must be on different branches.
p and q have the same feature value for all features previously used. But we already said p and q share all the same remaining feature values. Therefore, they share all feature values which is a contradiction to our assumptions.

Therefore, at no point in the creation of the decision tree is ID3 allowed to create a leaf that has data points that are of different classes, but can't be separated on remaining features. And thus, ID3 must create a solution that perfectly separates the data and has 0 training error.

Something to note however, is that allowing ID3 to grow the decision tree in this way will likely cause us to overfit the training set. To combat this we set a maximum depth that we allow ID3 to grow to.


What would be the state of the art image captioning deep learning model? I saw a couple of architectures, like CNN-LSTM, with and without attention model, use of Glove vector, self-critical models, etc. I am overwhelmed looking at different notebooks and architectures, came here for a guidance.  I am looking to build a personal project on image annotations. Also, if I wanted to use this deep learning model together with TFX pipeline, what would be the best type of architecture I can go with?

Here are a couple of Kaggle Kernels, Notebooks and Tutorials for Image Captioning.

Kaggle Kernel | Neural Image Captioning:  ->

Kaggle Kernel | Show Attend and Tell

Kaggle Kernel | Flickr Image Captioning : TPU, TF2 & Glove

Tensorflow Tutorial | Image captioning with visual attention

Show and Tell: A Neural Image Caption Generator by Vinyals et al.


How is uniqueness quantification translated in First Oder Logic I have this following natural language statement:

"There is only one house in area1 the size of which is less than 200m²."

which is mistranslated to FOL:

∃x.(house(x) ∧ In(x,area1) ∧ ∀y.(house(y) ∧ In(y,area1) ∧ size(y) < 200 -> x=y))

This translation is wrong according to my lecturer, because it is not necessary that the size of x must be less than 200. The statement is true if there only houses which are bigger.

I have two questions:

I don't get the FOL translation at all and don't see where the uniqueness part is expressed :
so translated it back : "if all houses in area1 have a size less then 200m² then there exists one house which equals to all houses ??"
why is not necessary that the size of x is less than 200, when it clearly says in the statement above that must exist one house with a size less then 200 ?

According to the Wikipedia entry on Uniqueness Quantification your lecturer is correct. There is no size requirement expressed in the FOL expression.

The point about the implication is that it can be true if the antecedent is false. So, there is a house in area1 (which we call x). And all houses in area1 which are smaller than 200 are the same as x. But if there aren't any, then the antecedent is false, and the consequence (x = y) is false, but the whole statement is still true.

As another example: "If Trump is the 31st president of the USA, then the moon is made of green cheese". Both antecedent and consequence are false, but the whole statement is still logically true. Same as "If there is a house in that area, and there are houses that are less than 200 (which there aren't), then that house is one of them."

Moving on to the correct expression: The unique quantifier (usually written as ∃!) can be rewritten using the existential and universal quantifiers as follows (see the above mentioned Wikipedia page):

  ∃x (P(x) ∧ ∀y (P(y) -> y = x))

This is not what you have got; you have got two different predicates, P1 and P2. Your P1(x) is (house(x) ∧ in(x, area1)), and your P2(x) is (house(x) ∧ in(x, area1) ∧ size(x) < 200)

The correct expression would require the same predicate for the quantifiers and would therefore be

  ∃x ((house(x) ∧ in(x, area1) ∧ size(x) < 200) ∧
    ∀y ((house(y) ∧ in(y, area1) ∧ size(y) < 200)
      -> y = x))

The difference is that you state that there is at least one house in the area with a size of less than 200. So the second predicate, that y is a house in the area with a size of less than 200, cannot be false.


What is difference between edge computing and federated learning? I recently read about federated learning introduced by Google, but it seems to be like edge computing.

What is the difference between edge computing and federated learning?

Federated systems or Fog models, basically push computation from the active system side to server or network side processes. This is commonly used when computationally expensive services are required on limited systems (such as running some AI or augmentation occlusion processing from on phone), allowing for distributed data processing. Here is a good paper on the matter-
https://arxiv.org/abs/1602.05629
What you might be caught up on is edge computing processes and Fog both operate the same. It's just the model differentiating where the services run.


Improving learning rate for classification task I am trying to perform classification task using Keras and tensorflow. However, the learning converges after achieving an accuracy of 57%. All my inputs and outputs are categorical data. Am I using the correct approach to train? Are there any other example in the web which is trying to solve the similar problem?

data.csv

religion,caste,qualification,marital_status,sex,nature_activity
2,20,5,1,1,10
2,20,5,1,1,10
2,20,5,1,1,10
2,20,5,1,1,10
1,3,5,1,1,13
1,4,4,1,2,3
1,3,4,2,1,1
1,3,3,2,1,1

Source code

from keras.models import Sequential
from keras.layers import Dense
import pandas
from sklearn.metrics import classification_report
from sklearn.preprocessing import LabelEncoder
from keras.utils import np_utils
import numpy
from sklearn.model_selection import train_test_split
from keras.wrappers.scikit_learn import KerasClassifier
import sys
import matplotlib.pyplot as plt
# fix random seed for reproducibility
numpy.random.seed(7)

# load pima indians dataset
dataframe = pandas.read_csv("data.csv",header=None)
dataset=dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:4].astype(float)
Y = dataset[:,4]
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.0015, random_state=42)
encoder = LabelEncoder()
encoder.fit(Y_train)
encoded_Y = encoder.transform(Y_train)
one_hot_enc_Y=np_utils.to_categorical(encoded_Y)
model=Sequential()
model.add(Dense(500,input_dim=4,activation='relu'))
model.add(Dense(100,activation='relu'))
model.add(Dense(50,activation='relu'))
model.add(Dense(3,activation='softmax'))

# Compile model
model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
# Fit the model
history=model.fit(X_train, one_hot_enc_Y, validation_split=0.33,epochs=300, batch_size=5)
# summarize history for accuracy
plt.plot(history.history['acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
# summarize history for loss
plt.plot(history.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper right')
plt.show()
# evaluate the model
scores = model.evaluate(X_train, one_hot_enc_Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
sys.exit()

You might find "What should I do when my neural network doesn't learn?" and "Reasons why your Neural Network is not working" interesting.