Are Rationality and Intelligence distinct? This just popped into my head, and I haven't thought it through, but it feels like a sound question.  The definition of intelligence might still be somewhat fuzzy, possibly a factor of our evolving understanding of "intelligence" in regard to algorithms, but rationality has some precise definitions.

Are Rationality and Intelligence distinct?

If not, explain.  If so, elaborate.

(I have some thoughts on the subject and would be very interested in the thoughts of others.)

From Norvig and Russel definitions of rationality:

  Thinking Rationally - The Greek philosopher Aristotle was one of the first to attempt to codify “right thinking,” that
  is, irrefutable reasoning processes. His syllogisms provided patterns for argument structures
  that always yielded correct conclusions when given correct premises—for example, “Socrates
  is a man; all men are mortal; therefore, Socrates is mortal.” These laws of thought were
  supposed to govern the operation of the mind; their study initiated the field called logic.
  Acting Rationally - An agent is just something that acts. Of course,
  all computer programs do something, but computer agents are expected to do more: operate
  autonomously, perceive their environment, persist over a prolonged time period, adapt to
  change, and create and pursue goals.
  A rational agent is one that acts so as to achieve the
  best outcome or, when there is uncertainty, the best expected outcome.
  In the “laws of thought” approach to AI, the emphasis was on correct inferences. Making
  correct inferences is sometimes part of being a rational agent, because one way to act
  rationally is to reason logically to the conclusion that a given action will achieve one’s goals
  and then to act on that conclusion. On the other hand, correct inference is not all of rationality;
  in some situations, there is no provably correct thing to do, but something must still be
  done. There are also ways of acting rationally that cannot be said to involve inference. For
  example, recoiling from a hot stove is a reflex action that is usually more successful than a
  slower action taken after careful deliberation.

Clearly and also intuitively, rationality is well defined.

Intelligence as seen form mathematical and computational approach:

Intelligence can be the ability for an agent to make rational or irrational decisions, on a varying time frame and also choose the level of rationality (strictly in a computational sense). For example, I have exams and I want to watch TV, on a time frame of a week/month the rational decision would be to study so that I can enjoy the fruits of my labor which will be much more than the instantaneous pleasure of TV (also I can watch reruns). But for a time frame of an hour watching TV is definitely the most rewarding thing. So intelligence can be defined as the capability in deciding the length of time frame to be rational (what we call visionaries those who can see rewards far in the future).

Also as Game Theory or economics suggest, we can have different definitions for rationality depending on our needs. Thus, watching TV to gain knowledge might be more important to someone than studying, so effectively he has a different rationality function (arbitrary made-up term) to satisfy. Thus, intelligence can be deciding our rationality function, based on our needs and external experiences (learning in a nutshell). Also we may decide to minimize our rationality cost function or leave it in an intermediate state (which can be thought of as the minima for a different rationality cost function, thus only rationality functions are the true variable and not the intelligent decision to minimize it or not).

Lets take an example of bees (I am not sure whether this is the correct interpretation though): Bees can hardly be called intelligent (no foresight), but they are rational. They perform the task assigned to them with efficiency and toil (even though this does not reward the bee itself, it rewards the genes carried by the bees and ensures it survival through the queen - can be thought of as evolutionary coded intelligence). Bees perform these jobs in an apparently rational way, which has been decided by 1000's of years of evolution. Though bees individually are of hardly any importance, together they create a truly intelligent community - taking smart decisions and actions albeit their farsightedness is only within a  smaller time frame compared to humans. Thus in common terms, it can be thought that rationality always almost leads to intelligence, but the same cannot be said for vice versa (as for intelligence you now have a choice for rationality function and you can choose not to satisfy it or choose an irrational function).

An important consequence of bees not being intelligent is that they are always performing rational actions as hard coded in their genes, which is causing the entire colony to behave in an intelligent and a very optimized energy efficient way (but there maybe better strategies, we can never know whether they follow the best strategy unless we account for all the variables).

TL;DR : Intelligence can be thought of the ability of an agent to choose the amount of rationality the agent wishes to satisfy. Using mathematics we can always find one or more completely rational methods of solving a problem with variables being time and the environment. But intelligent beings can add more variables like their experience, needs and motivation. Intelligent beings to some extent can single-handedly manipulate the external environment to suit their needs.

From psychological viewpoint:

Here are a few definitions of different types of intelligence and learning - Quite good and concise.

IQ - In science, the term intelligence typically refers to what we could call academic or cognitive intelligence. In their book on intelligence, professors Resing and Drenth (2007)* answer the question 'What is intelligence?' using the following definition: "The whole of cognitive or intellectual abilities required to obtain knowledge, and to use that knowledge in a good way to solve problems that have a well described goal and structure."

Intelligence - Wikipedia - Actually has some good definitions.

Intelligence Quotient - Wikipedia

Emotional Intelligence - Emotional intelligence is the ability to identify and manage your own emotions and the emotions of others. It is generally said to include three skills: emotional awareness; the ability to harness emotions and apply them to tasks like thinking and problem solving; and the ability to manage emotions, which includes regulating your own emotions and cheering up or calming down other people.

Emotional Intelligence - PsychCentral

Emotional Intelligence - Wikipedia

Hope this is of some insight!


Are fully connected layers necessary in a CNN? I have implemented a CNN for image classification. I have not used fully connected layers, but only a softmax. Still, I am getting results.

Must I use fully-connected layers in a CNN?

Are fully connected layers necessary in a CNN?

No.  In fact, you can simulate a fully connected layer with convolutions. A convolutional neural network (CNN) that does not have fully connected layers is called a fully convolutional network (FCN). See this answer for more info.

An example of an FCN is the u-net, which does not use any fully connected layers, but only convolution, downsampling (i.e. pooling), upsampling (deconvolution), and copy and crop operations. Nevertheless, u-net is used to classify pixels (more precisely, semantic segmentation).

Moreover, you can use CNNs only for the purpose of feature extraction, and then feed these extracted features in another classifier (e.g. an SVM). In fact, transfer learning is based on the idea that CNNs extract reusable features.


Wouldn't an AI that specializes in making other AI be an AGI if they can cooperate? If said AI can assess scenarios and decide what AI is best suited and construct new AI for new tasks. In sufficient time would the AI not have developed a suite of AIs powerful/specialized for their tasks, but versatile as a whole, much like our own brain’s architecture? What’s the constraint ?

If the AI can indeed assess arbitrary scenarios and come up with solutions to handle them, then it would indeed be an AGI.

  What’s the constraint ?

It doesn't exist.  Current programmers are very good at developing AI that can handle specific tasks ("narrow AIs"), but it is currently impossible to build an AI that can assess and handle "general" situations (unlike your proposed algorithm, which possess that capacity).

Theoretically, we can have a program that can build other programs (genetic algorithms are arguably one such example), but handling arbitrary scenarios and problems requires a form of "general intelligence", which we don't know how to program. Therefore, we can't build this machine.

It's possible that we can built this machine, but we must first figure out  the hard problem of "general intelligence". We're nowhere near reaching that level.

If we figure out how to program "general intelligence", then it should be fairly simple to use your approach (building an AGI to assess scenarios and then build "narrow AIs" that can handle Those scenarios). Only then we can understand the AGI's limitations and weaknesses, and be able to identify probable constraints to its power. For example, it's possible that such an AGI may be slow in handling arbitrary scenarios and developing the "narrow AIs"...in which case, it may take an absurdly long period of time to develop "a suite of AIs powerful/specialized for their tasks".

But until we build the AGI itself, we won't be able to identify its faults or weaknesses. Going beyond that would be science-fiction speculation.


What AI designs are suited for producing title replacements? Problem: "For a given news article, generate another title for the article if the article is to be published under a different Publication."

Which algorithm will be well suited for this? Should I use naive Bayesian or any NLP algorithm?

You can use a seq2seq LSTM model. Giving a article will generate a title for it.

You need to train it on data which comprises news articles and their corresponding titles.
You can refer here and here.


When to use AND and when to use Implies in first-order logic? I am trying to learn the theory behind first-order logic (FOL) and do some practice runs of converting statements into the form of FOL.
One issue I keep running into is hesitating on whether to use an AND ($\land$) statement or an IMPLIES ($\rightarrow$) statement.
I have seen examples such as "Some boys are intelligent" turned into:
$$
\exists x \text{boys}(x) \land \text{intelligent}(x)
$$
Can I make a general assumption that when I see $x$ is/are $y$, I can use an AND?
With a statement such as "All movies directed by M. Knight Shamalan have a supernatural character", I feel that that statement can be translated to either:
$$
\forall x, \exists y \; \text{directed}(\text{Shamalan}, x) \rightarrow \text{super-natural character}(y)
$$
or
$$
\forall x, \exists y \; \text{directed}(\text{Shamalan}, x) \land \text{super-natural character}(y)
$$
Is there a better way to distinguish between when to use one or the other?

When considering the sentence "Some boys are intelligent", it makes sense to express it by ∃x boys(x) ∧ intelligent(x). This is because the existential quantifier makes sure to express that at least some, but not necessarily all boys are intelligent. More specifically, you say that there exists something which is a boy and which is intelligent. But if such a something exists, then there are some boys which are intelligent. Your statement is satisfied.
Using a simple implication → here wouldn't work semantically in that context. For example, ∃x boys(x) → intelligent(x) expresses that: For some x, if x is a boy, then x is intelligent. By rewriting, we would get ∃x ¬boys(x) ∨ intelligent(x). This statement would even be true if all x were not boys, because ¬boys(x) would be true in these cases. So, ∃x boys(x) → intelligent(x) would hold as soon as something in your model wasn't a boy. In that case, ∃x boys(x) → intelligent(x) would be true, but "Some boys are intelligent" would not really be satisfied, since there are simply no boys in your model which could be intelligent. So, you will have to resort to something like ∃x boys(x) ∧ intelligent(x), since otherwise there is no guarantee that boys (even intelligent ones) are present in the resulting model.


Apart from the state and state-action value functions, what are other examples of value functions used in RL? In reinforcement learning, we often define two functions, the state-value function

$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty}
\gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$

and the state-action-value function

$$Q^\pi(s,a) = \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}\Bigg|S_t=s, A_t=a \right]$$

where $\mathbb{E}_{\pi}$ means that these functions are defined as the expectation with respect to a fixed policy $\pi$ of what is often called the return, $\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}$, where $\gamma$ is a discount factor and $R_{t+k+1}$ is the reward received from the environment (while the agent interacts with it) from time $t$ onwards.

So, both the $V$ and $Q$ functions are defined as expectations of the return (or the cumulative future discounted reward), but these expectations have different "conditions" (or are conditioned on different variables). The $V$ function is the expectation (with respect to a fixed policy $\pi$) of the return given that the current state (the state at time $t$) is $s$. The $Q$ function is the expectation (with respect to a fixed policy $\pi$) of the return conditioned on the fact that the current state the agent is in is $s$ and the action the agent takes at $s$ is $a$.

Furthermore, the Bellman optimality equation for $V^*$ (the optimal value function) can be expressed as the Bellman optimality equation for $Q^{\pi^*}$ (the optimal state-action value function associated with the optimal policy $\pi^*$) as follows

$$
V^*(s) = \max_{a \in \mathcal{A}(s)} Q^{\pi^*}(s, a)
$$

This is actually shown (or proved) at page 76 of the book "Reinforcement Learning: An Introduction" (1st edition) by Andrew Barto and Richard S. Sutton.

Are there any other functions, apart from the $V$ and $Q$ functions defined above, in the RL context? If so, how are they related?

For example, I've heard of the "advantage" or "continuation" functions. How are these functions related to the $V$ and $Q$ functions? When should one be used as opposed to the other? Note that I'm not just asking about the "advantage" or "continuation" functions, but, if possible, any existing function that is used in RL that is similar (in purpose) to these mentioned functions, and how they are related to each other.

Advantage function: $A(s,a) = Q(s,a) - V(s)$
More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $V(s)$. The GVF is defined on page 459 of the 2nd edition of Sutton and Barto's RL book as
$$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$
where $C_t \in \mathbb{R}$ is the signal being summed over time.
$\gamma(S_t)$ is a function $\gamma: \cal{S}\to[0,1]$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. Some call it the continuation function.
Also of note are the differential value functions. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $R_{t+1}-r(\pi)$, where $r(\pi)$ is the average reward under policy $\pi$.
$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$
$$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$
The differential value functions assume that a single fixed value of $r(\pi)$ exists. That is, they assume the MDP is "ergodic." See section 10.3 of Sutton and Barto for details.


How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)? I am currently using the PPO method from the Nvidia's Isaac gym to train an agent for my robot. Below, you can see the plot which corresponds to a training process.
I know that something is massively wrong with my training process (since the robot does not manage to get a nice policy), so I am trying to understand the training process more with the help of the below values being logged during the problematic training.

So far, I could find out what this value function means: the reward function stabilizes, thus the loss value function also stabilizes, which means that my robot should explore more to fail and learn from the fail!
But what about the other two plots, surrogate and mean_noise_std? How should one interpret those values?

The ideal training process

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to maximize the expected return - while most of the Deep Learning classification problems are minimizing some loss.
Most of the modern Deep Learning programming frameworks was created with the classification problems in mind, so they call the optimization goals "losses": tf.keras, pytorch. The good news is that you can easily convert return-maximization problem to loss-minimization problem by just changing the sign of your target function.
Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign.
Value Function. Given an agent's state, Reinforcement Learning tries to find the best agent's action that maximized the expected return. The problem is that the expected return is also unknown. Note on the difference between "reward" and "return". Rewards $r_t$ are immediate bits of positive/negative reinforcement that the agent receives obtain after taking an action. The "return" $G_t$ is the function of a particular trajectory - it is the discounted sum of future rewards starting from step $t$:
$$G_t = \sum_k \gamma^k r_{t+k+1}$$
The expectation of the return at state $s$ given a policy $\pi$ is called value $V^\pi(s)$:
$$V^\pi(s) = \mathbb{E}_{a\sim\pi}\left[ G_t\right|\left. s_t=s\right]$$
In principle, it is possible to estimate $V^\pi(s)$ by doing Monte Carlo: start from the state $s$ and run many simulations following $\pi$ and average all the returns you've got. For most practical cases this approach is infeasible. Modern Deep Reinforcement Learning algorithms try to reduce computational load by using neural networks to approximate $V(s)$.
Having the value function approximation $V(s)$, one can then try to optimize the policy approximation function $\pi(a|s)$ using the policy gradient update. Such setup: with one approximator for policy and another approximator for value function is called an "Actor-Critic" family of Reinforcement Learning algorithms.
BTW, it doesn't look like your value function "stabilizes" during training. And it also doesn't change too much if you look at the vertical scale of your plot.
Surrogate Loss. In practice, the policy gradient optimization step above suffers from instabilities. The gradient step tends to change the policy too strongly, which makes the approximation error of the value function too large.
To deal with this, the TRPO/PPO algorithms introduce a kind of clipping of the loss gradient that constrains the policy gradient changes from being too large. The new function that produces these clipped/constrained gradients is called a "surrogate loss".
mean_noise_std. This doesn't look like a standard name for anything, but I suspect that this is a version of PPO with artificial noise added for better exploration of the agent.


How should I generate datasets for a SARSA agent when the environment is not simple? I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. Datasets generation seems a common feature in this specific subject as stated in [1]

When a dataset is not available, learning is performed through experience.

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem and what the experience means.
PS: The environment consists of 15 mobile users and 3 edge servers, each of which covers a number of mobile users. Each mobile user might generate a computationally heavy-load task and at the beginning of each timestep and can process the task itself or requests its associated edge server to do the processing. If the associated edge server is not capable of processing, due to some reasons, it requests a nearby edge server to lend it a hand. The optimization problem (reward) is to reduce time and energy consumption (multi-objective optimization). Each server has a DRL agent that makes offloading decisions.
I'd really appreciate your suggestions and help.

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem

There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach.
As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you are using SARSA then you would not expect to use any record of previous experience either because SARSA is designed to work on-policy and online - which means that data needs to be generated during training. Training data for SARSA is typically stored only temporarily before being used, or is used immediately (you might keep a log of it for analysis or to help document your thesis, but that log will not be used for further training by the agent). This is different to Q-learning and DQN, which could in theory make use of longer-term stored experience.
You have two main choices for acquiring data:

Use a real environment. In your case, set up 15 mobile users and 3 edge servers. Instrument the environment to collect state and reward data for the agent. Implement the agent as the real decision maker in this environment.

Simulate the environment. Write a simulation that models user behaviour and server loading. Instrument that to provide state and reward data, and integrate your learning agent with it. Typically the agent will call the environment's step function, passing the action choice as an argument and receiving reward and state data back.

If you can simulate the environment, this is likely to be preferable to you since you will likely use less compute resources (than 3 servers and 15 mobile phones) and can run the training faster than real time. Deep reinforcement learning can use a large amount of experience to converge on near-optimal policies, and fast simulations can help because they generate experience faster than reality.
You can also do both approaches. Train an initial agent in simulation, then implement the real version once it reaches a good level of performance in simulation. You can even have the agent continue to learn and refine behaviour in production. Given that you are working with SARSA, this may be an important part of the intent of your project, that the agent continues to adapt to changes in user behaviour and server load over time. In fact this is a key advantage of SARSA over Q-learning, that it should be more reliable and safe to use in such a continuous learning scenario deployed to production.

and what the experience means.

The experience in reinforcement learning is the record of states, actions and rewards that the agent encounters during training.


What is the space complexity of iterative deepening search? When using iterative deepening, is the space complexity, $O(d)$, where $b$ is the branching factor and $d$ the length of the optimal path (assuming that there is indeed one)?

As stated in my other answers here and here, the space complexity of these search algorithms is calculated by looking at the largest possible number of nodes that you may need to save in the frontier during the search.
Iterative deepening search (IDS) is a search algorithm that iteratively uses depth-first search (DFS), which has a space complexity of $O(bm)$, where $m$ is the maximum depth, with progressively bigger depths, and it turns out that its space complexity is $O(bd)$.
(Note that there is also the informed version of IDS, which uses the informed search algorithm A*, rather than DFS, known as iterative deepening A*).


Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate? It is known that if $\alpha$ is set to high, then the cost function of the model may not converge.
However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually $\alpha$ value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?

Setting too high a learning rate will extend the time to get a good result.

In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.


What are the most recent and influential breakthroughs in NLP? I'm looking at the history of NLP, starting in the 1950s, with the Georgetown–IBM experiment.

What are examples of the most recent (e.g. in the last 5-10 years) and influential breakthroughs in natural language processing?

I suggest you take a look at the syllabus of http://cs224d.stanford.edu/syllabus.html and see what was invented in the last ~20 years. They focus on deep learning methods, and they include there almost everything important.

Among the things, which I would include into the review are:

Word2Vec models
Seq2Seq models

Concerning what happened in the last 5 years, i.e. since 2013, I suggest you mentioning super human performance on the SQuAD 1.1 dataset: https://rajpurkar.github.io/SQuAD-explorer/


Looking for NLP technique and drawing a blank Within a piece of text, I'm trying to detect who did what to whom.

For instance, in the following sentences:

  CV hit IV. CV was hit by IV.

I'd like to know who hit whom.

I can't remember what this technique is called. Thanks for any help!

You might be referring to Semantic role labeling. SRL is the task of assigning labels to words or phrases in a sentence that shows their semantic role in that sentence.

In your example CV was hit by IV, the task is to identify the verb "hit" carried out by the actor "CV" affected "IV" the recipient.

Note: If you're only interested in the syntactic relationship among words or phrases in a sentence, not the semantic relationship between them, simple dependency parsing would do the job.


Did Minsky and Papert know that multi-layer perceptrons could solve XOR? In their famous book entitled Perceptrons: An Introduction to Computational Geometry, Minsky and Papert show that a perceptron can't solve the XOR problem. This contributed to the first AI winter, resulting in funding cuts for neural networks. However, now we know that a multilayer perceptron can solve the XOR problem easily.

Backprop wasn't known at the time, but did they know about manually building multilayer perceptrons? Did Minsky & Papert know that multilayer perceptrons could solve XOR at the time they wrote the book, albeit not knowing how to train it?

There does not appear to be a historical consensus on this.
The Wikipedia page on the Perceptrons book (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts).
However, this page gives an account by someone present at the MIT AI lab in 1974, claiming that this was not common knowledge there, alluding to documentation in "Artificial Intelligence Progress Report: Research at the Laboratory in Vision, Language, and other problems of Intelligence" (p31-32) which is claimed to support this.


Are there benchmarks for assessing the speed of the forward-pass of neural networks? I have a task where I would like to use a convolutional neural network (CNN). I would like to incrementally start from the fastest models, fine-tune and see whether they fit my "budget". At the moment, I'm just looking at object detection CNN-based feedforward models.

I'm curious to know if there is any article, blog, web page or gist that benchmarks the popular CNN models based on the forward-pass speed. If there is back-propagation time and dataset-wise performance, even better!

https://github.com/jcjohnson/cnn-benchmarks might be a good start. It mostly focuses on GPUs, but there is also one CPU (Dual Xeon E5-2630 v3).


Could an AI be killed in an infinite loop? Currently, we use control flow statements (such as loops) to program the artificially intelligent systems. Could an AI be killed in an infinite loop (created by itself, for example, while manipulating its source code)?

The question isn't baseless, it really questions the kind of computational infrastructure a modern AGI would require, and would it be able to alter itself without any intervention.

This is a hard to answer question because a truly correct answer would involve static analysis of a given Intelligence to determine whether it has the computational capability to generate a looping state (e.g. some state which reproduces itself in the next instance), and in fact whether these looping states can even exist in the given architecture.

Generally speaking, assuming we can build AI, there is no reason it would be impossible build an AI that has the potential to get stuck in a computational loop and become non-functional. You can imagine a recurrent neural network where the continuous state inside the ANN dominates its behavior causing it to reproduce states with small variety (as input will always be relevant in a connected ANN) but are converged around a particular state space. Whether that is akin to death is a philosophical question for your own peace of mind.

However, if we designed an AI agent that commonly did this, that would reflect poor design; that wouldn't necessarily devalue the contribution of the designed agent but it would bring the agent's other activities under scrutiny. If this is a bi-product of some simulated intelligence, is the process going on really that intelligent?


Reinforcement learning with uniformly random dynamics Suppose I have an MDP $(S, A, p, R)$ where the $p(s_j|s_i,a_i)$ is uniform, i.e given an state $s_i$ and an action $a_i$ all states $s_j$ are equally probable.

Now I want to find an optimal policy for this MDP. Can I just apply the usual methods like policy gradients, actor-critic to find the optimal policy for this MDP? Or is there something I should be worried about?

At least, in theory, it shouldn't make any difference. But I'm wondering are there any practical considerations I should be worried about? Should the discount factor, in this case, be high?

The reward function here depends both on states and actions and is not uniformly random.

Convergence guarantees for basic RL algorithms like policy gradient / actor-critic methods make no assumptions about the dynamics of the MDP. So, theoretically, you don't need to change much.

Practically, when the number of possible trajectories from any given state is so high, the return from each state will have high variance. This means you'll have to collect much more experience for your estimates of expected return to converge to their true values. Intuitively, an environment with high uncertainty requires the agent to do more knowledge-gathering to behave optimally.

My real advice to you depends on what exactly you're trying to do. If you want to have the kind of agent that could learn to behave well in an extremely random environment, then all you need to worry about is giving it enough experience to learn from.

(Your agent should also take a little longer before deciding it's "confident" in its evaluation of different states. That is, don't behave greedily before you're sure your estimates are accurate. Explore adequately. This advice is only relevant if your MDP dynamics aren't actually completely uniform.)

If, however, you want to train an RL agent specifically to solve a problem formulated as an MDP with uniformly random dynamics, then I would tell you to not waste your time. We know before spending the computation that all policies would be equally good/bad in this setting. Since actions are irrelevant to the environment, it would be inefficient to deploy an RL agent that will only learn that which action it takes doesn't matter.

As noted in the comments, the last paragraph is only true when reward from each state-action pair $(s,a)$ is also uniformly random. If it is not, just being aware of the high variance and giving your agent a lot of experience should do the trick.


Applications of CNN for detecting crime from video surveillance cameras Inspired by this discussion about recognizing human actions, I have found the Fall-Detection project which detects humans falling on the ground from a CCTV camera feed, and which can consider alerting the hospital authorities.

My question is, are there any existing real-life implementations or research projects which specifically use live video feed from the surveillance cameras in order to detect crime using convnets (or similar approaches)? If so, how do they work, briefly? Do they automatically inform the police about the crime with the details what happened and where?

For example car accidents, physical assaults, robberies, violent disturbances, weapon attacks, etc.

After a bit of research I found something kind of close:

Artificially intelligent security cameras are spotting crimes before they happen
New surveillance cameras will use computer eyes to find 'pre crimes' by detecting suspicious behaviour and calling for guards
CCTV 'fightcams' detect violence 'before it happens'  at Dailymail, also check at Telegraph

They, however, makes no mention of what specific methods they use.

So a crime detection system as that is written does not exist, but abnormal behaviour detection systems do.

An accurate generalized system seems intuitively infeasible, however. Commiting a crime, unlike falling, is a complex behavior, and takes so many forms. A camera watching a store's counter like at a 7-11 could perhaps see that the 'customer''s arm is strangely reaching across the counter, and the attendant is moving a lot more than usual suddenly, but aside from very specific cases like this such a system is currently quite unfeasible.

Crimes are unusual, relatively speaking, and their dramatic nature means that even the simplest crimes play out in very different ways. Perhaps you could in this case you could try to look for images of a gun, or someone with their hands up. So, looking for unusual, detectable behavioural mannerisms may be possible, but not crime detection.

Ultimately, while you may be able to make (possibly pretty good) systems to detect specific crimes in specific environments, that's all we got for now.

P.S. - Do these camera's also get audio signals? That is also an interesting facet to consider ("PUT YOUR HANDS UP / GIVE ME ALL YOUR MONEY")


How to model categorical variables / enums? I am new to the field and I am trying to understand how is possible to use categorical variables / enums?

Lets say we have a data set and 2 of its features are home_team and away_team, the possible values of these 2 features are all the NBA teams.
How can we "normalize" these features to be able to use them to create a deep network model (e.g. with tensorflow)?

Any reference to read about techniques of modeling that are also very appreciated.

Authors use many different approaches.

One approach is to have a different input neuron for each possible category, and then use a "1-hot" encoding. So if you have 10 categories, then you can encode this as 10 binary features.

Another is to use some sort of binary encoding. If you have 10 categories, it is sufficient to use 4 neurons to represent all possible categories by using binary numbers.

A third approach is to convert your categories to cardinal values, and then normalize them. This may be more effective if your categories really are cardinal (i.e. orderable). If there isn't a natural ordering to them though, this might lead to strange results or make the problem difficult to learn (since it ends up embedding non-linear relationships in the learning problem that don't need to exist).


Does adding a model complexity penalty to the loss function allow you to skip cross-validation? It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.

Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit.
However, this regularisation is not a magic bullet. The degree of regularisation that you achieve will vary, it may depend on hyper-parameters in the regularisation technique. In your case the relative weightings of objective functions for the accuracy and simplicity of the model will have a sweet spot that minimises overfit without compromising too much and under-fitting instead.

It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

Typically using regularisation requires cross-validation in order to find good values for your regularisation hyperparameters. If the regularisation technique you have chosen is a good fit for your problem, then you may not have to search too much - there may be a broad set of values that work well for you.
In turn, a good choice of regularisation may mean that your model's accuracy is less sensitive to other hyperparameters of your model, so searching for an accurate model becomes a little easier.
However:

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

No. Assuming you want to find the best performing model, you still have to perform cross-validation. At best you may have to perform a little less than without the added objective because it has stabilised your model against other factors that can affect generalisation. However, you might have to perform more cross-validation, at least initially, in order to establish useful values of the new relative weighting hyperparameters you added with the secondary objectives. In addition the new simplicity objective function will likely change the best choices for other hyperparameters, such as number of free parameters, learning rate and length of training time.
If you were previously performing cross-validation after every epoch of training and picking the best model after the epoch that gave the best accuracy on the cv set, then that is often considered a different form of regularisation called early stopping. You may find you could relax this and test less often during training, because with regularisation based on complexity objectives, the training will tend towards a more stable end point and be less likely to overfit through additional training epochs. Although in my experience cross-validation during training is usually left on by default, in order to plot learning curves and ensure this stability really holds.


Are there any rules for choosing batch size? I am training a CNN with a batch size of 128, but I have some fluctuations in the validation loss, which are greater than one. I want to increase my batch size to 150 or 200, but, in the code examples I have come across, the batch size is always something like 32, 64, 128, or 256. Is it a rule? Can I use other values for it?

Data that has a size of a factor of $2$ (aka $2^n$ for some integer $n$) allows for easier memory management because data can be organized contiguously (without gaps). This allows for faster memory reading and thus faster iteration time in general. From a computational point of view, this is important as it can be taken advantage of by the compiler and speed up iteration loops. This is why batch sizes are chosen as such in practice. However, this does not necessarily imply better training results.
In regards to your question of "Can I use other values for batch size?":
Yes, you can use different values, and, for the most part, you probably won't see the difference in computational performance because of the speed of modern training APIs. So, until training a large model with a lot of compute where this optimization has a larger impact, feel free to experiment :)


What is the difference between the $\epsilon$-greedy and softmax policies? Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the main difference between these two algorithms, but I didn't understand all the combinations between algorithm and policy

SARSA + $\epsilon$-greedy
SARSA + Softmax
Q-Learning + $\epsilon$-greedy
Q-Learning + Softmax

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $\epsilon $. The problem with $\epsilon$-greedy is that, when it chooses the random actions (i.e. with probability $\epsilon$), it chooses them uniformly (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others. Of course, this approach is not ideal in the case certain actions are extremely worse than others. Therefore, a natural solution to this problem is to select the random actions with probabilities proportional to their current values. These policies are called softmax policies.

Q-learning is an off-policy algorithm, which means that, while learning a so-called target policy, it uses a so-called behaviour policy to select actions. The behaviour policy can either be an $\epsilon$-greedy, a softmax policy or any other policy that can sufficiently explore the environment while learning.

The figure below shows the pseudocode of the Q-learning algorithm. In this case, the $\epsilon$-greedy policy is actually derived from the current estimate of the $Q$ function. The target policy, in this context, is represented by the $\operatorname{max}$ operator, which is used to select the highest $Q$ value of the future state $s'$, which is the state the RL agent ends up in after having taken the action $a$ selected by the $\epsilon$-greedy behaviour policy, with respect to another action $a'$ from state $s'$. This may sound complicated, but if you read the pseudocode several times, you will understand that there are two different actions (and states). The target policy (i.e. the policy that the RL agent wants to learn) is represented by the $\operatorname{max}$ operator in the sense that the so-called target of the Q-learning update step, i.e. $r + \gamma \operatorname{max}_{a'} Q(s', a')$, assumes that the greedy action is taken from the next state $s'$. For this reason, Q-learning is said to learn the greedy policy (as a target policy), while using an exploratory policy, usually, the $\epsilon$-greedy, but it can also be the softmax. Note that, in both cases, the policies are derived from the current estimate of the Q function.

On the other hand, SARSA is often considered an on-policy algorithm, given that there aren't necessarily two distinct policies, i.e. the target policy is not necessarily different than the behaviour policy, like in Q-learning (where the target policy is the greedy policy and the behaviour policy is e.g. the softmax policy derived from the current estimate of the Q function). This can more easily be seen from the pseudocode.

.

In this case, no $\operatorname{max}$ operator is used and the $\epsilon$-greedy policy is mentioned twice: in the first case, it is used to choose the action $a$ and indirectly $s'$, and, in the second case, to select the action $a'$ from $s'$. In Q-learning, $a'$ is the action that corresponds to the highest Q value from $s'$ (i.e. the greedy action). Clearly, you are free to choose a different policy than the $\epsilon$-greedy (in both cases), but this will possibly have a different effect.

To conclude, to understand the difference between Q-learning and SARSA and the places where the $\epsilon$-greedy or softmax policies can be used, it is better to look at the pseudocode.


How is it possible that deep neural networks are so easily fooled? The following page/study demonstrates that the deep neural networks are easily fooled by giving high confidence predictions for unrecognisable images, e.g.

How this is possible? Can you please explain ideally in plain English?

First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network.

  The deep neural network is the pre-trained network modeled on AlexNet provided by Caffe. To evolve images, both the directly encoded and indirectly encoded images, we use the Sferes evolutionary framework. The entire code base to conduct the evolutionary experiments can be download [sic] here. The code for the images produced by gradient ascent is available here.

Images that are actually random junk were correctly recognized as nothing meaningful:

  In response to an unrecognizable image, the networks could have output a low confidence for each of the 1000 classes, instead of an extremely high confidence value for one of the classes. In fact, they do just that for randomly generated images (e.g. those in generation 0 of the evolutionary run)

The original goal of the researchers was to use the neural networks to automatically generate images that look like the real things (by getting the recognizer's feedback and trying to change the image to get a more confident result), but they ended up creating the above art. Notice how even in the static-like images there are little splotches - usually near the center - which, it's fair to say, are triggering the recognition.

  We were not trying to produce adversarial, unrecognizable images. Instead, we were trying to produce recognizable images, but these unrecognizable images emerged.

Evidently, these images had just the right distinguishing features to match what the AI looked for in pictures. The "paddle" image does have a paddle-like shape, the "bagel" is round and the right color, the "projector" image is a camera-lens-like thing, the "computer keyboard" is a bunch of rectangles (like the individual keys), and the "chainlink fence" legitimately looks like a chain-link fence to me.

  Figure 8. Evolving images to match DNN classes produces a tremendous diversity of images. Shown are images selected to showcase diversity from 5 evolutionary runs. The diversity suggests that the images are non-random, but that instead evolutions producing [sic] discriminative features of each target class.

Further reading: the original paper (large PDF)


What are the features get from a feature extraction using a CNN? I've just started to learn CNN and somewhere I have read if I remove the last FCL I will get the features extracted from the input image but... what are those features?

Are they numbers? Labels? An image location (x,y) where there is a line.

I want to use these features on a one shot network, but I can't imagine how to use them if I don't know what they are.

You get what we call high-level features, which are basically abstract representations of the parts that carry information in the image you want to classify.

Imagine you want to classify a car. The image you feed your network could be a car on a road with a driver and trees and clouds, etc. The network, however, if you've trained it to recognize cars, will try to focus on parts of the image regarding a car. The final layers will have learned to extract an abstract representation of a car from the image (this means a low-resolution car-like shape). Now your final FC layers will try to classify the image from these high-level features. In this example, you would have an FC layer that learns to classify a car if this this abstract car-like figure is present in the image. Likewise, if it isn't present it won't classify it as a car. By accessing these high-level features, you essentially have a more compact and meaningful representation of what the image represents (based always on the classes that the CNN has been trained on).

By visualizing the activations of these layers we can take a look on what these high-level features look like.

The top row here is what you are looking for: the high-level features that a CNN extracts for four different image types.


What does the symbol $\mathbb E$ mean in these equations? I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it:

$Q^\pi \left(s,a \right) = \mathbb E \left[R_t|s_t = s, a_t = a, \pi \right]$ ,

$V^\pi \left(s \right) = \mathbb E_{a\backsim\pi\left(s \right)} \left[Q^\pi \left(s, a\right) \right]$ ,

$Q^\pi \left(s, a \right) = \mathbb E_{s'} \left[r + \gamma\mathbb E_{a'\backsim\pi \left(s' \right)} \left[Q^\pi \left(s', a' \right) \right] | s,a,\pi \right]$

$\nabla_{\theta_i}L_i\left(\theta_i \right) = \mathbb E_{s, a, r, s'} \left[\left(y_i^{DQN} - Q \left(s, a; \theta_i \right) \right) \nabla_{\theta_i} Q\left(s, a, \theta_i \right) \right]$

Could someone explain to me what is the purpose of $\mathbb E$?

That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that expression involves some random variables, which means that there may be a wide range of different values the expression may take in any concrete, single event. Taking the expectation basically means that you "average" over all the values the expression could potentially take, appropriately weighted by the probabilities with which certain events occur.

You'll often find assumptions under which the expectation is taken after a vertical line ($\mid$) inside the brackets, and/or in the subscript to the right of the $\mathbb{E}$ symbol. Sometimes, some assumptions may also be left implicit.

For example:

$$\mathbb{E} \left[ R_t \mid s_t = s, a_t = a, \pi \right]$$

may be read in english as "the expected value of the discounted returns from time $t$ onwards ($R_t$), given that the state at time $t$ is $s$ ($s_t = s$), given that our action at time $t$ is $a$ ($a_t = a$), given that we continue behaving according to policy $\pi$ after time $t$".

I would say that, in this case, the expected value also relies on the transition dynamics of the Markov decision process (i.e. the probabilities with which we transition between states, given our actions). This is left implicit.

Second example:

$$V^{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} \left[ Q^{\pi}(s, a) \right]$$

may be read as "$V^{\pi}(s)$ is equal to the expected value of $Q^{\pi}(s, a)$, under the assumption that $a$ is sampled according to $\pi(s)$".

In theory, something like this could be computed by enumerating over all possible $a$, computing $Q^{\pi}(s, a)$ for every such $a$, and multiplying it by the probability that $\pi(s)$ assigns to $a$. In practice, you could also approximate it by running a large number of experiments in which you sample $a$ from $\pi(s)$, and then evaluate a single concrete case of $Q^{\pi}(s, a)$, and average over all the evaluations.


How to write a C decompiler using AI? I would like to learn more about whether it is possible and how to write a program that decompiles executable binary (an object file) to the C source. I'm not asking exactly 'how', but rather how this can be achieved.
Given the following hello.c file (as example):
#include <stdio.h>
int main() {
  printf("Hello World!");
}

Then after compilation (gcc hello.c) I've got the binary file like:
$ hexdump -C a.out | head
00000000  cf fa ed fe 07 00 00 01  03 00 00 80 02 00 00 00  |................|
00000010  0f 00 00 00 b0 04 00 00  85 00 20 00 00 00 00 00  |.......... .....|
00000020  19 00 00 00 48 00 00 00  5f 5f 50 41 47 45 5a 45  |....H...__PAGEZE|
00000030  52 4f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |RO..............|
00000040  00 00 00 00 01 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  19 00 00 00 d8 01 00 00  |................|
00000070  5f 5f 54 45 58 54 00 00  00 00 00 00 00 00 00 00  |__TEXT..........|
$ wc -c hello.c a.out
  60 hello.c
8432 a.out

For the learning dataset, I assume I'll have to have thousands of source code files along with its binary representation, so the algorithm can learn about moving parts on certain changes.
How would you tackle this problem?
My concerns (and sub-questions) are:

Does my algorithm need to be aware of the header file, or it's "smart" enough to figure it out?

If it needs to know about the header, how do I tell my algorithm "here is the header file"?

What should be input/output mapping (whether some section to section or file to file)?

Do I need to divide my source code into some sections?

Do I need to know exactly how decompilers work or AI can figure it out for me?

Should I have two neural networks, one for header, another for body it-self?

or more separate neural networks, each one for each logical component (e.g. byte->C tag, etc.)

In-between your input and desired output, there's obviously a huge space to search. The more relevant domain information you include as features, the higher chance that the Deep Learning (DL) algorithm can find the desired mapping.

At this early stage in DL research, there aren't so many rules of thumb to tell you what features to explicitly encode - not least because it depends on the size of your training corpus. My suggestion would be: obtain (or generate) a large corpus of C code, train on that with the most naive feature representation that you think might work, then repeatedly gather data and add more feature preprocessing as necessary.

This following paper describes a DL approach to what is almost the 'reverse problem' to yours - generating the source code for a program described in natural language.

I found the strength of the results reported in this paper surprising, but it does give me some hope that what you are asking might be possible.


Can the hidden state of an RNN be a matrix? If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?

Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are:
$$
h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h)
$$
$$
y_t = \sigma_y(W_y h_t + b_y)
$$
Here $W_h$ is input to hidden state matrix, and $U_h$ is hidden state to hidden state, $W_y$ is hidden state to output. One can simply upgrade these matrix products to more general case of tensor products, such that they could handle multidimensional input data and hidden state.
The contraction of the $m$-dimensional tensor $X$ with $n$-dimensional Y (we assume $m \geq n$) by maximal amount of indices will be $m-n$ dimensional tensor $Z$:
$$
X_{i_1 \ldots i_m} Y_{i_1 \ldots i_n} = Z_{i_{n+1} \ldots i_m}
$$
Namely, imagine, that the $x_t = (x_t)_{i_1 \ldots i_X}$ is an $d_x$-dimensional tensor. Then take $W_h$ to be  $d_{wh}$ dimensional tensor, hidden state $h$ to be $d_h = d_{wh} - d_{x}$ -dimensional, $U_h$ is $d_{uh}$ - dimensional, $W_y$ is $d_{wy}$ -dimensional. The output $y_t$ of the RNN will be $d_{wy} - d_{h}$ dimensional tensor.
Here note, that in order for the sum to be make sense in the first equation, dimensions need to match:
$$
d_{uh} - d_{h} = d_{h}
$$


What model to use for fully unbalanced data? I am working on an anti-fraud project. In the project, we are trying to predict the fraud user in the out time data set. But the fraud user has a very low ratio, only 3%. We expect a model with a precision more than 15%.

I tried Logistic Regression, GBDT+LR, xgboost. All models are not good enough. Step wise Logistic Regression performs best, which has a precision of 9% with recall rate 6%.

Is there any other models that I can use for this problem or any other advise ?

You can balance your data-set.
Many models work with batches of samples. If you have a very unbalanced dataset, you can simply split it and ensure your batches are balanced (for example, for a Neural Network, using minibatches of 32 samples, you could draw 16 from your fraud users, and 16 from non-fraud users).
During the learning phase, this ensures the model doesn't just output the most common class, but instead tries to learn to distinguish both.


Given enough computational resources, do we currently have any algorithms which could achieve AI? Hypothetically, assume that you have access to infinite computing power. Do we have designs for any brute-force algorithms that can find an AI capable of passing traditional tests (e.g. Turing, Chinese Room, MIST, etc.)?

What 'infinite' means here could possibly be debated at some length, but that notwithstanding, here are two conflicting answers:

'Yes': Simulate all possible universes. Stop when you get to one containing a flavor of intelligence that passes whatever test you have in mind. Steven Wolfram has suggested something broadly along these lines. Problem: the state of computational testing for intelligence e.g. Winograd schema would then be the bottleneck. In the limit, testing for intelligence requires intelligence and creativity on behalf of the questioner.

'No': It may be that, even with infinite ability to simulate, there may be some missing aspect of our simulation that is necessary for intelligence. For example, AFAIK quantum gravity (for which we lack an adequate theory) is involved in Penrose's "Quantum Microtubules" theory of consciousness (*). What if that was needed, but we didn't know how to include it in the simulation?

The reason for talking in terms of such incredibly costly computations as 'simulate all possible universes' (or at least a brain-sized portion of them) is to deliberately generalize away from the specifics of any techniques currently in vogue (DL, neuromorphic systems etc). The point is that we could be missing something essential for intelligence from any of these models and (as far as we know from our current theories of physical reality) only empirical evidence to the contrary would tell us otherwise.

(*) No-one knows if consciousness is required for Strong AI, and physics can't distinguish a conscious entity from a Zombie.


Can we detect the emotions (or feelings) of a human through conversations with an AI? Can we detect the emotions (or feelings) of a human through conversations with an AI?
Something like a "confessional", disregarding human possibilities to lie.
Below, I have the categories joyful, sadness, anger, fear and affection. For each category, there are several words that can be in the texts that refer to it.

Joy: ( cheerful, happy, confident, happy, satisfied, excited, interested, dazzled, optimistic, relieved, euphoric, drunk, witty, good )

Sadness: ( sad, desperate, displeased, depressed, bored, lonely, hurt, desolate, meditative, defrauded, withdrawn, pitying, concentrated, depressed, melancholic, nostalgic )

Anger: ( aggressive, critical, angry, hysterical, envious, grumpy, disappointed, shocked, exasperated, frustrated, arrogant, jealous, agonized, hostile, vengeful )

Fear: ( shy, frightened, fearful, horrified, suspicious, disbelieving, embarrassed, embarrassed, shaken, surprised, guilty, anxious, cautious, indecisive, embarrassed, modest )

Affection: ( loving, passionate, supportive, malicious, dazzled, glazed, homesick, embarrassed, indifferent, curious, tender, moved, hopeful )

Flow Example
Phrase 1: "I'm very happy! It concludes college."
Categorization 1:
 - Joy (+1)

Sadness (-1)

Phrase 2: "I'm sad, my mother passed away."
Categorization 2:
 - Sadness (+1)

Joy (-1)

Phrase 3: "I met a girl, but I was ashamed."
Categorization 3:
 - Fear (+1)
Is this a clever way to follow and / or improve, or am I completely out of the way?
I see that there is a Google product that creates parsing according to the phrases. I do not know how it works, because I like to recreate the way I think it would work.
Remembering that this would not be the only way to categorize the phrase. This would be the first phase of the analysis. I can also identify the subject of the sentence, so we would know if the sadness is from the creator of the message or from a third party, in most cases.

NLTK
Sentiment Analysis Python Example

I think you are definitely on a very sensible track. No one defines right or wrong in emotion field. It's not hard science. It's all theories.

I have recently read a paper regarding emotions in Reinforcement Learning (RL). It has explained briefly emotion from 3 perspectives: psychology, neuroscience and computer science. In particular, your way of emotion definition matches one of the categorical emotion theory in the psychology perspective. The other theories in psychology perspective and componential emotion. You can try to implement them and try out which one works well. The paper has also introduced ways to measure the level of emotions (emotion elicitation).

Here is the link for the paper I have mentioned. I am sure you will receive lots of inspiration. I have also written a summary of this paper. Take a look if the original paper is too long to read.

I don't have any concrete solution for implementing. But the general idea is always trying to categorize abstract concepts and quantify them. And try some thing, and iteratively modify and improve it. All the best!


How can a genetic algorithm adapt and get better in a changing environment? I've just started studying genetic algorithms and I'm not able to understand why a genetic algorithm can improve if, at each learning, the 'world' that the population encounters change. For example, in this demo (http://math.hws.edu/eck/js/genetic-algorithm/GA.html), it's pretty clear to me that the eating statistics will improve every year if bunches of grass grow exactly in the same place, but instead they always grow in different positions and I can't figure out how it  can be useful to evaluate (through the fitness function) the obtained eating stats given that the next environment will be different.

There are a couple of ways of dealing with this. Very often, the approach is just to design your representation and operators in a way to account for the fact that the world changes. The idea is to give the algorithm something that can be used to learn general behaviors or solutions rather than specific ones.
Take an example of learning to steer a race car around a track. You want to represent the state of the world and have the GA learn to select an appropriate action. You might choose to represent the state of the world as a vector of $[x, y, v, a]$, where $(x, y)$ is your location on the track, $v$ is your current velocity vector, and $a$ is an acceleration to apply. The fitness function could return how "good" it was to apply that acceleration. If you do this, your algorithm can probably learn to navigate this track, but a different track will be hopeless, as the locations aren't corresponding to the same turn locations on the new track.
However, what if you encode the world as $[s, v, a]$, where instead of an $(x, y)$ pair representing your current position, you have $s$ as a vector of sensor readings? Is there a wall coming up in front of you or is the track starting to bank? Now, your algorithm can learn to be more general. It doesn't need to be the exact track it learned on, because what it's learning is not to brake at a specific point, but to brake when it detects a wall coming up.
I didn't do a lot of digging into the example you linked, but if you run it for a few years, you can see evidence of this. You see agents that appear to have learned to move in lines horizontally or vertically until they encounter a green square, and then they'll stick around and eat in the patches around that square. That behavior is general, because every environment it encounters has lots of blank space with clusters of green. It's not learning "go to square $(20, 30)$". It's learning to move in a pattern until it finds green and then move around that location.
You can do this in a lot of cases where the specific environment can change, but the objective is the same. There are problems where the actual fitness function changes over time, however. For those problems, there are specific techniques to deal with dynamic fitness functions. Generally, this involves doing something to maintain diversity so that your whole population isn't getting stuck on whatever the current "best" looks like. That's a bit more advanced topic though, and I think your question was really more about the former type of problems.


How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs? I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition.

Can I know what are the differences between $Q(\lambda)$ and Q-learning for this problem (SMDP)? I actually want to extend the pseudo-code presented here to an SMDP problem with discretization of time horizon.

If you really just want an SMDP-version of the algorithm, which only needs to be capable of operating on the "high-level" time scale of macro-actions, you can relatively safely take the original pseudocode of whatever MDP-based algorithm you like, replace every occurrence of "action" with "macro-action", and you're pretty much done.

The only caveat I can think of in the case of $Q(\lambda)$ is that the "optimal" value for $\lambda$ is probably somewhat related to the amount of time that expires... so intuitively I'd expect it to be best if the value for $\lambda$ decreases as the amount of time expired during execution of the last macro-action increases. A constant $\lambda$ probably still works fine as well though.

If you actually want your algorithm to also be aware of lower-time-scale MDP underlying an SMDP, and not only treat macro-actions as "large actions" and be done with it... I'd recommend looking into the Options framework. There you get interesting ideas like intra-option updates, which may allow you to also perform learning whilst larger macro-actions (or options) are still in progress.

Last time I looked there hasn't been a lot of work involving the combination of eligibility traces and options, but there has been some: Eligibility Traces for Options. This paper doesn't specifically apply the algorithm you mentioned ($Q(\lambda)$), but it does discuss a bunch of other -- much more recent, and likely better -- off-policy algorithms with eligibility traces.


How exactly does SGD with momentum works in keras This is my first question in this community (Artificial Intelligence) and I hope I can get some answers to a troublesome issue of mine.

I am trying to understand Stochastic Gradient Decent (SGD) (with possibly momentum) and modify the keras version to an accumulative one. I have asked a similar question on Stack Overflow but here I am interested in the actual function (and secondarily to the code itself).

For simplicity let's consider I have a neural network with just 10 layers and I am training for 100 iterations (regardless of epochs).
For easier reference I post the keras code here (and also skipped nesterov and constraints lines of code):

def get_updates(self, loss, params):
    grads = self.get_gradients(loss, params)
    self.updates = [K.update_add(self.iterations, 1)]

    lr = self.lr
    if self.initial_decay > 0:
        lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))
    shapes = [K.int_shape(p) for p in params]
    moments = [K.zeros(shape) for shape in shapes]
    self.weights = [self.iterations] + moments
    for p, g, m in zip(params, grads, moments):
        v = self.momentum * m - lr * g  # velocity
        self.updates.append(K.update(m, v))
        new_p = p + v

        self.updates.append(K.update(p, new_p))
    return self.updates

As far as I understand the mechanism of SGD works as follows:

The gradients are updated at every iteration, which in the keras code is performed by calling the get_updates(self, loss, params) function.
There is a gradient that correspond to each layer (in the keras code I think a list of Tensors are produced to keep the gradients).
So, in that sense the for p, g, m in zip(params, grads, moments) loop just iterates through all layers (and their corresponding gradients and moments) and calculates new values.

The really confusing part for me is that if the above function is being called at every iteration then the moments = [K.zeros(shape) for shape in shapes] basically initializes moments to a zero Tensor at every iteration. Wouldn't make more sense if moment was initialized once and then keep updating with new values?

I have noticed that despite my understanding the moments are actually used (as it would be more expected for such a big project to have missed a large bug such as this one). I cannot really understand how though.

I noticed the same issue. Also, it seems that the equation for applying momentum does not exactly match up with the standard one:

$$ v_{d\theta} := \beta v_{d\theta} + (1 - \beta)d\theta \\
 \theta := \theta - \alpha v_{d\theta}$$


How to manually draw a $k$-NN decision boundary with $k=1$ given the dataset and labels? How to manually draw  a $k$-NN decision boundary with $k=1$ knowing the dataset

the labels are

and the euclidean distance between two points is defined as

This is a rather involved task. What to do from a high-level theoretical perspective might be easy to see, but it's difficult putting that into code from scratch.
Doing this in Python using existing libraries in not too complicated, though.
See for example this tutorial or this StackOverflow post.
Edit:
Theoretically, I would first plot (draw) the points from your dataset in a graph and then watch out for "decision points" half-way in between any two near-by points (from the dataset) from distinct classes.
For the next step, keep in mind those decision points. Given close-by data points from classes (e.g.) A and B, imagine a straight line connecting these two points from separate classes. Next, take the point half-way along that imaginary line (i.e. your decision point) and draw a "soft" line (maybe using pencil instead of pen) orthogonal/perpendicular to that imaginary line which intersects the imaginary line in the decision point.
Do that for all combinations of "reasonably" close points from different classes.
Parts of the lines you have just drawn will define the final decision boundary. Next, think of each line as consisting of multiple elements, which are separated from one another by means of the points of intersection with other drawn lines. In other words, split lines into elements wherever they intersect other lines. Now, decide which of these elements to outline as the eventual decision boundary (finally using a pen instead of pencil). This step simply involves human intelligence and is difficult to describe. Whenever a line's element accurately separates (logically speaking) two classes, indicate it using a pen. Otherwise, if it does not contribute to separating two classes, don't indicate it.
After having indicated the final decision boundary using a pen, simply erase the pencil drawings. Now, you should be left with a decision boundary.
I hope this is clear and accurate enough.


In how few updates can a multi layer neural net be trained? A single iteration of gradient descent can be parallelised across many worker nodes. We simple split the training set across the worker nodes, pass the parameters to each worker, each worker computes gradients for their subset of the training set, and then passes it back to the master to be averaged. With some effort, we can even use model parallelism.

However, stochastic gradient descent is an inherently serial proces. Each update must be performed sequentially. Each iteration, we must perform a broadcast and gather of all parameters. This is bad for performance. Ultimately, number of updates is the limiting factor of deep model training speed.

Why must we perform many updates?
With how few updates can we achieve good accuracy?

What factors affect the minimum number of updates requires to reach some accuracy?

In Don't Decay the Learning Rate, Increase the Batch Size, Smith et al. train ResNet-50 on ImageNet to 76.1% with only 2500 updates. Has anyone done it in less?

In The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent Sankararaman et al. present the concept of gradient confusion which slows convergence, and show that depth increases gradient confusion while overparameterization on width and skip connections reduces gradient confusion.

In Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash, Mikami et al. train ResNet-50 on ImageNet to 75.29% with some number of updates, but I can't do the math to compute out how many updates they must have used.

To summarise:

Larger mini-batches help, but give diminishing returns and cause other issues.
Deeper models tend to require more updates.
Wider layers need fewer updates.
Skip connections, label smoothing, and batch norm might help.
The best I have found so far on ImageNet/ResNet-50 to >75% is 2500 updates.


How to export Pytorch Deep Neural Networks trained model to C++ program to use it from C++? I want to build a DNN model that I will later integrate into a C++ program.

I heard that PyTorch model is hard to load it on C++ and the integration requires extra code, and it's complicated.

I have searched a bit on the internet and I have found this post: Loading a PyTorch Model in C++. I am still unsure whether or not I will have any problems in practice if I opt for PyTorch.

PyTorch now has a C++ frontend. I haven't tried it, but I'm sure you could use that.

Another option, which is more production-tested, is using a message passing framework such as ZeroMQ to communicate requests and results between Python and C++ executables.


Is case-based reasoning a machine learning technique? A few years ago when I was in university, I had implemented (for my final year project) an Itinerary Planning System, which incorporates an AI technique called "case-based reasoning".
Is case-based reasoning a machine learning technique or an AI technique (that is not machine learning)?

The book Machine Learning (1997) by Tom Mitchell covers case-based reasoning (CBR), a form of instance-based learning (nearest neighbor is the typical example of IBL) in chapter 8 (p. 230).
T. Mitchell  writes

Instance-based methods such as $k$-NEAREST NEIGHBOR and locally weighted regression share three key properties. First, they are lazy learning methods in that they defer the decision of how to generalize beyond the training data until a new query instance is observed. Second, they classify new query instances by analyzing similar instances while ignoring instances that are very different from the query. Third, they represent instances as real-valued points in an $n$-dimensional Euclidean space. Case-based reasoning (CBR) is a learning paradigm based on the first two of these principles, but not the third. In CBR, instances are typically represented using more rich symbolic descriptions, and the methods used to retrieve similar instances are correspondingly more elaborate. CBR has been applied to problems such as conceptual design of mechanical devices based on a stored library of previous designs (Sycara et al. 1992), reasoning about new legal cases based on previous rulings (Ashley 1990), and solving planning and scheduling problems by reusing and combining portions of previous solutions to
similar problems (Veloso 1992).

He then goes on and gives the example of a CBR system: the CADET system. He also formulates CBR as a learning problem and uses the term "learn" to refer to a search process that CADET goes through, which is similar to what k-NN does.
He then writes

To summarize, case-based reasoning is an instance-based learning method in which instances (cases) may be rich relational descriptions and in which the retrieval and combination of cases to solve the current query may rely on knowledge-based reasoning and search-intensive problem-solving methods.

To conclude, yes, CBR can be considered a machine learning technique (if you also consider k-NN a learning algorithm, which people often do), even though it may rely on knowledge-based reasoning and search-intensive problem-solving methods.
You may also be interested in the paper Representation in case-based reasoning (2005) by Ralph Bergmann et al. Moreover, the famous AIMA book (3rd edition) mentions case-based reasoning in chapter 19 (p. 799), which is dedicated to knowledge and learning.


Artifacts After pruning Unet CNN I'm trying to make a dark image brighter using CNN-UNet architecture.
When I train the network, I get the following results:

When I cut the features in half for pruning, and do full train again, I get the following result:

There are those bubble artifacts near the light.
Why are they happening?

How can I locate the problematic neurons that cause it?

Can I train the network (change the loss or anything) to not generate those made up bubbles?

Did you retrain the network after pruning the weights?

If not, 1) the bubbles are probably appearing because of that, and 2) you definitely should retrain your network.

Did you prune output features only or weights inside the network also?

The standard procedure is to retrain the network after pruning, and the best results can be achieved with iterative pruning: you prune small part of weights (the loss of accyracy is inevitable here), then retrain to get the accuracy back, and repeat until you won't be able to get the accuracy back to high level.

Here Pruning Convolutional Neural Networks for Resource Efficient Inference is an article about (among others) iterative pruning.


What is the relevance of AIXI on current artificial intelligence research? From Wikipedia:

  AIXI ['ai̯k͡siː] is a theoretical mathematical formalism for artificial general intelligence. It combines Solomonoff induction with sequential decision theory. AIXI was first proposed by Marcus Hutter in 2000[1] and the results below are proved in Hutter's 2005 book Universal Artificial Intelligence.[2]

Albeit non-computable, approximations are possible, such as AIXItl. Finding approximations to AIXI could be an objective way for solving AI.

Is AIXI really a big deal in artificial general intelligence research? Can it be thought as a central concept for the field? If so, why don't we have more publications on this subject (or maybe we have and I'm not aware of them)?

"Current artificial intelligence research" is a pretty broad field. From where I sit, in a mostly CS realm, people are focused on narrow intelligence that can do economically relevant work on narrow tasks. (That is, predicting when components will fail, predicting which ads a user will click on, and so on.)

For those sorts of tools, the generality of a formalism like AIXI is a weakness instead of a strength. You don't need to take an AI that could in theory compute anything, and then slowly train it to focus on what you want, when you could just directly shape a tool that is the mirror of your task.

I'm not as familiar with AGI research itself, but my impression is that AIXI is, to some extent, the simplest idea that could work--it takes all the hard part and pushes it into computation, so it's 'just an engineering challenge.' (This is the bit about 'finding approximations to AIXI.') The question then becomes, is starting at AIXI and trying to approximate down a more or less fruitful research path than starting at something small and functional, and trying to build up?

My impression is the latter is much more common, but again, I only see a small corner of this space.


How are these equations of SGD with momentum equivalent? I know this question may be so silly, but I can not prove it.
In Stanford slide (page 17), they define the formula of SGD with momentum like this:
$$
v_{t}=\rho v_{t-1}+\nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t},
$$
where:

$v_{t+1}$ is the momentum value
$\rho$ is a friction, let say it's equal 0.9
$\nabla f(x_{t-1})$ is the gradient of the objective function at iteration $t-1$
$x_t$ are the parameters
$\alpha$ is the learning rate

However, in this paper and many other documents, they define the equation like this:
$$
v_{t}=\rho v_{t-1}+\alpha \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}- v_{t},
$$
where $\rho$ and $\alpha$ still have the same value as in the previous formula.
I think it should be
$$v_{t}=\alpha \rho v_{t-1}+\alpha \nabla f(x_{t-1})$$
if we want to multiply the learning rate inside the equation.
In some other document (this) or normal form of momentum, they define like this:
$$
v_{t}= \rho v_{t-1}+ (1- \rho) \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t}
$$
I can not understand how can they prove those equations are similar. Can someone help me?

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately.
Equation 1
Consider the equation from the Stanford slide:
$$
v_{t}=\rho v_{t-1}+\nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t},
$$
Let's evaluate the first few $v_t$ so that we can arrive at a closed form solution:
$v_0 = 0 \\
v_1 =  \rho v_0 + \nabla f(x_0) = \nabla f(x_0)\\
v_2 = \rho v_1 + \nabla f(x_1) = \rho \nabla f(x_0) + \nabla f(x_1)\\
v_3 = \rho v_2 + \nabla f(x_2) = \rho^2 \nabla f(x_0) + \rho \nabla f(x_1) + \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \nabla f(x_i)
$
So the closed form update is:
$$x_t = x_{t-1} - \alpha \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \nabla f(x_i)$$
Equation 2
Now consider the equation from the paper:
$$
v_{t}=\rho v_{t-1}+\alpha \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}- v_{t},
$$
We again evaluate the first few $v_t$ to arrive at a closed form solution:
$v_0 = 0 \\
v_1 =  \rho v_0 + \alpha \nabla f(x_0) = \alpha \nabla f(x_0)\\
v_2 = \rho v_1 + \alpha \nabla f(x_1) = \rho \alpha \nabla f(x_0) + \alpha \nabla f(x_1)\\
v_3 = \rho v_2 + \alpha \nabla f(x_2) = \rho^2 \alpha \nabla f(x_0) + \rho \alpha \nabla f(x_1) + \alpha \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \alpha \nabla f(x_i)
$
So the closed from update is:
$$x_t = x_{t-1} - \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \alpha \nabla f(x_i)$$
As you can see, this is equivalent to the previous closed form update. The only difference is if $\alpha$ is inside or outside the summation, but since it is a constant, it doesn't really matter anyways.
Equation 3
As for the last equation
$$
v_{t}= \rho v_{t-1}+ (1- \rho) \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t}
$$
Let's do the same thing:
$v_0 = 0 \\
v_1 =  \rho v_0 + (1-\rho) \nabla f(x_0) = (1-\rho) \nabla f(x_0)\\
v_2 = \rho v_1 + (1-\rho) \nabla f(x_1) = \rho (1-\rho) \nabla f(x_0) + (1-\rho) \nabla f(x_1)\\
v_3 = \rho v_2 + (1-\rho) \nabla f(x_2) = \rho^2 (1-\rho) \nabla f(x_0) + \rho (1-\rho) \nabla f(x_1) + (1-\rho) \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} (1-\rho) \nabla f(x_i)
$
And so the closed form update is:
$$x_t = x_{t-1} - \alpha \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} (1-\rho) \nabla f(x_i)$$
This equation is equivalent to the other two as long as you scale $\alpha$ by a factor of $\displaystyle \frac{1}{1-\rho}$.


Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture? In the paper Human-level control through deep reinforcement learning, the DQN architecture is presented, where the loss function is as follows

$$
L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \theta_i^-)  - Q(s, a; \theta) \right)^2\right]
$$

where $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ approximates the "target" of $Q(s, a; \theta)$. But it is not clear to me why. How can existing weights approximate the target (ground truth)? Isn't $r$ is a sample from a the experience replay dataset? Is $r$ a scalar value?

This is the problem that reinforcement learning (RL) is trying to solve: What is the best way to behave when we don’t know what the right action is and only have a scalar (the reward (r) is a scalar) reward of how well we have done?

RL approaches this problem by utilizing temporal difference learning and makes predictions based on the previous experience. An RL agent is trying to maximize the sum of future discounted rewards, called the return.

The term $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ is essentially saying "the reward I just saw + $\gamma$ * (my prediction of the return given that I take what I think is the best action in the current state and follow my policy from then on)".

By updating the Q function, the agent can better predict the consequences of its actions and can then choose the best action with greater probability.

The $\gamma$ (gamma) helps to balance between the immediate reward and future rewards. A $\gamma = 0$ makes the immediate reward the only thing that matters but usually the reward for a good action is delayed so values of gamma which put a higher importance on later rewards ($\gamma = .8, .9, .99$, etc) are used.


Understanding how the loss was calculated for the SQuAD task in BERT paper In the BERT paper, section 4.2 covers the SQuAD training.

From my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End). For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.

I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph

  and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions.

What do they mean by "maximum scoring span is used as the prediction"?

Furthermore, how does that play into

  The training objective is the loglikelihood of the correct start and end positions

From this source: https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

It says the log-likelihood is only applied to the correct classes. So, we are only calculating the softmax for the correct positions only, not any of the incorrect positions.

If this interpretation is correct, then the loss will be

Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )

These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help:

What do they mean by "maximum scoring span is used as the prediction"?

As you know in SQuAD the input sequence is divided to 2 parts: Question and Document (from which we extract the answer if possible).

Sometimes the input length exceeds the max_seq_length parameter, in this case the document is truncated to as many parts as needed and we end up having more than one input for the same question/document. Mention that the question is replicated in all the resulting inputs (see line_350 for details).

So in such cases, in order to determine the predicted span among all the generated-inputs we use the maximum scoring i.e (max_start + max_end) / 2 = ( max(Softmax(S*Ti)) + max(Softmax(E*Ti)) ) / 2 of all the inputs related to the same question.

"The training objective is the loglikelihood of the correct start and end positions"

The loss is the average of the start_position loss start_loss and the end_postition loss end_loss. Each loss is computed in the same way: after applying the Softmax to the final output (i.e S*Ti or E*Ti) we use the real start/end postitions to compute the loss (see code below).

From run_squad.py in Bert_repo:

 def compute_loss(logits, positions):
        one_hot_positions = tf.one_hot(positions, depth=seq_length, dtype=tf.float32)
        log_probs = tf.nn.log_softmax(logits, axis=-1)
        loss = -tf.reduce_mean(tf.reduce_sum(one_hot_positions * log_probs, axis=-1))
        return loss

 start_positions = features["start_positions"]
 end_positions = features["end_positions"]

 start_loss = compute_loss(start_logits, start_positions)
 end_loss = compute_loss(end_logits, end_positions)

 total_loss = (start_loss + end_loss) / 2.0


In a neural network, can colors be used for neurons in place of floating points and would there be any benefit in doing so? Firstly, some context. I have been reading and watching videos on the subject for around 3 years, but I am still very much a beginner in machine learning and artificial intelligence. That said, I might not know what I'm even talking about here. So bear with me.

If I understand correctly, each node in a neural network (neuron) is represented by some floating point number between 0 and 1, that are arranged in layers and have corresponding weights. Right? While a color has RGB values, CMYK values, and HSV values that are all interrelated to each other.

My question is would there be any benefit to having each node represented by a color instead of a single floating point number?

My thinking is that each neuron could select any of the values (r, g, b, c, m, y, k, h, s, or v) contained within the color in some meaningful way, while the Alpha value could possibly represent the weight associated with that neuron.

Thoughts? Would it not work like that? Could you use it to have multiple congruent networks running on 3 different channels? Again, would there be any benefit to doing this than just using a single number? Or would it over-complicate (or even break) the network? Would it be useless?

Although I've also dabbled in Unity3D (which is how I got the idea in the first place), I'm too much of a beginner to know how to even begin an attempt at testing this myself.

To answer your question, I'll just simplify it and assume you are representing the activation of a neuron (the value it produces) by an RGB value. So a tuple with 3 values ranging from 0 to 1.

It's important to remember that every single machine learning model could be computed by a human on pen and paper using just numbers. So keeping that in mind, this neural network would essentially just be a combination of multiplying tuples of size 3 in a certain order (ignoring activation functions).

I will also assume that each RGB value has it's own unique set of incoming weights, like following:

this essentially amounts to just have 3 unique neurons. There's no difference:

All I did there was get rid of the circle. The calculations are no different. If you changed the calculation, say values are shared between the 3 RGB nodes, then you now just have 3 duplicate nodes. In all cases, you can just represent the equivalent calculation as a standard neural network.

However, in the same vein, in a convolutional neural network, if you create conv layers with an output of 3 channels, you can visualise their outputs as a colour image (assuming you are using an activation function that outputs numbers between 0 and 1), which can be cool to look at.


Can a neural network be used to predict the next pseudo random number? Is it possible to feed a neural network the output from a random number generator and expect it learn the hashing (or generator) function, so that it can predict what will be the next generated pseudo-random number?

Does something like this already exist? If research is already done on this or something related (to the prediction of pseudo-random numbers), can anyone point me to the right resources?

Currently, I am looking at this library and its related links.
https://github.com/Vict0rSch/deep_learning/tree/master/keras/recurrent

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random.

When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course be easily predicted by a neural network with little training. But in the real world things look different.

The neural network could be trained to find certain patterns in the history of random numbers generated by a PRNG to predict the next bit. The stronger the PRNG gets, the more input neurons are required, assuming you are using one neuron for each bit of prior randomness generated by the PRNG. The less predictable the PRNG gets, the more data will be required to find some kind of pattern. For strong PRNGs this is not feasable.

On a positive note, it is helpful that you can generate an arbitrary amount of training patterns for the neural network, assuming that you have control over the PRNG and can produce as many random numbers as you want.

Because modern PRNGs are a key component for cryptography, extensive research has been conducted to verify that they are "random enough" to withstand such prediction attacks. Therefore I am pretty sure that it is not possible with currently available computational resources to build a neural network to successfully attack a PRNG that's considered secure for cryptography.

It is also worth noting that it is not necessary to exactly predict the output of a PRNG to break cryptography - it might be enough to predict the next bit with a certainty of a little more than 50% to weaken an implementation significantly. So if you are able to build a neural network that predicts the next bit of a PRNG (considered secure for cryptography) with a 55% success rate, you'll probably make the security news headlines for quite a while.


What do you call a machine learning system that keeps on learning? As I understand it from this video lecture, there are three types of deep learning:

Supervised
Unsupervised
Reinforcement

All these can serve to train a neural network either only prior to its deployment or during its operating.

For the latter case, it is referred to as continuous learning here and here and as dynamic learning here and here.

Which term should I use to refer to a machine learning algorithm that keeps on learning (even after deployment)? If it's "continuous", is there an opposing term (such as "static" for "dynamic") for those systems that stop learning before being deployed?

There are several terms or expressions related to such systems, such as online learning, incremental learning, continuous learning, continual learning, and lifelong learning. They are sometimes used interchangeably, but some of them have slightly different meanings. For example, online learning does not need to be incremental, which refers to algorithms that attempt not to forget previously learned information.

The opposite of online is offline. However, the expression batch learning is sometimes used as an antonym for online learning.