What is the best approach for multivariable and multivariate regression? I want to build a multivariable and multivariate regression model in Keras (with TensorFlow as backend), that is, a regression model with multiple values as input (multivariable) and output (multivariate).

The independent variables are, for example, the length, weight, force, etc., and the dependent variables are the torque, friction, heat, temperature, etc.

What is the best approach to achieve that? Any guidance before I start?  (If anyone can share any example code/notebook/code would be great as well).

You create a model like this very easily with keras. Follow these steps.
Data Exploring
Before building a machine learning model, you must explore your data first. An approach is to use libraries to visualize the data as graphs. You can use the tool pandas-profiling
https://towardsdatascience.com/exploring-your-data-with-just-1-line-of-python-4b35ce21a82d
Points to look for when exploring the data
You should look for the following:

Distribution of labels and input features
Range of input features and labels
Missing label or input feature
Nan on value
Outliers

You should look out for them and use appropriate data cleaning or filtering method to clean them. For example, Nan or missing value may be substituted with 0 and outliers will be removed. Input features may also be normalized.
Building the model
Keras is a easy tool for building machinea learning model. For how to build a basic MLP ( Multi-Layer Perceptron), you can refer to the example code and the resource.
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
Experimenting
After creating the basic model, you can start experimenting with stuff like:

Different model structure like adding layers and changing hidden layer size
Different Optimizer
Different hyperparameters like learning rate and batch size
Different input features

You will gain more experience as you go. There is a lot of ways to improve the performance of the model.
Bewares
Here are things to beware:

If your model doesn't work out, don't worry. Try another method or tune parameters until it work.
If the testing loss become very high while training loss is low, you can add dropout to prevent this from happening. This is called overfitting.

Hope I can help you and have fun!


Why do Decision Tree Learning Algorithm preferably outputs the smallest Decision Tree? I have been following the ML course by Tom Mitchel.
The inherent assumption while using Decision Tree Learning Algo is: The algo. preferably chooses a Decision Tree which is the smallest.

Why is this so when we can have bigger extensions of the tree which could in principle perform better than the shorter tree?

The bigger your tree is the more overfitting your model is. In machine learning, we always prefer a simpler model unless there is good reason to go for complication.


What would be a good internal language for an AI? For an AI to represent the world, it would be good if it could translate human sentences into something more precise.

We know, for example, that mathematics can be built up from set theory. So representing statement in language of set theory might be useful.

e.g. "All grass is green" is something like: $\forall x\in grass: isGreen(x)$

But then I learned that set theory is built up from something more basic. And that theorem provers use a special form of higher order logic of types. Then there is propositional logic.

Basically what the AI would need would be some way representing statements, some axioms, and ways to manipulate the statements.

Thus what would be a good language to use an internal language for an AI?

This is (even though it doesn't look like it at first glance) a deeply philosophical question about the nature of 'meaning'. This answer is necessarily limited in scope.

There are many ways of representing knowledge, and countless formalisms have been developed since the early days of AI. Many of them are based on some kind of predicate calculus, ontologies, semantic networks (providing eg inheritance of features and part-of relationships), and they seem to work fine for limited domains.

One problem is the grounding: if you have a predicate isGreen(x), what does that actually mean? How is it related to isBlue(x)? Do you want to treat them similarly? If so, you need to represent this somehow. You quickly come to the point where you will need to encode all the world's knowledge in some generalised way. An impossible task.

Linguists have struggled with this for decades: what is the meaning of a particular sentence? Apart from the fact that every individual human will interpret a given sentence differently (based on their own life experience and culture), there are many aspects to 'meaning' that need representing: the 'factual' meaning, but also pragmatic, evaluative, and all sorts of other nuances. An innocent utterance, That's a nice Apple you've got there, could have a whole raft of meanings packed into it, all implicit. For example, the person probably likes apples, that one in particular, that apple looks like a tasty piece of fruit, the other person is the owner of the apple, and it might also be a request which is intended to prompt the other person to offer it to you. How are you going to represent all that meaning?

One area that interests me personally is representing narrative events. This can — up to a point — be done using Conceptual Dependency, which uses a limited set of semantic primitives. While useful to encode basic stories, you cannot easily use it to represent the fact that grass is green.

So the answer is: there is no answer. AI is too broad a field, and you need to look at a particular application to decide which knowledge is relevant to it, and then how it can best be represented. There is a reason why there are so many ways of representing knowledge.

PS: You suggest this would be more precise. My personal view is that precision here is a red herring. The word green is not precise, as it covers a range of wave lengths, and different people would disagree on whether something is green or not. So a predicate isGreen(x) is not any more precise than that. Hence the appeal of fuzzy logic, which allows computation to be based on less precision.


Is it a valid Deep Neural Network? For a regression task, I have sequences of training data and if I define the layers of deep neural network to be:

Layers=[ sequenceInputLayer(featuredimension) reluLayer dropoutLayer(0.05) fullyConnectedLayer(numResponse) regressionLayer]

Is it a valid deep neural network? Or do I need to add LSTM layer too?

Yes, it is a very common practice to use some RNN when your input data is a sequence.
Besides, your network has some shape issue if your input data is 2D. You should, at least flatten your input data to a vector to be able to forward propagate to the dense layer, but instead of this use some kind of RNN.
To the best of my knowledge, that dropout value doesn't good. It seems too low.


How would researchers determine the best deep learning model if every run of the code yields different results? There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network.

Since results might be stochastic, how would researchers know what their best performing model is? I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?

I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?

That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is that it is the best one given the hyperparameters you have selected and that specific seed. Sometimes the value of the seed is highly relevant, other times less so.

  Since results might be stochastic, how would researchers know what their best performing model is?

In general, as with any experiment where measurements are variable, by running the experiment multiple times and taking statistics over the set of results. This will give you a much better sense of how the algorithm does in general, independently of specific seeds. You can still fix your RNG seeds for repeatability, but you will need multiple sets of them.

For certain goals, such as making the best possible model that you can, independently of whether the approach you take is "best in general", this is not necessary. A single run which creates a state-of-the-art performance is still of interest, for instance. Or if you are creating a model that you want to use in production, you may care less about the stability of the technique (and being "lucky") than being in possession of a high performing agent for the task.


How to avoid being stuck local optima in q-learning and q-network When using Bellman equation to update q-table or train q-network to fit to greedy max values, the q-values very often get to the local optima and get stuck although randomisation rate ($\epsilon$) already applied since start.
The sum of q-values of all very first steps (of different actions at the original location of agent) increases gradually until a local optimum is reached. It gets stuck and this sum of q-values starts decreasing slowly a bit by a bit.
How to avoid being stuck in a local optimum? and how to know if the local optimum is already the global optimum? I may think of this but it's chaotic: Switch on randomisation again for a while, worse values may come at first but may be better in future.

I found out the problem why the optimisation process got stuck and never moved closer to global optimum. It's because of the rate between 'explore' or 'exploit'.
Basically, in RL, agent 'explore's by doing a random action and to find new solutions, 'exploit's the existing so-called known max future rewards to do the max action.
Initially, I put the agent to explore when $random() < 1/(replay\_index+1)$, exploration rate reduces too quick (<10% after 10 iterations), and when the number of replays (number of times to play again from start) is not enough, the explore rate at the end of the loop is almost zero, and nothing new learnt.
The solution opted is allowing 'explore' and 'exploit' have the same rate (or lower exploration a bit is also ok), pseudo-code:
# Part 1 in a step: Choose action
if random() < 0.5: # 0.25 is also good, 25% for exploration
    action = random_action()
else:
    action = choose_best_known_action()

Explore rate can be reduced correctly this way:
if random() < 1-i/NUM_REPLAYS: # i is current train step index
    action = random_action()
else:
    ...

With the half-explore/half-exploit scheme above, the agent will learn to infinity, so, it is kinda sure that global optimum would be reached. When knowing from practice the number of iterations should be used, 'exploit' may be utilised more for faster convergence.
Note that the 'explore' and 'exploit' rates are put equal above, the but q-table or q-network is still better and better due to having another 'exploit'-kind when updating q-table or fitting q-network with Bellman equation, there's another 'exploit' here, the 'max' in Bellman equation:
Pseudo-code:
# Part 2 in a step: Update q-table or q-network
q[s][a] += learning_rate * (reward + max(q[sNext][aNext]) - q[s][a])

# Q-network
# target = r + max(...
```


How can Image Caption work? I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be .pkl files + caption file. Right?
If someone can help me in getting the link between the two models, I will appreciate it.

The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.
The training loop in PyTorch would look something like this:
# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()

# Forward pass
features = encoder(image)
outputs = decoder(features, captions)

# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size),
                         captions.view(-1))

# Backward pass.
loss.backward()

# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.
I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.


What event would confirm that we have implemented an AGI system? I was listening to a podcast on the topic of AGI and a guest made an argument that if strong music generation were to happen, it would be a sign of "true" intelligence in machines because of how much creative capability creating music requires (even for humans).
It got me wondering, what other events/milestones would convince someone, who is more involved in the field than myself, that we might have implemented an AGI (or a "highly intelligent" system)?
Of course, the answer to this question depends on the definition of AGI, but you can choose a sensible definition of AGI in order to answer this question.
So, for example, maybe some of these milestones or events could be:

General conversation
Full self-driving car (no human intervention)
Music generation
Something similar to AlphaGo
High-level reading/comprehension

What particular event would convince you that we've reached a high level of intelligence in machines?
It does not have to be any of the events I listed.

It is a difficult question to answer, as — for a start — we still don't really know what 'intelligence' means. It's a bit like Supreme Court Justice Potter Stewart declining to define 'pornography', instead stating that [...]I know it when I see it. AGI will be the same.

There is no single event (almost by definition), as that's not general. OK, we've got machines that can beat the best human players at chess and go, two games that were for centuries seen as an indication of intelligence. But can they order a takeaway pizza? Do they even understand what they are doing? Or, even more fundamental, know what they means in the previous sentence?

In order for a machine to show a non-trivial level of intelligent behaviour, I would expect it to interact with its environment (which is more social intelligence, an aspect that seems to be rather overlooked in much of AI). I would expect it to be aware of what it's doing/saying. If I have a conversation with a chatbot that really understands what it's saying (and can explain why it came to certain conclusions), that would be an indication that we're getting closer to AGI. So Turing wasn't that far off, though nowadays it's more achieved with smoke and mirrors rather than 'real' intelligence.

Understanding a story: being able to finish a partial story in a sensible way, inferring and extrapolating the motives of characters, being able to say why a character acted in a particular way. That for me would be a better sign of AGI than beating someone at chess or solving complex equations. Jokes that are funny; breaking rules in story-telling in a sensible way.

Writing stories: NaNoGenMo is a great idea, and throws up lots of really creative stuff, but how many of the resulting novels would you want to read instead of human-authored books? Once that process has generated a best-seller (based on the quality of the story), then we might be getting closer to AGI.

Composing music: of course you can already generate decent music using ML algorithms. Similar to stories, the hard bit is the intention behind choices. If choices are random (or based on learnt probabilities), that is purely imitation. An AGI should be able to do more than that. Give it a libretto and ask it to compose an opera around it. Do this 100 times, and when more than 70-80 of the resulting operas are actually decent pieces of music that one would want to listen to, then great.

Self-driving cars? That's not really any more intelligent (but a lot sexier!) than to walk around in a crowd without bumping into people and not getting run over by a bus. In my view it's much more a sign of intelligence if you can translate literature into a foreign language and the people reading it actually end up enjoying it (instead of wondering who translated that garbage).

One aspect we need to be aware of is anthropomorphising. Weizenbaum's ELIZA was taken for more than it was, because its users tried to make sense of the conversations they had and built up a mental model of Eliza, which clearly wasn't there on the other side of the screen. I would want to see some real evidence of intentionality of what an AGI was doing, rather than ascribing intelligence to it because it acts in a way that I'm able to interpret.


How can I implement policy evaluation when reward is tied to an action outcome? I'm following Stanford reinforcement learning videos on youtube. One of the assignments asks to write code for policy evaluation for Gym's FrozenLake-v0 environment.

In the course (and books I have seen), they define policy evaluation as

$$V^\pi_k(s)=r(s,\pi(s))+\gamma\sum_{s'}p(s'|s,\pi(s))V^\pi_{k-1}(s')$$

My confusion is that in the frozen lake example, the reward is tied to the result of the action. So, for each pair state-action, I have a list that contains a possible next-state, the probability to get to that next-state and the reward. For example, being in the target state and performing any action brings a reward of $0$, but being in any state that brings me to the target state gives me a reward of $1$.

Does this mean that, for this example, I need to rewrite $V^\pi_k(s)$ as something like this:

$$V^\pi_k(s)= \sum_{s'} p(s'|s,\pi(s)) [r(s,\pi(s), s')+ \gamma V^\pi_{k-1}(s')]$$

The renowned book Reinforcement Learning: An Introduction (2nd edition), by Sutton and Barto, provides a different update rule than your first update rule for policy evaluation. Their update rule is more similar to your second update rule. See section 4.1. They also provide the pseudocode for policy evaluation on page 75 of the book. You can also find the pseudocode here.

Moreover, note that the update rule doesn't need to change only because the rewards are tied to the outcome of an action. This information is encoded in the functions $p$ (the transition function) and $r$ (the reward function) of the Markov decision process, which is incorporated in the update rule. If you want to understand the update rule, you should read the relevant pages (especially, chapter 4) of the cited book.


Why the non-exploitation of edge labels in current graph convolutions "results in an overly homogeneous view of local graph neighborhoods"? I am currently reading a paper called Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, CPPR), and I cannot understand the following sentence:

We identify that the current formulations of graph convolution do not
exploit edge labels, which results in an overly homogeneous view of
local graph neighborhoods, with an effect similar to enforcing
rotational invariance of filters in regular convolutions on images.

What does this sentence mean?

Consider a two-dimensional convolution layer with 3x3 kernels. The 2d inputs of this layer can be seen as a particular graph with each pixel being a graph node, that is connected to 8 of his neighbors:

The 3x3 kernels of the convolutional layer not only process the information about neighborhood relation between pixels, but also about their relative orientation. For example the [0,0] element of the kernel might represent the weight of the node to NW. And the [1,2] element of the kernel represent the weight the node across the S edge:

Now, if we make a convolution that "doesn't exploit edge labels" then we'll have to forget the labels on the picture above, making us loose directional information:

All we can say now is that the "red" node has those neighbors, but we don't really know how they are oriented relative to it. Since now the sub-graph does not provide any directional information, the learned convolution kernels will be direction-agnostic - in other words, they will be rotation-invariant.


Can we give a command to an AI and wait for it to do the job without explicitly telling it how to do it? I am a computer science student. I learned about programming languages recently, but I don't know much about artificial intelligence.
I want to know, why don't we program something in a way that we could tell the program

Hey! Do this for me!

And then just sit down and wait that the AI does the job?
Is this currently possible to do?

Normally when you write a program, you are acting like a boss that micromanages the job, telling the workers how to accomplish a task, perhaps without even letting them know what the purpose is. What you are hoping to be is a boss that gives the workers a goal and allows them to determine how to accomplish it.
In many ways, that is one of the aims of AI.
We already have small examples, such as a smart washing machine that weighs the clothes, monitors the amount of dirt in the water, and continually decides how much water to add or drain, when to agitate, when to rinse, and when to spin.  All you had to tell it was "clean these", and perhaps say what kind of material they are made of.
As a much larger example, automobiles were traditionally operated by turning the steering wheel to cause it to change direction and by pressing the pedals to increase or decrease the speed, but now we are working on car controllers that can respond to "go to Cleveland" by determining the best speed and direction by itself.
But note that those two examples (the first requiring only a little "intellegence", the second a lot) were for very specific requests that could be expressed in a few words.
As soon as the request becomes even slightly more complicated, the difficulty of creating a solution becomes much much more difficult.
Ask your "Do this for me!" request of a human being.
The first thing they'll do is ask "what does this mean?".
And then you'll have to give a lot of details.
And then they'll have questions about what you really want.
And so on.
Providing the requirements for "this" will be neither simple nor easy.
Human intellegence is still vastly superior to current AI.
In particular, it is capable of not only recognizing that additional information is required, but of having the intuitive ability to know what that missing information must be like.


Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process? If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).
Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?
From my current train of thought, the only thing that is different is that the policy is not changing (the agent is not learning the policy). Everything else is the same.
How is it that it switches to an MRP, which is not affected by actions?
I am reading "Deep Reinforcement Learning Hands-On" by Maxim Lapan, which states this. I have also found this statement in online articles, but I cannot seem to wrap my head around it.

If a policy is fixed, it is said that an MDP becomes an MRP.

I would change the phrasing slightly here, to:

If a policy is fixed, an MDP can be accurately modeled as an MRP.

  Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?

The transition and reward matrices remain the same in the MDP, but it is possible to flatten them into an equivalent MRP, because in terms of observations of next state and reward, the action that is taken is just part of the transition rules - if the policy is fixed, then so are all the probabilities for next state and reward.

More concretely, if you have an MDP with $|\mathcal{A}|$ transition matrices $P_{ss'}^a$ and a fixed policy $\pi(a|s)$, then you can create a combined transition matrix with a sum:

$$P_{ss'} = \sum_{a \in \mathcal{A}} \pi(a|s) P_{ss'}^a$$

and you can similarly reduce the reward function. Once you have done so you have data that describe an MRP.

  How is it that it switches to an MRP which is not affected by actions?

If the MDP represents a real system where actions are still being taken by an agent, then of course those are still present within the system, and still affect it. The difference is that if you know the agent's policy, then the action choice is predictable, and the MRP representation covers the full definition of probabilities of observed state transitions and rewards.


How does t-SNE preserves embedding orders? According to the triplet loss Wikipedia page:

t-SNE (t-distributed Stochastic Neighbor Embedding) preserves embedding orders via probability distributions, whereas triplet loss works directly on embedded distances.

I don't understand how does t-SNE preserves embedding order from the description given by its Wikipedia page.
I am trying to understand this claim in order to translate the page into other languages. I don't have a very quick understanding of maths, don't be afraid to explain it as if I was a teenager.

You just to look at the method of T-SNE. It "computes probabilities $p_{ij}$ that are proportional to the similarity of objects ${x} _{i}$ and ${x} _{j}$". This means the same as the sentence that you have already quoted:

t-SNE (t-distributed Stochastic Neighbor Embedding) preserves embedding orders via probability distributions, whereas triplet loss works directly on embedded distances.


What are the domains where SVMs are still state-of-the-art? It seems that deep neural networks and other neural network based models are dominating many current areas like computer vision, object classification, reinforcement learning, etc.

Are there domains where SVMs (or other models) are still producing state-of-the-art results?

State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask when you might prefer to try an SVM.

SVMs have several advantages:

Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear combinations of features, rather than the original feature set. In contrast, a more modern approach like a deep neural network will need to get deeper or wider to model the same patterns, which will increase its training time.
SVMs have an inherent bias towards picking "conservative" hypotheses, that are less likely to overfit the data, because they try to find maximum margin hypotheses. In some sense, they "bake-in" Occam's razor.
SVMs have only two hyperparameters (the choice of kernel and the regularization constant), so they are very easy to tune to specific problems. It is usually sufficient to tune them by performing a simple grid-search through the parameter space, which can be done automatically.

SVMs also have some disadvantages:

SVMs have a runtime that scales cubically in the number of datapoints you want to train on (i.e. $O(n^3)$ runtime)1. This does not compare well with, say, a typical training approach for a deep neural network which runs in $O(w*n*e)$ time, where $n$ is the number of data points, $e$ is the number of training epochs, and $w$ is the number of weights in the network. Generally $w, e << n$.
To make use of the Kernel trick, SVMs cache a value for the kernelized "distance" between any two pairs of points. This means they need $O(n^2)$ memory. This is far, far, more trouble than the cubic runtime on most real-world sets. More than a few thousand datapoints will leave most modern servers thrashing, which increases effective runtime by several orders of magnitude. Together with point 1, this means SVMs will tend to become unworkably slow for sets beyond maybe 5,000-10,000 datapoints, at the upper limit.

All of these factors point to SVMs being relevant for exactly one use case: small datasets where the target pattern is thought, apriori, to be some regular, but highly non-linear, function of a large number of features. This use case actually arises fairly often. A recent example application where I found SVMs to be a natural approach was building predictive models for a target function that was known to be the result of interactions between pairs of features (specifically, communications between pairs of agents). An SVM with a quadratic kernel could therefore efficiently learn conservative, reasonable, guesses.

1 There are approximate algorithms that will solve the SVM faster than this, as noted in the other answers.


Why isn't my implementation of A2C for the the atari pong game converging? I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.

https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing

The above code is from the following Github repository: https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py It converged perfectly well!

You can find an explanation in Maxim Lapan's book Deep Reinforcement Learning Hands-on page 269

Here is the mean reward curve :

https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing

The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.

Here is the mean reward curve :

Can someone tell me the problem portion of the code and how can I fix it?

UPDATE 1

I have decided to test my code with a simpler environment, i.e. Cartpole-v0.

Here is the code : https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing

Even that code doesn't seem to converge. Still can't see where is my problem.

UPDATE 2

I think the bug might be in the ExperienceSource class or in the Agent class.

UPDATE 3

The following question will help you understand the classes ExperienceSource and ExperienceSourceFirstLast.

Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4

In [3]: histories
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)

In [5]: histories
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]

So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]

In [8]: histories
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)

In [10]: histories
Out[10]: [deque([1]), deque([]), deque([]), deque([])]

The curve representing the mean reward looks like


Can Q-learning be used for continuous (state or action) spaces? Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning for continuous spaces like the control of a pendulum?

Q-learning for continuous state spaces

Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the space to a small number of states. Provided you can sample and update enough times, then a few million states is not a major problem.

However, with large state spaces it is more common to use some form of function approximation for the action value. This is often noted $\hat{q}(s,a,\theta)$ to show that it is both an estimate (the circumflex over $\hat{q}$) and that you are learning some function parameters ($\theta$). There are broadly two popular approaches to Q-learning using function approximation:

Linear function approximation over a processed version of the state into features. A lot of variations to generate features have been proposed and tested, including Fourier series, tile coding, radial basic functions. The advantage of these methods are that they are simple, and more robust than non-linear function approximations. Which one to choose depends on what you state space represents and how the value function is likely to vary depending on location within the state space.
Neural network function approximation. This is essentially what Deep Q Networks (DQN) are. Provided you have a Markov state description, you scale it to work sensibly with neural networks, and you follow other DQN best practices (experience replay table, slow changing target network) this can work well.

Q-learning for continuous action spaces

Unless you discretise the action space, then this becomes very unwieldy.

The problem is that, given $s,a,r,s'$, Q-learning needs to evaluate the TD target:

$$Q_{target}(s,a) = r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$$

The process for evaluating the maximum becomes less efficient and less accurate the larger the space that it needs to check.

For somewhat large action spaces, using double Q-learning can help (with two estimates of Q, one to pick the target action, the other to estimate its value, which you alternate between on different steps) - this helps avoid maximisation bias where picking an action because it has the highest value and then using that highest value in calculations leads to over-estimating value.

For very large or continuous action spaces, it is not usually practical to check all values. The alternative to Q-learning in this case is to use a policy gradient method such as Actor-Critic which can cope with very large or continuous action spaces, and does not rely on maximising over all possible actions in order to enact or evaluate a policy.

Controlling a pendulum

For a discrete action space e.g. applying one of a choice of forces on each time step, then this can be done using a DQN approach or any other function approximation. The classic example here might be an environment like Open AI's CartPole-v1 where the state space is continuous, but there are only two possible actions. This can be solved easily using DQN, it is something of a beginner's problem.

Adding continuous action space ends up with something like the Pendulum-v0 environment. This can be solved to some degree using DQN and discretising the action space (to e.g. 9 different actions). However, it is possible to make more optimal solutions using an Actor-Critic algorithm like A3C.


How to let an RL Agent move the mouse? Gday guys,

I'am building a game enviroment (picture) where an agent should position the mouse on the screen (cords upper right corner) and then click to shoot a canonball. If the goal (left) is hit. The agent gets a reward based on the elapsed time between this strike and the last one. If three shots missed. The game is done and the environment will reset.

The env is done so far. But now I wonder what the action space should look like. How can I make the agent choose some x and y coordinates? And how can I combine this whith a "shoot" action?

Thanks in advance for any help

You could make the actions

move up
move down
move left
move right
shoot

Then you can declare the speed of this movement. If you want to go more in depth you can do

Accelerate up
Accelerate down
Accelerate left
Accelerate right
Shoot

This will give control of the speed to the neural networkm but is harder to train, and you should give the speed of the mouse as an input.


What should I do with the flatten layer during back-propagation? I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on.
During the forward pass, the Flatten layer reshapes the previous layer's activation.  I know there are a lot of questions about it, but what should I do with the Flatten layer during back-propagation? Should I compute the derivative of $dA$ and reshape it for the next layer or just reshape $dA$ of the previous layer?

The Flatten layer has no learnable parameters in itself (the operation it performs is fully defined by construction); still, it has to propagate the gradient to the previous layers.
In general, the Flatten operation is well-posed, as whatever is the input shape you know what the output shape is.
When you backpropagate, you are supposed to do an "Unflatten", which maps a flattened tensor into a tensor of a given shape, and you know what that specific shape is from the forward pass, so it is also a well-posed operation.
More formally
Say you have Img1 in input of your Flatten layer
$$
\begin{pmatrix}
f_{1,1}(x; w_{1,1}) & f_{1,2}(x; w_{1,2}) \\
f_{2,1}(x; w_{2,1}) & f_{2,2}(x; w_{2,2})
\end{pmatrix}
$$
So, in the output you have
$$
\begin{pmatrix}
f_{1,1}(x; w_{1,1}) & f_{1,2}(x; w_{1,2}) & f_{2,1}(x; w_{2,1}) & f_{2,2}(x; w_{2,2})
\end{pmatrix}
$$
When you compute the gradient you have
$$
\frac{df_{i,j}(x; w_{i,j})}{dw_{i,j}}
$$
and everything in the same position as in the forward pass, so the unflatten maps from the (1, 4) tensor to the (2, 2) tensor.


What is self-supervised learning in machine learning? What is self-supervised learning in machine learning? How is it different from supervised learning?

Introduction
The term self-supervised learning (SSL) has been used (sometimes differently) in different contexts and fields, such as representation learning [1], neural networks, robotics [2], natural language processing, and reinforcement learning. In all cases, the basic idea is to automatically generate some kind of supervisory signal to solve some task (typically, to learn representations of the data or to automatically label a dataset).
I will describe what SSL means more specifically in three contexts: representation learning, neural networks and robotics.
Representation learning
The term self-supervised learning has been widely used to refer to techniques that do not use human-annotated datasets to learn (visual) representations of the data (i.e. representation learning).
Example
In [1], two patches are randomly selected and cropped from an unlabelled image and the goal is to predict the relative position of the two patches. Of course, we have the relative position of the two patches once you have chosen them (i.e. we can keep track of their centers), so, in this case, this is the automatically generated supervisory signal. The idea is that, to solve this task (known as a pretext or auxiliary task in the literature [3, 4, 5, 6]), the neural network needs to learn features in the images. These learned representations can then be used to solve the so-called downstream tasks, i.e. the tasks you are interested in (e.g. object detection or semantic segmentation).
So, you first learn representations of the data (by SSL pre-training), then you can transfer these learned representations to solve a task that you actually want to solve, and you can do this by fine-tuning the neural network that contains the learned representations on a labeled (but smaller dataset), i.e. you can use SSL for transfer learning.
This example is similar to the example given in this other answer.
Neural networks
Some neural networks, for example, autoencoders (AE) [7] are sometimes called self-supervised learning tools. In fact, you can train AEs without images that have been manually labeled by a human. More concretely, consider a de-noising AE, whose goal is to reconstruct the original image when given a noisy version of it. During training, you actually have the original image, given that you have a dataset of uncorrupted images and you just corrupt these images with some noise, so you can calculate some kind of distance between the original image and the noisy one, where the original image is the supervisory signal. In this sense, AEs are self-supervised learning tools, but it's more common to say that AEs are unsupervised learning tools, so SSL has also been used to refer to unsupervised learning techniques.
Robotics
In [2], the training data is automatically but approximately labeled by finding and exploiting the relations or correlations between inputs coming from different sensor modalities (and this technique is called SSL by the authors). So, as opposed to representation learning or auto-encoders, in this case, an actual labeled dataset is produced automatically.
Example
Consider a robot that is equipped with a proximity sensor (which is a short-range sensor capable of detecting objects in front of the robot at short distances) and a camera (which is long-range sensor, but which does not provide a direct way of detecting objects). You can also assume that this robot is capable of performing odometry. An example of such a robot is Mighty Thymio.
Consider now the task of detecting objects in front of the robot at longer ranges than the range the proximity sensor allows. In general, we could train a CNN to achieve that. However, to train such CNN, in supervised learning, we would first need a labelled dataset, which contains labelled images (or videos), where the labels could e.g. be "object in the image" or "no object in the image". In supervised learning, this dataset would need to be manually labelled by a human, which clearly would require a lot of work.
To overcome this issue, we can use a self-supervised learning approach. In this example, the basic idea is to associate the output of the proximity sensors at a time step $t' > t$ with the output of the camera at time step $t$ (a smaller time step than $t'$).
More specifically, suppose that the robot is initially at coordinates $(x, y)$ (on the plane), at time step $t$. At this point, we still do not have enough info to label the output of the camera (at the same time step $t$). Suppose now that, at time $t'$, the robot is at position $(x', y')$. At time step $t'$, the output of the proximity sensor will e.g. be "object in front of the robot" or "no object in front of the robot". Without loss of generality, suppose that the output of the proximity sensor at $t' > t$ is "no object in front of the robot", then the label associated with the output of the camera (an image frame) at time $t$ will be "no object in front of the robot".


Why does the machine learning algorithm need to learn a set of functions in the case of missing data? I am currently studying the textbook Deep Learning by Goodfellow, Bengio, and Courville. Chapter 5.1 Learning Algorithms says the following:

  Classification with missing inputs: Classification becomes more challenging if the computer program is not guaranteed that every measurement in its input vector will always be provided. To solve the classification task, the learning algorithm only has to define a single function mapping from a vector input to a categorical output. When some of the inputs may be missing, rather than providing a single classification function, the learning algorithm must learn a set of functions. Each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing. This kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. One way to efficiently define such a large set of functions is to learn a probability distribution over all the relevant variables, then solve the classification task by marginalizing out the missing variables. With $n$ input variables, we can now obtain all $2^n$ different classification functions needed for each possible set of missing inputs, but the computer program needs to learn only a single function describing the joint probability distribution. See Goodfellow et al. (2013b) for an example of a deep probabilistic model applied to such a task in this way. Many of the other tasks described in this section can also be generalized to work with missing inputs; classification with missing inputs is just one example of what machine learning can do.

I was wondering if people would please help me better understand this explanation. Why is it that, when some of the inputs are missing, rather than providing a single classification function, the learning algorithm must learn a set of functions? And what is meant by "each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing."?

I would greatly appreciate it if people would please take the time to clarify this.

@The Pointer the $2^n$ came from the question: How many function do we need to have if each of the $n$ inputs can be missing?
example: $f_1(\text{missing}, x_2, x_3, \dots, x_n)$ for $x_1$ missing
$f_2(x_1, x_2, \text{missing}, x_4, \text{missing}, \dots, x_n)$ for $x_3$ and $x_5$ missing.
So this problem is a combinatorial one and the event for each $x_i$ is Missing or Not.
Each function corresponds 1-by-1 with a possible set $(x_1, \text{missing}, x_2, \dots, x_n)$. So how many sets can you form $2^n$. Why?
This formula comes back to like tree developing. First variable $x_1$ missing or not (2 possbile events). Now after that, for EACH of these events 2 possbile events for $x_2$ so in total 2 \times 2 (2 for $x_1$ , 2 for $x_2$ for each $x_1$'s event) and etc., $2 \times 2 \times 2 \times \dots$ $n$ times = $2^n$.


Can artificial intelligence applications be hacked? Can artificial intelligence (or machine learning) applications or agents be hacked, given that they are software applications, or are all AI applications secure?

To answer your question, it really depends on the purpose of the Artificial Intelligence program.

For example, 4Chan has hacked a number of "Artificial Intelligent" bots, most notably was Microsoft's Twitter bot Tay. The general purpose of the bot was to parse what was tweeted at it and respond in kind, learning and evolving with each and every interaction.

Within 24 hours, 4Chan had corrupted Tay beyond repair, by teaching it racist and sexist terminology, ironic memes, sending it to shitpost tweets, and otherwise attempting to alter its output so much so that Microsoft had to remove it.

Now, the flaw with Tay was that it accepted any input, and learned off of that exclusively, without any interaction from the developers. Other bots have similar features, but they have checks in place that require human intervention to determine what is "quality" information to learn, and what is "bad" information to learn as to not pollute the global knowledge base of the bot.

These are just two examples of how Artificial Intelligence can be "hacked", but it ultimately comes down to how the programs are implemented.

You mention in one of your comments about Cellphone Artificial Intelligence such as Siri, and whether this technology can be hacked. The answer is - not really.

Siri learns based off of her global interactions - with limited user input allowed. You can ask Siri how to pronounce a name. When she pronounces it incorrectly, you can say to her "Siri that's now how you pronounce that". And she will provide you with a limited set of options of how you pronounce that name, and you have to choose which option sounds the best.

There was no option to allow a user to give Siri "bad" information, as she already populates the results for you, and you have to teach her based off of her list of options. To give Siri bad input, you would have to have access to Siri's global learning base, which we do not have access to, and alter how she accepts human interactions within the program - which would never happen due to too many moving parts within the iPhone update process, and would be caught before you were able to deploy your update.


If two objects are too close to each other, would an object detector do a poor job of correctly classifying them? Suppose we have an object detector that is trained to detect $20$ products. If two objects are too close to each other, in general, would an object detector do a poor job of correctly classifying them? If they were far apart in the scene, would the object detector to a better job of correctly classifying them?

As you ask, "in general...", I will answer generally, however this changes a lot from model to model and the way they handle close objects.
In general, yes, they would do a poor job detecting very close objects, switch to segmentation models for that (for class or better, instance segmentation).
In general, objects detectors learn to tell an object from other based in 2 criterion:

Intersection over union: for object of the same class
Class probability: for objects of different class

So, if two objects of the same class are very close, the 2 detected bounding boxes will be highly overlapping, then, the Non Maximal Suppression filter will remove one of them. This is where objects detector, in general, perform worse.
Similarly, if two objects belong to different classes the 2 detected bounding boxes will be highly overlapping but the NMS filter won't remove them (again, in general, NMS is set only for same class objects). However when 2 objects are very close, there is a high chance they are partially occluded. Objects detectors, in general, don't handle occlusions very well.
So, in conclusion, objects detectors will perform better detecting far-away objects.


Should I use leave-one-out cross-validation for testing? I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
Here's what I was thinking:

train/test split the data, with only one data point in the test set.

train the model on training data, potentially with grid-search/cross-validation

use the best model from step 2 to make a prediction on the one data point and save the prediction in an array

repeat the previous steps until all the data points have been in the test set

calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions

The pros of this approach would be to:

You don't have to split the data into train/test so you can train
with more data points.

The cons would be:

This approach suffers from potential(?) data leakage.

You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.

I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.
I would really appreciate some feedback about whether this approach is actually feasible or not and why.

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves.

The important aspect when choosing $k$ is a bias-variance tradeoff. Note that in LOOCV you train each model using almost as many samples as there are available ($n-1$), so the validation step should give you an unbiased estimate of the real test error. However, each model in LOOCV is trained each time on almost exactly same dataset!. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, LOOCV will suffer from huge variance.

On the other hand, in 2-fold CV the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).

What to do in this scenario? Choose something in the middle. Usually $k=5$ and $k=10$ should be a good choice.


Does Q Learning learn from an opponent playing random moves? I've created a Q Learning algorithm to play Connect Four against an opponent who just chooses a random free column. My Q Agent is currently only winning about 0.49 games on average (30,000 episodes). Will my Q Agent actually learn from these episodes, seeing as its opponent isn't 'trying' to beat it, as there's no strategy behind its random choices? Or should this not matter – if the Q Agent is playing enough games, it doesn't matter how good/bad its opponent is?

It should be possible to train an agent using some variant of DQN to beat a random agent around 100% of the time within a few thousand games.

It may require one or two more advanced techniques to get the learning time down to a low number of thousands. However, if your agent is winning ~50% of games against a random agent, something has gone wrong, since that is the performance you would expect of another random agent. Even simple policies, such as always play in same column, will beat a random agent a significant fraction of the time.

First thing to consider is that there are too many states in Connect 4 to use tabular Q learning. You have to use some variant of DQN. As a grid-based board game where winning patterns can repeat, some form of convolutional neural network (CNN) for the Q function is probably a good start.

I think for a first step, you should double-check that you have implemented DQN correctly. Check the TD target formula is correct, and that you have implemented experience replay. Ideally you will also have a delayed-update target network for calculating the TD targets.

As a second step, try some variations of hyper-parameters. The learning rate, exploration rate, size of replay table, number of games to play before starting learning etc. A discount factor $\gamma$ slightly below 1 can help (despite this being an episodic problem) - it makes the agent forget more of the initial bias for early time steps.

  Or should this not matter – if the Q Agent is playing enough games, it doesn't matter how good/bad its opponent is?

Up to a point this is true. It is hard to learn against a perfect agent in Connect 4, because it always wins as player one, which means all policies are equally good and there is nothing to learn. Other than that, if there is a way to win, eventually a Q learning agent with exploration should find it.

Against a random agent, you should be seeing some improvement if your agent is correctly set up for the problem, after a few thousand games. As it happens I am currently training Connect 4 agents using variants of DQN for a Kaggle competition, and they consistently beat random agents with 100% measured success rate after 10,000 training games. I have added a few extras to my agents in order to achieve this - there are some discussions of approaches in the forums at https://www.kaggle.com/c/connectx


How can I generate What, why, who types of questions from "Gap-fill-in" type of questions? I was wondering if in any way it is possible to generate W questions based on gap-fill-in type questions (e.g "______ is a process in which plants generate energy."   --->   "What is the process in which plants generate energy called?")

If so, how can I achieve this? I am familiar with working with natural language processing and have no problem with implementing an algorithm for this but I do not know where to start with this.

Any help would be appreciated!

This seems tricky. It seems that any "surface level" transformation wouldn't give adequate results and any working solution would need to properly capture the sentence structure, and generate a gramatically correct transformed sentence.

One possible option is to use a "traditional pipeline" - e.g. you run a NLP pipeline up to syntactic parsing, which for general domain english is quite accurate (you'd need some special handling for the gap "____" part though), then implement some heuristic rules to transform the syntax tree, and regenerate a sentence from the transformed tree. There are a lot of publications about similar transformations in machine translation domain, used as a way to preprocess data before running statistical machine translation for language pairs with very different word(or sentence part) ordering.

A second option that may work is to look into the field of controlled natural languages, or something like http://www.grammaticalframework.org/ that can be used as a toolkit to help generating new sentences.

Current fashion also suggests a very different option that might work - you could train a character-level recurrent neural network with an attention mechanism (look into recent neural machine translation publications for details) to do this transformation, but I'm not sure of how much training data it will need for decent accuracy.


Why do we need weights when training a perceptron as an OR gate? Without using any of Matlab's neural network tools, I'm writing a program to simulate an OR gate with a perceptron. I have seen many tutorials, but I still can't understand why we need weights to train a perceptron for such a simple purpose.

One way is to program the perceptron with the conditions (0,0)=0. (1,0)=1. (0,1)=1. (1,1)=1. So the two inputs to the perceptron would be either zeroes or ones. I don't see the purpose of weights here. Assuming weights are 1, for the second training example, the output would be 1*1 + 0*1 = 1. For the last example, it would be 1*1 + 1*1 = 2. So an activation function which says if output >= 1, output = 1 else output =0; end should suffice. This would successfully simulate an OR gate. So why do I need to "train" any weights?

This would successfully simulate an OR gate.

Of course. In fact, hardware implementation of an OR gate needs just a few transistors. It may sound surprising, but the best python implementation of OR is a or b.

  So why do I need to "train" any weights?

It's one of the simplest functions out there, everyone understands how OR works just from the boolean table. The task shows that a neural network can learn it as well, just from examples and without specification what a logical operation is and how to wire it.


Checking Whether Given Logistic Regression Classifier classifies data A bank wants to decide whether a customer can be given a loan,
  based on two features related to (i) the monthly salary of the customer, and (ii) his/her
  account balance. For simplicity, we model the two features with two binary variables $X1$, $X2$
  and the class $Y$ (all of which can be either 0 or 1). $Y=1$ indicates that the customer can be
  given loan, and Y=0 indicates otherwise.
  Consider the following dataset having four instances:
  ($X1 = 0$, $X2 = 0$, $Y = 0$)
  ($X1 = 0$, $X2 = 1$, $Y = 0$)
  ($X1 = 1$, $X2 = 0$, $Y = 0$)
  ($X1 = 1$, $X2 = 1$, $Y = 1$)
  Can there be any logistic regression classifier using X1 and X2 as features, that can perfectly classify the given data?

The approach followed in the question was to calculate respective probabilities for Y=0 and Y=1 respectively.  The value of $p$ obtained was $0.25$ and $(1-p)$ as $0.75$.The $log(p/1-p)$ is coming as negative. But I am stuck at this place as to which parameter to be calculated which decides that logistic regression would classify/not classify the data ?


Evaluate model multiple times in loss function? Is this reinforcement learning? I am interested in models that exhibit behavior. My goal is a model that survives indefinitely on a two dimensional resource landscape. One dimension represents the location (0 to 1) and the second says if there is a resource available at that location (-1 = resource one, 0 = no resource, 1 = resource two).
The landcape looks like this:
location = [0, 0.2, 0.4, 0.6, 0.8, 1]
resource = [-1, 0,   0,   0,   0,  1] (I added spaces so the elements line up)

My model represents an organism deciding if it will move or rest on the landscape at each time step. The organism has reserves of each resource. The organism fills its reserve of a resource if it rests on the resource and loses 1 unit of both resources at each time step. I am considering neural networks to represent my organisms. The input would be 4 values; The location on the landscape, the resource value at that location, and the reserve levels of resource one and two. The output would be 3 values; move right, rest, move left. The highest value decides what happens. To survive indefinitely the model will have to bounce between the ends of the landscape, briefly resting on the resource. Model evaluation would go like this: start the model in the middle of the landscape with full resource reserves. Allow time to pass until one of the resource reserves is depleted (the organism dies).
My question is this: Can my loss function be evaluating the model until it dies? 1/survival time could be the loss value to be minimized by gradient descent. Is this a reinforcement learning problem (I don't think so..?) Thanks!!

Can my loss function be evaluating the model until it dies? 1/survival time could be the loss value to be minimized by gradient descent.

In order to use backpropagation and gradient descent, you have to relate the loss function directly to the output of the neural network. Your proposed loss function is too indirect, it is not possible to turn it directly into a gradient that could be used to alter the neural network weights.
In addition, the specific function of time you have chosen will be difficult to optimise, as incrememental improvements from e.g. 10 to 11 time steps surviving will provide a much lower signal for adjusting behaviour than the improvement from  e.g. 2 to 3 lifetime. If the environment has enough randomness (and typically these kind of a-life scenarios do), then the signal here could be swamped by random events and very hard to optimise, requiring a larger number of samples in order to extract expected improvements to the loss function.

Is this a reinforcement learning problem (I don't think so..?)

It is very close to a definition of reinforcement learning (RL) problem. For a RL problem you need the following things:

An environment in which an agent exists, and which has a measurable state.

A set of actions = sequential decisions that need to be made, and that have consequences.

The consequences of any action are:

A consistent (but allowing for some randomness) change to the state of the environment. Here you have change to agent's location and resource levels.
A consistent scalar reward value that measures how well the agent is achieving a goal. This can also be partially random, and can also be sparse, only achieved in certain specific states.

In your problem definition you don't have a reward signal, but it would be easy to add one. A suitable one would be $+1$ per time step.
Technically your problem would also be partially observable (sometimes called a POMDP), in that the agent does not get to see resources available in other locations. It only knows its current location, its internal state and resources available at its current location. This is not a major issue, although you should note that adding some kind of memory (either open-ended memory as in a recurrent neural network, or explicitly added to the state) would allow for more efficient agents. That's not a RL issue as such, any learning process without ability to form or use memories would be limited in this environment, and you might want to look into that as a later experiment.
How RL helps is that it provides a framework to convert your problem definition into measurements and gradients for the neural network to improve its performance.
As you have set up your neural network to predict best action choice, this would naturally lend itself to policy gradient methods in RL, such as REINFORCE. As it is a simple problem, I would expect REINFORCE with baseline to perform well enough for it.
I will not explain the algorithm here in full detail, but the basic approach is to have the agent act with the current network for a few episodes, collecting data on its choices and performance. You will get a dataset of (state, action, return = sum of rewards to end of episode). You then use that as a labelled dataset to train a minibatch as if the action choice was correct ground truth, but multiply each gradient by (return - baseline) where the baseline is typically the average return seen so far from that state. You may need a second neural network during training to estimate that expected return (aka state value). After using a minibatch once, you will need to discard it, as it represents results for the previous iteration of the network before the weight updates. There are ways around this, but not typically done in REINFORCE - instead the approach is to just keep generating data and train on the new mini-batch as fast as you can go.
RL offers other methods which may work just as well to solve your problem, but I suspect REINFORCE will serve you well since it will allow the agent to randomise direction choice which is important when it cannot see where the resources are and has no memory of where it has searched.
You don't have to use RL for this problem. An alternative that may work for you is using genetic algorithms to tune the network architecture and weights. It avoids using gradients, but I would still recommend a simple fitness function equal to number of time steps survived. There is a framework called NEAT which is ideal for this sort of a-life control problem.


How do I locate a specific object in an image? Some pictures contain an elephant, others don't. I know which of the pictures contain the elephant, but I don't know where it is or how does it look like.

How do I make a neural network which locates the elephant on a picture if it contains one? There are no pictures with more than one elephant.

so assuming your not allowed to use transfer methodologies (like take an already exisiting elephant object detector) my recommendation is to train a CNN classifier (labels are binary-- elephant exist, elephant doesnt exist) and then use strategies founded in like grad cam. Note there does exist a gradcam++ but because you can assure theres only one instance, it isnt necessary and is just more complicated.

Note that since you just need the location and not the pixel specificity, you dont even need to do the guided backprop, but just the relation with respect to the last convoluitional map.

A quick description is that its using the gradient of the class loss w.r.t the last feature map to see which locations helped make the classification, and from there you can upscale to the receptive field that those neurons touch

Hope this helped!


How does best-first search differ from hill-climbing? How does best-first search differ from hill-climbing?

Best-first search
BFS is a search approach and not just a single algorithm, so there are many best-first (BFS) algorithms, such as greedy BFS, A* and B*. BFS algorithms are informed search algorithms, as opposed to uninformed search algorithms (such as breadth-first search, depth-first search, etc.), i.e. BFS algorithms make use of domain knowledge that can be encoded into a so-called heuristic function (that's why they are informed!).
Every BFS algorithm defines a so-called evaluation function $f$ of the following form
$$f(n) = g(n) + h(n)$$
for all nodes $n \in V$ (where $V$ is the set of nodes or states of the search space), where

$g(n)$ is the cost from the start node (of the search) to the node $n$, and

$h(n)$ (the so-called heuristic, which is the way of including domain knowledge to solve the search problem) is an estimate of the cost of the cheapest path from $n$ to the goal node.

Depending on how you define $f$ and, in particular, the heuristic function $h$, you get different BFS algorithms. For instance, A* is a BFS algorithm where $h$ is an admissible heuristic (i.e. it never overestimates the cost to the goal node). Because of this admissibility property, A* is guaranteed to find the globally optimal solution (i.e. the cheapest path from a start node to the goal node among all paths), but you can ignore this detail.
To apply a BFS algorithm, you need to define the evaluation function and the search space, i.e. the states (or nodes) and their connections. For example, if you want to find the cheapest path from Paris to Madrid, you need to define that Paris is the start node, Madrid the goal node and then you need to define all intermediate nodes between Paris and Madrid, but you also need to define $g$ and $h$.
Hill climbing
Hill climbing (HC) is a general search strategy (so it's also not just an algorithm!). HC algorithms are greedy local search algorithms, i.e. they typically only find local optima (as opposed to global optima) and they do that greedily (i.e. they do not look ahead). The idea behind HC algorithms is that of moving (or climbing) in the direction of increasing value. HC algorithms can be used to solve optimization problems and not just well-defined search problems, i.e. you start from some solution and you move to the best neighboring solution, and then loop.
Best-first search vs hill climbing

BFS algorithms are informed search algorithms (as opposed to uninformed)
BFS algorithms need to define the search space and the evaluation function
Some BFS algorithms (such as A*) are guaranteed to find the best global solution
HC algorithms are general (i.e. widely applicable) but local and greedy search and optimization algorithms; consequently, they are not generally guaranteed to find the global optimum (but, in practice, they may work well, depending also on the problem).
HC algorithms do not need to define the search space explicitly (i.e. you do not need to define the start and goal nodes, and so on), but you just need to a way of determining the best neighboring solution

Further reading
The book Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig provides more details about these two search approaches. (You can find free copies of this book online).


keras ValueError: Error when checking model target: expected activation_4 to have shape (None, 19) but got array with shape (100, 1) I'm trying to create simple keras NN which will learn to make addition on numbers between 0 and 10. But I am getting the error:

ValueError: Error when checking model target: expected activation_4 to have shape (None, 19) but got array with shape (100, 1)

here is my code:

from keras.models import Sequential
from keras.layers import Dense, Activation
import numpy as np

keras.optimizers.SGD(lr=0.01, momentum=0.0, decay=0.0, nesterov=False)

model = Sequential()
model.add(Dense(output_dim=50, input_dim=2))
model.add(Activation("relu"))
model.add(Dense(output_dim=50))
model.add(Activation("softmax"))
model.add(Dense(output_dim=50))
model.add(Activation("softmax"))
model.add(Dense(output_dim=19))
model.add(Activation("softmax"))

model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

x = []
y = []

for i in range(0, 10):
    for j in range(0, 10):
        x.append((i, j))
        y.append(i + j)

x = np.array(x)
y = np.array(y)
print(x)
print(y)

model.fit(x, y, nb_epoch=5, batch_size=32)

how to fix that?

Try to use the model like this, for example:

model = Sequential()
model.add(Dense(50, input_shape=(2,)))
model.add(Activation("relu"))
model.add(Dense(50, activation='softmax'))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='sgd', loss='mse', metrics=["accuracy"])

This means that first layer will have 50 neurons and can receive data in form of matrix with 2 columns and an unspecified number of rows.
So you can prepare your data in this form – 2 numbers for adding in each row.

Dense(50, input_shape=(2,))

At the end, you need a layer with 1 neuron and the 'linear' activation, because you expect one simple number as a result.

Dense(1, activation='linear')

And finally, use 'mse' loss function or something similar. 'categorical_crossentropy' is needed for classification tasks, not regression as needed for you.
See: https://keras.io/objectives/


Are there any reliable ways of modifying the reward function to make the rewards less sparse? If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like

\begin{align}
R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{everything} \\
R(\text{state})& = 0\ \ \text{if not terminal}
\end{align}
i.e. when it reaches the terminal state, it receives a reward but one that decreases if it is slower. Actually I'm not sure if this is better or worse than $R(\text{terminal}) = 1 / \text{time}$, so please correct me if I'm wrong.

However, if the maze is really big, it could spend a long time wandering around before even encountering that reward. Are there any reliable ways of modifying the reward function to make the rewards less sparse? Assume that the agent knows the Euclidean distance between itself and the exit, just not the topography of the maze.

Is it at all sound to simply do something like

\begin{align}
R(\text{current}) = (d_E(\text{start}, \text{exit}) - d_E(\text{current}, \text{exit})) + (\text{terminal}==True)*(N-\text{time})?
\end{align}

Or if not, what kind of dense heuristic reward or other techniques might be better?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to reinforce unwanted behaviour.

For example, the way I read that reward function you propose, it provide a positive reward for any steps taken by the agent, with larger rewards for steps that get you closer to the goal (except for steps moving back into the start, those would have a reward of $0$. There does not appear to be any "compensation" with negative rewards for moves that take you back away from the goal; in fact, such steps also still seem to carry positive rewards! This means that the optimal behaviour that your agent can end up learning is to keep moving around in circles (somewhat close to the goal, but never quite stepping into the goal) for an infinite amount of time, continuously racking up those positive rewards.

The idea of adding some extra (heuristic) rewards to speed up learning is referred to as "reward shaping". Naive approaches to reward shaping often end up unintentionally modifying the "true" objective, as highlighted above. The correct way to implement reward shaping, which provably does not modify the optimal policy, is Potential-Based Reward Shaping. The basic intuition behind this is that, if you use reward shaping to encourage "movement" in one "direction", you should also provide equivalent (taking into account discount factor $\gamma$) discouragement for subsequent "movement" in the other "direction".

Now, there is this really cool paper named "Expressing Arbitrary Reward Functions as Potential-Based Advice" which proposes a method that can automatically convert from additional reward shaping functions specified in the more "natural" or "intuitive" manner like you did, into (approximately) a potential-based one that is more likely to actually function correctly. This is not quite straightforward though, and the approach involves learning an additional value function which makes additional predictions used to implement the "conversion". So... in practice, in a simple grid-world like yours, I think it's going to be simpler to just figure out the correct potential-based definition yourself than trying to learn it like this, but it's cool stuff nevertheless.


Dealing with empty frames in MRI images I started working on the application of deep learning in medical imaging recently. While dealing with MRI images in the BraTS dataset, I observe that first and last few frames are always completely empty (black). I want to ask those who are already working in the field, is there a way to remove them in a procedural manner before training and add them correctly after the training as a postprocessing step (to comply with the ground truth segmentations' shape)? Has anyone tried that?
I could not find any results on Google. So asking here.

Edit: I think I did not make my point clear enough. I meant to say first and last few frames of each MRI scan are empty. How to deal with those is what I intended to ask.

I've worked on the BRATS dataset and I can verify that this is pretty much standard process. Besides throwing the totally blank images, I also throw away the images in the beginning and ending of the sequence that show the tip of the scull and the base of the neck.

Generally when dealing with MRIs,
I do this with a script (think of is as a preprocessing step) I run on each image that counts the amount of pixels that have a positive intensity (I actually add a small value to this to account for noise). Let's say for images with values 0-255 I count the amount of pixels with an intensity of over 10-15, let's call this $x$. After that I set a threshold (empirically), let's call it $t$ and discard images with $x < t$.

Specifically for BRATS, because you have the labels, you can see which of these have the desired classes and discard most of the rest. If you try to train a network on the dataset as is you face an enormous imbalance ratio. I've had trouble training networks due to this and the most success I got was when I threw away most of the irrelevant images.


Q table not converging for an arbitrary experiment This is an experiment in order to understand the working of Q table and Q learning.

I have the states as

states = [0,1,2,3]

I have an arbitrary value for each of these states as shown below (assume index-based mapping) -

arbitrary_values_for_states = [39.9,47.52,32.92,37.6]

I want to find the minimum of the state which will give me the minimum value.
So I have complimented the values to 50-arbitrary value.

inverse_values_for_states = [50-x for x in arbitrary_values_for_states]

Therefore, I defined reward function as -

def reward(s,a,s_dash):
    if inverse_values_for_states[s]<inverse_values_for_states[s_dash]:
        return 1
    elif inverse_values_for_states[s]>inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0

Q table is initialized as -
Q = np.zeros((4,4)) (np is numpy)

The learning is carried out as -

episodes = 5
steps = 10
for episode in range(episodes):
    s = np.random.randint(0,4)
    alpha0 = 0.05
    decay = 0.005
    gamma = 0.6
    for step in range(steps):
        a = np.random.randint(0,4)
        action.append(a)
        s_dash = a
        alpha = alpha0/(1+step*decay)
        Q[s][a] = (1-alpha)*Q[s][a]+alpha*(reward(s,a,s_dash)+gamma*np.max(Q[s_dash]))

        s = s_dash

The problem is, the table doesn't converge.

Example. For the above scenario -

np.argmax(Q[0]) gives 3
np.argmax(Q[1]) gives 2
np.argmax(Q[2]) gives 2
np.argmax(Q[3]) gives 2

All of the states should give argmax as 2 (which is actually the index[state] of the minimum value).

Another example,

when I increase steps to 1000 and episodes to 50,

np.argmax(Q[0]) gives 3
np.argmax(Q[1]) gives 0
np.argmax(Q[2]) gives 1
np.argmax(Q[3]) gives 2

More, steps and episodes should assure convergence, but this is not visible.

I need help where I am going wrong.

PS: This little experiment is needed to make Q-learning applicable to a larger combinatorial problem. Unless I understand this, I don't think I will be able to do that right.
Also, there is no terminal state because this is an optimization problem. (And I have heard that Q-learning doesn't necessarily needs a terminal state)

If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit.

The current reward structure provides the incentive to just move to a stage where it gets a reward.

For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a higher inverse value.

To make the agent learn to move to state 2, you would have to provide it with more incentives to go to state 2.

def reward(s,a,s_dash):
    if s_dash == 2:
        return 5
    elif inverse_values_for_states[s]<inverse_values_for_states[s_dash]:
        return 1
    elif inverse_values_for_states[s]>inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0

I tried using this and it converges to 2. This is a hard-coded version, but I guess you get the idea.


Is the following neural network architecture considered deep learning? I am working in the following neural network architecture, I am using keras and TensorFlow as a back-end.

It is composed by the following, embedding of words, then I added a layer of
Long Short-Term Memory (LSTM) neural networks, one layer of output and finally. I am using the softmax activation function.

model = Sequential()
model.add(Embedding(MAX_NB_WORDS, 64, dropout=0.2))
model.add(LSTM(64, dropout_W=0.2, dropout_U=0.2))
model.add(Dense(8))
model.add(Activation('softmax'))

I have the following question, if I am getting a model through this code, could the final product be called a deep learning model?,
I know that this code is very small however there is a lot of computations that the machine is making on the background.

"Deep learning" is not formally defined. However, typically even simple RNNs are taught as advanced neural network subject alongside other topics labelled "deep learning".

Technically, given the time dimension, the depth of a RNN can include many layers of processing (as opposed to many layers of parameters). As such, some of the knowledge and experience used to help with deep feed-forward networks also applies to RNNs. You could consider the LSTM architecture one such thing, because it is designed to address the vanishing gradient problem that plagues simpler RNN architectures.

So, yes you can call your model a "deep learning model" and have that generally accepted.

I'd be slightly concerned if anyone important to the success of a project thought that label was a big deal - either placed on your CV as or as a buzzword on a resulting product. However, it is not unrealistic marketing  because it is essentially true.


Understanding the W term in off policy monte carlo learning In Sutton and Barto's RL textbook they included the following pseudocode for off policy Monte Carlo learning. I am a little confused, however, because to me it looks like the W term will become infinitely large after a couple thousand iterations (and this is exactly what happens when I implement the algorithm).

For example, say that the MC algorithm always follows the behavioral policy for each episode (ignoring epsilon soft/greedy for examples sake). If the probability of the action specified by the policy is 0.9, then after 10,000 iterations W would have a value of 1.11^10,000. I understand that the ratio of W to C(a,s) is what matters, however this ratio cannot be computer once W becomes infinite. Clearly I am misunderstanding something.

The pseudocode you have copied looks incorrect to me, and I think it is from the first edition.
The main issue is at the end of the loop. Where the book has

$\qquad W \leftarrow W \frac{1}{\mu(A_t|S_t)}$
$\qquad \text{If } W = 0 \text{ then ExitForLoop}$

It should have either

$\qquad W \leftarrow W \frac{1}{\mu(A_t|S_t)}$
$\qquad \text{If } \pi(S_t) \neq A_t \text{ then ExitForLoop}$

or

$\qquad W \leftarrow W \frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}$
$\qquad \text{If } W = 0 \text{ then ExitForLoop}$

This latter one is more general - it covers situations where the target policy can be stochastic - but doesn't fit with the notation used elsewhere for a deterministic policy. For some reason the first edition book had a mistake using hybrid algorithm which adjusted to a deterministic target policy except for the exit loop statement. This is fixed in the second edition (page 111).

after 10,000 iterations

Are your episodes really 10,000 time steps long? If so, the chances of off-policy MC control learning anything for early time steps seems remote unless $\epsilon$ is really low (and in which case $W$ will not get too high). If not, have you missed that $W \leftarrow 1$ occurs at the start of each episode?


Decision boundary figure in Least square GAN paper I currently reading Least Square GAN paper. But, I cannot interpret the one of the its figures. .
Explanation of the figure goes like this:

Figure 1: Illustration of different behaviors of two loss functions. (a): Decision
boundaries of two loss functions. Note that the decision boundary should go
across the real data distribution for a successful GANs learning. Otherwise,
the learning process is saturated. (b): Decision boundary of the sigmoid cross
entropy loss function. It gets very small errors for the fake samples (in magenta)
for updateing G as they are on the correct side of the decision boundary. (c):
Decision boundary of the least squares loss function. It penalize the fake samples
(in magenta), and as a result, it forces the generator to generate samples toward
decision boundary.

I already knew the vanilla GAN structure. But I could not understand why decision boundary looks like this. Any help will be appreciated


Is unsupervised disentanglement really impossible? In Locatello et al's Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations he claims to prove unsupervised disentanglement is impossible.

His entire claim is founded on a theorem (proven in the appendix) that states in my own words:

Theorem: for any distribution $p(z)$ where each variable $z_i$ are  independent of each other there exists an infinite number of transformations $\hat z = f(z)$ from $\Omega_z \rightarrow \Omega_z$ with distribution $q(\hat z$) such that all variables $\hat z_i$ are entangled/correlated and the distributions are equal ($q(\hat z) = p(z)$)

Here is the exact wording from the paper:

(I provide both because my misunderstanding may be stemmed from my perception of the theorem)

From here the authors explain the straightforward jump from this to that for any unsupervised learned disentangled latent space there will exist infinitely many entangled latent space with the exact same distribution.

I do not understand why this means its no longer disentangled? Just because an entangled representation exists, does not mean the disentangled is any less valid. We can still conduct inference of the variables independently because they still follow that $p(z) = \prod_i p(z_i)$, so where does the impossibility come in?

The impossibility is referring how to learn the disentangled representations from the observed distribution or to know whether you have a disentangled representation in the first place.

Basically, an unsupervised learning agent tasked with learning a disentangled transformation of some features $\mathbf{z}$ needs to infer a set of features from the data which are not entangled, but the supplied data will always have many equally valid entangled solutions - valid from the point of view of describing the distribution of $\mathbf{z}$ accurately.

An analogy would be "I have observed the value 50, and know it is the sum of 3 numbers. What are those numbers?". Whilst the correct answer exists, and it is possible to guess it, it cannot be inferred from the supplied information.

The part of the proof you quote shows that the multiple equivalent entangled feature sets exist, and theoretically cannot be separated from a "true" disentangled feature set on the basis of knowing the distribution. Once you accept this, it is indeed just a short hop in logic to say that the disentangled features are not learnable - there is no way for a learning system to differentiate between the entangled and disentangled features that explain the distribution, and a large (infinite) set of valid entangled features are guaranteed to exist, which will confound attempts to find a perfect solution.

It is worth noting that the impossibility refers to learning perfect solutions, and that the proof does not rule out useful or practical approximate solutions, or solutions that augment unsupervised learning by applying some additional rules or a semi-supervised approach.


How do layers in an artificial neural network transform inputs to outputs? To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, actually, in each neuron, convert the input to the output?

Putting activation function aside, is the neuron simply doing $\text{input}*a+b=\text{output}$, and try to find the correct $a$ and $b$? If it's true, then how about where you have two neurons and their output ($c$ and $d$) is pointing to one neuron? Do you first multiply $c$ and $d$ then feed it in as input?

The basic calculation for a single neuron is of the form

$$\sigma\left(\sum_{i} x_i  w_i \right),$$

where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be

$$c\,a_c + d\,a_d + b$$

Note, that the bias term $b$ is just a weight that gets multiplied by the input $1$, thus it appears to have no input.

If you want to develop a further understanding for this, you should try to get familiar with matrix and vector notations and the basic linear algebra that underlies the feed-forward neural networks. If you do, an entire layer of neurons on a whole batch of data will suddenly simply look like this:

$$\sigma(WX)$$

and a FFNN with say 3 layers will look like this:

$$\sigma_{3}(W_3\sigma_2(W_2\sigma_1(W_1X)))$$


Is there a bias-variance equivalent in unsupervised learning? In supervised learning, bias, variance are pretty easy to calculate with labeled data. I was wondering if there's something equivalent in unsupervised learning, or like a way to estimate such things?
If not, how do we calculate loss functions in unsupervised learning?

Yes, the concept applies but it is not really formalized.  Consider unsupervised learning as a form of density estimation or a type of statistical estimate of the density.
Variance: You will train on a finite sample of data selected from this probability distribution and get a model, but if you select a different random sample from this distribution you will get a slightly different unsupervised model.  This variation caused by the selection process of a particular data sample is the variance.
Bias: This is a little more fuzzy depending on the error metric used in the supervised learning.  An unsupervised learning algorithm has parameters that control the flexibility of the model to 'fit' the data.  For example,  k means clustering you control the number of clusters.  Simple example is k means clustering with k=1.  You could imagine a distribution where there are two 'clumps' of data far apart.  The mean would land in the middle where there is no data.  This model is biased to assuming a certain distribution.  For a higher k value, you can imagine other distributions with k+1 clumps that cause the cluster centers to fall in low density areas.  For a low value of parameters, you would also expect to get the same model, even for very different density distributions. This is also a form of bias. This unsupervised model is biased to better 'fit' certain distributions and also can not distinguish between certain distributions.
You can see that because unsupervised models usually don't have a goal directly specified by an error metric, the concept is not as formalized and more conceptual.


Can PDDL be utilized for action recognition? The Planning Domain Definition Language (PDDL) is known for its capabilities of symbolic planning in the state space. A solver will find a sequence of steps to bring the system from a start state to the goal state. A common example of this is the monkey-and-banana problem. At first, the monkey sits on the ground and, after doing some actions in the scene, the monkey will have reached the banana.
The way a PDDL planner works is by analyzing the preconditions and effects of each primitive action. This will answer the question of what happens if a certain action is executed.
However, will a PDDL domain description work the other way around as well, not for planning, but for action recognition?
I've searched in the literature to get an answer, but all the papers I've found are describing PDDL only as a planning paradigm.
My idea is to use the given precondition and effects as a parser to identify what the monkey is doing and not what he should do. That means, in the example, the robot ape knows by itself how to reach the banana and the AI system has to monitor the actions. The task is to identify a PDDL action that fits the action by the monkey.

I don't know of any work on this with respect to PDDL, but this is very similar to a conceptual dependency application called SAM (Script Applier Mechanism). Conceptual Dependency (CD) models actions using a number of primitives (which could be seen as equivalent to PDDL primitive actions):  PTRANS for physical transfer, PROPEL for application of a physical force to an object, GRASP for grasping an object, etc. Their number varies around 12 or so, depending on the version of CD.

Stories are described by a sequence of primitive acts, which have slots for actor, object, etc. They are supposed to enable a program to draw inferences about what happens. A common problem when trying to understand stories is that often common knowledge about a situation is omitted; the standard example here is going to a restaurant. It is usually assumed that the listener/reader knows what commonly happens when the protagonist enters a restaurant, so that when they leave without paying this is recognised as something unusual.

The approach used to solve this problem is to encode such knowledge in scripts, sequences of primitive acts. When triggered, eg by "Manuel went to a restaurant", this script is retrieved, and the following actions are looked for in the script. Anything that is recognised is used to fill gaps in the story, eg sitting down at a table, or looking at a menu. This was the task of the SAM program.

Basically you have a sequence of primitive actions, and you try to recognise a more abstract event "going to a restaurant" from that. Obviously you'd need to have a script to recognise, but one could presumably use this to derive a sequence of more generalised events, such as "retrieving an object from a high place", or "standing on top on another object".

The theory of using scripts, plans, and goals to describe human reasoning is detailed in Schank, Roger; Abelson, Robert P. (1977). Scripts, plans, goals and understanding: An inquiry into human knowledge structures. New Jersey: Erlbaum. ISBN 0-470-99033-3.