Are neural networks the only way to reach "true" artificial intelligence? Currently, most research done in artificial intelligence focuses on neural networks, which have been successfully used to solve many problems.  A good example would be DeepMind's AlphaGo, which uses a convolutional neural network. There are many other examples, such as Google Translate, which uses transformers, or DQN, which has been used to play Atari games.
So, are any of the variants of neural networks the only way to reach "true" artificial intelligence (or AGI)?

If by true AI, you mean 'like human beings', the answer is - no-one knows what the appropriate computational mechanisms (neural or otherwise) are or indeed whether we are capable of constructing them.

What Artificial Neural Nets (ANNs) do is essentially 'nonlinear regression' - perhaps this is not a sufficiently strong model to express humanlike behaviour.

Despite the 'Universal function approximation' property of ANNs, what if human intelligence depends on some as-yet-unguessed mechanism of the physical world?

With respect to your question about "the only way":
Even if (physical) neural mechanisms somehow actually were the only route to intelligence (e.g. via Penrose's quantum microtubules), how could that be proved?

Even in the formal world of mathematics, there's a saying that "Proofs of non-existence are hard". It scarcely seems conceivable that, in the physical world, it would be possible to demonstrate that intelligence could not arise by any other mechanism.

Moving back to computational systems, note that Stephen Wolfram made the interesting observation in his book 'A New Kind of Science' that many of the apparently distinct mechanisms he observed seem to be capable of 'Universal Computation', so in that sense there's nothing very particular about ANNs.


Are AI winters inevitable? According to Wikipedia (citations omitted):

  In the history of artificial intelligence, an AI winter is a period of reduced funding and interest in artificial intelligence research. The term was coined by analogy to the idea of a nuclear winter. The field has experienced several hype cycles, followed by disappointment and criticism, followed by funding cuts, followed by renewed interest years or decades later.

The Wikipedia page discusses a bit about the causes of AI winters. I'm curious, however, whether it is possible to stop an AI Winter from occurring. I don't really like the misallocation of resources that are caused by over-investment followed by under-investment.

One of the causes of the AI winter listed on that Wikipedia page is "hype":

  The AI winters can be partly understood as a sequence of over-inflated expectations and subsequent crash seen in stock-markets and exemplified by the railway mania and dotcom bubble. In a common pattern in the development of new technology (known as hype cycle), an event, typically a technological breakthrough, creates publicity which feeds on itself to create a "peak of inflated expectations" followed by a "trough of disillusionment". Since scientific and technological progress can't keep pace with the publicity-fueled increase in expectations among investors and other stakeholders, a crash must follow. AI technology seems to be no exception to this rule.

And it seems that this paragraph indicates that any new technology will be stuck in this pattern of "inflated expectations" followed by disillusionment.

So, are AI winters inevitable? Is it inevitable that AI technologies will always be overhyped in the future and that severe "corrections" will always occur? Or can there a way to manage this hype cycle to stop severe increases/decreases in funding?

I think that by strict definition of the word inevitable, no, future AI Winter events are not inevitable. However likely or unlikely it may be, it is possible to control research spending and to create a more stable plan of funding research in Artificial Intelligence. Because it is possible to avoid an AI Winter event, an event is not inevitable.


Do RNNs/LSTMs really need to be sequential? There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.
Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.
Is my thought correct?

You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
Can RNNs/LSTMs do that? No.
Can Attention (i.e. Transformers) do it? Yes.
The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.
In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.
Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.


How fast are autoencoders? I was exploring image/video compression using Machine Learning. In there I discovered that autoencoders are used very frequently for this sort of thing. So I wanted to enquire:-

How fast are autoencoders? I need something to compress an image in milliseconds?
How much resources do they take? I am not talking about the training part but rather the deployment part. Could it work fast enough to compress a video on a Mi phone (like note8 maybe)?

Do you know of any particularly new and interesting research in AI that has enabled a technique to this fast and efficiently?

Actually it depends on the size of your AE, if you use a small AE with just 500.000 to 1M weigths, the inferencing can be stunningly fast. But even large networks can run very fast, using Tensorflow lite for example, models are compressed and optimized to run faster on Edge-devices (Handys for example, end-user devices). You can find a lot of videos on Youtube, where people test inferencing large networks like Resnet-51 or Resnet-101 on a raspberrypi, or other SOC Chips. Handys are comparable to that, but maybe not that optimized.

For example,I have an Jetson Nano (SOC of Nvidia costs arround 100 euro) and i tried to inference a large Resnet with arround 30 million parameters over my fullHD Webcam. Stable 30 FPS, so speaking in milliseconds its around 33 ms per image.

To answer your question, yes Autoencoders can be fast, also very fast in combination with an optimized model and hardware. Autoencoder structures are quite easy, check out this medium,keras example


What are the likely AI advancements in the next 10 years? What are the likely AI advancements in the next 5-10 years?

I first want to specify that I have nearly no knowledge about How AI works. I just have interest to know more and more about it.

Some examples of Weak AI at present are like Siri and Cortana, those are pretty interesting! But how high levels is it going to reach (likely) in future years?

The determination of likelihood of advancements in any science or technology over a decade are dependent upon several technical features of culture. Not all are technical because they are elements in hardware and software design. Some are.

Market forces
Perpetuation of traditions in education
Feasibility
Availability of resources
Cost of research
Cost of deployment and sustenance
Natural cultural balance of curiosity and fear

The low level cybernetic conversations with service applications like Siri and Cortana were correctly identified in the question, and Google Assistant should be included. If a survey is done of these conversations by working around public relations, perhaps by using a competitor search engine to locate PR unfriendly records of conversations, the low level becomes immediately apparent1. Anyone could confirm this by asking these service applications a set of the popular questions in a set of the StackExchange sites. Most if not all of them would produce answers that would be down voted if not deleted.2

These service applications are efforts to meet the criteria of Alan Turing's thought experiment, which Turing called The Immitation Game3.

  How high levels is it going to reach (likely) in future years?

Prediction of AI milestones is not possible without defining first the milestone criteria to be met. Let's pick seven coveted ones.

A service application the answers of which can gain StackExchange reputation faster than humans, which is a more demanding version of Alan Turing's thought experiment
A robot that can mow our lawns safely
A car without steering wheel or pedals that can take us to places we typically go as safely as a median human driver can
A-pets
Automated theorem proving of currently problematic proofs, and not just verification
Practical goal driven underwater robotic navigation
A home tutor that produces a better distribution of college board exam results in students than public or private educational institutions, better meaning higher mean and lower standard deviations

Based on the seven factors already listed above plus one more, the instantaneous state of AI research and its publication during each moment of the search for probabilities regarding the above seven milestone, some probabilistic predictions can be made about 2029. The bullet items primarily driving the numbers are at the top of this answer4, 5, 6.

< 1% — bullet item 2 too low, and 7 too skewed toward fear
40% — bullet item 1 too low
50% — bullet items 1, 2, 3, and 7 all favorable
2% — bullet item 2 too low, and 7 too skewed toward fear
< 1% — bullet items 2 too high, 3 too low, and 7 too skewed toward fear
80% — bullet items 1 and 3
30% — bullet item 3 high, but 1, 2, 3, 5, and 7 are unfavorable

There are a few that some will find unpalatable. Service applications and automatic theorem provers are not at all likely to become smart in the way defined in the milestones within the next ten years. Robots will likely be swimming and assisting with mining, drilling, and troubleshooting. Level 5 automated vehicles may already be driving around town.

Footnotes

[1] All intelligent systems, including biological ones, have weaknesses and strengths, and the strengths are not perfectly reliable. For instance AI on a CAD application may select a screw fastener to attach one part to another in a sub-assembly with 100% accuracy 99.99999% of the time, seven 9s of accuracy. Much of the talk of general, strong, and weak AI fails the burden of mathematical rigor or scientific verifiability and is largely fabrication and self-deluding conjecture to get cushy, high profile jobs. When published or aired, it is done so not because of credibility but to get paid for ad impressions.

[2] Please don't run this experiment fully, since it would burden moderators that are volunteering to keep the sites clean. Ask the questions but do not post the answers you find. The superficiality will be obvious enough without a focus group.

[3] See the section The Turing Challenge to Cartesian Thought in the post Can a brain be intelligent without a body? for more information on Turing's work in this space and its limitations.

[4] The parenthetic (likely) in the question in the body of the question is correctly placed. The question's tag predicting-ai-milestones is also correct. The likelihoods listed will change because, as the search for likelihoods of AI milestones progresses, the eight factors listed, which are involved in the predictions, will change. This answer author may or may not change the predictions accordingly.

[5] The reasons behind these probabilities cannot be fully disclosed for contractual reasons, but what is disclosed is explained where it can be. Furthermore, the results of prediction are based on the information available to this particular answer author. Others may have other numerical numbers to offer based on other quantitative or qualitative information and can disclose what they wish and are legally allowed to do in another answer.

[6] There is no guarantee made or implied in this answer.


How to prevent overfitting in stacked models? I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.

The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.

As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN.  All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting.  A few can be eliminated up front.  For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.

H. Hutson, S. Geva, and P. Cimiano wrote, in their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting."  Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.

Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation."  Reading their paper, Deep Forest, may give you some causes to evaluate.

Over-fitting is usually the application of too sophisticated a model to which data is fit.  In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.

Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.


Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$? Ok, due to previous question I was pointed to use reinfrocement learning.

So far what I understood from random websites is the following:

there is a Q(s,a) function involved
I can assume my neural network ~ Q(s,a)
my simulation has a state (N input variables)
my actor can perform M possible actions (M output variables)
at each step of the simulation my actor perform just the action corresponding to the max(outputs)
(in my case the actions are 1/2/3 % increase or decrease to propellers thrust force.)

From this website I found that at some point I have to:

Estimate outputs Q[t] (or so called q-values)
Estimate outputs over next state Q[t+1]
Let the backpropagation algorithm perform error correction only on the action performed on next state.

The last 3 points are not clear at all to me, infact I don't have yet the next state what I do instead is:

Estimate previous outputs Q[t-1]
Estimate current outputs Q[t]
Let backpropagation fix the error for max q value only

Actually for code I use just this library which is simple enough to allow me understand what happens inside:

NeuralNetwork library

Initializing the neural network (with N input neurons, N+M hidden neurons and M output neurons) is as simple as

Network network = new NeuralNetwork( N, N+M, M);

Then I think to understand there is the need for an arbitrary reward function

public double R()
{
     double distance = (currentPosition - targetPosition).VectorMagnitude();
     if(distance<100)
         return 100-distance; // the nearest the greatest the reward
     return -1; // too far
}

then what I do is:

// init step
var previousInputs = ReadInputs();
UpdateInputs();
var currentInputs = ReadInputs();

//Estimate previous outputs Q[t-1]
previousOutputs = network.Query( previousInputs );

//Estimate current outputs Q[t]
currentOutputs = network.Query( currentInputs);

// compute modified max value
int maxIndex = 0;
double maxValue = double.MinValue;
SelectMax( currentOutputs, out maxValue, out maxIndex);

// apply the modified max value to PREVIOUS outputs
previousOutputs[maxIndex] = R() + discountValue* currentOutputs[maxIndex];

//Let backpropagation fix the error for max q value only
network.Train( previousInputs, previousOutputs);

// advance simulation by 1 step and see what happens
RunPhysicsSimulationStep(1/200.0);
DrawEverything();

But it doesn't seem to work very nice. I let simulation running for over one hour without success. Probably I'm reading the algorithm in a wrong way.

What normally happens in DQN is the following:

First, the NN is used to estimate an approximation of Q for each state-action pair by using a vector of weights:
$$Q(s,a,w) \simeq Q(s,a)$$

There are two options: You can either feed it a certain state, and get as output the value of each possible action, or, you can input a state-action pair and get as output the value of that pair. (basically the NN gives you the policy)

After having this output you can now use your environment to simulate the next state. So basically you apply the action given by the NN (maybe using $\epsilon -greedy$), and you get the next state, and your next reward from your environment by observing it.

Where does the Q function comes in all of this? The Q-function
$$Q(s,a) = Q(s,a) + \alpha [r(s,a) + \gamma  max_a{Q(s',a)} - Q(s,a)]$$
that in standard Q-learning is used to update the values after each simulation step, is now used as the Target of the neural network, to make the back propagation.

There are some considerations you have to make to help this converge, mainly experience replay and fixed Q-targets.

So when you say " I don't have yet the next state", is because you need to simulate the action that is selected to get this next state.

EDIT: The main thing you might be missing is that the right part of the equation you present in the title is called TD-Target, and it is not used to make the updates, you might see it as the final result you want to achieve $Q^*(s,a)$ (the optimal values, after convergence). But to make the update you need to use the weighted function with the previous value of $Q(s,a)$ as I showed above, using a learning rate $0< \alpha < 1$


How do I keep track of already visited states in breadth-first search? I was trying to implement the breadth-first search (BFS) algorithm for the sliding blocks puzzle (number type). Now, the main thing I noticed is that, if you have a $4 \times 4$ board, the number of states can be as large as $16!$, so I cannot enumerate all states beforehand.

How do I keep track of already visited states? I am using a class board each class instance contains a unique board pattern and is created by enumerating all possible steps from the current step.

I searched on the net and, apparently, they do not go back to the just-completed previous step, BUT we can go back to the previous step by another route too and then again re-enumerate all steps which have been previously visited.

So, how to keep track of visited states when all the states have not been enumerated already? Comparing already present states to the present step will be costly.

Dennis Soemers' answer is correct: you should use a HashSet or a similar structure to keep track of visited states in BFS Graph Search.

However, it doesn't quite answer your question. You're right, that in the worst case, BFS will then require you to store 16! nodes. Even though the insertion and check times in the set will be O(1), you'll still need an absurd amount of memory.

To fix this, don't use BFS. It's intractable for all but the simplest of problems, because it requires both time and memory that are exponential in the distance to the nearest goal state.

A much more memory-efficient algorithm is iterative deepening. It has all the desirable properties of BFS, but uses only O(n) memory, where n is the number of moves to reach the nearest solution. It might still take a while, but you'll hit memory limits long before CPU-related limits.

Better still, develop a domain specific heuristic, and use A* search. This should require you to examine only a very small number of nodes, and allow the search to complete in something much closer to linear time.


How should I detect an object in a camera image? I would like to create a model, that will tell me if one type of object is in an image or not.

So, for example, I have a camera and I would like to see when one object gets into the shot.

Object detection: This could be an overkill, because I don't need to know the bounding box around. Also, this means that I would need to label a lot of images, and draw the bounding box to have train data (a lot of time)
Image classification: This doesn't solve the problem, because I don't know what else could not be an object. It would be impossible to train for 2 classes: object / not object.

My idea is to have Autoencoder. Train it only on data with the object. Then, if Autoencoder produces a result with a high difference with the original, I detect it as an anomaly - no object.

Is this a good approach? Will I have a lot of trouble with different backgrounds?

Is this a good approach? Will I have a lot of trouble with different backgrounds?

A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without the object in frame. My gut feeling is that your system will have poor performance compared to a properly trained classifier, as the autoencoder will naturally have to get good at constructing background elements in order to score well, so unless your object is very consistently always in a similar place with radically different appearance to the background then the autoencoder will get good reconstructions of background-only frames, and your anomaly detection would need to be set too sensitive. That would cause the detector to fail to spot the object when in frame.

There is a catch-all answer to "how well will my idea work" with ML projects. You should try it and see. Data science is essentially an empirical approach, and building practical models is an engineering discipline where testing is a core part of the process.

In order to test your model, you are going to need a lot of images of just background, including all the kinds of background that you expect the system to be used. Which begs the question, why not collect a range of suitable images without your target object, and use them as the second class?

  This doesn't solve the problem, because I don't know what else could not be an object. It would be impossible to train for 2 classes: object / not object.

Actually, it is easy. Just collect suitable images from locations where you expect your system to be used, without the object in frame. These are your "Not Object" class. A "Not Object" does not have to include some substitute foreground object. Although I recommend that you do have some images like this to prevent accidentally creating an "object is in the foreground" detector*. The primary goal should be to be collect data that matches how the model will be used once deployed. That will depend a lot on how much control and consistency you get over the production cameras where the trained detector will be put to use.

I would do this, then train a standard binary classifier.

If you are still interested in how well your auto-encoder idea could work, you then have plenty of test data to evaluate it.

*  This is one area where your auto-encoder idea might do better than a classifier - reconstructing unfamiliar foreground objects should be hard for it leading to high error.

It is too difficult to tell in advance whether this effect is strong enough to make your auto-encoder approach better than a classifier.


Do I need an encoder-decoder architecture to predict the next item of a sequence? I am trying to understand how RNNs are used for sequence modelling.

On a tutorial here, it mentions that if you want to translate say a sentence from English to French you can use an encoder-decoder set-up as they described.

However what if you want to do a sequence to sequence modelling where your inputs and outputs are of the same domain but you just want to predict the next output of a sequence.

For example if I want to use sequence modelling to learn the sine function. So say I have 20 y-coordinates from $y = sin(x)$ from 20 evenly spaced out x-coordinates and I want to predict the next 10 or so y-coordinates. Would I use an encoder-decoder setup here?

You don't need to Encoder-Decoder here. When using seq2seq learning for text (for example, for translation) you need encoder-decoder to encode the words into the numeric vectors and decode the vectors into the words. Therefore, for your numerical case, you don't need an encoder or decoder to train the RNN.


Why different images of the same person, under some restrictions, are in a 50 dimension manifold? In this lecture (starting from 1:31:00) the professor says that the set of all images of a person lives in a low dimensional surface (compared the the set of all possible images). And he says that the dimension of that surface is 50 and that they get this number by adding the three translations of the body, the three rotations of the head and the independent movements of the face's muscles. He also adds that it may be more than 50 but less than 100. How do we get the number 50 ?
The professor previously said (in the same lecture, 1:29:00) that the set of all the images that we could describe as natural and that we could interpret are in a manifold. I try to understand how the number 50 came up like the following: let's take an image of a person, since it's "natural" then it belongs to that manifold. Hence there is an open set to which this image belongs to and there is a homeomorphic map from this open set to an euclidean space. Let's suppose (I don't know why but it's the only possible thing I could come up with to understand) that all the images of that same person, regardless of his position and expressions..., are in that open space then through the homeomorphic mapping we have the "same points" in an euclidean space, do we get the base of it by decomposing all the possible movements of the person?
I hope someone can clarify things for me, it seems this doesn't only work with images but all types of non-structured types of data.

The number 50 is essentially just a guess based on results when compressing and/or generating data of a certain type. The variables such as "the three translations of the body, the three rotations of the head and the independent movements of the face's muscles" are examples only. There is no known formal map with well-defined parameters that defines a well understood manifold of "clear images of this person" in natural images. The lecturer has not constructed such a map as far as I can tell, but has done some related experiments.
Experimentally, it is possible to establish parameter vectors that work, with models like Variational Autoencoders and Generative Adversarial Networks. Depending on the size of the target image, and amount of variation in subject matter that you want to allow for (pose, lighting, clothing, hair style, makeup, camera properties etc), you will end up with different sizes of embedding vectors that appear to capture the important variations. When dealing with multiple people, it is common to see vector sizes of 64, 128, 256.
The lecture suggests compressing images with a clear background, consistent lighting, same person with only changes being in pose. Around 50 dimensions for this relatively simple image space seems reasonable, given facial recognition engines that work well in a more complex domain using 128 dimensional embeddings.
I expect that the lecturer has seen experimental evidence that a vector of 50 dimensions performs well at representing all variations in these images, plus smaller vectors perform measurably worse and larger vectors do not perform better. This experiment is possible by constructing something like a VAE with a specific size of embedding vector, training it, then measuring loss when reconstructing a set of test images.


Effect of adding an Independent Variable in Multiple Linear Regression I am new in machine learning and learning linear regression concept. Please help with answers to below queries.
I want to understand effect on existing independent variable(X1) if I add a new independent variable(X2) in my model.
This new variable is highly correlated with dependent variable(Y)

Will it have any effect on beta coefficient of X1?
Will relationship between X1 and Y become insignificant?
Can adjusted R-square value decrease?

Yes, the addition of the independent variable ($X_2$) will have an impact on the beta coefficient of $X_1$.
Maybe, it depends on the relationship between $X_1$ and $Y$, and whether you are using any regularization or not.
If the independent variable $X_2$ is more correlated than $X_1$, then the value of $r^2$ should be higher (but I am not sure on this)


What is predicate argument recognition? There is a study about The Necessity of Parsing for Predicate Argument Recognition, however I couldn't find much information about 'Predicate Argument Recognition' which could explain it.

What is it exactly and how does it work, briefly?

In grammar, a predicate-argument relationship is one which is implied from text but not expressed in the syntactic structure. (Asher S)

Predicate logic or first order logic is a collection of formal systems used in mathematics, philosophy, linguistics and computer science.

The NLP community is interested in recognizing, representing and classifying predicate arguments for shallow semantic parsing which can be viewed as the process of assigning a WHO did WHAT to WHOM, WHEN, WHY, HOW e.t.c. structure to plain text.

This process entails identifying groups of words in a sentence that represent these semantic arguments and assigning specific labels to them. This could play a key role in NLP tasks like information extraction, question answering and summarization.

Automatic and accurate techniques that can annotate naturally occurring text with semantic/predicate argument structure can facilitate the discovery of patterns of information in large text collections.

For reference I recommend this paper that uses Support Vector Machine algorithms for predicate argument classification. (Sameer P Kadri H 2005) https://link.springer.com/article/10.1007/s10994-005-0912-2#enumeration


What technology do people use to create bots for games like LOL or Runescape? I was curious about how people make AI to play games. Does anyone know of the AI used to play these games? What allows the AI to see/click the screen in real-time? Even just direction on what libraries for such tasks would be helpful. I can't imagine game developers make an API for creating bots in their games like browsers use with selenium.

Bot development is more about 'hacking' than AI in a way that in the very first place you need to read and (over) write game data which you are not supposed to (and thereby potentially violating the Terms and Conditions - so be aware of that). The AI part is fairly simple for most hack/bot applications that I can think of.

Read data

To read game data you can for example:

use an assembly code debugger like Ollydbg to locate relevant data in the memory, e.g. amount of gold
Observe graphical objects being rendered, e.g. a unit being drawn
Intercept network packages containing all kinds of game information, e.g. a unit appearing on your screen

Write data

In a similar way there are multiple ways to write data:

overwrite game data in memory
Use the Windows API SendInput() function to emulate keyboard inputs
Use the Windows API SendMessage() function to send messages to the game
Manipulate network traffic

These lists are not comprehensive but to give you an idea of how it is being done.

AI-wise an A* algorithm for example can be deployed to do path finding.

If you are interested in the topic I suggest to read 'Game Hacking' by Nick Cano. The book provides a good introduction.


How can the multiple intelligences model be incorporated into AI? I have been wondering since a while ago about the theory of multiple intelligences and how they could fit in the field of Artificial Intelligence as a whole.

We hear from time to time about Leonardo Da Vinci being a genius or Bach's musical intelligence. These persons are commonly said to be (have been) more intelligent. But the multiple intelligences speak about cooking or dancing or chatting as well, i.e. coping with everyday tasks (at least that's my interpretation).

Are there some approaches on incorporating multiple intelligences into AI?

Here's a related question: How could emotional intelligence be implemented?.

If anything, multiple intelligences are much more obvious in AI than in other fields, because we haven't yet unlocked how to do transfer between domains.

As an example, AlphaGo is very, very good at playing Go, but it's got basically nothing in the way of bodily-kinesthetic intelligence. But other teams have built software to control robots that does have bodily-kinesthetic intelligence, while not being good at the tasks that AlphaGo excels at.

This sort of modular intelligence is typically referred to as 'narrow AI,' whereas we use the term 'general AI' (or AGI, for Artificial General Intelligence) to refer to intelligence that we've built that can do roughly as many different kinds of things as people can do.


How does non-max suppression work when one or multiple bounding boxes are predicted for the same object? My understanding of how non-max suppression works is that it suppresses all overlapping boxes that have a Jaccard overlap smaller than a threshold (e.g. 0.5). The boxes to be considered are on a confident score (may be 0.2 or something). So, if there are boxes that have a score over 0.2 (e.g. the score is 0.3 and overlap is 0.4) the boxes won't be suppressed.
In this way, one object will be predicted by many boxes, one high score box, and many low confident score boxes, but I found that the model predicts only one box for one object. Can someone enlighten me?
I currently viewing the ssd from  https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection
Here is the code.
#Finding Jaccap Overlap and sorting scotes
class_scores, sort_ind = class_scores.sort(dim=0, descending=True)
class_decoded_locs = class_decoded_locs[sort_ind]  # (n_min_score, 4)
overlap = find_jaccard_overlap(class_decoded_locs, class_decoded_locs)
suppress = torch.zeros((n_above_min_score), dtype=torch.uint8).to(device)

for box in range(class_decoded_locs.size(0)):
# If this box is already marked for suppression
    if suppress[box] == 1:
    continue
    suppress = torch.max(suppress, overlap[box] > max_overlap)
    suppress[box] = 0

I might be able to help with the theory, but the coding... it is a non standard API such as Tensorflow or Pytorch (it might be custom code for what I can tell).
The key element here is that the bouding boxes are removed only if they hold a prediciton for the same class that the box that is overlapping with (but with less confidence, that is why it gets removed).
Here is an example, where we have:

Two classes $c \in [c_1, c_2] = [$ "star"$, $ "moon" $]$
Three bounding boxes

The blue bounding boxes holds prediction for the class $c_1$ so their predictions are $p(c_1)_{box1} = 0.8$ and $ p(c_1)_{box2} = 0.9$. On the other hand, the green box holds a prediction for the class $c_2$.
The three boxes are highly overlaping so the overlap between any box $x$ with any box $y$ will be above the IoU threshold: $IoU(box_x, box_y) > 0.5$. So in principle all boxes are suceptible to be removed.
However the NMS only applies for boxes predicting the same class (in the case the blue one). So the NMS algorihtm is: if the boxes are overlapping, $IoU(box_1, box_2) > 0.5$, which is true, remove all non maximal class probability boxes. Said differently, take just the box with highest $p(c_1)$ and remove the rest. So the $box_1$ with class probability $p(c_1) = 0.8$ would be removed.
So what happens with green box? Isn't it overlapping as well? Yes, but consider that the green box is not trying to predict the same object, is trying to predict another object, $c_2$, which happens to be very close to the first object, $c_1$. This way object detectors support detection of different overlapping objects.


CNN's vs Densely Connected NN's In image classification we are generally told the main reason of using CNN's is that densely connected NN's cannot handle so many parameters (10 ^ 6 for a 1000 * 1000 image). My question is, is there any other reason why CNN's are used over DNN's (densely connected NN)?

Basically if we have infinite resources will DNN trump CNN's or are CNN's inherently well suited for image classification as RNN's are for speech. Answers based either on mathematics or experience on the field is appreciated.

The keyword here is Parameter Sharing or Weight Sharing across various image portions.

If we take a simple example of grayscale binary image of an alphabet 'F', it is a combination of multiple patterns. The patterns here are vertical lines and horizontal lines. These patterns are based on relation between intensities of contiguous cells. This relation between contiguous cells is established using a weight matrix.

Also, for identifying multiple horizontal lines, we dont need multiple node-sets in hidden dense layer trying to identify different horizontal lines in the image. The pattern is same but present in different locations. Hence the sharing of weights comes into picture.

In the 1st hidden layer, encode the pattern horizontal line in a weight matrix(learnt during training and used in testing). Place it over small grid and check for presence. As this matrix is slided and tested across the image, the presence of horizontal lines is marked in various locations. This weight matrix is called a kernel.

Combining the above points, kernel provides a way of handling parameter / weight sharing between contiguous cells to identify patterns. Dense layer instead of kernels would solve it eventually but start in a random manner. Since a efficient way was identified, it is being used.

Next to identify vertical lines, another kernel needed and slide across.

Suppose next we have dense layer as 2nd hidden layer. This layer looks for combination of patterns ('p' horizontal lines and 'q' vertical lines in this case for 'F') present and learns combinations to identify output.

Just to compare with traditional programming, kernels are like regular expressions. dense layers are like loops. Just sharing my thoughts. Any better explanation is welcome.


How can computers beat humans at image recognition? For supervised learning, humans have to label the images computers use to train in the first place, so the computers will probably get wrong the images that humans get wrong. If so can computers beat humans?

When researchers claim "better than human accuracy", they are demonstrating that a computer can beat an individual human on a test. And that is because the ground truth labels are actually higher accuracy than a single human could label the images individually.

There are at least two major ways that ground truth labels can beat an individual human on image tasks.

Additional information is available from the same source as the image. For instance many pictures of pets in the ImageNet database are labeled with a specific breed of animal, due to how they are sourced. Most people
who are not experts at pet breeds will score quite badly on a test to identify dog breeds at the fine grained level that ImageNet presents.
Ground truth based on expert opinion can be sourced from multiple experts and their opinions combined. This approach can independently be shown to be more reliable than the opinion of a single person.

So in short yes computers can beat humans when they have had access to better original ground truth, and that is possible, even if that ground truth is generated by humans.

However, in general your concern stands. Ground truth data is a limiting factor. It might be possible in theory for a computer model to have an even better accuracy at the "real" task than the ground truth for a supervised learning task. However, this is next to impossible to prove, and other concerns, such as changes to distribution of real data as opposed to training data, are usually more important at that level of accuracy.


Is this LSTM layer learning anything? I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:

I don't understand. Is this layer learning anything? Or no?
Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got:

So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.

What may be more informative in terms of whether it is learning or not, is to track gradients.
Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.
You are developing with tf.keras, in which case you can add to your tensorboard callback:
tf.keras.callbacks.TensorBoard(write_grads=True)
Additional experiments may include:

trying shorter length sequences and compare grad flow
try replacing tanh with alternative activation functions in the LSTM layers of your model tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu()) (see tanh saddle points problem)

(graph borrowed from d2l)


What techniques can be used to predict future attendance of students for a particular subject lecture session? I have data of 30 students attendance for a particular subject class for a week. I have quantified the absence and presence with boolean logic 0 and 1. Also, the reason for absence are provided and I tried to generalise these reason into 3 categories say A, B and C. Now I want to use these data to make future predictions for attendance but I am uncertain of what technique to use. Can anyone please provide suggestions?

I suggest you should use AI Regression Model for future predictions for an attendance of students. Because of this technique or model design for future predictions.

Follow this to get more information about regression type and methodology


Why are the terms classification and prediction used as synonyms in the context of deep learning? Why are the terms classification and prediction used as synonyms especially when it comes to deep learning? For example, a CNN predicts the handwritten digit.

To me, a prediction is telling the next step in a sequence, whereas classification is to put labels on (a finite set of) data.

Many people confuse and misuse the two terms, classification and prediction (or classify and predict).  This is because in many cases classification techniques are being used for prediction purposes which creates part of the confusion to others who then use the term ‘prediction’ (or ‘predict’) inappropriately.

Your understanding of the definitions of classification and prediction is mostly correct and you are absolutely correct that there are many people using the terms synonymously, sometimes correctly but I believe mostly erroneously.  There are many good articles elaborating on the two and I have added some links and excerpts at the end of this answer.  What these articles don't cover is that many forecasting (i.e. prediction) researchers and practitioners will use conventional classifiers to predict the future state of a time series or data sequence.  More advanced researchers and practitioners will use time-recurrent models, which learn temporal patterns.  These are still called classifiers but the for purpose of prediction.

There are more papers written on this use of classifiers, conventional and time-recurrent type classifiers, for time series than the use of regressor models!

This adds to the confusion in the data science and machine learning community in the usage of the terms ‘classify’ and ‘predict'.

Galit Shmueli sums it up best in his paper, “To Explain or to Predict?”, where he states: “Conflation between explanation and prediction is common, yet the distinction must be understood for progressing scientific knowledge.”

There is also the opposite problem where people will confuse regression models with classification.  See the first article below.

Classification vs. Prediction, by Professor Frank Harrell

Excerpt:
By not thinking probabilistically, machine learning advocates frequently utilize classifiers instead of using risk prediction models. The situation has gotten acute: many machine learning experts actually label logistic regression as a classification method (it is not). It is important to think about what classification really implies. Classification is in effect a decision. Optimum decisions require making full use of available data, developing predictions, and applying a loss/utility/cost function to make a decision that, for example, minimizes expected loss or maximizes expected utility. Different end users have different utility functions. In risk assessment this leads to their having different risk thresholds for action. Classification assumes that every user has the same utility function and that the utility function implied by the classification system is that utility function.

To Explain or to Predict?, by Galit Shmueli

Abstract. Statistical modeling is a powerful tool for developing and testing
theories by way of causal explanation, prediction, and description. In many
disciplines there is near-exclusive use of statistical modeling for causal explanation and the assumption that models with high explanatory power are
inherently of high predictive power. Conflation between explanation and prediction is common, yet the distinction must be understood for progressing
scientific knowledge. While this distinction has been recognized in the philosophy of science, the statistical literature lacks a thorough discussion of the
many differences that arise in the process of modeling for an explanatory versus a predictive goal. The purpose of this article is to clarify the distinction
between explanatory and predictive modeling, to discuss its sources, and to
reveal the practical implications of the distinction to each step in the modeling process.

What is the difference between classification and prediction?, from KDnuggets

If one does a decision tree analysis, what is the result? A classification? A prediction?

Gregory Piatetsky-Shapiro answers:
The decision tree is a classification model, applied to existing data. If you apply it to new data, for which the class is unknown, you also get a prediction of the class.

The assumption is that the new data comes from the similar distribution as the data you used to build your decision tree. In many cases this is a correct assumption and that is why you can use the decision tree for building a predictive model.

When Classification and Prediction are not the same?

Gregory Piatetsky-Shapiro answers:
It is a matter of definition. If you are trying to classify existing data, e.g. group patients based on their known medical data and treatment outcome, I would call it a classification. If you use a classification model to predict the treatment outcome for a new patient, it would be a prediction.

gabrielac adds
In the book "Data Mining Concepts and Techniques", Han and Kamber's view is that predicting class labels is classification, and predicting values (e.g. using regression techniques) is prediction.

Other people prefer to use "estimation" for predicting continuous values.


Why does unsupervised pre-training help in deep learning? What is the effectiveness of pre-training of unsupervised deep learning?

Does unsupervised deep learning actually work?

Unsupervised pre-training was done only very shortly, as far as I know, at the time when deep learning started to actually work. It extracts certain regularities in the data, which a later supervised learning can latch onto, so it is not surprising that it might work. On the other hand, unsupervised learning doesn't give particularly impressive results in very deep nets, so it is also not surprising that with current very deep nets, it isn't used anymore.

I was wondering whether the initial success with unsupervised pre-training had something to do with the fact that the ideal initialization of neural nets was only worked out later. In that case, unsupervised pre-training would only be a very complicated way of getting the weights to the correct size.

Unsupervised deep learning is something like the holy grail of AI right now and  hasn't been found yet. Unsupervised deep learning would allow you to use massive amounts of unlabeled data and let the net form its own categories. Later you can just use a little bit of labeled data to give these categories their proper labels. Or just train it immediately on some task, in the conviction that it has a huge amount of knowledge about the world already. This is also what the problem of common sense comes down to: a huge and detailed model of the world, that could only be acquired by unsupervised learning.


Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine? My company has full access to beta testing for GPT-3. We wanted to try it for some games or game mechanics within Unity3D. Is it possible to use it for dialogues or with unity scripts?
The Documents of OpenAI does not say anything about this possibility, so I'm not sure.

Yes, OpenAI will release an API for GPT-3, so any developer can integrate it into their application. I don't believe the document for their API is public yet, so we don't know what the final interface will look like, but it's likely to be a simple REST API. In the future, I imagine your developers can take advantage of their API, or alternatively there will be community-made scripts for you to use/copy.
The pricing for using their API is explained here. Note that they charge per token, which might be important in case your game plans to make live calls to GPT-3 during gameplay (as opposed to mining a huge corpus of answers to build an offline database).
The use cases of GPT-3 suggests that you can legally use them for commercial products, although I couldn't find a definitive license or user agreement document.


What are finite horizon look-ahead policies in reinforcement learning? I was reading the paper How to Combine Tree-Search Methods in Reinforcement Learning published in AAAI Conference 2019. It starts with the sentence

  Finite-horizon lookahead policies are abundantly used in Reinforcement Learning and demonstrate impressive empirical success.

What is meant by "finite horizon look-ahead"?

Per this paper a look ahead policy is a policy that will make decisions based on some 'horizon'. Here horizon means some time steps into the future, and so a finite horizon is simply a finite amount of time steps into the future. For example, as we are typically concerned with maximising returns in RL, a 10-step look ahead policy would choose an action at time $t$ that maximises the (expected) rewards at time $t+1, ... t+10$.


Necessity of GPUs and hardware minimum specs for Deep Learning? I’m doing some research into what hardware I need and what hardware I have available in college for a final year project.

The project is designing a self driving car/computer vision system inside a simulation (instead of using an actual car and track as that’s out of scope for a final year project)
The simulation will be an emulator of either N64 or SNES.

What would be the recommended hardware specs?
How will having many GPUs benefit the project?

I can also get the specs of some of the available computers I have in college?
I may even be able to get the college to fund new hardware.

For a simulation of SNES or N64, the resolution is probably not that high. You can use either online credits or buy new hardware. For online credits, it is recommended if you do teh simulation/training for only several dozens hours, as eache hour costs around 5-10 dollars. AWS is a good choice as it have a large variety of choice for hardware. For higher performance, choose multi GPU setup.

However, if the simulation requires a long time, or you wish to own the hardware, you should buy your own hardware. Multiple GPU will scale quite nicely, so you may go for multi GPU if performance is needed. For single GPU, a RTX 2000 series is recommended as it have tensor cores to accelerate training of CNN. For a budget option, consider an RTX 2060 SUPER. It have 8 GB of VRAM with a relatively low price and yet it have tensor core functionality. For more powerful GPU, choose the RTX 2080 TI. It have 11GB of VRAM which is enough in most use case.
For multiple GPU, you might want several RTX 2060 SUPER as they are cheap but the performance is quite good. The 8GB VRAM helps as well. For the highest performance, choose 2 or more RTX 2080 TI. It is enough for nearly most task. If this is not enough, renting an online instance is a better idea. Hope it helps and have a nice day.


How do I check that the combination of these models is good? I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a GridSearchCV, and all of them placed within a VotingClassifier object.
Alone, they all bring around 70% accuracy, on a data set which is about half normal/uniform distributed, and half one-hot distributed. Together, they provide 80% accuracy, which isn't good enough, given that I was told that 95%< is achievable.
The models: DecisionTreeClassifier, ExtraTreesClassifier, KNeighborsClassifier, GradientBoostingClassifier, LogisticRegression, SVC, Perceptron, and a few more classifiers.
How do I check if the combination is good?

Goodness is subjective.  Reliable knowledge isn't possible with that flimsy a quality objective.

The sturdy objective criteria you gave is 95%, so it is bad by that criteria.  (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)

However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy.  (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.)

I have four questions.  (My apologies that this question leads to the two assumptions above and a bunch more questions.)

Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?
Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?
Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?
What economic, contractual, or operational hard stop is dictating the 90%?

If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.


What kind of neural network should I build to classify each instance of a time series sequence? Let's say I have the time-series dataset below-left. I would like to train a model in such a way that, if I feed the model with an input like the test sequence below, it should be able to classify each sample with the correct class label.
    Training Sequence:              Test Sequence:

Time,   Bitrate,   Class           Time,   Bitrate    Predicted Class
 0,       312,       1              0,       234      -----> 0
 0.3,     319,       1              0.2,     261      -----> 0
 0.5,     227,       0              0.4,     277      -----> 0
 0.6,     229,       0              0.7,     301      -----> 1
 0.7,     219,       0              0.8,     305      -----> 1
 0.8,     341,       1              0.9,     343      -----> 1
 0.9,     281,       0              1.0,     299      -----> 0
          ...                                ...

So, what kind of neural network should I build to classify each instance of a time series sequence?

A time series, usually, requires regular time intervals, but, from looking at your example, it seems that's not the case. You could try to use a MLP and give it as input the Time and Bitrate pairs and make it output the Class.
The activation function is what makes your neural network produce its output, i.e. activate the neurons. The loss function calculates the error of your model's predictions. This error is used by the backpropagation algorithm to adjust the model when training.
If your data can only be in one of two classes, for example 0 or 1, you have a binary classification problem. I recommend using the Sigmoid as the activation function. If you look at the Sigmoid's plot, you'll notice that, no matter the input, the output will always be within 0 and 1. You can think of this as the probability of the input being in class 0 or 1. So, for example, any input that produces a probability under 0.5 is class 0. Of course, if you want to be more precise you can simply procure the probabilities without the making it fit the labels. For example, an output of 0.7 represents that there is a 70% probability of the input belonging to class 1. Another term for this is Logistic Regression.
The Binary cross-entropy calculates the error between the model's prediction and the real class label. This loss function can be applied to many classes, but the binary version keeps it to 0 and 1.
Choosing the right activation functions and loss functions will depend on the problem you are trying to solve. Here are some useful guides  on loss functions and activation functions.
Additional to  the above, try to use scaling(normalize) the inputs before feeding to the network, so that the network may generalize, as it looks like bitrate value above 300 is likely to be associated with class 1. It may improve performance.


Why are we asking, "How can we simulate the brain?" As I've thought about AI, and what I understand of the problems that we face in the creation of it, I've noticed a recurring pattern: we always seem to be asking ourselves, "how can we better simulate the brain?"

Why are we so fascinated with simulating it? Isn't our goal to create intelligence, not create intelligence in a specific medium? Isn't growing and sustaining living brains more inline with our goals, albeit a bit of an ethical controversy?

Why is this exchange's description: "For people interested in conceptual questions about life and challenges in a world where 'cognitive' functions can be mimicked in a purely digital environment?"

To condense these feelings in a more concise question: Why are we trying to create AI in a computer?

Human intelligence is very general / broad in its scope. This is self-evident, and whatever AI ends up to be, we'd like it to be a general problem solver as well (cf. Simon and Newell). Taking liberal interpretations of your question...

  Why AI in a computer?

Computers, to the extent that we can frame problems in general as a solvable computational problem, are also general problem solvers. Wether this is actually the case (can you compute meaning or feels?) is up for debate (cf. computational functionalism, hyper-computation), but it is part of the artificial intelligence project to make a claim on this statement.

  Why do we think a computational framework brings us any closer to an understanding of cognition / consciousness?

Good question, and frankly there is no good answer to that asides from "its the best thing we got".

TL;DR "computational functionalism", a lot of the literature in psychology and philosophy seems to converge towards an understanding of cognition as "computational" (as in information processing: the V1 stream in the brain processes "early visual information") and functional (goal directed grounded on "meaning", ex: "i scratch itch because itchy", as opposed to "i am moving atoms").

However the two theories don't mesh together well (cf. Chinese Room Argument, and the many other arguments in a similar flavour) despite their independent successes in the theory of mind. Why this is the case nobody quite knows...

  Why not AI in something that isnt a computer?

I don't know, but to the extent that our understanding of the world is grounded in math, then it being in a computer is sufficient anyways.

Maybe there are other paradigms of understanding the world though. Fingers crossed

  Why are we asking, “How can we simulate the brain?”

Because its the best tentative understanding we have of an "intelligent faculty", though it should be noted that various methods in machine learning don't seem to be directly inspired by biological implementation  (kNN, statistical methods, as opposed to neural nets)

Further reading: http://www.scaruffi.com/nature/mach01.html


Can an artificial intelligence suffer? As a piece of computer code, it's tempting to think that artificially intelligent systems can't have feelings. So, can an artificial intelligence suffer?

AFAIK, pain is produced by the nervous system (after the realease of chemicals), so no AI suffers. AI does not (yet) possess a life (and I don't think this will ever happen).

A definition of life from the dictionary:

  the condition that distinguishes animals and plants from inorganic matter, including the capacity for growth, reproduction, functional activity, and continual change preceding death.

The current AI definitely does not possess a life according to this definition. AI is inorganic (like a rock): it is based on silicon (not carbon atoms).


Can CNNs be applied to non-image data, given that the convolution and pooling operations are mainly applied to imagery? When using CNNs for non-image (times series) data prediction, what are some constraints or things to look out for as compared to image data?
To be more precise, I notice there are different types of layers in a CNN model, as described below, which seem to be particularly designed for image data.
A convolutional layer that extracts features from a source image. Convolution helps with blurring, sharpening, edge detection, noise reduction, or other operations that can help the machine to learn specific characteristics of an image.
A pooling layer that reduces the image dimensionality without losing important features or patterns.
A fully connected layer also known as the dense layer, in which the results of the convolutional layers are fed through one or more neural layers to generate a prediction.
Are these operations also applicable to non-image data (for example, times series)?

Usually, you need to ensure that your convolutions are causal, meaning that there is no information leakage from the future into the past. You could start by looking at this paper, which compares Temporal Convolutional Networks (TCN) with vanilla RNNs models.


Are all fully observable environments episodic? According to the definition of a fully observable environment in Russell & Norvig, AIMA (2nd ed), pages 41-44, an environment is only fully observable if it requires zero memory for an agent to perform optimally, that is, all relevant information is immediately available from sensing the environment.

From this definition and from the definition of an "episodic" environment in the same book, it is implied that all fully observable environments are, in fact, episodic or can be treated as episodic, which doesn't seem intuitive, but logically follows from the definitions. Also, no stochastic environment can be fully observable, even if the entire state space at a given point in time can be observed because rational action may depend on the previous observation that must be remembered.

Am I wrong?

No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book:
Fully Observable Environment (section 2.3.2)

If an agent’s sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable. A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action

Episodic Environment (section 2.3.2)

In an episodic task environment, the agent’s experience is divided into atomic episodes. In each episode the agent receives a percept and then performs a single action. Crucially, the next episode does not depend on the actions taken in previous episodes.

Take note of the "crucial" part at the end of the definition of episodic environment. A fully observable environment that is not episodic (and therefore sequential in the book's taxonomy) is chess. Chess is fully observable because the player can view the positions of all active pieces on the chess board, and that is all the information that needs to be known in order to take the optimal action. But chess is not episodic, because the player's current move depends on all previous moves, and the current move will have downstream effects in later turns.
In fact, if you look at Figure 2.6 in the book on pg. 45, they provide three examples of fully observable sequential (i.e. not episodic) environments: crossword puzzles, chess, and backgammon. There are of course many more. Most games are sequential as that is the main appeal of them - how to best sequence my moves now in order to ensure victory over my opponent at a future time?


A deep learning algorithm to optimize the outcome I'm am quite new to deep learning but I think I found just the right real-world situation to start using it. The problem is that I have only used such algorithms to predict outcomes.  For my new project, I need information to feed a machine with to optimize outcomes. Could someone explain briefly how I should proceed? I'm stuck.

Here's the situation:

I have a machine that takes planks of wood with different grades of wood available throughout its length and has to cut it into blocks provided in a cut list. This machine will always choose the highest score it can get from a given plank. The score is obtained by multiplying each block's area by its multiplicator. The algorithm I want to build has to give that machine a multiplicator for each block listed in a cut list. All of the physical output from this machine will be stocked on shelves by a robot until needed. The cutting machine is allowed to downgrade parts of a plank if it helps it reach a higher score.

The value has to act as an incentive for the machine to give me the block I need the most without downgrading too much wood.

OPTIMIZATION GOALS

Make sure each block is in stock by the time it is needed, but not too early without reason
Downgrade as little area of wood as possible (some species are very expensive)

INPUT NODES

Amount of time before this block is needed
Grade of wood for this block
Amount of this block needed
Block's area (Maybe?)

FEEDBACK PROVIDED TO THE ALGORITHM

Amount of time in advance that the block was ready (must be as low as possible)
Area of wood downgraded * number of grades skipped

EXPECTED RETURN DATA

A multiplicator that will give that block an optimal its priority relative to others

INFORMATION I DON'T HAVE BUT COULD GATHER

Mean ratio of each grade for each species of wood

What I've figured out so far is that I may need my feedback to be smashed in only one value in order to make it the output node. The problem is that I can't understand how to make this algorithm to determine a multiplicator. Am I wrong in trying to solve this through deep learning?

Deep learning models for regression tasks are quite difficult to train so I would suggest not to start with them. Instead I would start with one of the approaches below and maybe try to use deep learning afterwards.

A classical approach to the problem could be to analyze your optimization software and this would probably lead you to some deterministic algorithm.

Different approaches can be to treat your optimization software as a black box - give it a wide variety of inputs, note down the variables you are interested in (execution time, cutting results, etc.) and try to fit some sort or regressor on it.

One option is to follow Kourosh's idea and formulate it as a classical optimization problem.

If you prefer to use machine learning tools than I'd suggest that you start with a simple model like linear regression just to verify that there is any signal in the data that you can use. Afterwards you can look at more powerful algorithms like xgboost, regression trees, etc.


Generalising performance of Q-learning agent through self-play in a two-player game (MCTS?) I'm using Q-learning (off-policy TD-control as specified in Sutton's book on pg 131) to train an agent to play connect four. My goal is to create a strong player (superhuman performance?) purely by self-play, without training models against other agents obtained externally.

I'm using neural network architectures with some convolutional layers and several fully connected layers. These train surprisingly efficiently against their opponent, either a random player or another agent previously trained through Q-learning. Unfortunately the resulting models don't generalise well. 5000 episodes seems enough to obtain a high (> 90%) win rate against whichever opponent, but after > 20 000 episodes, they are still rather easy to beat by myself.

To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent, i.e.:

I train a batch of models against a completely random agent (let's call them the generation one)
Then I train a second generation of agents against this first generation
Then I train a third generation against generation two
...

So far this helped in creating a slightly stronger/more general connect four model, but the improvement is not as good as I was hoping for. Is it just a matter of training enough models/generations or are there better ways for using Q-learning in combination with self-play?

I know the most successful techniques (e.g. alpha zero) rely on MCTS, but I'm not sure how to integrate this with Q-learning? Neither how MCTS helps to solve the problem of generalisation?

Thanks for your help!

To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent,

This seems quite a reasonable approach on the surface, but possibly the agents will still lose generalisation if the solutions in each generation are too similar. It also looks like from your experiment that learning progress is too slow.

One simple thing you could do is progress through the generations faster. You don't need to train until agents win 90% of games before upping the generation number. Yuo could set the target as low as 60% or even 55%.

For generalisation, it may also help to train against a mix of previous generations. E.g. if you use ten opponents, have five from previous generation, two from each of two iterations before that, and one even older one.

Although the setup you have created plays an agent you are training against another agent that you have created, it is not quite self-play. In self-play, an agent plays against itself, and learns as both players simultaneously. This requires a single neural network function that can switch its evaluation to score for each player - you can either make it learn to take the current player into account and make the change in viewpoint itself, or in zero-sum games (which Connect 4 is one) it can be more efficient to have it evaluate the end result for player 1 and simply take the negative of that as the score for player 2. This is also equivalent to using $\text{max}_a$ and $\text{argmax}_a$ for player 1's action choices and  $\text{min}_a$ and $\text{argmin}_a$ for player 2's action choices - applying the concept of minimax to Q learning.

You can take minimax further to improve your algorithm's learning rate and performance during play. Essentially what Q learning and self-play does is learn a heuristic for each state (or state/action pair) that can guide search. You can add search algorithms to your training and play in multiple ways. One simple approach during training is to perform some n-step look ahead using negamax with alpha-beta pruning (an efficient variant of minimax in zero-sum games), and if it finds the end of the game:

when training, use the result (win/draw/lose) as your ground truth value instead of the normal Q-learning TD target.
when evaluating/playing vs human, prefer the action choice over anything the Q function returns. In practice, only bother with the Q function if look-ahead search cannot find a result.

In the last few months, Kaggle have been running a "Connect X" challenge (which is effectively only Connect 4 at the moment). The forums and example scripts (called "Kernels") are a good source of information for writing your own agents, and if you choose to compete, then the leaderboard should give you a sense for how well your agent is performing. The top agents are perfect players, as Connect 4 is a solved game. I am taking part in that competition, and have trained my agent using self-play Q-learning plus negamax search as above - it is not perfect, but is close enough that it can often beat a perfect playing opponent when playing as player 1. It was trained on around 100,000 games of self-play as I described above, plus extra training games versus previous agents.

  I know the most successful techniques (e.g. alpha zero) rely on MCTS, but I'm not sure how to integrate this with Q-learning? Neither how MCTS helps to solve the problem of generalisation?

MCTS is a variant of search algorithm, and could be combined with Q-learning similarly to negamax, although in Alpha Zero it is combined with something more like Actor-Critic. The combination would be similar - from each position in play use MCTS to look ahead, and instead of picking the direct action with the best Q value, pick the one with the best MCTS score. Unlike negamax, MCTS is stochastic, but you can still use its evaluations as ground truth for training.

MCTS does not solve generalisation issues for neural networks, but like negamax it will improve the performance of a game-playing agent by looking ahead. Its main advantage over negamax in board games is a capability to scale to large branching factors. MCTS does work well for Connect 4. Some of the best agents in the Kaggle competition are using MCTS. Howver, it is not necessary for creating a "superhuman" Connect 4 agent, Q-learning plus negamax can do just as well.


Is training a CNN object detector on an image containing multiple targets that are not all annotated will teach it to miss targets? I want to train a convolutional neural network for object detection (say YOLO) to detect faces. Consider this image:

In this training image, I have many people, but only 2 of them are annotated. Is having this kind of images (where target classes are not all annotated) will train the network to ignore positives?
If yes, are there any techniques to solve the issue apart from annotating the data (I don't have enough resources to do that).

The neural network will learn what we teach it, for example with that image only, after finish training, your model will hard to recognize humans with dark skin, glasses, big eyes, etc, the features that two annotated targets don't have.
If your data is big enough, and contain all the feature of humans face, the result should be good.
If not, I recommend a semi unsupervised learning method which called Noisy Student. Quick explanation, you take a part of data, add noise (augmentation, drop-out, stochastic depth), and train. Then use that model to label the rest of the dataset and train a new bigger teacher (such as YOLOv3 > YOLOv1) and repeat it. They told that this method was even better than you labeled all the dataset.
You will need to choose the data to train the teacher really carefully, then pray for the result.


How can I implement the reward function for an 8-DOF robot arm with TRPO? I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to write the code for the reward functions and the algorithm for the joint space motion.

The most important part of the RL is a reward function. If we want agent to do something speciﬁc, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour.
Assume, the robot's goal is to reach desired position as fast as possible. You can construct you reward function so, that it will take into account the Euclidean distance to the position. If the arm moves to the position directly, you will reward the agent with a positive value, otherwise you will punish it with a deviation from the direct line. You probably have other parameters of the joints, such as position and velocity. It can be also included in your reward function, in order to find optimal movements.

Check out this video from the free udacity overview course on RL and this paper "Setting up a Reinforcement Learning Task with a Real-World Robot"
Here is also related DeepMind's article and paper
I also have a project on github, where I implemented custom Gazebo environment for OpenAI Gym. This allows you to run the test even on a Jupyter Notebook. Check out my example


Is there a way to define the boundaries of the optimal size of a training set? At a related question in Computer Science SE, a user told:

  Neural networks typically require a large training set.

Is there a way to define the boundaries of the "optimal" size of a training set in general case?

When I was learning about fuzzy logic, I've heard some rules of thumb that involved examining the mathematical composition of the problem and using that to define the number of fuzzy sets.

Is there such a method that can be applicable for an already defined neural network topology?

For a finite value to be 'optimal,' typically you need some benefit from more paired up with some cost for more, and eventually the lines cross because the benefit decreases and the cost increases.

Most models will have a reduction in error with more training data, that asymptotically approaches the best the model can do. See this image (from here) as an example:

The costs of training data are also somewhat obvious; data is costly to obtain, to store, and to move. (Assuming model complexity stays constant, the actual cost of storing, moving, and using the model remains the same, since the weights in the model are just being tuned.)

So at some point the slope of the error-reduction curve becomes horizontal enough that more data points are costlier than they're worth, and that's the optimal amount of training data.


GPT-2: (Hardware) requirements for fine-tuning the 774M model I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts.
I'm running TensorFlow 1.14.0 with CUDA V 9.1 on Ubuntu 18.04. For fine-tuning I'm using gpt-2-simple.
When fine-tuning using the 77M model, I keep running into OOM errors, such as:
W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.25MiB (rounded to 6553600).  Current allocation summary follows.
So far I've tried:

Using different a optimizer (RMSPropOptimizer instead of AdamOptimizer)
Setting batch-size to 1
use_memory_saving_gradients
only_train_transformer_layers

Fine-tuning works smoothly on the 355M model.
So what I'm really asking is:

is it possible to fine-tune GPT-2's 774M model without industrial-sized hardware?
if so, please tell me about your successful attempts
apart from hardware-recommendations, how could fine-tuning be optimized to make 77M fit in memory?

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 easily takes more than 10GB of VRAM. You might be able to run GPT-2 774M if you run it in FP16.
Alternatively, you can use Google Collab TPUs which provide at 11GB+ VRAM. Here's a good source listing a few posts about fine tuning GTP-2 1.5B on Google Collab TPUs: https://news.ycombinator.com/item?id=21456025
And here's the notebook itself demonstrating the process: https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a


Can anyone find the source code for the chatbot Luna? This AI is really human-like and allegedly doesn't give pre-programmed responses. It's makers Robots without Borders say the project is open source but I couldn't find the code anywhere.

You can't find the source code because it doesn't exist. The whole thing was a 2015 kickstarter scam, and now it's a patreon scam. All the tech "demos" are obviously pre-scripted videos.

There was actually a question on Quora about it, in which one of the answerers managed to find, among other things, the question and answer that his fake AI gave in one of the videos.


Preventing bias by not providing irrelevant data This seems like such a simple idea, but I've never heard anyone that has addressed it, and a quick Google revealed nothing, so here it goes.

The way I learned about machine learning is that it recognizes patterns in data, and not necessarily ones that exist -- which can lead to bias. One such example is hiring AIs: If an AI is trained to hire employees based on previous examples, it might recreate previous, human, biases towards, let's say, women.

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Again, this seems like such an easy topic, so I apologize if I'm just being ignorant. But I have learned a bit about AI and machine learning specifically for some time now, and I'm just surprised this hasn't ever been mentioned, not even as a "here's-what-won't-work" example.

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Yes. The AI/ model still can learn those prejudiced connections. Consider that you have a  third variable which is a confounding variable or has spurious relationship that is correlated with the bias variable (BV) and the dependent variable (DV). And, the analyst removed the BV but failed to remove the third variable from the data that is fed to the model. Then the model will learn the relationships the analyst didn't want it to learn.

But, at the same time the removal of the variables could lead to omitted variable bias, which occurs when a relevant variable is left out.

Ex:

Suppose that the goal is prediction of salary ($S$) of an individual and the independent variables are age ($A$) and experience ($E$) of the individual. The analyst wants to remove the bias that could come in because of age. So, she removes age from one of the models and comes up with two competing linear models:

$S = \beta_0 + \beta_1E + \varepsilon$

$S = \beta_0 + \beta_1^*E + \beta_2A + \varepsilon$

Since, experience is highly correlated with age, in presence of age in the model, it is very likely that $\beta_1^* < \beta_1$. $\beta_1$ will be a bogus estimate of a person's experience on salary as the first model suffers from the omitted variable bias.

At the same time the predictions from the first model would be reasonably good although the second model is very likely to beat the first model. So, if the analyst wants to remove any 'bias' that might come in because of age i.e. $A$ she must also remove $E$ from the model.


How do I know when to use which Monte Carlo method? I'm a bit confused with extensive number of different Monte Carlo methods such as:

Hamiltonian/Hybrid Monte Carlo (HMC),
Dynamic Monte Carlo (DMC),
Markov chain Monte Carlo (MCMC),
Kinetic Monte Carlo (KMC),
Dynamic Monte Carlo (DMC)
Quasi-Monte Carlo (QMC),
Direct Simulation Monte Carlo (DSMC),
and so on.

I won't ask for the exact differences, but why are all of them called Monte Carlo? What do they all have in common? Can they all be used for AI? E.g. which one can be used for gaming (like Go) or image recognition (resampling)?

They are all called Monte Carlo because all of them are a different version of the canonical Monte Carlo algorithm.

The canonical version of Monte Carlo algorithm is a stochastic algorithm to determine an action based in a tree representation. The differences among all these version are their exploration and exploitation mechanisms, and it is necessary to analyse each of them to define which one fits in your case.


Can an evolving AI be programmed not to lie? It seems that certain AI are able to lie or deceive. See e.g. Evolving Robots Learn To Lie To Each Other (Popsci), Robots Evolve to Deceive (MIT Tech Review) or Robots 'Evolve' the Ability to Deceive (MIT Tech Review).

However, can evolving AI be programmed in such a way that they don't lie?

If a machine learning based AI is "sufficiently smart enough" to be able lie then there is nothing preventing it from lying. This does not mean it can't be persuaded from lying.

So just make the AI simple enough to not be able to lie.

The reasoning here is that in order for a system to be able to lie, a system must be able to recognize an incentive to lie. Recognizing this incentive is a challenging function and would be impossible to code manually into a computer. Machine leaning can be applied to problems such as these where the function is hard to code manually. Although there has been promising work on understanding what the representations/features learned by machine learning actually represent, it may not be possible in general to have an understanding of what a lie in the agent's representation looks like. Because of this, having a hand coded rule to catch when an agent is lying is not possible and thus being able to prevent an agent from (or catch an agent when) lying isn't possible when using machine learning.


Obtain the most important input data for binary classification on a neural network I have a simple neural network for a binary classification.
Input features include:
age, sex, economic_situation, illness, disability, etc.
Output is simply 1 and 0
I would like to order the features for each input from greatest to least impact it had on the classification.
An example answer could look like this:
Classification: 1
1) illness
2) economic_situation
3) disability
4) sex
5) age

Another example:
Classification: 0
1) economic_situation
2) age
3) disability
4) sex
5) illness

Two popular methods I’ve seen done:

1) For each feature, remove it and run the model and see the impact it has on the result. The idea is that the larger the impact, the more pertinent it was to the result.

2) Look at the gradients magnitude $|\nabla_f {y} |$. You can either look at the raw gradient or look at the guided back-propagation which is just the back props  product rule, but you only look at when the nodes positively help trigger a neuron by taking only the positive gradients at each step.

There’s probably also more methods. Hope this helped.


What is the best algorithm to solve the regression problem of predicting the number of languages a Wikipedia article can be translated to? I'm doing a student project where I construct a model predicting the number of languages that a given Wikipedia article is translated into (for example, the article TOYOTA is translated into 93 languages). I've tried extracting basic info (article length, number of links, etc.) to create a simple regression model, but can't get the $R^2$ value above $0.25$ or so.
What's the most appropriate NLP algorithm for regression problems? Almost all examples I find online are classification problems. FYI I'm aware of the basics of NLP preprocessing (tokenization, lemmatization, bag of words, etc).

I think it's difficult to tell wich algorithm is "the best" or "the simplest".
I had the same issue of choosing the suited NLP algorithm for my dataset and I used :
https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html
Then I recommanded you to test many algorithms as you can to find the best for your needed.


Are current AI models sufficient to achieve Artificial General Intelligence? I read an interesting essay about how far we are from AGI. There were quite a few solid points that made me re-visit the foundation of AI today. A few interesting concepts arose:

  imagine that you require a program with a more ambitious
  functionality: to address some outstanding problem in theoretical
  physics — say the nature of Dark Matter — with a new explanation that
  is plausible and rigorous enough to meet the criteria for publication
  in an academic journal.

  Such a program would presumably be an AGI (and then some). But how
  would you specify its task to computer programmers? Never mind that
  it’s more complicated than temperature conversion: there’s a much more
  fundamental difficulty. Suppose you were somehow to give them a list,
  as with the temperature-conversion program, of explanations of Dark
  Matter that would be acceptable outputs of the program. If the program
  did output one of those explanations later, that would not constitute
  meeting your requirement to generate new explanations. For none of
  those explanations would be new: you would already have created them
  yourself in order to write the specification. So, in this case, and
  actually in all other cases of programming genuine AGI, only an
  algorithm with the right functionality would suffice. But writing that
  algorithm (without first making new discoveries in physics and hiding
  them in the program) is exactly what you wanted the programmers to do!

The concept of creativity seems like the initial thing to address when approaching a true AGI. The same type of creativity that humans have to ask the initial question or generate new radical ideas to long lasting questions like dark matter. Is there current research being done on this? I've seen work with generating art and music, but it seems like a different approach.

  In the classic ‘brain in a vat’ thought experiment, the brain, when
  temporarily disconnected from its input and output channels, is
  thinking, feeling, creating explanations — it has all the cognitive
  attributes of an AGI. So the relevant attributes of an AGI program do
  not consist only of the relationships between its inputs and outputs.

This is an interesting concept behind why reinforcement learning is not the answer. Without input from the environment the agent has nothing to improve upon however with the actual brain if you had no input or output it is still in a state of "thinking".

I am focusing on what you have posted here without going through (or having read) the whole essay you linked:

In the classic ‘brain in a vat’ thought experiment, the brain, when temporarily disconnected from its input and output channels, is thinking, feeling, creating explanations — it has all the cognitive attributes of an AGI. So the relevant attributes of an AGI program do not consist only of the relationships between its inputs and outputs.

The classic brain in a vat experiment does not disconnect the brain from all inputs and outputs but replaces "real" connections with its environment with "fake" connections, e.g. by connection the brain to a computer. This is what Wikipedia says:

In philosophy, the brain in a vat (BIV; alternately known as brain in a jar) is a scenario used in a variety of thought experiments intended to draw out certain features of human conceptions of knowledge, reality, truth, mind, consciousness, and meaning. It is an updated version of René Descartes's evil demon thought experiment originated by Gilbert Harman.1 Common to many science fiction stories, it outlines a scenario in which a mad scientist, machine, or other entity might remove a person's brain from the body, suspend it in a vat of life-sustaining liquid, and connect its neurons by wires to a supercomputer which would provide it with electrical impulses identical to those the brain normally receives.2 According to such stories, the computer would then be simulating reality (including appropriate responses to the brain's own output) and the "disembodied" brain would continue to have perfectly normal conscious experiences, such as those of a person with an embodied brain, without these being related to objects or events in the real world.

Following that, the argument of the author referring to the brain in a vat scenario does not hold.
If you ignore that problem for a moment the next inconsistency arises: The author assumed that because a brain would act "so and so" an AGI would need to act "so and so" as well. That however is not in line with how artificial intelligence is usually defined. Even if you consider the broad range of definitions:

(source: "Artificial Intelligence: A Modern Approach"; Russel, Norvig, 3rd Ed, 2010)
None of these definitions directly references AI to the brain. Therefore, the assumption that an AGI (as a subset of AI) would need to act like a brain in a vat is, without further assumptions,  generally flawed.
Another problem with the author's argumentation lies here:

to address some outstanding problem in theoretical physics — say the nature of Dark Matter — with a new explanation that is plausible and rigorous enough to meet the criteria for publication in an academic journal.
Such a program would presumably be an AGI (and then some).

The assumption that "creativity" (in quotation marks since we actually need to precisely define that in the first place) requires an AGI does not hold either. Let's stick to the authors example related to dark matter:

This is per definition a specialized field of application of an AI. Accordingly, there is not necessarily a need for an AGI since "specialization" is exactly what an AGI does not have.
Moreover, you could make an argument that coming up with new explanations related to dark matter is somewhat related to automated theorem proving. Accordingly, we might already have AI which is, in principle, capable of "solving" this task as of today.


Why is reinforcement learning not the answer to AGI? I previously asked a question about How can an AI freely make decisions?. I got a great answer about how current algorithms lack agency.
The first thing I thought of was reinforcement learning, since the entire concept is oriented around an agent getting rewarded for performing a correct action in an environment. It seems to me that reinforcement learning is the path to AGI.
I'm also thinking: what if an agent was proactive instead of reactive? That would seem like a logical first step towards AGI. What if an agent could figure out what questions to ask based on their environment? For example, it experiences an apple falling from a tree and asks "What made the Apple fall?". But it's similar to us not knowing what questions to ask about say the universe.

Some AI researchers do think RL is a path to AGI, and your intuition about how an agent would need to be proactive in selecting actions to learn about is exactly the area these researchers are now focused on.

Much of the work in this area is focused on the idea of curiosity, and since 2014 this idea has gained a lot of traction in the research community.

So, maybe RL can lead to AGI. We don't know for sure yet.

However, many of the classic arguments against AGI aren't addressed by the RL approach. For instance, if like Searle, you think computers just don't have the right kind of hardware to do thinking, then running an RL algorithm on that hardware isn't going to yield AGI, just ever increasingly robust narrow AI. Ultimately Searle's arguments get into issues of metaphysics, so it isn't clear that there exists any argument that would convince someone like Searle that a particular computer-based technique is AGI-capable.

There are also other arguments. For example, the cognativist school of thought thinks that statistical learning approaches to AI, and in particular, the black-box approaches of statistically-driven RL, are unlikely to lead to general intelligence because they do not engage in the kind of systematic reasoning process that proponents of cognativism assume is necessary for general intelligence. Some more extreme proponents of this school might say that a logical planning algorithm like STRIPS is innately more intelligent than any approach based on deep learning, because it involves sound logical deduction rather than mere statistical calculation. In particular, STRIPS can correctly generalize to any new domain, as long as it is fed the correct sense data, while an RL approach will need to learn how to act there.

So, while there are definitely reasons to be optimistic about RL as a direction for achieving AGI, it's definitely not yet settled.