Is it feasible to use GAN for high-quality image synthesis other than human faces? The famous Nvidia paper Progressive Growing of GANs for Improved Quality, Stability, and Variation, the GAN can generate hyperrealistic human faces. But, in the very same paper, images of other categories are rather disappointing and there hasn't seemed to be any improvements since then. Why is it the case? Is it because they didn't have enough training data for other categories? Or is it due to some fundamental limitation of GAN?

I have come across a paper talking about the limitations of GAN: Seeing What a GAN Cannot Generate.

Anybody using GAN for image synthesis other than human faces? Any success stories?

Generative Adversarial Networks, basically boil down to a combination of a generic Generator and a Discriminator trying to beat each other, so that the generator tries to generate much better images (usually from noise) and discriminator becomes much better at classification. So, no it is not just suited for only synthesis high quality human face synthesis but any image type.

In fact, not only can it be used for any high quality image synthesis, it can work on non-image data types as well (such as text, etc). It all depends on the type of neural network you are using for the discriminator and generator at the end of the day.

http://openaccess.thecvf.com/content_iccv_2017/html/Osokin_GANs_for_Biological_ICCV_2017_paper.html

The above refers to a paper synthesizing cell image through GANs, as I haven't personally used GANs on a practical level.

General blog explaining GANs:
https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/

Human faces are much more frequently tackled, for many reasons, usually that human faces are highly symmetric and have a lot of different features, usually more than other types of images, with the added difficulty of being that we as actual humans are usually good at recognizing faces - making a neural net to fool ourselves, makes it a challenging area of research.

Hope it helped! Do let me know if I'm wrong somewhere.


What is Imagination Learning and Imagination machines? I recently came across a Quora post, where I saw the term "Imagination Learning". It seems to be based on something called "Imagination Machines" (the link is based on a guy's work profile as of now; subject to change).

The only thing that I could find on Internet about it is this paper: Imagination-Augmented Agents for Deep Reinforcement Learning. (But I'm not sure if it's related to that concept.)

Any ideas on this would be appreciated.

The whole idea of Imagination Learning seems to be in its infancy. The author of the response you linked wrote a paper on the subject, but as he notes the paper is more of an outline for "a new overarching challenge for AI." The second link you posted is not directly related to the ideas referenced in the paper on Imagination Learning, other than that both use the idea of imagination in humans as inspiration. In short, there does not seem to be much information on this topic and I suspect it will stay that way until we learn some of the underlying processes that go along with imagination in humans and then maybe Imagination Machines may come to fruition.


Cnn for Combination of both digits and letters(small and capital) Hi I am new to machine learning can anyone suggest open dataset consists of both digits and letters(small,capital)
I want images consisisting of both digits and letters to train my cnn model and make the model recognize the real time images
Can anyone please suggest me that dataset link
Thanks in advance

As for me, the easiest path to what you are asking for is generating them yourself.
An Example
I usually grab some TTF fonts and put them into a directory so that I have variety for character identification.  Begin by importing dependencies and creating a generator function:
from PIL import Image, ImageDraw, ImageFont
from os import listdir

WIDTH       =   200
HEIGHT      =   100
MINX        =    20
MINY        =    20
MAXX        =    WIDTH-60
MAXY        =    HEIGHT-60
MINSIZE     =    24
MAXSIZE     =    48

fonts = [i for i in filter(lambda i:i[-3:]=="ttf", listdir("../data/fonts"))]

def generate_character_image():
    fonts = [i for i in filter(lambda i:i[-3:]=="ttf", listdir("../data/fonts"))]
    charset=list("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
    while 1:
        img = Image.new('RGB', (WIDTH,HEIGHT), color = (255,255,255))
        font = ImageFont.truetype(f'../data/fonts/{fonts[randrange(len(fonts))]}', randrange(MINSIZE,MAXSIZE))
        canvas = ImageDraw.Draw(img)
        where = (randrange(MINX,MAXX), randrange(MINY,MAXY))
        character = charset[randrange(len(charset))]
        coords = canvas.textbbox(where, character, font)
        # The coordinates are the top left corner
        bounding_box = (coords[0], coords[1], coords[2]-coords[0], coords[3]-coords[1])
        #canvas.rectangle(coords, outline=0, width=1)
        canvas.text(where, character, font=font, fill=(0,0,0), anchor="la")
        yield (character, coords, img)

for i in range(5):
    character,coords,image = next(generate_captcha())
    print(f"{character} is in Bounding Box: {coords}")
    display(image)
    print(image)

You can then use this to generate an infinite number of training samples pretty trivially.
Why This Might Be Good
Personally, I prefer to generate my datasets whenever possible.  It allows me to do the following:

Train without ever repeating a sample
The validation data changes every time (in fact, I'll often not bother with a validation step since every epoch uses different data)
Zero storage required

Hope this helps!


Would AI be appropriate for converting unstructured text into an XML? I need to understand whether it is better to use AI algorithms (ML, DL, etc.) instead of the classic parser (based onto grammars with regular expression and automaton) for the following task: structuring in XML an unstructured plain text.

The text is a legal document, so the structure is well defined and a classic parser could do a good job.

In the case AI could be a viable way, what would be an appropriate approach for the task?

A rule-based approach will guarantee a correct result, and it works perfectly fine. On the other hand, an AI based approach will introduce errors as AI cannot produce result with 100% accuracy, and also decreasing speed. As the document you are processing is a legal document, it would be better to use a parser as AI is only adding unnecessary wasted time and non-accurate results.

Hope I can help you.


Is it possible to use the GPT-2 model for time-series data prediction? Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.


Could the deployment of self-driving cars make rides free? As self-driving technology is improving, there are so many companies developing  self-driving cars like Google, Uber, etc. Is it possible that we won't need any private/paid self-driving cars and the "self-driving taxi" becomes ubiquitous in the city? If we assume that there are such taxis everywhere, would transportation become extremely low cost or free? (The self-driving car company could benefit from broadcasting advertisements for advertising agencies.)

You ask an interesting question.  There have been many discussions by industry on this topic.  A company called Vugo has their business model based on advertising to passengers.  An article about Vugo, "The Quest to Make Ridesharing Free", states:

Flessner is co-founder and CEO of Minneapolis-based rideshare advertising platform Vugo, which uses its patented TripIntent technology to display targeted advertisements to Uber and Lyft passengers on a tablet attached to the back of the vehicle’s headrests.
The company’s founders predict that within the next few years, all ridesharing vehicles in the U.S. will be driverless, and in-vehicle advertising tailored to passengers’ destinations and interests will lead to free transportation. “The idea is to put brands in front of passengers who are en route to make purchases,” says Flessner. For example, retailers will sponsor transportation to their stores so they can preview products to customers who are headed their way.

This article "How free self-driving car rides could change everything" makes these comments:

Car data is so lucrative that Ben Volkow -- CEO of otonomo, an Israeli startup that sells vehicle data -- expects automakers to make more money selling data than vehicles by 2020.
If the money made off self-driving vehicle data outweighs the costs of offering rides, then it becomes reasonable for a business to offer free rides broadly.

A paper entitled "Leveraging Adverts in the Coming
Autonomous Car Eco-system" published by Berkeley, University of California, proposes "a world where rideshares are free, or in some cases, heavily subsidized through the use of advertisements".


Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA) I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results shown in Figure 1 (a) in [1] (with the same hyperparameters). Here is the result I got:

The issue I found is that the learning procedure is stuck due to bad PBA in the early stages of training. Without PBA, Sarsa can converge well.

Did anyone try the method before? I am really puzzled and how the authors obtain these good results? There are some top conference papers using the same method in [1], for example, [2] and [3].

[1]  Expressing Arbitrary Reward Functions as Potential-Based Advice

[2]  Learning from demonstration for shaping through inverse reinforcement learning

[3]  Policy Transfer using Reward Shaping

Is the method itself defective or anything wrong with my code? Here is part of my codes:

import copy
import numpy as np
import pandas as pd

def expert_reward(s, action):
    if (action == RIGHT) or (action == DOWN):
        return 1.0
    return 0.0

class DynamicPBA:
    def __init__(self, actions, learning_rate=0.1, reward_decay=0.99):
        self.lr = learning_rate
        self.gamma = reward_decay
        self.actions = actions
        self.q_table = pd.DataFrame(columns=self.actions, dtype=np.float64) #q table for current time step
        self.q_table_ = pd.DataFrame(columns=self.actions, dtype=np.float64) #q table for the current time step
        self.check_state_exist(str((0,0)))

    def learn(self, s, a, r, s_, a_): #(s,a) denotes current state and action, r denotes reward, (s_, a_) denotes the next state and action
        self.check_state_exist(s_)
        q_predict = self.q_table.loc[s, a]
        q_target = r + self.gamma * self.q_table.loc[s_, a_]
        self.q_table.loc[s, a] = self.q_table.loc[s, a] + self.lr * (q_target - q_predict)

    def update(self):
        self.q_table = copy.deepcopy(self.q_table_)

    def check_state_exist(self, state):
        if state not in self.q_table.index:
            # append new state to q table
            self.q_table = self.q_table.append(
                pd.Series(
                    [0]*len(self.actions),
                    index=self.q_table.columns,
                    name=state,
                    )
                )
            self.q_table_ = self.q_table_.append(
                pd.Series(
                    [0]*len(self.actions),
                    index=self.q_table_.columns,
                    name=state,
                    )
                )

#######Main part

RL = SarsaTable(actions=list(range(len(actions_dict))), reward_decay=0.99, learning_rate=0.05)
expert = DynamicPBA(actions=list(range(len(actions_dict))), learning_rate=0.1, reward_decay=0.99)
for episode in range(100):
    # initial observation
    s = (0,0)
    env.reset(s)
    action = RL.choose_action(str(s))

    r_episode_s = 0
    r_episode = 0

    current_step = 0
    while True:

        # RL take action and get next observation and reward
        s_, _, reward, status = env.step(action)
        current_step += 1

        action_ = RL.choose_action(str(s_))
        # update dynamic potentials
        expert_r = -expert_reward(s, action)
        expert.learn(str(s), action, expert_r, str(s_), action_)

        # compute PBA
        F = expert.gamma * expert.q_table_.loc[str(s_), action_] - expert.q_table.loc[str(s), action]

        #update expert PBA table
        expert.update()

        RL.learn(str(s), action, reward+F, str(s_), action_, status)

        # swap observation
        s = s_
        action = action_

        # break while loop when end of this episode
        if status != 'not_over':
            break
        if current_step>10000:
            print(episode, r_episode, r_episode_s, current_step)
            break
        # learning rate decay
        RL.lr = RL.lr*0.999
#     expert.update()

Is the method itself defective or anything wrong with my code?

There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust their writing too :) ).

The first issue I see, and likely the most important, is that the update() calls of DynamicPBA frequently update the contents of self.q_table to those of self.q_table_, but the contents of self.q_table_ are never updated. So, your q_table is essentially always filled with a bunch of $0$ values that never get a chance to properly learn.

I did not check the rest of the implementation in detail, but at a glance it all looks fine to me. So I guess that changing the last line of learn() to update an entry in self.q_table_ rather than self.q_table should go a long way towards fixing your issue.


How is the reCaptcha useful for Google? I am wondering where Google uses the result from deep learning of reCaptcha (how can a system that knows to recognize street signs is useful somewhere? how they profit from it?)

At this time, Google is aggressively pursuing research into AI of many sorts. All AI at this point is constrained by the size and accuracy of available training data. Getting human curated data is expensive and time consuming. Crowdsourcing through Captcha gets access to that without paying directly.

Why roads specifically? It can hardly be a coincidence that Google's holding company also owns a self driving car company Waymo. They need machines to learn what road signs look like in real life, so that they can respond to them.


Can AlphaFold predict proteins with metals well? There are certain proteins that contain metal components, known as metalloproteins. Commonly, the metal is at the active site which needs the most prediction precision. Typically, there is only one (or a few) metals in a protein, which contains far more other atoms. So, the structural data that we could be used to train AlphaFold will contain far less information about the metal elements. Not to mention most proteins don't have metals at all (it is estimated that only 1/2-1/4 of all proteins contain metals [1]).
Given that maybe there is not enough structural data about protein local structure around metal atoms (e.g. Fe, Zn, Mg, etc.), then AlphaFold cannot predict local structure around metals well. Is that right?
I also think that the more complex electron shell of metal also makes the data less useful, since its bounding pattern is more flexible than carbon, etc.

Let me address first some of the things you wrote in your question:

There are certain proteins that contain metal components, known as
metalloproteins.

Natural proteins do not ever contain metal components as far as we know. Natural proteins are composed of natural amino acids which only contain H,C,O,N,S. Selenocysteine contains Se (also a non-metal!) but it's a proteinogenic amino acid, which means it's a precursor to proteins but doesn't typically show up in the protein itself. From the Wikipedia page that you gave us in your question: "Metalloprotein is a generic term for a protein that contains a metal ion cofactor" and "A cofactor is a non-protein chemical compound or metallic ion".
But your question still deserves an answer, because even though it would incorrect to call a co-factor "part" of the protein, they can still affect the folding and the overall shape. But let's first address one last part of your question:

Given that maybe there is not enough structural data about protein
local structure around metal atoms (e.g. Fe/Zn/Mg), then AlphaFold
cannot predict local structure around metals well. Is that right?

The first sentence of the Wikipedia article that you linked in your question, says "For instance, at least 1000 human proteins (out of ~20,000) contain zinc-binding protein domains [3] although there may be up to 3000 human zinc metalloproteins [4]." Therefore, while metalloproteins might not be the majority of all proteins, there's a decent enough number of them that are of relevance to the human body, and therefore constructing training databases that contain enough metalloproteins (or even 100% metalloproteins, if desired) is not difficult.
I mentioned a bit elsewhere that AlphaFold was used to predict protein structures in the CASP competition, for which you can see for yourself that many/most of the proteins for which contestants (such as DeepMind) need to predict the structure, come from studies of proteins of relevance to humans because the CASP structures typically come from X-ray crystallography studies, which are typically done on proteins of relevance to humans.
You can also see for yourself not only the "target list" that I showed above, but also the results of the competition which will show how well AlphaFold performed in CASP13 (2018) and CASP14 (2020) for metalloproteins.
Finally:

I also think that the more complex electron shell of metal also makes
the data less useful, since its bounding pattern is more flexible than
carbon, etc.

It is true that metals are typically harder to model than C,H,O,N,S, and even Se, if doing ab initio calculations on the metals or metail-containing complexes. However, the purpose of machine learning in protein folding studies is to skip ab initio, statistical-dynamical and/or molecular-dynamical calculations of the relevant structures and simply use training data to predict the protein structures. That being said, there needs to be enough training data available (as you correctly pointed out) to learn what happens near the metal co-factors: The answer to this is that there are indeed enough metalloproteins to sufficiently populate a training set, but they won't contain enough of the specific metals involved in every metalloprotein. For example, lots of data will be available for proteins containing Fe since Fe is in hemoglobin (for example) which is essential to the functioning of red blood cells to absorb oxygen, but the protein vanabins contains vanadium which is much rarer and therefore training data involving it will be much less available. You're also correct that metal elements can form more bonds than typical elements found in organic compounds.
So it depends on the metal in the relevant co-factor. Fe-based co-factors will have quite a lot of training data available, as well as Mg-based ones, Zn-based ones, and a lot of other ones which contain the "more common" metals. For proteins like vanabins which contains vanadium, you are quite correct that training data will be limited, but also keep in mind that vanabins is a very rare protein found in sea squirts and we already know more about its structure (through X-ray crystallography, which means we don't need machine learning for it) than we even know about what it even does. The chances of other vanadium-containing co-factors in metalloproteins being very significant is too low to justify working on protein folding algorithms specifically for them.


How does one start learning artificial intelligence? I am a software engineering student and I am complete beginner to AI. I have read a lot of articles on how to start learning AI, but each article suggests a different way. I was wondering if some of you experts can help me get started in the right way.

A few more specific questions

Which language should I focus on? A lot of articles suggest Python, C++ or Lisp for AI. Can I use Java instead of any of the other languages mentioned?
What kind of mathematical background should I have? During the first year, I did discrete mathematics, which included the following topics: sets, matrices, vectors, functions, logic and graph theory (They taught these topics briefly). Are the are there any more topics that I should learn now? For example, calculus?

If possible, I would appreciate any resources or books I could use in order to get started, or maybe you guys can give me a detailed procedure I can follow in order to catch up with to your level.

Note: For now I would like to focus on neural networks and machine learning. After I that I would like to explore robotics and natural language processing.

What is artificial intelligence?

Artificial Intelligence is a very broad field and it covers many and very deep areas of computer science, mathematics, hardware design, and even biology and psychology.

What math do you need?

As for math, I think calculus, statistics, and optimization are the most important topics, but learning as much math as you can won't hurt.

Free resources for beginners

There are many good free introductory resources about AI for beginners.
I highly recommend starting with this website AI Playbook, which is intended to help newcomers (both non-technical and technical) begin exploring what's possible with AI. They also published two videos about the general concepts of AI, you can find them on Youtube AI, Deep Learning, and Machine Learning: A Primer and The Promise of AI.

What is your goal?

Once you have a clear understanding of the basic AI terms and approaches, you have to figure out what your goals are. What kind of AI software do you want to develop? What industries are you interested in? What are your chances to get involved in projects of big companies? It's easier to pick up the right tools when you know exactly what you want to achieve.

AI, machine learning and deep learning

For most newcomers to AI the most interesting area is deep learning.
Just to make it clear, there are many areas of AI outside of machine learning and there are many areas of machine learning outside of deep learning.

  Artificial Intelligence > Machine Learning > Deep Learning

Most of the recent developments and hyped news are about DL.

Neural networks

If you got interested in deep learning too, you have to start with learning about the concepts of artificial neural networks. Fortunately, it's not too difficult to understand the basics and there are lots of tutorials, code examples and free learning resources on the web and there are many open-source frameworks to start experimenting with.

Frameworks and libraries

Python

The most popular such deep learning framework is TensorFlow. It's backed by Google. Love it or hate it, it's a Python-based framework. There are many other Python-based frameworks and libraries, as well. Scikit-learn, theano, Keras are frequently mentioned in tutorials too. (A tip: if you use Windows you can download WinPython that includes all of these frameworks.)

Java

As for Java frameworks, unfortunately, there are not so many options. The most prominent Java framework for DL is Deeplearning4j. It's developed by a small company and its user base is much smaller than the crowd around TensorFlow. There are fewer projects and tutorials for this framework. However, industry specialists say Java-based frameworks eventually integrate better with Java-based big data solutions and they may provide a higher level of portability and easier product deployment. Just a sidenote: NASA's Jet Propulsion Laboratory used Deeplearning4j for many projects.

Resources

If you decide to go with the flow and want to start learning more about TensorFlow, I recommend you check out the YouTube channels of DeepLearning.TV, sentdex and Siraj Raval. They have nice tutorials and some cool demos. And if you decide to take a deeper dive, you can sign up for an online course at Udacity or Coursera.

Other programming languages

It also may be interesting to you to know that there are other deep learning frameworks for the Java Virtual Machine with alternative languages, for example, Clojure. (Clojure is a dialect of LISP and it was invented by John McCarthy, the same computer scientist who coined the term "artificial intelligence". In other words, there are more modern and popular programming languages and tools, but it's still possible, and kinda cool, to use the language for AI that was originally designed for AI. ThinkTopic in Boulder and Freiheit in Hamburg are two companies that use Clojure for AI projects. And if you want to see something awesome to get inspiration to use Clojure in AI and robotics, I recommend you to check out the YouTube video OSCON 2013: Carin Meier, The Joy of Flying Robots with Clojure. (Mentioning Clojure in this answer was just an example to show you there is life outside of the bubble of Python-based AI frameworks.)


What happens if there is no activation function in some layers of a neural network? What if I don't apply an activation function on some layers in a neural network. How will it affect the model?
Take for instance the following code snippet:
def model(x):
    a = Conv2D(64, (3, 3))(x)
    x = Conv2D(64, (3, 3), activation = 'relu')(x)
    b = Conv2D(128, (3, 3))(x)
    x = Conv2D(128, (3, 3), activation = 'relu')(b)
    return x, a, b

If you do not specify an activation for a layer you are effectively creating a linear transformation through that layer.  From the documentation:

activation: Activation function to use. If you don't specify anything, no activation is applied (see keras.activations).


In 2016, can $1000.00 buy enough operations per second to be approximately equal to the computational power of a human brain? In The Age of Spiritual Machines (1999), Ray Kurzweil predicted that in 2009, a \$1000 computing device would be able to perform a trillion operations per second. Additionally, he claimed that in 2019, a \$1000 computing device would be approximately equal to the computational ability of the human brain (due to Moore's Law and exponential growth.)
Did Kurzweil's first prediction come true? Are we on pace for his second prediction to come true? If not, how many years off are we?

Edit (12.19.2020)
According to the answer here, estimates of the processing power of the human brain range at least as widely as from $10^{17}$ flops to $10^{28}$ flops.

The development of CPUs didn't quite keep up with Kurzweil's predictions. But if you also allow for GPUs, his prediction for 2009 is pretty accurate.
I think Moore's law slowed down recently and has now been pretty much abandoned by the industry. How much that will affect the 2019 prediction remains to be seen. Maybe the industry will hit its stride again with non-silicon based chips, maybe not.
And, of course, whether hitting Kurzweil's estimate of the computing power of the human brain will make an appreciable difference for the development of Artificial General Intelligence (AGI) is another question altogether.


Why is automated theorem proving so hard? The problem of automated theorem proving (ATP) seems to be very similar to playing board games (e.g. chess, go, etc.): it can also be naturally stated as a problem of a decision tree traversal. However, there is a dramatic difference in progress on those 2 tasks: board games are successfully being solved by reinforcement learning techniques nowadays (see AlphaGo and AlphaZero), but ATP is still nowhere near to automatically proving even freshman-level theorems. What does make ATP so hard compared to board games playing?

There are two ways to look at the problem, one in terms of logic and the other in terms of psychology.
To get any -start- on automation of mathematics, you need to formalize the part you want. It has only been since the early part of the 20th c that most day to day math had been formalized with logic and set theory. And even though Gödel's incompleteness theorems say (very loosely) that there is no algorithm to decide theorem-hood for mathematical statements (that include a theory of arithmetic), that still leaves a lot of math that -can- be decided. But that has taken the Reverse Mathematics program (still ongoing) to say specifically what subsets of math are decidable or to what degree (what logical assumptions are necessary) they are undecidable.
So theorems in arithmetic of just '+' (that is, dropping '*') can be decided, Euclidean geometry can be decided, single variable differential calculus can be decided but not single variable integral calculus. These examples show that what we know to be decidable is pretty elementary. And most of the things we care about are very un-elementary (almost by definition).
As to psychology, the theorems and proofs that you learn in mathematics classes are nowhere near like their formalizations. Most mathematicians aren't pushing symbols around in their heads like a computer does. A mathematician is more like an artist, visualizing dreams and connecting metaphors just on their barely conscious images borne out of repetition. That is, machines and mathematicians just work on different representations (despite what non-mathematicians might imagine).

To address your specific question, yes, mathematical theorems and the systems to prove them are very similar in a technical sense. Games (often, not always) can be modeled as trees, And likewise proofs can often be modeled as trees. Without writing you a library of books about games and proofs, let's just say that the mathematical proofs that are like games that are won by Alpha Zero are not for particularly interesting theorems. Winning a game of go is more like proving that a very very large boolean formula. Most mathematical theorems require a lot of ingenuity in introducing steps in their proof trees. It may be mechanical after the fact to check that a proof is correct, but discovering the proof almost needs magic to come up with a step in the game. Sure, some things in math are automatable (as mentioned before, derivatives), but some mathematical systems (such as integration) are provably impossible to find proofs of all true statements.
Another difference between theorem proving and games is that proofs have to be air tight on all paths, whereas with games one side just has to eke out a single win over the other side.

A separate issue entirely that may contribute to the difficulty is that we just may not yet have the tooling available, ie editors, notation, proof assistants that make it easy to do what should be easy. Or it could just be that mathematicians don't have the fluency with theorem proving systems.
Or it could be that if there were automated theorem provers good enough, mathematicians just wouldn't care too much for them because they'd take away the fun of finding the proofs themselves.


How to fine tune BERT for question answering? I wish to train two domain-specific models:

Domain 1: Constitution and related Legal Documents
Domain 2: Technical and related documents.

For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.
Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?
If yes, what are the resources to achieve this task?
Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.

The answer is yes but 'lightweight' will require a 'lightweight' model.
Your application for 'domain one' is called open domain question answering (ODQA).  Here is a demonstration of ODQA using BERT: https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
Your application for 'domain two' is a little different.  It is about learning sequences from sequences.  More specifically these are called sequence to sequence models.  Here is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
In both applications, the resources required are going to rely on the performance you require.  There are many sizes of BERT models.  Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.).  For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.
This article, NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.
There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones.  Here is an article on using that with SQuAD: https://www.tensorflow.org/lite/models/bert_qa/overview
cdQA-suite is a good package.  The following should help in fine-tuning on your own corpus:

How to create your own Question-Answering system easily with python
cdQA examples


What exactly is a hidden state in an LSTM and RNN? I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.
Does this make sense?
I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know.

Lets take this simple sequence first,

   X = [a,b,c,d,.......,y,z]
   Y = [b,c,d,e,.......,z,a]

Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details doesn't matter.

We can write this relationship in maths as

  $f(x)\rightarrow y$

where x is an element of X and y is an element of Y and f() is our neural network.

After training, if given input x = a our neural network will give an output b because f() learned the mapping between the sequence X and Y.

Now instead of the above sequence try teach this sequence to the same neural network.

 X = [a,a,b,b,c,c,.........y,z,z]
 Y = [a,b,c,,...z,a,b,c......y,z]

More than likely this neural netwok will  not be able to learn the relationship between X and Y. This is because a simple neural network can't learn and understand the relationship between the previous and current characters.

Now we train the same sequence in RNN, in RNN we take two inputs one for our input and a previous hidden values and two outputs one for output and next hidden values

  $f(x,h)\rightarrow (y,h+t)$

Important: here  h+t represents next hidden value and not the arithematic addition with time step t.

We will execute some sequence of this RNN model, at the start hidden values are considerd as zeros

   x = a and h = 0
   f(x,h) = (a,next_hidden)
   prev_hidden = next_hidden

   x = a and h = prev_hidden
   f(x,h) = (b,next_hidden)
   prev_hidden = next_hidden

   x = b and h = prev_hidden
   f(x,h) = (c,next_hidden)
   prev_hidden = next_hidden

   and so on

If we look at the above process we can see that we are taking previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.


Dropout causes too much noise for network to train I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:

The following runs were seeded.

key: dropout = 0.3, means 30% chance of dropout

graph x axis: iteration

y axis: error

dropout=0

dropout = 0.001

dropout = 0.1

dropout = 0.5

I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:

3x3x3                    Input image
3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
20x1x1                   Flatten layer: 27 -> 20
20x1x1                   Fully connected layer: 20
10x1x1                   Fully connected layer: 10
2x1x1                    Fully connected layer: 2

But I have tested a CNN with architecture:

10x10x3                  Input image
9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
8x8x12                   Max pooling layer: 2x2, stride = 1
6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
5x5x24                   Max pooling layer: 2x2, stride = 1
300x1x1                  Flatten layer: 600 -> 300
300x1x1                  Fully connected layer: 300
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2

overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:

activation = relu(val)*(random.random() > self.dropout)

then at test time:

activation = relu(val)*(1-self.dropout)

why dropout of 0.5 effectively kills the networks ability to train.

because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25%  remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network.
Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers.  And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.


How is simulated annealing better than hill climbing methods? In hill climbing methods, at each step, the current solution is replaced with the best neighbour (that is, the neighbour with highest/smallest value). In simulated annealing, "downhills" moves are allowed.

What are the advantages of simulated annealing with respect to hill climbing approaches? How is simulated annealing better than hill climbing methods?

In the least technical, most intuitive way possible: Simulated Annealing can be considered as a modification of Hill Climbing (or Hill Descent). Hill Climbing/Descent attempts to reach an optimum value by checking if its current state has the best cost/score in its neighborhood, this makes it prone to getting stuck in local optima.

Simulated Annealing attempts to overcome this problem by choosing a "bad" move every once in a while. The probability of choosing of a "bad" move decreases as time moves on, and eventually, Simulated Annealing becomes Hill Climbing/Descent.

If configured correctly, and under certain conditions, Simulated Annealing can guarantee finding the global optimum, whereas such a guarantee is available to Hill Climbing/Descent iff the all local optima in the search space have equal scores/costs.

For more, go through Wolfram Mathworld's entry here.


What is a RAM state in the gym's breakout-ram environment? I have encountered the gym environment and decided to create AI that plays breakout. Here is the link: https://gym.openai.com/envs/Breakout-ram-v0/.

The documentation says that the state is represented as a RAM state, but what is the RAM in this context? Is it the random access memory? What does the RAM state represent?

Yes, it is the state of the memory; this would mainly involve variables, since the code would be in ROM. Since it is only 128 bytes in size, the screen memory would also not be included in this.

The idea is that all information relevant to the game is captured in these 128 bytes; they represent the state of the game world at any given time. Movements of the ball, the game controller position, etc are all encoded there.

For machine learning it is not actually relevant which bytes are representing which value, as any optimisation of outcomes will treat all these 128 bytes as parameters describing the state. A machine learning system will pick up the optimal configuration through the learning process, eg that the 'racket' position should always near the ball x-coordinate. That will just be a correlation of two bytes, no matter which ones they actually are.


Which approach can I use to generate text based on multiple inputs? I have a little experience in building various models, but I've never created anything like this, so just wondering if I can be pointed in the right direction.

I want to create (in python) a model which will generate text based on multiple inputs, varying from text input (vectorized) to timestamp and integer inputs.

For example, in the training data, the input might include:

eventType = ShotMade

shotType = 2

homeTeamScore = 2

awayTeamScore = 8

player = JR Smith

assist = George Hill

period = 1

and the output might be (possibly minus the hashtags):
JR Smith under the basket for 2! 8-4 CLE. #NBAonBTV #ThisIsWhyWePlay #PlayByPlayEveryDay #NBAFinals

or

JR Smith out here doing #WhateverItTakes to make Cavs fans forgive him. #NBAFinals

Where is the best place to look to get a good knowledge of how to do this?

Generally, text generators work by modeling the joint distribution of the text by its Bayesian forward decomposition

$
\begin{align*}
p(w_1, w_2, ..., w_n) &= p(w_1) * p(w_2|w_1) * p(w_3|w_2, w_1) *\ ...\ * p(w_n|\{w_i\}_{i<n})\\
&= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i})\\
\end{align*}
$

From a modeling perspective, this looks right up RNN's ally, where you can have a state holding information from $\{w_k\}_{k<i}$ to learn a representation of $w_i$

Now, in your specific case, you're interested in a conditional text-generator, so you are trying to model $p(w_1, w_2, ..., w_n | \{v_j\}_j)$, but this same tactic works.

$
\begin{align*}
p(w_1, w_2, ..., w_n| \{v_j\}_j) &= p(w_1|\{v_j\}_j) * p(w_2|w_1, \{v_j\}_j) * p(w_3|w_2, w_1, \{v_j\}_j) *\ ...\ * p(w_n|\{w_i\}_{i<n}, \{v_j\}_j)\\
&= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i}, \{v_j\}_j)\\
\end{align*}
$

So, in your RNN or forward-based model, you can use the exact same approach just additionally embed the conditional inputs you have and somehow infuse it into the model (in practice, I have seen this through attention, concatenation, or some other common approach).

My recommendation (depending on the computational power you have) is to take advantage of the recent fad of pre-trained language models. Specifically, ones trained on next word prediction will probably do the job best. A good example is gpt-2, and, if you check out their GitHub, their code is very readable and easy to adjust for adding conditional input in the ways I have described.


Are CRUD webapps today the modern version of the "expert system"? From Wikipedia, citations omitted:

In artificial intelligence, an expert system is a computer system that emulates the decision-making ability of a human expert. Expert systems are designed to solve complex problems by reasoning about knowledge, represented mainly as if–then rules rather than through conventional procedural code. The first expert systems were created in the 1970s and then proliferated in the 1980s. Expert systems were among the first truly successful forms of artificial intelligence (AI) software.
An expert system is divided into two subsystems: the inference engine and the knowledge base. The knowledge base represents facts and rules. The inference engine applies the rules to the known facts to deduce new facts. Inference engines can also include explanation and debugging abilities.

CRUD webapps (websites that allows users to Create new entries in a database, Read existing entries in a database, Update entries within the database, and Delete entries from a database) are very common on the Internet. It is a vast field, encompassing both small-scale blogs to large websites such as StackExchange. The biggest commonality with all these CRUD apps is that they have a knowledge base that users can easily add and edit.
CRUD webapps, however, use the knowledge base in many, myriad and complex ways. As I am typing this question on StackOverflow, I see two lists of questions - Questions that may already have your answer and Similar Questions. These questions are obviously inspired by the content that I am typing in (title and question), and are pulling from previous questions that were posted on StackExchange. On the site itself, I can filter by questions based on tags, while finding new questions using StackExchange's own full-text search engine. StackExchange is a large company, but even small blogs also provide content recommendations, filtration, and full-text searching. You can imagine even more examples of hard-coded logic within a CRUD webapp that can be used to automate the extraction of valuable information from a knowledge base.
If we have a knowledge base that users can change, and we have an inference engine that is able to use the knowledge base to generate interesting results...is that enough to classify a system as being an "expert system"? Or is there a fundamental difference between the expert systems and the CRUD webapps?
(This question could be very useful since if CRUD webapps are acting like "expert systems", then studying the best practices within "expert systems" can help improve user experience.)

The key feature of an expert system is that the knowledge base is structured to be traversed by the inference engine.  Web sites like Stack Exchange don't really use an inference engine; they do full-text searches on minimally-structured data.  A real inference engine would be able to answer novel queries by putting together answers to existing questions; Stack Exchange sites can't even tell if a question is duplicate without human confirmation.


What is the difference between artificial neural network (ANN) and deep learning? I have read many mixed definitions around these two terms. For example, is it right to say deep learning is any ANN with more than two hidden layers?
What are formal definitions for these two?

A few years ago, deep learning was a buzz word, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!).
From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, deep learning typically refers to the subset of machine learning techniques (specifically, gradient descent and back-propagation) applied to neural networks, and, in particular, neural networks with multiple layers. However, there is no consensus on what multiple refers to. In fact, Schmidhuber (co-author of the LSTM), in [1], writes

At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. Instead of committing myself to a precise answer, let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.

In other words, although there is typically no agreement on the definition of deep or multiple, there are several problems (such as the vanishing or exploding gradient problems) that arise as the number of layers in a neural network increases, so there are reasons behind the distinction of deep and non-deep learning.
Note that some research papers (e.g. [1]) distinguish between deep learning applied to neural networks and other types of deep learning, so that not to exclude the layered composition of other models (e.g. SVMs) other than neural networks and the use of machine learning algorithms (e.g. gradient descent) to train them.
To answer your question more directly, neural networks are models or function approximators, while deep learning is the set of machine learning techniques applied to the layered composition of models (typically, neural networks).


Do we need multiple parallel environments to train in batches an on-policy algorithm? When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that to apply batching to an on-policy method you have to have multiple parallel environments?
As an extension of this, if only one environment is available when using on-policy methods, does that mean batching isn't possible? Doesn't that limit the power of such algorithms in certain cases?

We don't need multiple environments.
On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.


DQN input representation for a card game In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory.
The problem is, I can't get the intuition behind how to represent the input to the neural network..

About the game

It's a fairly simple 2-players game. There is a deck of 40 unique cards (4 types of cards, 10 numbered cards in each type).
Each player gets 4 cards and each turn a player must put a card on the table.
If a player puts a card and there is already a card with the same number on the table, the player wins both cards.
If for example a player plays Card 2 and on the table there is Cards 2, 3, 4, 5 then the player wins all those cards (sequence).
Cards won don't go back to the hand nor to the deck, they are just kept as like a score.
When the players have 0 cards in hand, another 4 cards are dealt to each one untile the deck has 0 cards left where then we decide who won based on the number of cards eaten/won.

Question

As the input, I will be using the following:

Current cards in the AI hand (40 one-hot-encoded features?)
Current cards on the table (40 one-hot-encoded features?)
History of played cards (40 one-hot-encoded features?)

This would give 120 columns/features in each state.
I am wondering wheter this is too much for the NN or wheter my input representation would be bad for the NN?
Should the features be represented as a (120,) vector or as a 3x40 matrix?

I am also wondering if it's a good idea to represent the current cards on the table as just a 10 one-hot-encoded features since the type of the cards don't matter and the same number can't exist 2 times in the table?

Thank you in advance.

120 inputs can be handled by a complex enough network. Dealing with high complexity is one of NN's strengths.

Using a (120,) vector or a (3,40) matrix is the same, they're still 120 inputs. Your binary encoding should work. Another option is a single (40,) vector, with 0 being "still in deck", 1 being "in hand", 2 being "on table", 3 being "already played".

If the types of cards are irrelevant, you could actually have a (3,10) matrix with counters of cards (1 in hand, 1 in table, 2 already played). You can try different approaches and see what works best.


What is a landmark in computer vision? I guess I understand the concept of face detection, a technique specifies the location of multiple objects in the image, and draws bounding boxes on the target.

The question is related to the concept of a landmark. For example, the bottom guy in the image above pointed out by the red arrow has 18 green dots on his face. Is anyone of the dots is a landmark?

What is the size of a landmark? What is the acceptable error of its position? For example, the landmark in the middle of his nose has to be in what kind of range?

Could someone please give a hint?

The paper A Brief Introduction to Statistical Shape Analysis (2002) by M. B. Stegmann and D. D. Gomez provides a definition of a landmark in the context of statistical shape analysis, which I will report below.

Definition 1: Shape is all the geometrical information that remains when location, scale and rotational effects are filtered out from an object.

For example, in the following diagram, the shape of a hand, after several transformations (i.e. translation, scaling and rotation), is illustrated. This definition captures the intuitive notion of the shape of an object. In fact, as expected, after these transformations, all of the objects below correspond to a hand.

Definition 2: A landmark is a point of correspondence on each object that matches between and within populations.

In your picture, all of the faces (apart from the occluded one) have 18 green points. Each of these points roughly corresponds to the same position in the face. For example, there's a green point on tip of the nose of each face. In your picture, the populations refer to the faces.
You can divide landmark into three groups

Anatomical landmarks Points assigned by an expert that corresponds between organisms in some biologically meaningful way.

Mathematical landmarks Points located on an object according to some mathematical or geometrical property, i.e. high curvature or an extremum point.

Pseudo-landmarks Constructed points on an object either on the outline or between landmarks.

The landmarks in your picture could be considered anatomical landmarks, so they might have been assigned by an expert (a human), so this should explain why they do not exactly correspond. Moreover, note that e.g. noses have different shapes across humans, so correspondences are unlikely to be exact in most cases.

What is the size of a landmark?

This is just an educated guess, but a landmark is probably just a point (i.e. a coordinate of a pixel in the image), although in your picture they look bigger, but this is probably to make them more noticeable.

What is the acceptable error of its position?

This is also a guess, but this might depend on the application and the used algorithms, i.e., in certain cases, your algorithms may be very sensitive to the position of the landmarks, in other cases, that might not be a big problem.


How can action recognition be achieved? For example, I would like to train my neural network to recognize the type of actions (e.g. in commercial movies or some real-life videos), so I can "ask" my network in which video or movie (and at what frames) somebody was driving a car, kissing, eating, was scared or was talking over the phone.

What are the current successful approaches to that type of problem?

This study from 2012 uses 3D convolutional neural networks (CNN) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning approach based on 3D CNN is demonstrated in the LIRIS and Orange Labs study from 2011.

This Oxford study from 2014 also uses a similar approach, but with two-stream CNN which incorporates spatial and temporal networks which can achieve good performance despite having limited training data. It recognises action from motion in the form of dense optical flow. For example:

Another study from 2007 demonstrates a method by detecting human falls based on a combination of motion history and human shape variation by analysing the video frames. It uses Motion History Image (MHI) to quantify the motion of the person.

Source: harishrithish7/Fall-Detection at GitHub

An alternative general approach could be action detection based on the posture using DNN. See: How to achieve recognition of postures and gestures?


Do neurons of a neural network model a linear relationship? I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past.  Also, my formal mathematics training is more than 30 years in the past, so please be kind. :)

As I'm reading François Chollet's book on Deep Learning with Python, I'm struck that it appears that we are effectively treating the weights (kernel and biases) as terms in the standard linear equation ($y=mx+b$). At page 72 of the book, the author writes

output = dot(W, input) + b
output = (output < 0 ? 0 : output)

Am I reading too much into this, or is this correct (and so fundamental I shouldn't be asking about it)?

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is

$$o_i^l = \sigma(\mathbf{x}_i^l \cdot  \mathbf{w}_i^l + b_i^l),$$

where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity function, $\mathbf{x}_i^l$ and $\mathbf{w}_i^l$ are the vectors that respectively contain the inputs and the weights for neuron $i$ in layer $l$, and $b_i^l \in \mathbb{R}$ is a bias. Similarly, the output of a layer of a feed-forward neural network (FFNN) is computed as

$$\mathbf{o}^l = \sigma(\mathbf{X}^l \mathbf{W}^l + \mathbf{b}^l).$$

In your specific example, you set the new weight to $0$, if the output of the linear combination is less than $0$, else you use the output of the linear combination. This is the definition of the ReLU activation function, which is a non-linear function.


What does a joint probability density function have to do with Stochastic Optimal Control and Reinforcement Learning? I stumbled upon a job offer from a company that was looking for someone who was good with Reinforcement Learning (applied to finance) and something in their offer caught my eye. It goes something like this:

  We want you to be able to study the price dynamic (of a stock I suppose) and its evolution in order to extract a Joint PDF that will be used in the Optimal Stochastic Control of a Loss Function (or gain)

The thing is I understand what each of these things mean and how they are used separately (from my background in Control theory & dynamical systems) and I worked with fitting Joint PDFs and Copulas before, but I don't understand how a Joint PDF would help with the "Optimal Stochastic Control of a Loss Function" ? Thanks.

Extracting a joint PDF just means that you create a model that models the behavior of several variables combined instead of in isolation.

If these variables aren't independent and your loss functions is influenced by all of them, you obviously have to learn this joint PDF to minimize your loss.

So I don't see this statement as particularly mysterious.


Should I grey-scale the coloured frames/channels to build the approximation of the state? I'm doing reinforcement learning, and I have a visual observation that I will use to build an input state for my agent. In the DeepMind's Atari paper, they greyscale the input image before they input it into the CNN to reduce the input space's size, which makes sense to me.
In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and green. This also makes intuitive sense to me since it's like a bit-encoding.
Any thoughts on what could be better? Greyscaling into 2 shades of grey and black and white also maintains the information, but feels somehow less direct, since my environment's visual space is categorical, which makes more sense in a categorical encoding.

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy?

In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and green. This also makes intuitive sense to me since it's like a bit-encoding.

Generally, if you have an environment like this, I would (without any other information) guess that each of the 5 colors have some meaning that may be relevant for your agent. That's just what I would guess though. In theory, it might be possible that white means one thing (e.g. "empty"), and every other colour means the same other thing (e.g. "not empty"). If you had domain knowledge like that, and knew that it is only important whether or not any given pixel is white, you could of course binarise your input.
But in general, if the colours might be important, I'd recommend including them. If you really only have a few distinct colours like that though, I would not recommend encoding them in some format like RGB where values can range from 0 to 1 or 0 to 255. I would recommend having 4 (or 5?) binary channels in your input:

Binary channel containing 1s for pixels that are black, and 0s for all other pixels.
Binary channel containing 1s for pixels that are white, and 0s for all other pixels.
Binary channel containing 1s for pixels that are blue, and 0s for all other pixels.
...
etc.

The reason for this is that deep neural networks often tend to have an easier time learning with binary inputs, and here you can completely binarise your inputs without requiring an excessively high number of channels. If you had hundreds or thousands of different possible colours, this would probably no longer be a good idea.


Are there reinforcement learning algorithms that scale to large problems? Given a large problem, value iteration and other table based approaches seem to require too many iterations before they start to converge. Are there other reinforcement learning approaches that better scale to large problems and minimize the amount of iterations in general?

This is a big question. I'm not going to try to cover the state-of-the-art, but I'll try to cover some of the main ideas.

Function Approximation [1]
An essential strategy for scaling up RL algorithms is to reduce the effective size of your state and/or action space through function approximation. For example, you could parameterize your value function using fewer parameters than there are states. Optimization would then take place in the much smaller parameter space, which can be substantially faster. Note that using function approximation almost always loses you any convergence guarantees you would have had otherwise in the tabular setting. It has been very successful in practice, though.

Sampling[2]
Value iteration and other dynamic programming algorithms sweep through the entire state space when computing value functions. Sample-based approaches instead update value functions for states as they are visited. These include Monte Carlo and Temporal Difference methods. Sampling allows us to focus on a subset of the states, freeing us from the computation required to get accurate value estimates of potentially irrelevant states. This is essential in real-world settings, where almost all possible states of the world are irrelevant or even impossible to reach.

Sample Efficiency/Experience Replay
All else equal, a sample efficient agent is one that learns more with the same experience. Doing this reduces learning time, especially if the time bottleneck is in interacting with the environment. One basic way of improving sample efficiency is to store and reuse experience with something like the experience replay buffer popularized in the DQN paper. Another, more recent, algorithm called Hindsight Experience Replay improves sample efficiency by allowing the agent to learn more from its failures (trajectories with no reward).

Model-Based Methods[3]
While technically also about sample efficiency (maybe all of these points are?), model-based methods are important enough to warrant their own section. Usually, the MDP dynamics aren't known to the agent beforehand. Learning and maintaining an estimate of the MDP is therefore often a good idea. If an agent can use its internal model of the world to simulate experience, it can then learn from that simulated experience (called planning) in addition to learning from actual experience. Because simulated experience is much cheaper to gather than actual experience, this can reduce the time needed to learn.

Search[4]
If our value estimates were perfect, then behaving optimally would only be a matter of moving to the neighboring state with the highest value. This is hardly ever the case, though, so we would like to make decisions more intelligently. One way, called forward search, is to use a model to consider many possible trajectories starting from the current state. The most popular and successful example of forward search is Monte Carlo Tree Search (MCTS), which was famously used in AlphaGo Zero. Because search allows us to make better decisions given imperfect value estimates, we can focus on more promising trajectories, saving time and computation.

Exploration
Only ever taking what we think is the "best" action in a given state is usually not a very good idea. When sampling trajectories through large state and/or action spaces, this strategy can completely fail. Taking exploratory actions can help ensure that high-value states are discovered at all. Deciding when to explore and which actions to take is active area of research. In general, though, exploratory actions are ones that reduce an agent's uncertainty of the environment.

Injecting Human Knowledge
Finally, and maybe obviously, reducing the time complexity of an RL algorithm can be accomplished by giving the agent more information about the world. This can be done in many ways. If using linear function approximation, for example, an agent could be given useful information through the features it uses. If it makes use of a model, the model could be initialized with reasonable priors for the reward and transition probability distributions. "Reward shaping", the practice of manually engineering a (dense) reward function to facilitate learning a specific task, is a more general approach. An agent could also learn directly from human demonstrations with inverse reinforcement learning or imitation learning.

References
All references not already linked to are chapters out of Sutton and Barto's RL book.
[1] Linear function approximation is discussed in depth in Chapter 9.
[2] Monte Carlo and Temporal Difference methods are discussed in Chapters 5 and 6.
[3] Model-based methods are discussed in the first part of Chapter 8.
[4] MCTS and search in general are discussed in the second half of Chapter 8.


What's the definition of "singularity" in the context of neural networks? The paper Skip connections eliminate singularities explains the use of skip connections to break the singularity in deep networks, but I have not fully understood what a singularity is.
Any easy-understanding explanation?

Firstly, I invite you to take this answer with a grain of salt (and possibly suggest edits) as I am not specifically familiar with the issue of singularities myself. Also, I appologise for all the links being at the bottom of the answer, I am unable to reuse links throughout my answer due to my reputation on this particular exchange.

This appears to refer to singular points in Algebraic Geometry. At singular points, no singlar tagent can be defined using the usual methods. This is as opposed to regular (i.e non-singular points), where one (and only one) tangent can be regularly defined.

The Easy Answer

In simpler terms, and particularly with respect to neural networks, it refers to the points on a hypersurface (e.g. a line, plane, surface, etc) at which the gradient/slope is ambigious.

I've taken example of this from the wikipedia page on singular points.

In the above image, the hyperspace is a line. On this line, there is a singularity at (0,0), where the line crosses itself. At this point, there are two slopes (one going up, and the other going down). There is no way to choose between the slopes without applying arbatrary rules, so the gradient is ambigous.

These singularities are problematic for some of the algorithms used in deep learning, such as Gradient Descent, which uses the the derivative of objective functions (e.g. loss functions) to search for the weights/parameters that yield the lowest error.

Gradient Descent In This Example

For anyone unfamiliar with gradient descent, the problem with singularaties can be understood with the following example:

Say that the above line represents a path. Starting at the top right, you want to find the lowest point on the path. At any time, all you can know is the slope at the current point. You can turn left or right freely, but will never move upwards, so you will stop as soon as you reach the bottom of a valley.

Initially moving down and to the left, you will eventually reach the singularity and need to choose to go either left or right. Because you have no information about the path beyond your current location, you have no way of knowing which way leads to the bottom of the path.

If you choose at random, you have a 50% chance of getting stuck at in the valley at the bottom of the loop on the left of the path.

You could impose a rule like "always follow the gradient most different to the current gradient", then you'd go right, and find the true lowest point. However, if the path was slightly different (e.g. if it curved up at (0.25, 0.25) this would be bad, as the correct path would then be on the left.

In practice, there is no way of knowing which way is better without a deep understanding of the hypersurface of your objective functions (and if you knew that much about the objective function, you likely wouldn't need to use machine learning in the first place).

Research/Resources

I discovered this in order to answer this questions so, as justification, here is the process and resources I used to come to this conclusion:

I dug through the references in the paper you linked. Of particular relevance was Amari et al. (2006), which mentions Hironaka’s
theorem of singularity resolution.

In the introductory paragraph, the wikipedia page mentions the concept of Non-Singularity Variety, and links to this page on singular points in Algebraic Geometry.

The example on singular points article is analogous to a 1d hypersurface version of many of the figures in Amari et al. (e.g. figures 1,2,4,5, and 6), and the discussion therein generally centers around the point at which the shapes converge.

My answer was composed based on this discovery process, as well my prior-knowledge of Gradient-Descent


Input optimization on a supervised learning system Problem

Given a collection of  pairs (X, y) where X belongs to R^n and y belongs to R, find the X such that the associated y would be maximum.

Example

Given:

(X=(1, 2), y=-9)
(X=(-2, 4), y=-36)
(X=(-4, 2), y=-24)
...

The algorithm should be able to detect that the function being applied to X is y=-(X[0]^2+2*(X[1]^2)) and find the input that maximizes this function, in this case X=(0,0) because y=0^2+2*0^2=0 and 0 is the maximum possible value, as all the other values are negative.

How I've tried to solve it

My first guess has been to create a neural network that predicts y given X, but, after that is done, I don't know how to go about optimizing the input.

Questions

Is there any algorithm that would help in this situation?

Also, would some other supervised learning algorithm fit better here than a neural network?

You can use a neural network for this (sort of), and you've got the first step right - train the network to predict $y$ given $x$ (i.e. train it to approximate the function $f$ such that $y = f(x)$). Because the entire neural network is differentiable (presumably), you can take gradients of the input (some $x$) with respect to the predicted output. Then you can use this to update $x$ the same way that you update the weights during training of the network. This let you find a local optimum from $x$. I don't know of a way to try to find a global optimum with a neural network like this except to find the local optimum near many $x$'s and then take the best of those.

If you want a concrete example, take a look at the tutorial on Neural Style Transfer with PyTorch: here you have a noise image as the input and you optimize it to minimize the "style distance" to a reference style image and the "content distance" to a reference content image (i.e. starting with noise, make it look like the content of one image but in the style of another). There's full code there, but here's a short PyTorch snippet that shows the main idea:

# I'm just optimizing one input; clone so that you don't modify the
# original tensor; let it know that we want gradients computed
inputs = train_inputs[0:1].clone().requires_grad_()
input_optimizer = torch.optim.Adam([inputs])

def optimize_input():
    input_optimizer.zero_grad()
    # where `model` is your trained neural network
    output = model(inp)
    (-output).backward()
    return output

output = []
for _ in range(10000):
    output.append(input_optimizer.step(optimize_input).detach().numpy())

output is a list at the end of what the predicted output was after each step of optimizing the input. You can use this to see if you've converged to a local optimum yet (and take more steps/fewer next time as necessary). inputs will be the optimized input at the end. Note that if there are constraints that your input should satisfy, you'll need to enforce those yourself (e.g. in neural style transfer, they have to enforce that the values are valid for an RGB image).

Note also that how well this works really depends on how well your network approximates $f$; you may well get some unreasonable $x$ for which the network predicts a very large $y$ because there was no training data similar to that $x$ so the network isn't constrained in that region. In general, you should probably be cautious of an $x$ you generate this way that doesn't seem similar to your training data (e.g. has larger/smaller value than any $x$ the network has seen before; interpolation is much easier than extrapolation).

I'm assuming that you have a single data set; if you're able to query for the $y$ values of given $x$'s, then you might want to take a look at Bayesian optimization - essentially the field of trying to find $x$ that maximizes $y = f(x)$ when $f$ is expensive to evaluate and you don't have gradients of it. Bayesian optimization seeks a global optimum.


Why do we need importance sampling? I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the practical example of importance sampling.
For instance, in a video, it is said that we need to calculate the expected value of a biased dice, here $g(x)$, in terms of the expected value of fair dice, $f(x)$. Here is a screenshot of the video.

Why do we need that, when we have the probability distribution of the biased dice?

Importance sampling is typically used when the distribution of interest is difficult to sample from - e.g. it could be computationally expensive to draw samples from the distribution - or when the distribution is only known up to a multiplicative constant, such as in Bayesian statistics where it is intractable to calculate the marginal likelihood; that is
$$p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)} \propto p(x|\theta)p(\theta)$$
where $p(x)$ is our marginal likelihood that may be intractable and so we can't calculate the full posterior and so other methods must be used to generate samples from this distribution. When I say intractable, note that
$$p(x) = \int_{\Theta} p(x|\theta)p(\theta) d\theta$$
and so intractable here means that either a) the integral has no analytical solution or b) a numerical method for computing this integral may be too expensive to run.
In the instance of your die example, you are correct that you could calculate the theoretical expectation of the bias dice analytically and this would probably be a relatively simple calculation. However, to motivate why importance sampling may be be useful in this scenario, consider calculating the expectation using Monte Carlo methods. It would be much simpler to uniformly sample a random integer from 1-6 and calculate the importance sampling ratio $x \frac{g(x)}{f(x)}$ than it would be to draw samples from the bias dice, not least because most programming languages have built in methods to randomly sample integers.
As your question is tagged as reinforcement learning I will add why it is useful in the RL domain. One reason is that it may be our policy of interest is expensive to sample from, so instead we can just generate actions from some other simple policy whilst still learning about the policy of interest. Second, we could be interested in a policy that is deterministic (greedy) but still be able to explore, so we can have an off-policy distribution that explores much more frequently.
NB: it may not be clear how you can use importance sampling if the distribution is only known up to a constant so see this answer for an explanation.


What are some examples of everyday systems that use AI? I would like to know some daily basis applications of AI. I think these might be relevant examples:

Google search engine
Face recognition on iPhone

Are my examples correct? Could you provide some more examples?

A good, recent, and accessible book which includes many case studies is Prediction Machines. Check it out for more details than I can provide in this answer.

Example applications are all around us, but one of the problems with recognizing them is that the bar for what we call AI is constantly being raised.

Consider that a few decades ago, directions from google maps would certainly be recognized as AI, whereas now most laypeople wouldn't make that association.

Some other commonplace examples:

Amazon.com can guess at what you'd like to buy next.
Facebook and many other companies have programs that decide which ads to show you, without human intervention.
Tesla's autopilot feature will drive your car down a highway.
Google search can instantly return the results you were looking for, even when you enter strange or ambiguous search terms.
Virtual Assistants from many companies, like Alexa recognize your voice, figure out what words you're saying, and then figure out what you want them to do.
StackOverflow determines which questions are hot, and which ones need moderator attention.

There are are also a lot of less obvious examples. I'll pick a few from my research area:

When you spot a sky marshal on a plane, or a dog patrol in the airport it's because an AI system put them there. Some subway systems now use the same technology for fare checkers.
When people trade agricultural products in Uganda, it's an AI system and AI techniques at work behind the scenes.
When you're assigned a donor for a kidney transplant (and now other organs too), an AI system did the heavy lifting in that decision.

Of course, there's lots more going on too, but those should give you some suggestions for everyday conversation.


Are there ensemble methods for regression? I have heard of ensemble methods, such as XGBoost, for binary or categorical machine learning models. However, does this exist for regression? If so, how are the weights for each model in the process of predictions determined?

I am looking to do this manually, as I was planning on training two different models using separate frameworks (YoloV3 aka Darknet and Tensorflow for bounding box regression). Is there a way I can establish a weight for each model in the overall prediction for these boxes?

Or is this a bad idea?

? This means that there are not promising versions of this algorithm fro regression until 2012.
After your question, I have found one of the survey research paper which is done or ensemple methods for regression. This table also extracted from this paper.
Read this paper, it will help you a lot more
This one is latest paper published on object detection with an ensemble approach


Are convolutional neural networks inspired by the human brain? The Deep Learning book by Goodfellow et al. states

Convolutional networks stand out as an example of neuroscientiﬁc principles inﬂuencing deep learning.

Are convolutional neural networks (CNNs) really inspired by the human brain?
If so, how? In particular, what structures within the brain do CNN-like neuron groupings occur?

In the eye, the retinal ganglion cells have a receptive field that is equivalent to some types of convolution filters, most of them edge detectors.
The brain is a big unknown, nobody knows how it does to organize, memorize, create concepts, learns the language, ... . Thus, it is not possible to establish a parallelism.
In particular, brain has a capacity of handle independence in scale and rotation that CNN's are not able to reproduce.
As general remark about NN and brain: even when it is always said that "neural network cells" are "inspired" in biological neurons, there are critical differences that made this similitude only a "inspiration". Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison. The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the learned parameters.


What is the cost function of a transformer? The paper Attention Is All You Need describes the transformer architecture that has an encoder and a decoder.
However, I wasn't clear on what the cost function to minimize is for such an architecture.
Consider a translation task, for example, where give an English sentence $x_{english} = [x_0, x_1, x_2, \dots, x_m]$, the transformer decodes the sentence into a French sentence $x_{french}' = [x_0', x_1', \dots, x_n']$. Let's say the true label is $y_{french} = [y_0, y_1, \dots, y_p]$.
What is the object function of the transformer? Is it the MSE between $x_{french}'$ and $y_{french}$? And does it have any weight regularization terms?

I took a look at the Tensor2Tensor's source code implementation as per @nbro's suggestion, and it seems like the loss function is the cross entropy between the predicted ($\|sentence length\|$ x $\|vocab\|$) probability matrix (right before taking the argmax to find the token to output), and the $\|sentence length\|$-length vector of token IDs as the true label.


Should I build an environment from scratch myself or it is not always needed? I am inspired by the paper Neural Architecture Search with Reinforcement Learning to use reinforcement learning for optimizing a child network (learner). My meta-learner (controller or parent network) is an MLP and will take as the reward function a silhouette score. Its output is a vector of real numbers between 0 and 1. These values are k different possibilities for the number of clusters (the goal is to cluster the result of the child network which is an auto-encoder, embedded images are the input to the meta-learner).
What I am confused about is the environment here and how to implement this network. I was reading this tutorial and the author has used gym library to set the environment.
Should I build an environment from scratch myself or it is not always needed?
I appreciate any help or hints or links to a source that helps me understand better RL concepts. I am new to it and easily gets confused.

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking.
If you want a custom environment, you can add an environment to gym like this.


How could we define passion in artificial intelligence? Mankind can create machines to do work. How could we define passion in artificial intelligence? How could we define a passionate AI?

Would passion cause the machine to do a better job? How could we compare the performance of a passionate machine, as opposed to a non-passionate one? How could we measure the level of passion?

An elementary approach to 'passion' would be to pre-assign different areas for the program to be 'passionate' about and associate different numeric 'drive strengths' with each (perhaps adaptively). Mechanisms of this sort were studied in Toby Tyrell's widely cited PhD thesis on 'Action Selection in Animals'

More recently, some more sophisticated AI architectures have been developed under the heading of 'Intrinsic Motivation'.

Here  is a link to a paper on the subject by Pierre-Yves Oudeyer, a leading expert in the field of Developmental Robotics.

With regard to the question "would this cause the machine to do a better job?", that would very much depend on how open-ended the architecture is:

It's clearly easier if, rather than having to spell everything out in detail to a machine, we can simply specify a problem at a high-level and let its own motivations cause it to explore promising avenues.

Conversely, if motivations are too open ended, it may well spend all its time doing the equivalent of 'doodling on its paper' (Hofstadter).

Hence, like people, the quality of the output will be a function of its internal dispositions and could be measured in the same way for a given task (e.g. quantitatively for scientific activities, qualatatively for the arts).


How to distinguish AI modeling from implementation? Quote from this Eric's meta post about modelling and implementation:

  They are not exactly the same, although strongly related. This was a very difficult lesson to learn among mathematicians and early programmers, notably in the 70s (mathematical proofs can demand a lot of non-trivial programming work to make them "computable", as in runnable on a computer).

If they're not the same, what is the difference?

How we can say when we're talking about AI implementation, and when about modelling? It's suggested above it's not easy task. So where we can draw the line when we talk about it?

I'm asking in general, not specifically for this site, that's why I haven't posted question in meta

One good way of differentiating modelling and implementation is to consider that models occupy a much higher level of abstraction.

To continue with the mathematical example: even though experimental mathematics might be dependent on computation, the program can be considered as one possible realization of the necessary conditions of a more abstract existence proof.

Over the last 25 years, software engineering methodologies have become quite good at separating models and implementations, e.g. by using interfaces/typeclasses/abstract base classes to define constraints on behavior that is concretely realized by the implementation of derived classes.

AI has always been a battle between the 'neats and the scruffies'. Neats tend to prefer working 'top down' from clean abstractions, 'scruffies' like to work 'bottom up', and 'bang the bits' of the implementation together, to see what happens.

Of course, in practice, interplay between both styles is necessary, but AI as a science progresses when we abstract mechanisms away from specific implementations into their most general (and hence re-useable) form.


Why is the Universal Declaration of Human Rights not included as statement on the AI? Lots of people are afraid of what strong AI could mean for the human race. Some people wish for a sort of "Asimov law" included in the AI code, but maybe we could go a bit more far with the UDHR.
So, Why is the Universal Declaration of Human Rights not included as statement of the A.I.?

As response to comment, response or edition:
The Universal Declaration of Human Rights is clear.
Homo sapiens sapiens (aka "mankind") needs some way to make sure AI evolution does not result in our extinction or enslavement to potentially superior algorithmic intelligences.

If I understand what you are asking, I think the simple answer would be that AI is nowhere near having demonstrated sentience, thus they do not qualify for any type of rights.

We won't have to "cross this bridge" until an AI demonstrates self-awareness and human-level-or-beyond intelligence, but it sure is interesting to think about!

(Also, the UDHR dates to the 1940's and seems to have had its last additions in 1966.  Computers weren't very "smart" back then so likely no on was even considering the question ;)

Value Alignment is a huge looming issue in regards to Artificial General Intelligence

Although you may also want to look at the grey good scenario, which posits inadvertent destruction of homo sapiens sapiens not as a factor not of too much, but of too little, intelligence.

The problem with an Asimov approach is highlighted by his book I, Robot, which is the potential pitfalls of pure logic.  The philosophy of Neo-Luddism is preoccupied with these problems in relation to technology--specifically that the threats posed by technology cannot be predicted.

The problem with the UDHR today is that there is no algorithm smart enough to understand it--we're not even close.  (There is something called the symbol grounding problem which demonstrates that meaning and understanding in relation to algorithms is still unsolved.)


Can neural networks efficiently solve the traveling salesmen problem? Can neural networks efficiently solve the traveling salesmen problem? Are there any research papers that show that neural networks can solve the TSP efficiently?
The TSP is an NP-hard problem, so I suspect that there are only approximate solutions to this problem, even with neural networks. So, in this case, how would efficiency be defined?
In this context, it seems that the time efficiency may be obtained by resource inefficiency: by making the neural network enormous and simulating all the possible worlds, then maximizing. So, while time to compute doesn't grow much as the problem grows, the size of the physical computer grows enormously for larger problems; how fast it computes is then, it seems to me, not a good measure of the efficiency of the algorithm in the common meaning of efficiency. In this case, the resources themselves only grow as fast as the problem size, but what explodes is the number of connections that must be built. If we go from 1000 to 2000 neurons to solve a problem twice as large and requiring exponentially as much time to solve, the algorithms requiring only twice as many neurons to solve in polynomial time seem efficient, but, really, there is still an enormous increase in connections and coefficients that need be built for this to work.
Is my above reasoning incorrect?

To the best of my knowledge, there isn't any difference between the algorithmic methods and the NN methods. Those that can solve in polynomial time do not give a precise solution. Those that do give a precise solution do not solve in polynomial time. Of those that give a precise solution, the fastest takes $2^N$, but it blows up in terms of memory. The fastest good algorithm I believe is Concorde.
The efficient algorithms solve in polynomial time, don't blow up in terms of memory, and give a solution close to perfect, say, within 2-3%. Again, to the best of my knowledge, no NN has beaten the best algorithmic solutions, but there are suggestions that some NN solution could be faster.


How is PDDL used in production AI systems? I can't find much information on modern PDDL usage. Are there more popular alternatives, maybe something more suited to modern neural network/deep learning techniques?

I'm particularly interested in PDDL or alternative's current usage in autonomous driving software.

I have seen it used in automated story-telling (or game AI to control NPCs), and in NLG systems, where the generation of text is reinterpreted as a planning task.

What these systems have in common is that they're either off-line or in a simple environment (NPC control). I'm not sure they would be suitable for real-time applications, unless you can be sure that a feasible plan exists which can be found within certain time bounds. I wouldn't want to sit in a car going at high speed on the motorway and waiting for the driving unit to work out a plan how to avoid an obstacle that suddenly appears on the road.


How can I incrementally train a Yolo model without catastrophic forgetting? I have successfully trained a Yolo model to recognize k classes. Now I want to train by adding k+1 class to the pre-trained weights (k classes) without forgetting previous k classes. Ideally, I want to keep adding classes and train over the previous weights, i.e., train only the new classes. If I have to train all classes (k+1) every time a new class is added, it would be too time-consuming, as training k classes would take $k*20000$ iterations, versus the $20000$ iterations per new class if I can add the classes incrementally.

The dataset is balanced (5000 images per classes for training).

I appreciated if you can throw some methods or techniques to do this continual training for Yolo.

There's something called Elastic Weight Consolidation to prevent neural networks from forgetting previous tasks as they train on new tasks. It might be helpful for your case too.

The main idea is to quantify the importance of parameters for task $t$ and penalize the model in proportion when it changes its parameters as it trains to learn task $t+1$. As you can see, this incentivizes model to change parameters that are less important for task $t$ which prevents the model from forgetting it.


Can rewards be decomposed into components? I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like  reward = distance_x + distance_y + standing_up_straight, and then it adds this reward to the replay buffer. However, I think that it would be more efficient if it can break the reward down by category, so it can figure out "that action gave me a good distance distance_x, but I still need work on distance_y and standing_up_straight".
Are there any existing algorithms that add rewards this way? Or have these been tested and proven not to be effective?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however  that many scientist will often follow the reward hypothesis (Sutton and Barto) which says that

All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

The argument for a scalar reward could be that even if you define your policy using some objective vector (as in MORL) - you will find a pareto bound of optimal policies, some of which favour one component of the objective over the other - leaving you (the scientist) responsible for making the ultimate decision concerning the objectives' tradeoff - thus eventually degenerating the reward objective into scalar.
In your example there might be two different "optimal" policies - one which results in a very high value of distance_x but relatively poor distance_y and a one that favours distance_y instead. It'll be up to you to find the sweet spot and collapse a reward function back to a scalar.


Could AI be used to generate questions from a database input? I confront to the next scenario:

Let's say I have stored data about football matches between different teams: lineups, scorers, yellow cards, and many other events.
I need to generate everyday some questions about the matches that will be played on that day. So, if I give an input of two teams, I would like a related question to be generated, based on previous data of matches between those two teams.
For example, if my input are "TeamA" and "TeamB", I would expect a question of the type:

"Will there be less than 2 goals scored in the match?""

"Will PlayerX score a goal during the match?"

Of course I expect these questions to make sense based on previous data from matches between the two given teams.

So, my questions are:

Would be a good solution to use AI to generate these questions? It would make sense?
What would be the best approach?

One simple approach to consider would be storing each statement as a template made in advance.

Will there be less/more than x goals scored in the match?
Will player score a goal during the match?
...

The system will pick a random statement and will fill the variable fields with some statistically generated data between teamA and teamB; here you have your question.

Example: Will there be less/more than x goals scored in the match?

less/more fragment may be random
x may be the mean of the goals scored considering all the matches between teamA and teamB

Example: Will player score a goal during the match?

player may be a random choice between the top-goalscorer of teamA or teamB