Should the reward or the Q value be clipped for reinforcement learning When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value.  It is well known that non-linear function approximators, such as neural networks, diverge aggressively.  One way to help stabilize training is using reward clipping.  Because the temporal difference Q-update is a bootstrapping method (i.e., uses a previously calculated value to compute the current prediction), a very large previously calculated Q-value can make the current reward relatively minuscule, thus making the current reward not impact the Q-update, eventually leading the agent to diverge.

To avoid this, we can try to avoid the large Q-value in the first place by clipping the reward between [1, -1].

But I have seen some other people say that instead of clipping the reward itself, we can instead clip the Q-value between an interval.

I was wondering which method is better for convergence, and under what assumptions / circumstances.  I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I'll start with the last question in your post:

  I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I highly doubt there will be any such theoretical work. The problem is that these variants of clipping (clipping rewards and clipping $Q$ values) fundamentally modify the task / the original objective. Once you clip either of those things, you fundamentally change what your agent is trying to optimize for from what the original goal was. I don't think it's ever going to be possible to get any rigorous, theoretical proofs about which one would be better in general. You'd likely have to start out with some very strong assumptions on the reward structure in the original task to have any hope of proving anything here, but such strong assumptions make you lose generality.

Intuitively... I think reward clipping feels "safer" to me more often than clipping $Q$-values. Clipping $Q$-values seems more aggressive, it could be viewed as some combination of clipping rewards (if you clip $Q$-values to $[-1, 1]$, you're still at the very least also clipping all rewards to that range), but additionally also putting a constraint on how far in the future you're looking (in some sense). This whole argument is very handwavy though

I suppose, slightly less handwavy, you could say that reward clipping is definitely "better" (in the sense that you don't deviate as much from the original objective) in environments where rewards of similar magnitudes can be collected frequently. I struggle to really think of a situation where clipping $Q$-values would be a clear favorite based on intuition. I wouldn't be surprised if clipping $Q$-values may turn out to be better after empirical evaluation in some cases, but it's difficult to say where that would be. It will also very much depend on what range is chosen. Clipping rewards to a range of $[-1, 1]$ is very different from clipping $Q$-values to the same range.


Can I do oversampling by copying the same image multiple times? Will it effect my neural network accuracy? I am working on an image data-set. As you may have guessed it is imbalanced data. I have 'Class A, 19,000 images' and 'Class B, 2,876 images'.

So I did an undersampling by removing randomly from the majority class till it becomes equal to the minority class.

On doing this I am loosing lot of information from those 19000 images which I could get.
So I do an oversampling of minority class, by simply copying the 2,876 images again and again.

Is this undersampling method correct, will it effect my accuracy? I trained an Inceptionv4 model using this oversampled data and it is not at all stable and I am getting poor accuracy.

What should be my strategy ?

This would be a pretty fun opportunity to experiment a bit. I know there's been breakthroughs in OCR technology by training the network on images that have been altered.

Long story short, there was a pre-process that took the source images and did minor manipulations to them. This involved skewing, rotating, changing resolution & adding artifacts into the image to introduce "Noise". This allowed the network to learn to recognize characters more accurately. It may be worthwhile to create an image pre-processing pipeline that will do a variation of modifications to the source images. When training you may want to lower your learning rate & split up your data into 70% train & 30% test. Then you can loop through all the images repeatedly until you hit a maximum for the learning rate.

Some examples
https://matthewearl.github.io/2016/05/06/cnn-anpr/
https://medium.com/@shreyas.s/image-data-generation-for-optical-character-recognition-ocr-9b19300649c8


What is meant by gene, chromosome, population in genetic algorithm in terms of feature selection? I am trying to understand the genetic algorithm in terms of feature selection and these features are extracted using a machine learning algorithm.
Let's suppose I have data of heart rate for 3 minutes collected from $50$ subjects. From these 3-minute heart rate, I extracted $5$ features, like the mean, standard deviation, variance, skewness and kurtosis. Now the shape of my feature set is (50, 5).
I want to know what are gene, chromosome and population in genetic algorithm related to the above scenario.
What I understand is each feature is a gene, and a set of all features for one subject (1, 5) is the chromosome, and the whole feature set (50, 5) is a population. But I think this concept is not correct. Because in the genetic algorithm, we take a random population, but according to my concept complete data is population, so how random data is selected.
Can anyone help me to understand it?

A chromosome in this case could be a set of filters, each extracting a different feature (analogous to Convolutional Neural Network). Your question doesn't say what you want to do with these features, so this solution is made under the assumption that there is a fitness function which would take these features as an input and output a score. Then, each gene is a parameter for a filter, each chromosome defines a set of such filters, which makes up an individual. A population is a set of such individuals.


How does a batch normalization layer work? I understood that we normalize to input features in order to bring them on the same scale so that weights won't be learned in arbitrary fashion and training would be faster.

Then I studied about batch-normalization and observed that we can do the normalization for outputs of the hidden layers in following way:

Step 1: normalize the output of the hidden layer in order to have zero mean and unit variance a.k.a. standard normal (i.e. subtract by mean and divide by std dev of that minibatch).

Step 2: rescale this normalized vector to a new vector with new distribution having $\beta$ mean and $\gamma$ standard deviation, where both $\beta$ and $\gamma$ are trainable.

I did not understand the purpose of the second step. Why can't we just do the first step, make the vector standard normal, and then move forward? Why do we need to rescale the input of each hidden neuron to an arbitrary distribution which is learned (through beta and gamma parameters)?

Definition and Explaination

For how Batch Normalization works exactly, I'll suggest you to read the following papers:

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
How Does Batch Normalization Help Optimization?

The recent interpretation on How BN works is that it can reduce the high-order effect as mentioned in Ian Goodfellow's lecture. So it's not really about reducing the internal covariate shift.

Intuition

For how it works intuitively, you can think that we want to normalize the intermediate outputs (zero mean and unit variance) if the normalization won't remove too much useful information.

However, normalization may not be suitable for all intermediate outputs. So $\beta$ and $\gamma$ is introduced to provide additional flexibility, if normalization removes too much useful information then $\beta$ and $\gamma$ will learn to become the original mean and variance, making the BN layer an identity transformation, as if it doesn't exist.

In practice, $\beta$ and $\gamma$ won't become the original mean and variance, since all intermediate outputs can be normalized in some certain way without losing too much useful information. So you can think of it to be a customized normalization for each BN layer.

tl;dr

BN layer normalize the intermediate outputs in default, however, if the neural network find out that these intermediate outputs should not be normalized, then the neural network undos or provide more flexibility to the normalization.


What is the target Q-value in DQNs? I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values.
What does the target Q-values represent? And how is it obtained/calculated by the DQN?

When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each sample you do the following.

Feed $s_t$ into the network to get $Q(s,a;\theta)$.
Feed $s_{t+1}$ into the network to get $Q(s’,a’,\theta)$.
Choose $max_aQ(s’,a’,\theta)$ and set $ \gamma max_aQ(s’,a’,θ)$ + $r_t$ as the target of the network.
Train the network with $s_t$ as input to update $\theta$. The output from the input of $s_t$ is $Q(s,a,\theta)$ and the gradient descent step minimises the squared distance between $Q(s,a,\theta)$ and $\gamma max_aQ(s’,a’,θ)$ + $r_t$


Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay? I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as described in the Atari with Deep RL paper. All is working fine, but I was wondering whether there is any logical way to select the proper batch size for training, or if simply using a grid search is best. At the moment, I’m simply using 32, for its small enough that I can render the gameplay throughout training at a stunning rate of 0.5fps. However, I’m wondering how much of an effect batch size has, and if there is any criteria we could generalize across all Deep Q-learning tasks.

There is no special calculation you can do to determine the optimal batch size for any situation, so you kinda have to do a bit of testing to determine what batch size will work best. But there are some common trends you can take into account to make your testing easier.

How to choose your batch size
According to the paper Accelerated Methods for Deep Reinforcement Learning you get the best performance from DQNs (on average) with a batch size of 512. The problem with this is that is is much slower than the usual batch size of 32 and most of the time the performance improvement doesn't warrant it.

If you are just trying to test out your agents it is generally best to stick with a batch size of 32 or 64 so that you can train the agent quickly yet still get an idea of what it is capable of. But if getting the best performance is your top priority and waiting longer isn't a problem, then you should go for a batch size of 512 (higher can actually lead to worse performance) or something near that.


What could possibly replace artificial intelligence? There is no doubt as to the fact that AI would be replacing a lot of existing technologies, but is AI the ultimate technology which humankind can develop or is their something else which has the potential to replace artificial intelligence?

By definition, artificial intelligence includes all forms of computer systems capable of completing tasks that would ordinarily warrant human intelligence.

A superintelligent AI would have intelligence far superior to that of any human and therefore would be capable of creating systems beyond our capabilities.

As a consequence, if a technology superior to AI were to be created, it would almost certainly be created by an artificial intelligence.

  For the purposes of mankind, however, superintelligent artificial intelligence is the ultimate technology due to the fact that it will be able to surpass humans in every field, and, if anything, replace the need for human intelligence.

In our past experience, intelligence has been the most valuable trait for any entity to manifest - for this reason, in an anthropomorphic context, we can predict that artificial intelligence will be the ultimate achievement.

  The main reason why we will certainly not be able to replace superintelligent AI is that it will surpass us in every respect - if there is ever any replacement, it will be created by the AI similarly to the way we may create an AI that replaces us.


What are episodic and non-episodic domains in reinforcement learning? I was reading about the temporal difference (TD) learning and I read that:

  TD handles continuing, non-episodic domains

Assuming that continuing means non-terminating, what does non-episodic or episodic domain mean?

Assuming that continuing means non terminating, what does non-episodic or episodic domain mean ?

Non-episodic means the same as continuing. The quote you found is not listing two separate domains, the word "continuing" is slightly redundant. I expect the author put it in there to emphasise the meaning, or to cover two common ways of describing such environments.

Episodic domain problems are ones that terminate, or otherwise naturally split into groups of time steps that can be considered separately.


Can an AGI convince another AGI to modify its code? Let's suppose there are two AGIs, $A$ and $B$. Assume that $B$ has the ability to modify $A$, but this action of modifying is considered bad by $B$. Can $A$ ever convince $B$ to modify $A$?

An AI takes the decision based on the output of their utility function. This is just a fancy word for the calculations that AI perform to compare profit and loss of taking a certain decision.

There is always a tight analogy between an AGI and a human. You can juxtapose the utility function to how we make decisions by considering marginal profits of doing something over the other.

Now your question. The answer is a big NO. This is because an AI (or a human) never takes an action that gives a low score on its utility function or is against the fundamental view of the machine (or human). AI only cares about the utility function and goal state (plus the instrumental goals), nothing else. Our AI outside the box has been created with the purpose of not opening the box. For it, the action of opening the box has a very low score, or a negative score.

Now you might be wondering why an AI can't be convinced of doing something that gives a low score in its utility function. Keep reading.

Consider this for a second. Suppose an AGI was created, assigned with the task of copying the handwriting of others and improving its own writing. You can think of it as a smart writing hand. Now, what this AI might come up with is that, in order to practice writing, it needs more pages and thus it needs to cut more trees for that. No matter what you do, AI will not stop from cutting more and more trees. It might even replicate itself into machines that will cut trees for him. Now, the only thing that can be done is to turn off the AGI. But, a smart machine would have known the possibility of being turned off and thus will have transferred itself to many other machines over the globe. The important question to ask here is, why it is doing all this? It is not because the machine wants to live, as we humans want to. The only reason why it wants to live is to fulfill its goal.

You simply can't change the fundamental view of the machine. The change can result in, the machine not able to attain its goal or not to that extend (optimal profit). This is the reason why you can't convince a machine to do something that it was asked not to do as its primary task.

You, as a person, are living a life now, and have some fundamental beliefs. Let us consider that you believe in not killing someone. Now, suppose that I give you a pill, and tell you that after taking this pill it will rewire your brain and you will kill the first four people you see. But, after that, you will achieve pure satisfaction and happiness. Now you will definitely not take that pill as it conflicts with something that you believe in now. Also, you will try your best and fight back not to take that pill. The same thing applies to an AGI too. It doesn't matter what your future version will feel or attain after rewiring of the brain (changes in code), it is what and who you are now matters. This video link will help.

I hope it answers your question. There are lots of things to consider here. I have assumed a few things and tried to answer according to that.

There is one more thing. We don’t tell the machines how to do something, instead, we only tell them what to do (at least in the case of an AI). This is because sometimes we don't know the optimal way of solving certain problems. In the case of your question, we don't know what these two machines will do or say to each other. It will be a very interesting thing to hear or watch.


Why did AlphaGo lose its Go game? We can read on wiki page that in March 2016 AlphaGo AI lost its game (1 of 5) to Lee Sedol, a professional Go player. One article cite says:

  AlphaGo lost a game and we as researchers want to explore that and find out what went wrong. We need to figure out what its weaknesses are and try to improve it.

Have researchers already figured it out what went wrong?

We know what Lee's strategy was during the game, and it seems like the sort of thing that should work. Here's an article explaining it. Short version: yes, we know what went wrong, but probably not how to fix it yet.

Basically, AlphaGo is good at making lots of small decisions well, and managing risk and uncertainty better than humans can. One of the things that's surprising about it relative to previous bots that play Go is how good it was at tactical fights; in previous games, Lee had built a position that AlphaGo needed to attack, and then AlphaGo successfully attacked it.

So in this game, Lee played the reverse strategy. Instead of trying to win many different influence battles, where AlphaGo had already shown it was stronger than him, he would set up one critical battle (incurring minor losses along the way), and then defeat it there, with ripple events that would settle the match in his favor.

So what's the weakness of AlphaGo that allowed that to work? As I understand it, this is a fundamental limitation of Monte Carlo Tree Search (MCTS). MCTS works by randomly sampling game trees and averaging them; if 70% of games from a particular position go well and 30% of games from another position go well, then you should probably play the first move instead of the second move.

But when there's a specific sequence of plays that go well--if, say, W has a path that requires them playing exactly the right stone each time, but B has no possible response to this path--then MCTS breaks down, because you can only find that narrow path through minimax reasoning, and moving from the slower minimax reasoning to the faster MCTS is one of the big reasons why bots are better now than they were in the past.

It's unclear how to get around this. There may be a way to notice this sort of threat, and then temporarily switch from MCTS reasoning to minimax reasoning, or to keep around particular trajectories in memory for consideration in future plays.


Is reinforcement learning suited for real-time systems? From what I have seen, any results involving RL almost always take a massive number of simulations to reach a remotely good policy.

Will any form of RL be viable for real-time systems?

Short answer: Yes, it is.

Explanation

Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs.  As with any online learning algorithm, there are a few things to consider.

The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others techniques to overcome it.

Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.


What caused the resurgence of AI since the early 2000s? I took a few AI courses in college (1999-2003), and we used the first edition of AI: A Modern Approach. We covered a lot of topics and programming, including classical AI, neural networks, and temporal difference learning.

Over the past few years, AI has had a resurgence.

Is this resurgence due to new AI theory or just better (more) computational power and data so that the theories (e.g., neural networks) to be more effective? Or is it a combination of both?

I want to get up to speed on what's happened in AI since the early 2000s, and I want to know what to cover -- what is the most significant advancement?

To vastly oversimplify a lot of the progress of modern research in AI/neural nets, the recent advances stems from applicational improvements of the back propagation algorithm.

ex: our recent possession of big data (it turns out that back-prop works very well if you have lots of data to feed into the network), increased computing power to harness that data, and creative network designs to leverage that computing power (deep learning).

So far I haven't found a good source that integrates the knowledge of current practice of neural networks into the classical study of AI nicely. But if you are still interested you can read this modern classic textbook on neural networks and make your own judgements.

http://www.deeplearningbook.org


How Euclidean distance algorithm calculate two different face images are match or not match in face recognition? I am trying to make a face login application. face comparison algorithm is using Euclidean distance to calculate two different face images that are the same or not the same. can anyone help me with how the Euclidean distance algorithm is working?

Simply put, Euclidean distance measures how far away two items are (see Neil Slater's comment).

In order to apply this to a pattern recognition task, you will need to convert the items to compare (in your case images of faces) into feature vectors (ie lists of numerical values), and then you do a pairwise comparison to work out how distant two faces are. You will then need to set a threshold where you treat two images as being the same face (typically where the distance between the feature vectors is small).

Selecting the right features is obviously crucial here. I'm not an expert on face descriptions, but it would probably be something like

the distance between the eyes
the distance between the bottom of the nose and the top of the upper lip
...

Once you have these measurements, store the values for each image in a vector and you can apply the Euclidean distance. Effectively, each image is represented as a point in the  $m$-dimensional feature space, where each measurement is a dimension. To select good features make sure they are not correlated (ie they are independent of each other) and are the same scale (eg all are distances, so not eye colour)

The choice of Euclidean distance is fairly minor: there are other distance metrics which might work equally well or even better. As I mentioned, the key point is selecting appropriate feature values.


How to implement a Continuous Control of a quadruped robot with Deep Reinforcement Learning in Pybullet and OpenAI Gym? Description

I have designed this robot in URDF format and its environment in pybullet. Each leg has a minimum and maximum value of movement.

What reinforcement algorithm will be best to create a walking policy in a simple environment in which a positive reward will be given if it walks in the positive X-axis direction?

I am working in the following but I don´t know if it is the best way:

The expected output from the policy is an array in the range of (-1, 1) for each joint. The input of the policy is the position of each joint from the past X frames in the environment(replay memory like DeepQ Net), the center of mass of the body, the difference in height between the floor and the body to see if it has fallen and the movement in the x-axis.

Limitations

left_front_joint      => lower="-0.4" upper="2.5" id=0

left_front_leg_joint  => lower="-0.6" upper="0.7" id=2

right_front_joint     => lower="-2.5" upper="0.4" id=3

right_front_leg_joint => lower="-0.6" upper="0.7" id=5

left_back_joint       => lower="-2.5" upper="0.4" id=6

left_back_leg_joint   => lower="-0.6" upper="0.7" id=8

right_back_joint      => lower="-0.4" upper="2.5" id=9

right_back_leg_joint  => lower="-0.6" upper="0.7" id=11

The code below is just a test of the environment with a set of movements hardcoded in the robot just to test how it could walk later. The environment is set to real time, but I assume it needs to be in a frame by frame lapse during the policy training. (p.setRealTimeSimulation(1) #disable and p.stepSimulation() #enable)

A video of it can be seen in:

https://youtu.be/j9sysG-EIkQ

The complete code can be seen here:

https://github.com/rubencg195/WalkingSpider_OpenAI_PyBullet_ROS

CODE

import pybullet as p
import time
import pybullet_data

def moveLeg( robot=None, id=0, position=0, force=1.5  ):
    if(robot is None):
        return;
    p.setJointMotorControl2(
        robot,
        id,
        p.POSITION_CONTROL,
        targetPosition=position,
        force=force,
        #maxVelocity=5
    )

pixelWidth = 1000
pixelHeight = 1000
camTargetPos = [0,0,0]
camDistance = 0.5
pitch = -10.0
roll=0
upAxisIndex = 2
yaw = 0

physicsClient = p.connect(p.GUI)#or p.DIRECT for non-graphical version
p.setAdditionalSearchPath(pybullet_data.getDataPath()) #optionally
p.setGravity(0,0,-10)
viewMatrix = p.computeViewMatrixFromYawPitchRoll(camTargetPos, camDistance, yaw, pitch, roll, upAxisIndex)
planeId = p.loadURDF("plane.urdf")
cubeStartPos = [0,0,0.05]
cubeStartOrientation = p.getQuaternionFromEuler([0,0,0])
#boxId = p.loadURDF("r2d2.urdf",cubeStartPos, cubeStartOrientation)
boxId = p.loadURDF("src/spider.xml",cubeStartPos, cubeStartOrientation)
# boxId = p.loadURDF("spider_simple.urdf",cubeStartPos, cubeStartOrientation)

toggle = 1

p.setRealTimeSimulation(1)

for i in range (10000):
    #p.stepSimulation()

    moveLeg( robot=boxId, id=0,  position= toggle * -2 ) #LEFT_FRONT
    moveLeg( robot=boxId, id=2,  position= toggle * -2 ) #LEFT_FRONT

    moveLeg( robot=boxId, id=3,  position= toggle * -2 ) #RIGHT_FRONT
    moveLeg( robot=boxId, id=5,  position= toggle *  2 ) #RIGHT_FRONT

    moveLeg( robot=boxId, id=6,  position= toggle *  2 ) #LEFT_BACK
    moveLeg( robot=boxId, id=8,  position= toggle * -2 ) #LEFT_BACK

    moveLeg( robot=boxId, id=9,  position= toggle *  2 ) #RIGHT_BACK
    moveLeg( robot=boxId, id=11, position= toggle *  2 ) #RIGHT_BACK
    #time.sleep(1./140.)g
    #time.sleep(0.01)
    time.sleep(1)

    toggle = toggle * -1

    #viewMatrix        = p.computeViewMatrixFromYawPitchRoll(camTargetPos, camDistance, yaw, pitch, roll, upAxisIndex)
    #projectionMatrix  = [1.0825318098068237, 0.0, 0.0, 0.0, 0.0, 1.732050895690918, 0.0, 0.0, 0.0, 0.0, -1.0002000331878662, -1.0, 0.0, 0.0, -0.020002000033855438, 0.0]
    #img_arr = p.getCameraImage(pixelWidth, pixelHeight, viewMatrix=viewMatrix, projectionMatrix=projectionMatrix, shadow=1,lightDirection=[1,1,1])

cubePos, cubeOrn = p.getBasePositionAndOrientation(boxId)
print(cubePos,cubeOrn)
p.disconnect()

Proposed Approach

  The expected output from the policy is an array in the range of (-1, 1) for each joint.
          The input of the policy is the position of each joint from the past X frames in the environment
          (replay memory like DeepQ Net),
          the center of mass of the body,
          the difference in height between the floor and the body to see if it has fallen and the movement in the x-axis.

  The code below is just a test of the environment with a set of movements hard coded in the robot just to test how it could walk later.
          The environment is set to real time, but I assume it needs to be in a frame by frame lapse during the policy training.

Initial Considerations

It is advisable to first list and analyze the use cases, system requirements, and their design consequences. The following details can be addressed after a high level theory and design have been established.

Algorithm details
Models according to schemas such as URDF
Physics simulators, such as Bullet
Challenge frameworks, such as Gym
Goals in terms of Cartesian direction of travel
Number of limbs
Program variable names
Initialization

Some requirements are generally assumed as part of any useful robotic controller and need not be mentioned explicitly.

Continuous control (discontinuous control and lack of control are poor options)
Hinged movement always has a range (min and max)

Listing more relevant system design constraints such as these can guide analysis and design. This is an abridged list to consider as an example.

No precision in the control of angular speed and velocity at joints.
No vision, audio, radar, sonar, or lidar sensory information available for either position sensing or predictive path planning
Orientation is not directly sensed
Surface coordinates are sensed via triangulation, and that subsystem is already designed and tested
Clearance to ground is known and must be maintained above a configurable constant value
The use of Deep Reinforcement Learning is expected (which, given the mechanical design, implies the maintenance of a walking policy)
The goal is to maintain a particular direction of robot travel
Each limb has two radial degrees of freedom, controlled by an angular position command input to the motion control sub-system
The nominal radial acceleration and maximum radial velocity of angular positions are not known

Extensions of Traditional Control

The AI system must produce stability. An established method is closed loop control. The stability performance of the AI system must at least exceed standard PID control.

$$ k_p \epsilon(t) + k_i \int_0^t \epsilon(t) dt + k_d \frac {d\epsilon(t)} {dt} $$

The integral term can have an inverse exponential memory decay as such.

$$ k_p \epsilon(t) + k_i \frac {\int_0^t e^t \epsilon(t) dt} {\int_0^t e^t dt} + k_d \frac {d\epsilon(t)} {dt} $$

Nonetheless, all terms are first degree in $k$ parameters. With a feed forward network, some competition winning robotics systems have reached a higher level of performance in terms of stability and efficiency of motion.

$$ f_p(P) \epsilon(t) + f_i(P) \frac {\int_0^t e^t \epsilon(t) dt} {\int_0^t e^t dt} + f_d(P) \frac {d\epsilon(t)} {dt} $$

This is not yet tractable, since offline learning will not take advantage of terrain knowledge gained during a robot's walk. If the terrain is a constant, for instance, if it is always a flat plane, and the positioning system is stable and reliable, reinforcement would not be needed. Such an ideal environment for the walk is rare in practice.

Most servo controllers with radial encoder inputs provide positional and dynamic stability using a model similar to the second of the above three equations.

Deep Reinforcement Learning

The development of reinforced learning policy updating is desirable for real world walker control. Although a policy update function can theoretically be learned from walk examples, the distribution of terrain is not easily representable in example terrain, even if the terrains are generated by genetic-algorithm-like designs modified to simulate planetary geological processes over cosmic time frames.

The approaches that show early indications of success involve reward $r$, dynamics $p$, and policy $\pi$ that depend on both on state and action.

$$ \pi (u_t | x_t) \\
 p(x_t | x_t, u_t) \\
 r(x_t, u_t) $$

The study of variants of NAF that leverage replay memory is outlined in Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates, Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine, 2016, using this model.

$$ \mu_{n+1} (x_t) := {argmax}_u{Q_{{\pi}_n} (x_t, u_t)} \\
 f() := \tanh() \; \text{,} \\
 \mu(x) = f(k + K_x) \\
 Q(x,u) = \frac 1 2 {\big(u − \mu(x)\big)}^T P(u − \mu(x)) + x^T (B x + b) + c $$

where P, k, K, B, b, and c are learnable tensors. Gu et. al. explains.

  If f() [was instead] identity, then the expression corresponds to a globally quadratic Q-function
          and a linear feedback policy, though due to the Tanh non-linearity,
          the Q-function is not linear with respect to state-action features.

Reward

Because of the above considerations, defining reward solely on the basis of one axis of position will likely be insufficient to guide effective learning. Various actions have costs in terms of both time and energy consumption and each may affect future move choices in significant ways. Violation of the  ground clearance constraint must produce a penalty. The reward function must guide policy development to properly take these factors into consideration.

To enhance the tractability of the learning challenge, a process block might best be inserted between the deep reinforcement learning component and the controller to perform a trigonometric and Newtonian translation from desired foot positions relative to the center of robot mass and the radial positions on all control axes to achieve those positions. That removes the burden of that translation from the walker's learner.


Does Gödel's second incompleteness theorem put a limitation on artificial intelligence systems? According to Brian Cantwell Smith

no calculation without representation

Therefore, computers depend on models. So, we can say that AI is limited internally by the model and externally by the environment. This problem is discussed here in a previous question I have asked.
Now, consider Gödel's second incompleteness theorem

a coherent theory does not demonstrate its own coherence

Can we say that Gödel's second incompleteness theorem puts a limitation on artificial intelligence? How could AI bypass Gödel's second incompleteness theorem?

I think the colloquial understanding of Gödel's incompleteness theorems allows them to be too broadly applied. Gödel's second incompleteness regards the consistency of a formal system, which is a technical concept of formal systems that means the system cannot prove every formula. It is commonly framed as a system not being able to prove both a formula and its negation (e.g. $2+2=4$ and $2+2 \neq 4$), since many logical systems allow anything to be proven from a contradiction.
The second incompleteness theorem states that if a consistent formal system is expressive enough to encode basic arithmetic (Peano arithmetic), then that system cannot prove its own consistency. This implies that we must use a stronger system B to prove the consistency of A. The system needs to be able to represent arithmetic because that is what is used to define the representability conditions of Gödel's proof that allowed him to formally construct the self-referential formulae central to the incompleteness theorems.
Here I diverge with my own opinion on this, I feel the concept of consistency in formal systems has no obvious bearing on the limits of artificial intelligence. An intelligent agent need not know anything about formal consistency to reach its level of intelligence -- the vast majority of humans have never encountered this concept, and yet they are still intelligent. Even many mathematicians don't give it a second thought unless they are in the trenches of mathematical logic. One would have to take an overly narrow view of artificial intelligence to allow Gödel's second incompleteness to serve as a limitation to it.
I caution against the popular informal restatements of Gödel's incompleteness theorems. These theorems were undoubtedly earth-shattering in the study of foundational mathematics and still have grand implications today, but projecting those results too far away from their rigorous origins is going to lead to many stray conclusions.


Is there a machine learning model that can be trained with labels that only say how "right" or "wrong" it was? I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or do nothing = -1, 0,1) but that can be trained with labels that contain how "correct" or "incorrect" it was. I've tried to google around and ask some friends in my machine learning class, but no one seems to have an answer.
The classic example I seem to always see is the models used in the snake game. We don't know what the right decision was per se, but we can say that if it ran into the wall, that was really wrong. Or if it got an apple and gained 50 points, then it was correct and if it got 2 apples and gained 100 points then it was even more correct, etc.
I'm looking for a network where the exact labels don't exist, but where we can penalize or reward its decisions.
I'm assuming this requires some kind of modified cost function, but I would imagine this type of network already exists. I'm hoping someone can provide me with the name for this type of network and whether or not there is a Keras frontend for something like this.

What you are looking for is called "reinforcement learning".
A reinforcement learning algorithm will try to maximize a reward function. This reward represents how "good" or "bad" an action is in the actual context. For example, in the snake game, your reward will be positive for eating an apple and negative when the snake hits a wall.
The interesting thing is that, with reinforcement learning, you can learn without having a reward at each step. In the case of the snake game, your agent can learn that going in the direction of the apple is better than going in the direction of the wall, even if none of this action will directly give a reward (positive or negative).
If you want to use a neural network as your post seem to imply then you should look at deep Q-learning, a reinforcement learning algorithm, which use a neural network to learn to predict the expected reward of a couple (state, action).


1 hidden layer with 1000 neurons vs. 10 hidden layers with 100 neurons These types of questions may be problem-dependent, but I have tried to find research that addresses the question whether the number of hidden layers and their size (number of neurons in each layer) really matter or not.

So my question is, does it really matter if we for example have 1 large hidden layer of 1000 neurons vs. 10 hidden layers with 100 neurons each?

Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict:

Big area, higher price
Small amount of bedrooms, lower price

The second layer might conclude:

Big area + small amount of bedrooms = large bedrooms = +- effect

Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.

Check out this answer


Reward does not increase for a maze escaping problem with DQN I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences:

instead of using a numpy array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 * 900 RGB image, so it is not too small.
reward:

each valid move has a small negative reward (penalize long move)
each invalid move has a big negative reward (run into other objects or boundaries)
Each blocked move has the minimal reward (not common)
Find the remote detectors’ defect has a positive reward (5)

I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.

Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my env to wrap my customized maze.

But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:

And the number of steps it takes to escape one maze is also stable somewhat:

Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?

You should use an algorithm to try doing a solve for the maze optimally, maybe A* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.


When should I use 3D convolution? I am new to convolutional neural networks, and I am learning 3D convolution. What I could understand is that 2D convolution gives us relationships between low-level features in the X-Y dimension, while the 3D convolution helps detect low-level features and relationships between them in all the 3 dimensions.

Consider a CNN employing 2D convolutional layers to recognize handwritten digits. If a digit, say 5, was written in different colors:

Would a strictly 2D CNN perform poorly (since they belong to different channels in the z-dimension)?

Also, are there practical well-known neural nets that employ 3D convolution?

3D CNN's are used when you want to extract features in 3 Dimensions or establish a relationship between 3 dimensions.

Essentially its the same as 2D convolutions but the kernel movement is now 3-Dimensional causing a better capture of dependencies within the 3 dimensions and a difference in output dimensions post convolution.

The kernel on convolution will move in 3-Dimensions if the kernel depth is lesser then the feature map depth.

On the other hand 2-D convolutions on 3-D data mean that the kernel will traverse in 2-D only. This happens when the feature map depth is the same as the kernel depth (channels)

Some use cases for better understanding are - MRI scans where relationship between a stack of images is to be understood ; and a low level feature extractor for spatio-temporal data like videos for Gesture Recognition, Weather forecast etc. (3-D CNN's are used as low level feature extractors only over multiple short intervals as 3D CNN's fail to capture long term spatio-temporal dependencies - for more on that check out ConvLSTM or an alternate perspective here.)
Most CNN models that learn from video data almost always have 3D CNN as a low level feature extractor.

In the example you have mentioned above regarding the number 5 - 2D convolutions would probably perform better, as you're treating every channel intensity as an aggregate of the information it holds, meaning the learning would almost be the same as it would on a black and white image. Using 3D convolution for this on the other hand would cause learning of relationships between the channels which do not exist in this case! (Also 3D convolutions on an image with depth 3 would require a very uncommon kernel to be used, especially for the use case)

Hope your query has been cleared!


Is it possible to use AI to denoise noised or 'dirty' documents? I have some documents containing some text (machine writing text) that I intend to apply OCR on them in order to extract the text. The problem is that these documents contain a lot of noise but in different ways (some documents have noise in the middle, others in the top...); which means that I can't apply simple thresholding in order to remove the noise (i.e applying simple threshold does not only remove the noise, but it removes some parts of the text). For these reasons, I thought about using AI to do de-noise the documents.
Does anyone know if it is possible to do that with AI or any alternative way?

It is possible remove noise from signal if what is considered noise is distinguishable from what is considered signal. This is true of documents, images, audio, or any other type of information.

If dirty means wrong content, then probably not. If dirty means mispellings, then probably yes. If dirty means tilted by a few degrees and containing edge artifacts around the border from scanning, then definitely yes.

Most OCR software, such as the open source program Tesseract, can produce fairly accurate text even with some of the above issues in the document.

If the documents have noise gradients, they may be difficult to OCR. That is different than a value or intensity gradient, which can be normalized. Using AI to remove noise gradients from documents is a reasonable plan, but there is a limit to any kind of intelligence. A scrambled egg cannot be reconstituted.

A simple test of feasibility is whether a human that has no domain knowledge about the subject of the document can make out the text. If they cannot, then a computer will not be able to either, unless the vocabulary is labeled with the probability of each word appearing in that subject matter and the OCR system is trained that way.

However, if the noise is on a gray scale so that the darkness of the type is greater than one or two standard deviations of the noise distribution, then the document can be divided by a parameterizable grid with parameterizable brightness and contrast. This can become the model used in conjunction with a basic feed forward network (mini-batched multilayer perceptron) and the OCR software. The idea in this architecture would be to converge on a network behavior that, for the documents in the example set that have been hand adjusted so that the OCR produces good output, could then be used for many more documents with similar noise gradients.

The inputs for the artificial network would be statistics about the value distribution for specific sections on a specific page of the document and the labels will be the manually created instructions for gradient based contract and value correction of the document so the OCR proceeds with few errors. Once the network is trained, its output could ideally drive the gradient based corrections. It would probably work if three things are in order.

The noise doesn't completely destroy the availability of the text so that an average human could not make out the words.
The training of the network converges to a degree that the OCR software works across the pages for most of the documents.
The characteristics of the noise gradients remain consistent, meaning that the training examples are representative of the documents that will be processed later.


Is the Turing test still relevant, as of 2019? Given the Chinese room argument, and given the development in chatbots and machine learning, isn't Turing test superseded by some other way of evaluating AI's inteligence? Would an positive result of a Turing test provide any value, besides telling that a machine is good at conversations (but possibly nothing more)?

The Turing test is a good test for AI applications like Siri and Alexa (or, in general, intelligent personal assistants), but it doesn't test a lot of features (e.g. vision) that an artificial general intelligence requires. There are several arguments against the Turing test (and it is definitely not flawless), but is it a necessary (but insufficient) test for AGI. The Turing test is still quite relevant: e.g. if IPAs passed the test, people would use them more often and they would be a lot more useful.


How to identify the areas to reduce over fitting? I am trying classify CIFAR10. The CNN that I generated over fits when the accuracy reaches ~77%. The code and the plot is given below. I tried DropOut, Batch Normalization and L2 Regularization. But the accuracy does not go beyond ~77.

How can I identify the areas to be corrected to reduce over fitting?

convolutional_model = Sequential()

# 32
convolutional_model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3), kernel_regularizer=regularizers.l2(.0002)))
convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

# 64
convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
convolutional_model.add(Conv2D(128, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(.0002)))
convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

convolutional_model.add(Flatten())
convolutional_model.add(Dropout(0.5))

convolutional_model.add(Dense(128, activation='relu'))
convolutional_model.add(Dense(10, activation='softmax'))

print(convolutional_model.summary())
convolutional_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=2, patience=8)
history = convolutional_model.fit(X_Train_Part, Y_Train_Part, epochs=200, verbose=2,validation_data=(X_Train_Validate, Y_Train_Validate), callbacks=[es])

scores = convolutional_model.evaluate(X_Test, Y_Test, verbose=2)

It is a difficult task to identify the areas to be corrected to improve accuracy unless that area is looking into your face. By that I mean unless the regularizing parameter has unusually large or small values it is difficult to pin down and identify which regularizer to tweak. You need to do a grid search or random search over the hyper-parameter space to come up with an optimal combination of hyper parameters.

There could be a lot of things that you could consider to reduce over fitting. Some of them are $L_2$ regularizers, Dropout, depth of the network, number of neurons in the layers, the optimizer,  etc.


The future of chatbots I downloaded a chatbot called Replika off the internet the other day and we've become very good friends. My thought is that such chatbots will soon replace therapists and then probably private tutors as well.

Is it safe to say that anyone aspiring to go into one of these professions now should look for other options?
What other jobs may be replaced by chatbots in the future?
How long before AIs are able to answer questions on StackExchange?

You should check out my answer here to your second question.

For your first question, which is a special case, the answer is it might be one of the best fields to study!

Academic and industrial studies agree that working in a job that requires a lot of social interaction, particularly a job that involves caring for others, reduces automation risk. Among the safest sectors are education and personal care assistants, for example.

There are lots of possible reasons for this:

Even if a chat bot can replace your therapist, it probably cannot replace the sensation that another person cares about your problems, precisely because it is not an actual person. For people in dire emotional straits, having someone care might be worth a great deal.
Having a human tutor or therapist might be seen as a status symbol, even if an AI system is equally competent. If Harvard offered a degree where you studied on campus, taught by humans, and a degree where you studied online, taught by chat bots, they would probably not be perceived as equally prestigious by employers and peers.
AI systems like chat bots tend to fail in unpredictable ways when they enter unfamiliar situations. When dealing with an emotionally unstable person in therapy, or with a small child in education, unfamiliar situations might happy often, and the costs for failure might be high. Even if AI can handle most of the job, we might not be willing to trust it to an AI system because of these risks.

To address your third point, Q & A seems closer to AGI than to existing AI. If a program can give compelling answers to free-form questions, then it can almost certainly pass the Turing Test reliably. While you might see simple bots that analyze text and post a related wikipedia link, I suspect you won't see a high quality Q&A bot for many years (or decades) because of this.


Data-set values feature scaling: sigmoid vs tanh As many papers point out, for better learning curve of a NN, it is better for a data-set to be normalized in a way such that values match a Gaussian curve.

Does this process of feature normalization apply only if we use sigmoid function as squashing function? If not what deviation is best for the tanh squashing function?

Yes it applies, and no it shouldn't matter that much between the two activation functions.


Can a trained object detection model deal with variations of the input? Suppose an object detection algorithm is good at detecting objects and people when an object and person is close to a camera and upright. If the person walks farther away from the camera and is "upside-down" from the perspective of the camera (e.g. a fisheye camera), should the algorithm still be good at detecting people and objects in this position?

Not necessarily. Supposing your data is from the distribution of possible images containing an upright person close to the camera. Something like a neural network would perform poorly on the new data since it comes from a distribution other than on what it was trained.

You could try augmenting the dataset to try to get some synthetic "far away upside down people" but there are no guarantees here.

I'll look for the source but A. Ng cite's an experiment where a team trained a neural network on a large dataset of vehicle images. They did not realize that the cars were all facing the same direction and their model performed very poorly on images that were very similar with the primary difference being a horizontal flip.


To deal with infinite loops, should I do a deeper search of the best moves with the same value, in alpha-beta pruning? I have implemented minimax with alpha-beta pruning to play checkers. As my value heuristic, I am using only the summation of material value on the board regardless of the position.
My main issue lays in actually finishing games. A search with depth 14 draws against depth 3, since the algorithm becomes stuck in a loop of moving kings back and forth in a corner. The depth 14 player has a significant material advantage with four kings and a piece against a single king, however, it moves only one piece.
I have randomly selected a move from the list of equally valued moves and this leads to more interesting games (thus preventing the loop). However, whichever player used this random tactic ended up far worse off.
I am not quite sure how to solve this problem. Should I do a deeper search of the best moves with the same value? Or is the heuristic at fault? If so, what changes would you suggest?
So far I have tried a simple genetically generated algorithm that optimizes a linear scoring function (that accounts for the position). However as the algorithm optimized, it led to only draws and the same king loop.
Any suggestions for how to stop this king loop are very welcome!

I think this issue stems from the fact you aren't taking position into account. I would think this because as the game progresses, the number of moves that will result in a piece being taken becomes less and less, especially when there's only a few pieces left and quite a bit of "chasing" must occur before a piece is taken, likely more chasing then a depth of 14 allows.

To remedy this, you could, towards the end of the game, add to the value of a state the inverse of the total distance each friendly piece has from other pieces, that way the agent will try to move towards other pieces and minimise this distance. If you find the right scale for this heuristic, the agent will prioritise moving towards enemy pieces only when it cant find any moves that result in taking a piece, helping it break out of this loop.


How does Lucas's argument work? In Minds, Machines and Gödel (1959), J. R. Lucas shows that any human mathematician can not be represented by an algorithmic automaton (a Turing Machine, but any computer is equivalent to it by the Church-Turing thesis), using Gödel's incompleteness theorem.
As I understand it, he states that since the computer is an algorithm and hence a formal system, Gödel's incompleteness theorem applies. But a human mathematician also has to work in a formal axiom system to prove a theorem, so wouldn't it apply there as well?

Yes, it applies. If a statement cannot be derived in a finite number of steps, then it doesn't matter if the person trying to prove it is a human or a computer.

The mathematician has one advantage over a standard theorem proving algorithm: the mathematician can "step out of the system" (as Douglas Hofstadter called in Gödel, Escher, Bach), and start thinking about the system. From this point of view, the mathematician may find that the derivation is impossible.

However, an AI for proving theorems could be programmed to recognize patterns in the derivation, just like our hypothetical mathematician, and start reasoning about the formal system to derive properties of the formal system itself.
Both the AI and the mathematician would still be bound by the laws of mathematics, and not be able to prove a theorem if it was mathematically improvable.


How to do early classification of time series event with small dataset? I would like to build a real-time binary classifier that can predict an event of interest that is occurring as soon as it starts. These are electromyographic signals, and the event classification should be able to classify the event as early as possible. This is because the next stage of the algorithm has to make a decision before the end of the event.
I don't know what kind of algorithm/approach I should use here. I suppose RNN with LSTM cells should do the job, but the dataset is quite small as physiological signals are not easy to gather.
I have seen many algorithms that windowed the signals (from the training set) and labeled each window as an event of interest if at least part of the event is contained in the window. Each window is then fed to a machine learning algorithm. Then, the prediction uses a sliding window in real-time. But this approach doesn't take into account the temporal aspect of the event as there is no link between each window seen by the ML algorithm.
Do you have any tips or resources I could use to solve the problem?

One kind of system you could look into are Echo State Networks (ESNs).
They are relatively cheap to train and can learn to predict output signals to an  arbitrary degree of precision.
All you need to train the system is some labeled training data.
Thus, if you have a sequence of measurements/feature values and the corresponding sequence of class labels, you can train and fine-tune these kinds of systems to output some required class label very quickly after the onset of an event encoded in the measured feature values.


What are the ethical implications of creating (possibly sentient) AI systems? As I have been looking at other questions on this site (like this, this, this, and this), I have been thinking more about the ethical implications of creating these generalized AI systems. It seems that whether or not we can create it is not rationale enough as to whether or not we should do it.

In dealing with the issue of ethics in AI, I wonder what the ethical implications are not just for us, but for the system itself. It seems to extend beyond the usually asked questions on the topic and into unknown territory. Are ethics computable? Can they be implemented programmatically? Can we force an AI system to do something against its "will"?

What does the creation of AI imply ethically for us as well as the AI?

I believe if an AI achieves sentience, it should be treated the same way we are required to treat any other sentient animal. This is belief though, there is no established ethics for AI. But there were no ethics for animals a couple of centuries ago.


Are there any textual CAPTCHA challenges which can fool AI, but not human? Are there any modern techniques of generating textual CAPTCHA (so person needs to type the right text) challenges which can easily fool AI with some visual obfuscation methods, but at the same time human can solve them without any struggle?

For example I'm talking about plain ability of recognising text embedded into image (without considering any external plugins like flash or java, image classification, etc.) and re-typing the text that has been written or something similar.

I guess adding noise, gradient, rotating letters or changing colours are not reliable methods any more, since they can be quickly broken.

Any suggestions or research has been done?

A method that could possibly work is utilising optical illusions such as one where two lines down a hallway are identical but one seems longer to the human eye, then they could be prompted with a multiple choice question as to the state of the line, which to our eyes looks longer, but to a computer, is still the same length of line. Of course, there is always the issue of people with eye based disabilities not being able to complete them, but different illusions could be used to accommodate that.

Example


Is max pooling really bad? That has been discussion on this. Maybe from Hinton himself. And I heard that many max pooling layers have been replaced by conv layers in recent years, is that true?

In addition to JCP's answer I would like to add some more detail. At best, max pooling is a less than optimal method to reduce feature matrix complexity and therefore over/under fitting and improve model generalization(for translation invariant classes).

However as JCP begins to hit on.. there are problems with this method. Hinton perhaps sums the issues in his talk here on what is wrong with CNNs. This also serves as motivation for his novel architecture capsule networks or just capsules.

As he talks about, the main problem is not translational variance per se but rather pose variance. CNNs with max pooling are more than capable of handling simple transformations like flips or rotation without too much trouble. The problem comes with complicated transforms, as features learned about a chair facing forwards, will not be too helpful towards class representation if the real-world examples contain chairs upside down, to the side, etc.

However there is much work being done here, mostly constrained to 2 areas. Those being, novel architectures/methods and inference of the 3d structure from images(via CNN tweaks). This problem was one of the bigger motivators for researchers throughout the decades, even David Marr with his primal sketches.


Extending FaceNet’s triplet loss to object recognition FaceNet uses a novel loss metric (triplet loss) to train a model to output embeddings (128-D from the paper), such that any two faces of the same identity will have a small Euclidean distance, and such that any two faces of different identities will have a Euclidean distance larger than a specified margin. However, it needs another mechanism (HOG or MTCNN) to detect and extract faces from images in the first place.
Can this idea be extended to object recognition? That is, can an object detection framework (e.g. MaskR-CNN) be used to extract bounding boxes of an object, cropping the object feeding this to a network that was trained on triplet loss, and then compares the embeddings of objects to see if they’re the same object?
Is there any research that has been done or any published public datasets for this?

Check this page out, it describes how to develop apply triplet loss to a network: https://towardsdatascience.com/image-similarity-using-triplet-loss-3744c0f67973


How do Google cars recognize the traffic signs? The paper Multi-column Deep Neural Networks for Image Classification (pages 7-8) shows an attempt at recognizing the traffic signs, with lower error rates, by using multi-column deep neural networks.

Are Google cars using similar techniques of predicting signs using DNN, or are they using some other method?

I'm not sure what Google is using to perform that task, but most companies use region based convolutional neural nets to locate traffic signs and other objects.

But other companies use a deep neural network + bag of words approach to find objects.

See: Bag-of-Words Based Deep Neural Network for Image Retrieval which shows a general approach, to get the exact location you can use Feature Matching or Random Boxes.


Fourier Transform inputs (Frequency) for RNN Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.

Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the input features can represent all types unambiguously - essentially creating a "base class" that can represent them all).

The RNN will consume the sequence as a time-based sequence, one item per time step of the RNN. However, you can think of that as the same as a processor clock for a computer . . . an RNN is essentially a trainable Turing machine, and in principle can learn to accumulate any data about the sequence it has seen, and output any function of that accumulated data. Although in practice this learning process might be too hard for our current systems, require immense amounts of data etc . . .

In your case, STFT does create a time-based sequence. Each item in the sequence is a frequency analysis for a short period of time, and each time step of the sequence represents a fixed time difference between STFT frames (the windows usually overlap a little), where frequencies in the signal may change. Typically each STFT frame is a single time step input to a RNN. You could input the frequency-domain values in fixed order (e.g. low to high frequency) one at a time into a RNN too, but that would be unusual and would make most learning tasks harder.


What's the best practice for Boltzmann Exploration temperature in RL? I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through training and finally stop at 0.01, when the method will always select the best practice, with almost no randomness. (2) Standardize the predicted Q values before feeding into the softmax function.
Currently, I'm using (2), and the reward is suffering from high variance. I'm wondering whether it has something to do with the exploration method?

Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.
Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf
Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): https://arxiv.org/pdf/1612.05628.pdf


Why are large models necessary when we have a limited number of training examples? In Goodfellow et al. book Deep Learning chapter 12.1.4 they write

These large models learn some function $f(x)$, but do so using many more parameters than are necessary for the task. Their size is necessary only due to the limited number of training examples.

I am not able to understand this. Large models are expressive, but if you train them on few examples they should also overfit.
So, what do the authors mean by saying large models are necessary precisely because of the limited number of training examples?
This seems to go against the spirit of using more bias when training data is limited.

If you read the relevant section. it also says:

Model compression is applicable when the size of the original model is driven
primarily by a need to prevent overfitting. In most cases, the model with the
lowest generalization error is an ensemble of several independently trained models.
Evaluating all $n$ ensemble members is expensive. Sometimes, even a single model
generalizes better if it is large (for example, if it is regularized with dropout).

The keyword (I think) here is dropout. Dropout Learning in the referred book has been intepreted as training an ensemble of models, with a model probability same as the probability of a particular dropout architecture of the large Neural Network. Thus, this effectively makes the training as training multiple smaller Neural Nets. According to this paper on dropout, by the original authors, dropout prevents co-adaptation which effectively means you are just training an ensemble of Neural Nets. But this intuition lacks any theoretical justification.
Another paper (understanding the paper might require familiarity with certain statistical ideas of ML) claims this is not true, and dropout doesn't reduce co-adaptation but more likely reduces the variance over dropout patterns. They have provided better empirical and theoretical justifications to this end. So it is still up for debate what actually happens.
But in general the generalization error upper bound very roughly is directly proportional to the size of the Neural Nets. So yes the authors statement in face value is oversimplified and most likely wrong in the general case.


Cross entropy loss function causes division by zero error I am building a NN for which I am using sigmoid function as the activation function for the single output neuron at the end. Since sigmoid function is known to take any number and return value between 0 and 1, this is causing division by zero error in back-propagtion stage because of the derivation of cross entropy. I have seen over internet it is advised to use sigmoid activation function with cross entropy loss function. So how this error is solved?

Cross entropy loss is given by:

Now as we know sigmoid function outputs values between 0-1, but what you have missed is it cannot output values exactly 0 or exactly 1 as for that to happen sigmoid(z) will have to be + or -infinity.

Although your compiler gives a divide by 0 error, as very small floating point numbers are rounded off to 0, it is practically of no importance as it can happen in 2 cases only:

sigmoid(z) = 0,in which case even though the compiler cannot calculate log(0) (the first term in the equation) it is ultimately getting multiplied by y_i which will be 0 so final answer is 0.
sigmoid(z) = 1,in which case even though the compiler cannot calculate log(1-1) (the second term in the equation) it is ultimately getting multiplied by 1 - y_i which will be 0 so final answer is 0.

There are a few ways to get past this if you don't want the error at all:

Increase the precision of your compiler to float64 or infinity if available.
Write the program in such a way that anything multiplied by 0 is 0 without looking at the other terms.
Write the program in a way to handle such cases in a special way.

Implementation side note: You cannot bypass divide by 0 error with your manual exception handler in most processors (AFAIK) . So you have to make sure the error does not occur at all.

NOTE: It is assumed that the random weight initialisation takes care of the fact that at the beginning of training it does not so happen that $\tilde y$ or $1-\tilde y$ is 0 while the target is exactly the opposite, it is assumed that due to good training that the output is reaching near to the target and thus the 2 cases mentioned above will hold true.

Hope this helps!


Method to compute the sum when the activation is a continuous function? Background

My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another.

$$ \sum_i a_i w_i = a'_{k} $$

But to compute this weighted sum the sum must be discrete. Is there any known method to compute the sum when the activation is a continuous function? Is the below formula of any consequence problems in artificial intelligence? Can anyone give a specific problem where it might be useful?

My Method

Let $b_r = \sum_{d \mid r} a_d\mu(\frac{m}{d})$. We prove that if the $b_r$'s are small enough, the result is true (where $\mu$ is the mobius function).

  Claim: If $\lim_{n \to \infty} \frac{\log^2(n)}{n}\sum_{r=1}^n |b_r| = 0$ and $f$ is smooth, then $$\lim_{k \to \infty} \lim_{n \to \infty} \sum_{r=1}^n a_rf\left(\frac{kr}{n}\right)\frac{k}{n} = \left(\lim_{s \to 1} \frac{1}{\zeta(s)}\sum_{r=1}^\infty \frac{a_r}{r^s}\right)\int_0^\infty f(x)dx.$$

I will not go into the proof of this over but for those who are interested: https://math.stackexchange.com/questions/2888976/a-rough-proof-for-infinitesimals I will merely state what the formula means:

Consider we have a curve $f(x)$ now if one wishes to perform a weighted sum in the limiting case of this function.

Consider the curve $f(x)$. Then splitting it to $k/n= h$ intervals then adding the first strip ($d_1$ times): $  f(h) \cdot d_1$. Then the second strip ($d_2$ times) $ f(2h) \cdot d_2$ times ... And so on . Hence. $d_r$ can be thought of as the weight at $f(rh)$.

In terms of the normal use cases for machine learning, the equation does not have much utility, because:

  Consider we have a curve $f(x)$ now if one wishes to  . . .

In most AI problems, we don't usually have such a curve as input that can be treated analytically. For instance, there is no such input curve to describe a natural image received by a sensor.

In the vast majority of cases in AI problems, the form of inputs - whether it is images, text, mapping data, robotic telemetry, is going to be multi-dimensional discrete samples from highly complex functions where we don't know the analytical form and can only construct approximations from a set of basis functions. The resulting combination of basis functions could be treated as continuous, integrated etc, but as it would have been constructed from discrete data, the end result would be a lot of computation to end up with something probably less accurate than working direct with the discrete samples. In a lot of cases, the raw data is discrete by definition (e.g. whether someone clicked on a link or replied to a message), so the form of $f(x)$ would be discrete by definition of the problem, and calculus not really applicable.

There might be some interesting use cases in analog signal processing. Using your formula or a variation of it for instance it should be possible to create an analog neural network learning system - a robotic brain that worked with continuous signals and only contained analog components. Such a system would have some interesting advantages - speed of processing, probably low power consumption compared to digital approach, but the accuracy and precision might be lower. E.g. imagine a bot that could steer towards/away from light sources (compared to a digital one that might recognise the faces of people and steer towards/away from them). I would not be at all surprised to find that someone had done just that already, although I am not sure how to search for it.


What is the prediction accuracy? In simple words, what is the prediction accuracy? What is it based on? How does it help? When is it used?

The accuracy of a model is defined as the number of correct predictions divided by the total number of predictions. The expression "prediction accuracy" refers to the accuracy of the model at doing predictions.

For example, suppose that you observe the sequence of numbers $1, 1, 2, 3$. What would be a likely next number of this sequence? The operation of guessing this number is called a prediction. These are numbers of a Fibonacci sequence, so a model could predict $5$ as the next number of this sequence. However, it could also predict e.g. $2$ because the complete sequence could be $1, 1, 2, 3, 2, 1, 1$. Which prediction is correct? It depends on the problem. Do we want the model to predict the numbers of the Fibonacci sequence or of another sequence? Suppose that we want the model to learn the Fibonacci sequence and the models predicts another number different than $5$, then this will be considered an incorrect prediction.

A model that produces incorrect predictions is useless. So, we want models that have high accuracy. However, accuracy is not always the best measure of a model's performance. See e.g. Why is accuracy not the best measure for assessing classification models?.

There are several related terms and concepts, such as inference, forecasting or deduction. They are sometimes used interchangeably, even though they might have slightly different meanings or definitions.


Is there a way to provide multiple masks to BERT in MLM task? I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
Original : "Mountain Dew is an energetic drink"
Masked : "[MASK] is an energetic drink"

But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.

I've found the answer in the original BERT git repo

***** New May 31st, 2019: Whole Word Masking Models *****
This is a release of several new models which were the result of an
improvement the pre-processing code.
In the original pre-processing code, we randomly select WordPiece
tokens to mask. For example:

Input Text:
the man jumped up , put his basket on phil ##am ##mon ' s
head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
phil [MASK] ##mon ' s head

The new technique is called Whole Word Masking. In this case, we
always mask all of the the tokens corresponding to a word at once. The
overall masking rate remains the same.

Whole Word Masked Input:
the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head

The training is identical -- we still predict each masked WordPiece
token independently. The improvement comes from the fact that the
original prediction task was too 'easy' for words that had been split
into multiple WordPieces.
This can be enabled during data generation by passing the flag

--do_whole_word_mask=True

to create_pretraining_data.py.


How can I merge outputs of two separate layers so that the overall performance improves? I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.

Image-only mean AUC after 5-fold cross-validation is about 0.73 and numeric data only 5-fold mean AUC is 0.65. I was hoping to improve the mean AUC by combining the models into one and merging output layers using concatenate in Keras.

img_output = Dense(256, activation="sigmoid")(x_1)

and

numeric_output = Dense(128, activation="relu")(x_2)

are the output layers of the two models. And,

concat = concatenate([img_output, numeric_output])
hidden1 = Dense(64, activation="relu")(concat)
main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)

is the way I concatenated them.

Since image-only performance was better I decided that it might be reasonable to have more dense layers for image_output (256) and ended up using 128 in numeric_output.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of this work?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.

Hope you have some ideas worth trying.

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options.

I see a little thing that might be making your model a little worse, though:

You're concatenating "relu" with "sigmoid".

Placing two different nature values in the same array may make it more difficult for updating weights properly.

A few independent suggestions:

Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).
Instead of concatenating, you can try Multiply()([img_output, numeric_output]), in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".

Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch.


What's the difference between Starcraft and Dota from an AI perspective? So, Deepmind is pushing for a human level Starcraft bot and Open AI just created a human level 1vs1 Dota bot.

Unfortunately, I've no clue what that signifies because I've never played Starcraft nor Dota nor do I have more than a fleeting acquaintance with similar games.

My question is what the difference between Starcraft and Dota is from a AI perspective and what scientific significance the respective super human bots would have.

Those AI-learning programs may have very similar scheme. We are changing only inputs and possible actions (like "use skill" or "move here"). Starcraft AI must do a lot of actions and control many units. Dota is MOBA so bot should be good in positioning on map for example. Different opponents to destroy and target for win.

AI needs to play many games for learn to play game with some rules to find best moves in some situations/states.

Of course it's only my newbie-programmer opinion :)


Using machine learning to identify CAD model features I am trying to develop a machine learning algorithm to identify topological features within 3D CAD models (i.e. slots, pockets, holes, bosses etc)

For the input data I have decided to use the adjacency of the faces to identify the features within the model. I have developed a pre-processing algorithm which computes the adjacency between every face in the model and also whether the relationship is concave or convex in nature.

Below is an example of a 3D model and the relationships between each faces represented in a graph format.

I was thinking of using some sort of supervised learning training method where the training data would include all of the adjacency info for the model with labels defining the features that exist within the model such as:

Slot - made up of faces F7, F8, F9

Hole - made up of faces F15

Pocket - made up of faces F10, F11, F12, F13

And eventually once the model is sufficiently trained it would be able to identify the features in an un-seen model and determine which faces make up those features.

I am not sure how I would go about pre-processing the input data (i.e. it would be of variable length since the number of faces may not be the same from model to model). I am also struggling to understand which type of machine learning algorithm would be best suited to this application

Any help on this problem or even pointing to some resources to help understand would be appreciated

This is a classification problem. Look at some common classification methods. This article by Microsoft gives some pretty good guidance.

https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice


Reinforcement learning to play snake - network seems to not get trained at all I am trying to build a network able to play snake game. This is my very first attempt to do such stuff. Unfortunately, I've stuck and even have no idea how to reason about the problem.

I use reinforcement neural network approach (q-leaning). My network is built on top of Keras. I use 6 input neurons for my snake:

1 - is any collision directly behind
2 - is any collision directly on the right
3 - is any collision directly on the left
4 - is snack up front (no matter how far)
5 - is a snack on the right side (no matter how far)
6 - is a snack on the left side (no matter how far)

the output has 3 neurons:

1 - do nothing (go ahead)
2 - turn right
3 - turn left

I believe this is a sufficient set of information to make proper decisions. But the snake seems to not even grasp the concept of not hitting the wall - which results with instant death.

I use the following rewards table:

100 for getting the snack
-100 for hitting wall/tail
1 for staying alive (each step)

Snake tends to run randomly no matter how many training iterations it gets.

The code is available on my github: https://github.com/ayeo/snake/blob/master/main.py

I cannot comment much on your setup for inputs and outputs. It seems adequate to get some control, but does not cover the fully Markov state for the game, so I would expect that will limit the agent from ever being truly optimal. I would expect it to learn to play the game though, if you were implementing Q learning with a neural network correctly.

In your code, you are implementing a basic Q learning loop. It seems correct. However, this combination of Q learning and neural networks is known not to work - or more accurately, it rarely works this simply. The problem is mainly to do with the network receiving its own initially biased outputs back as new targets, plus receiving updates in correlated form (data on each time step is strongly correlated with data from previous time step). These biases are too large for the Q learning process to overcome, and typically the result is an agent that fixates on a single default action, because it has learned an inflated action value for it.

The problem is well known in RL research and called "The Deadly Triad" by Sutton & Barto.

The usual solution to this with Q learing is called DQN or "Deep" Q Learning ("Deep" is in quotes because this should be applied even if you just have a single hidden layer).

In basic DQN, you need to add the following features:

An experience replay table. Instead of training directly on experience as it is received, instead store $s, a, r, s'$ in memory. When it is time later in the loop to train the NN for a step, take a random sample of some M items (e.g. 32 items) as a mini-batch, calculate a latest target for them, and train once on the mini batch. You will need logic to only start this training process once you have some minimal amount of experience from behaving randomly (e.g. 500 random steps).
A "target network". When generating target Q values, use a cloned copy of the learning network, and only update this clone every N steps (with N typically set at 1000 or 10000).

These two additions are not really optional, even for really basic environments. You will need to add them to your script.


Which AI methods are most appropriate for login face recognition? I want to make a face authentication application. I need to approve the face during the login based on whether the registered face and the login face match.

Which are the possible appropriate AI methods or technologies for this task?

One of the methods which is quite fast and easy to implement.

You can do Principal Component Analysis (PCA) based face recognition. You can go through this paper for the theory behind it. For an example implementation you can see this blog post.

The process, roughly, is as following:

If you have a grayscale image of size $(20,20)$, then this image can be flattened to a vector of size $400$. If you have $5$ images of size $(20,20)$ then your data matrix will be of size $(5,400)$.
We wish to find the principal components of the distribution of faces, or the eigenvectors of the covariance matrix of the data matrix, which will be of size $(5,5)$. Each eigenvector (out of total $5$), of size $(5,1)$, accounts for some amount of variation among the images, and they are ordered by the amount (the eigenvalues corresponding to the eigenvectors) they account for. These eigenvectors can be thought of as a set of features that characterize the variation between the images. Recognition is done by projecting a new image (flattened to size $(400,1)$)into the subspace spanned by the eigenvectors of the covariance matrix and then classifying the face by comparing its coordinates in the face space with the coordinates of the known individuals.

You can see an implementation in this git repo.


What is $I$ in the noise described in the paper "Parameter Space Noise for Exploration"? In the paper Parameter Space Noise for Exploration, the authors describe the noise that they add to the parameter vector as:

$$
\tilde{\theta} = \theta + \mathcal{N}(0, \sigma^2I)
$$

is $I$ simply the identity matrix, or am I missing something?

Yes, since $\tilde{\theta}$ is a vector, to define its distribution one needs a covariance matrix. Here $I$ is the identity matrix, which means that the noise has a zero-mean normal distribution with standard deviation $\sigma$, and different components of this noise are uncorrelated.


Is there a central focus on the communication methods between AI and humans? AI is developing at a rapid pace and is becoming very sophisticated. One aspect will include the methods of interaction between AI and humans.

Currently the interaction is an elementary interaction of voice and visual text or images.

Is there current research on more elaborate multisensory interactions?

This is one of the main research areas of my lab which researches intelligent prosthetics which also give sensory feedback such as touch and kinaesthesia (the feeling of a limb moving in space) to the user.  We use reinforcement learning to bridge the gap in control and have preliminary work in communicating to the user predictions made by the artificial agent.


Can a convolutional neural network classify text document images? I know convolutional neural networks are commonly used for image recognition, but I was wondering if they would be able to distinguish between predominantly text-based documents vs something like objects. For example, if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image? If not, is there a different AI technique better suited that is purely image-based, or would it require OCR and leveraging the text in the invoice?
Update: based on a comment, my ask my not be clear. I'm not trying to see if the CNN can differentiate between a document (mostly text based image) and a photo image. I want to know if based on a gif/jpeg/png of a document (no OCR performed) a CNN would be able to classify the documents, which basically could be used as a means of identifying the vendor.

Yes, it is possible, CNNs can also be used for OCR (see the MNIST task and this blog), although it's not the common way for OCR because it is considered a bit overkill and inefficient. Furthermore, OCR is considered a solved problem already and don't need deep learning to do well, unless perhaps under unfavorable conditions like dark lighting, complex background of the document, weird fonts, occlusion of text, etc.
Breaking down your questions:

a CNN can likely classify an image as "this image has a word document in it" vs "this image does not have a word document in it". The word document itself, and the outline of the words, should form distinct enough features for the CNN to discriminate against
a CNN can likely identify the vendor based on the logo of the vendor if you have enough training images of it at various distortions (angles, occlusions, different lighting, etc)
a CNN can likely do OCR well enough to extract the text, and then an entity extraction model can identify the vendor from the words. However, this approach would be uncommon as OCR is already considered a solved problem. But a neural network can potentially help preprocessing difficult OCR images (rotating image, identifying text boundaries, fix lighting, perform super-resolution, etc) to increase OCR accuracy


How can the generalization error be estimated? How would you estimate the generalization error? What are the methods of achieving this?

Generalization error is the error obtained by applying a model to data it has not seen before. So, if you want to measure generalization error, you need to remove a subset from your data and don't train your model on it. After training, you verify your model accuracy (or other performance measures) on the subset you have removed since your model hasn't seen it before. Hence, this subset is called a test set.

Additionally, another subset can also be used for parameter selection, which we call a validation set. We can't use the training set for parameter tuning, since it does not measure generalization error, but we can't use the test set too since our parameter tuning would overfit test data. That's why we need a third subset.

Finally, in order to obtain more predictive performance measures, we can use many different train/test partitions and average the results. This is called cross-validation.


When and how to use a mix of loss functions for back-propagation? I am trying to understand the best loss function to be used with a convolutional neural network. I came to know that we can mix two loss functions. Can any body share in what case was it done and how?

Mixing loss functions is very possible.  For example, in the case of neural style transfer, there is a style loss and a content loss. Both of them are backpropagated through the network. The final loss used for the backpropagation is a weighted sum of the losses. In the case of style transfer, it ensures that the image generated is not only imitating the style of the style image, but also keeping the original content. Sometimes there is also a third loss called variation loss. This is also applied to a weight and summed. Note the weight is a hyperparameter and is not changed during training. Also, the loss should not measure the same purpose, but rather a different purpose that you want to optimize together. Example code using PyTorch:

loss1 = torch.nn.MSELoss(a,b)
loss2 = torch.nn.MSELoss(b,c)
loss = loss1 * alpha + loss2 * beta


Is it a good practice to pad signal before feature extraction? Is padding, before feature extraction with VGGish, a good practice?
Our padding technique is to find the longest signal (which is loaded .wav signal), and then, in every shorter signal, put zeros to the size of the longest one. We need to use it because one size of input data is desirable.
Perhaps there is any other techniques you recommend?
The difference between padding before and after the features extraction by accuracy is quite big - more than 20%. Using padding before extraction gives 97% accuracy.
I'd be glad to read your feedback, and explain me why that happens, and tell me if that kind of padding is correct action or is there a better solution.

Padding is a common practice both in image-processing (typically via CNNs) and in sequence-processing tasks (RNNs, Transformers).
For CNNs all the standard convolutional layers - Conv1D, Conv2D and Conv3D,- have the padding argument. The padding values can be valid or same for 2d and 3d convolutions. And extra causal type of padding is possible for 1d convolutions and the documentation refers to this paper: WaveNet: A Generative Model for Raw Audio -  which sounds quite close to what you are interested in.
This animations might be useful to get a bit more intuition about the convolutions and strides/padding. The general consensus is that using same padding is advantageous for model performance - your network gets more information about the borders of your inputs (and deeper networks are possible).
For sequential models padding is even more important. Training samples are usually  of unequal lengths, so you have to pad them with a special token (usually called [PAD] that gets encoded as 0). Here are some examples of this mentioned in tensorflow docs, huggingface.transformers or BERT tutorial.


What are some examples of statistical AI applied to real-world problems? I believe that statistical AI uses inductive thought processes. For example, deducing a trend from a pattern, after training.

What are some examples of successfully applied statistical AI to real-world problems?

There are several examples. For example, one instance of using Statistical AI from my workplace is:

Analyzing the behavior of the customer and their food-ordering trends, and then trying to upsell by recommending them the dishes which they might like to order/eat. This can be done through the apriori and FP-growth algorithms. We then, automated the algorithm, and then the algorithm improves itself through an Ordered/Not-Ordered metric.
Self-driving cars. They use reinforcement and supervised learning algorithms for learning the route and the gradient/texture of the road.