Question classification according to chapters I have a corpus, say an instruction manual. The text in this manual is grouped into chapters and each chapter is split up into sections. For example, Chapter 1/Section 1, Chapter 1/Section 2 and so on.
Assume the corpus has C chapters and each chapter has S sections. My goal is, given a sentence or question, to classify this sentence/question. In other words I want to compute three most probable chapters to which this sentence or question belongs to.
I tried MultinomialNB model using sklealrn, but it did not give me the desired result. I want to try another approach, for example using a Neural Network and compare it with the MultinomialNB model. I have Googled and found Doc2Vec but haven't tried yet.

Can anyone suggest a better or another possible approach so that I could try and compare? What is the standard approach to such kind of problem?

You have got text which have words. Now each of the words have dependency with words that are occurring previously or/and with the ones occurring later. So to capture the "context" in which a particular word occurs is done by sequence models in deep learning. Sequence models that are quite used often are Recurrent Neural Networks (RNN), Gated Recurrent Units (GRU) and Long Short Term Memory (LSTM) listed in the order of their functionality and complexity. To know more about them visit Jonathan Hui's blog.

Your problem is a classification problem i.e. given a sentence the classifier should tell which chapter and section it belongs to. You can use one of the sequence models to "encode" the sentence as a vector (based on the context of words in it and meaning of sentence) and pass that vector to a fully-connected neural network that has a softmax layer at the end to tell which chapter and section the text belongs to. The number of classes you are having are C * S (one for each chapter-section pair). From the output of softmax layer you can pick up the top 3 classes that got the highest probability for a sentence.

Doc2Vec converts a paragraph (can also be a sentence) into a vector representation. The Doc2Vec model has been trained over a huge corpus of text to be able to capture the context of a text and represent it as a high dimensional vector.

The corpus of text that you have, is from a probability distribution is quite different from that used for training Doc2Vec model. Sequence models (mentioned above) can learn that probability distribution for your text corpus exclusively. In other words, you have a tailor-made representation for your text so that the context can be understood much better.

That is why I suggested you to use sequence models. You can also compare the performance obtained by using pretrained Doc2Vec as text representation with that obtained by using one of the sequence models.

To know about the effectiveness of RNN visit Andrej Karpathy's blog.


Appropriateness of 3D Convolutional Neural Network for segmentation of medical image data I have a couple different segmentation tasks that I would like to perform on medical imaging data using CNN's. I'm currently trying to wrap my head around how well a 3D network might work, using a U-Net architecture, but I have some hesitation.

My specific questions are as follows:

Question 1 - Say we have medical imaging taken at different slices (different heights/depths) of the patient's body. In order for a 3D CNN to work properly on such data, do the images need to be taken at a heights that are close together, i.e. does the 3rd dimension need to be fairly continuous? For example, if you had a 3D stack/volume of 5 images that you wanted to feed to a 3D CNN like so,

img_1_depth_0cm.png
img_2_depth_5cm.png
img_3_depth_10cm.png
img_4_depth_15cm.png
img_5_depth_20cm.png

and the images were taken 5 cm apart from one and other, I would imagine that a 3D convolution might not perform very well because of the 5 cm of depth in between images? Is this an incorrect assumption?

(As a reference point, this nice repository on GitHub was designed for training on images of the brain, but the images do appear to be fairly continuous/close together, almost like a video: https://github.com/ellisdg/3DUnetCNN)

Question 2 - For a 3D network to properly segment non-labelled volumes after training is finished, I know that the input data dimensions would have to match those of the training data.

But I would also think that the new images must have been taken in a similar fashion (taken at similar depths and in general a similar orientation to the training data) in order for the NN to be able to perform its task. So unless the medical imaging processes are always performed similarly across machines and hospitals, I'm guessing that the NN performance might vary pretty wildly when it tries to segment new data. Is this correct?

In CNN, different height/width are not a problem. You can choose one specific size for your model and pads zero around the slices to make them of equal size.

No, your assumption is not correct. You are doing end to end training, so it does not matter, which sequence are you following.


Can AutoKeras be used for neural networks of PyTorch I use PyTorch, bauces AllenNLP is built on it and good libraries are for it. But can AutoKeras be used for PyTorch based ML pipelines, or am I required to switch to Keras? Google is quite silent when asked for tutorials for this combination. Maybe PyTorch is lacking automated ML framework?

"Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano."

But I found a useful tool which collects different deep learning model convertors.

Hope this will be useful for you :D


Why does nobody use decision trees for visual question answering? I'm starting a project that will involve computer vision, visual question answering, and explainability. I am currently choosing what type of algorithm to use for my classifier - a neural network or a decision tree.
It would seem to me that, because I want my system to include explainability, a decision tree would be the best choice. Decision trees are interpretable, whereas neural nets are like a black box.
The other differences I'm aware of are: decision trees are faster, neural networks are more accurate, and neural networks are better at modelling nonlinearity.
In all of the research I've done on computer vision and visual question answering, everyone uses neural networks, and no one seems to be using decision trees. Why? Is it for the accuracy? I think a decision tree would be better because it is fast and interpretable, but if no one's using them for visual question answering, they must have a disadvantage that I haven't noticed.

For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better to inputs and maintain performance when inputs have undergone certain linear transformations (e.g. some scaling or a translation along the x-axis). These properties, along with the robust frameworks that exist for developing and deploying neural nets, and the fact that they have been shown to widely produce very good results, are some of the reasons they’re used.

In terms of being a black box, while this is true for a lot of applications it’s actually less true for image-based tasks. The layers of a well designed and trained convolutional neural network model can actually be visualized and made quite interpretable; from these visualizations, it’s often clear how the representation roughly works. In contrast, I’d argue that while a decision tree is theoretically easier to interpret for some tasks (say medical decision making), this is less the case for vision tasks because we don’t interpret images one pixel at a time. People are almost always interested in the higher level representation within an image (say, a cat, a leaf, or a face). Decision trees tend to have trouble capturing these higher level representations.

Distill.pub has a nice explanation of feature visualization that may be of interest.


Are perfect and imperfect information games modelled as fully and partially observable environments, respectively? In perfect information games, the agent can see all the moves performed in the past. Besides, it can observe the next action that will be put into practice by the opponent.

In this case, can we say that perfect information games are actually a fully observable environment? If we reach this conclusion, I guess that imperfect information becomes a partially observable environment?

There is indeed a close parallel here, but the concepts are distinct. Every perfect information game is fully observable, but not every fully observable game is a game of perfect information.

A game of imperfect information is one in which you lack knowledge of any of the following:

The state of the game (e.g. current market prices).
The rewards you will receive from various states (i.e. utility and cost functions).

In contrast, in partially observable process (specifically, a POMDP), the requirement is that you must not know which state you are in.

This is a subtle distinction, so here are some examples:

A multi-armed bandit game with stationary distributions. Here, you know which state you are in (in fact, if the distributions are stationary, you know that the state doesn't change, except for the value of your winnings). You are not in a POMDP (the game is fully observable), but you are operating with imperfect information, because you don't know the utility function associated with different actions. You are operating in a regular MDP.
The game of chess has perfect information, and is also thus fully observable.
The game of poker has imperfect information because you cannot observe the current state of the game (you can't see the cards in your opponent's hand). It is thus a POMDP.


What are some examples of intelligent agents for each intelligent agent class? There are several classes of intelligent agents, such as:

simple reflex agents
model-based reflex agents
goal-based agents
utility-based agents
learning agents

Each of these agents behaves slightly different from the other agents. There are certain diagrams that describe each of these IA classes. However, all the descriptions are usually quite abstract.

What are some examples of intelligent agents for each intelligent agent class? Optionally, I would also like to see a compact definition of each class (and maybe how they are related to the diagrams).

There are no distinguishable hardware examples for each IA class. The same mobile robot architecture with proper sensors can be implemented to behave as any IA class. The way you can determine the class of an intelligent agent is from the way it processes the percept. Based on chapter 2 of Artificial Intelligent: A Modern Approach I will try to give a concise explanation for each class:

A simple reflex agent takes an action based on only the current environmental situation; it maps the current percept into proper action ignoring the history of percepts. The mapping process could be simply a table-based or by any rule-based matching algorithm. An example of this class is a robotic vacuum cleaner that deliberate in an infinite loop, each percept contains a state of a current location [clean] or [dirty] and, accordingly, it decides whether to [suck] or [continue-moving].

A model-based reflex agent needs memory for storing the percept history; it uses the percept history to help to reveal the current unobservable aspects of the environment. An example of this IA class is the self-steering mobile vision, where it's necessary to check the percept history to fully understand how the world is evolving.

A goal-based reflex agent has a goal and has a strategy to reach that goal. All actions are taken to reach this goal. More precisely, from a set of possible actions, it selects the one that improves the progress towards the goal (not necessarily the best one). An example of this IA class is any searching robot that has an initial location and wants to reach a destination.

An utility-based reflex agent is like the goal-based agent but with a measure of "how much happy" an action would make it rather than the goal-based binary feedback ['happy', 'unhappy']. This kind of agents provide the best solution. An example is the route recommendation system which solves the 'best' route to reach a destination.

A learning agent is an agent capable of learning from experience. It has the capability of automatic information acquisition and integration into the system. Any agent designed and expected to be successful in an uncertain environment is considered to be learning agent.


Is the policy gradient expression in Fundamentals of Deep Learning wrong? I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning.
Here is the whole paragraph:
Policy Learning via Policy Gradients

In typical supervised learning, we can use stochastic gradient descent to update our parameters to minimize the loss computed from our network's output and the true label. We are optimizing the expression:
$$
\arg \min _{\theta} \Sigma_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right)
$$
In reinforcement learning, we don't have a true label, only reward signals. However, we can still use SGD to optimize our weights using something called policy gradients. We can use the actions the agent takes, and the returns associated with those actions, to encourage our model weights to take good actions that lead to high reward, and to avoid bad ones that lead to low reward. The expression we optimize for is:
$$
\arg \min _{\theta}-\sum_{i} R_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right)
$$
where $y_{i}$ is the action taken by the agent at time step $t$ and where $R_{i}$ is our discounted future return. A In this way, we scale our loss by the value of our return, so if the model chose an action that led to negative return, this would lead to greater loss. Furthermore, if the model is very confident in that bad decision, it would get penalized even more, since we are taking into account the log probability of the model choosing that action. With our loss function defined, we can apply SGD to minimize our loss and learn a good policy.

The first expression about the loss computed in a network already seems false since the log of a probability is always negative, and taking the $θ$ (weights) for which the expression is minimal doesn't seem right because it would favor very unsure answers.
The same goes with the next expression on policy gradient. A very negative $R_i$ and very unsure $p(y_i)$ would both be big negatives and multiplied together give a big positive value. Since there is a - sign in front of the expression, this would be the best configuration for the argmin. Meaning we are looking for weights in the policy that give highly negative rewards and for highly unsure actions. This just doesn't make sense to me.
Is it just a sign error (or we could just change to argmax)? Or is there more to it?

There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
$$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim\pi, s\sim\mu}\left[ G_t \nabla_\theta \log\pi(A_t|S_t)\right]\;;$$
where $\mu$ is our state distribution induced by the policy $\pi$ and $G_t$ are the discounted returns.
Now, as we want to maximise our objective $J(\theta)$, we want to perform gradient ascent. That is, our parameters should be updated according to
$$\theta_{t+1} = \theta_t + \alpha \nabla_\theta J(\theta)\;.$$
The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of $f$ is just the same as finding the minimum of $-f$. Using this, we can then say our parameters can be updated using gradient descent on the negative of our objective, i.e. we get our parameter update rule to be
$$\theta_{t+1} = \theta_t - \alpha \nabla_\theta (-J(\theta))\;;$$
which is exactly what you have in the book.


Does backpropagation update weights one layer at a time? I am new to Deep Learning.
Suppose that we have a neural network with one input layer, one output layer, and one hidden layer. Let's refer to the weights from input to hidden as $W$ and the weights from hidden to output as $V$. Suppose that we have initialized $W$ and $V$, and ran them through the neural network via the forward algorithm/pass. Suppose that we have updated $V$ via backpropagation.
When estimating the ideal weights for $W$, do we keep the weights $V$ constant when updating $W$ via gradient descent given we already calculated $V$, or do we allow $V$ to update along with $W$?
So, in the code, which I am trying to do from scratch, do we include $V$ in the for loop that will be used for gradient descent to find $W$? In other words, do we simply use the same $V$ for every iteration of gradient descent?

The answer is implied in the term "backpropagation". All gradients are calculated at the same time. That is, the error from your loss function is propagated backwards from the output and through your whole network. This propagation results in an error associated with each weight in the network, which determines how you change each weight in order to minimize your loss function.

  or do we allow $V$ to update along with $W$?

Yes. This saves time, since many of the results of intermediate computations used to update $V$ can be reused in the update of $W$.

See http://neuralnetworksanddeeplearning.com for a detailed description of the backpropagation algorithm.


What to do when PDFs are not Gaussian/Normal in Naive Bayes Classifier While analyzing the data for a given problem set, I came across a few distributions which are not Gaussian in nature. They are not even uniform or Gamma distributions(so that I can write a function, plug the parameters and calculate the "Likelihood probability" and solve it using Bayes classification method). I got a set of a few absurd looking PDFs and I am wondering how should I define them mathematically so that I can plug the parameters and calculate the likelihood probability.

The set of PDFs/Distributions that I got are the following and I am including some solutions that I intend to use. Please comment on their validity:

1)

The distribution looks like:

$ y = ax +b $ from $ 0.8<x<1.5 $

How to programmatically calculate

1. The value of x where the pdf starts
2. The value of x where the pdf ends
3. The value of y where the pdf starts
4. The value of y where the pdf ends

However, I would have liked it better to have a generic distribution for this form of graphs so that I can plug the parameters to calculate the probability.

2)

This PDF looks neither uniform nor Gaussian. What kind of distribution should I consider it roughly?

3)

I can divide this graph into three segments. The first segment is from $2<x<3$ with a steep slope, the second segment is from $3<x<6$ with a moderate sope and the third segment is from $6<x<8$ with a high negative slope.

How to programmatically calculate

 1. the values of x where the graph changes its slope.
 2. the values of y where the graph changes its slope.

4)

This looks like two Gaussian densities with different mean superimposed together. But then the question arises, how do we find these two individual Gaussian densities?

The following code may help:

variable1=nasa1['PerihelionArg'][nasa1.PerihelionArg>190]
variable2=nasa1['PerihelionArg'][nasa1.PerihelionArg<190]

Find mean and variance of variable1 and variable2, find the corresponding PDFs. Define the overall PDF with a suitable range of $x$.

5)

This can be estimated as a Gamma distribution. We can find the mean and variance, calculate $\alpha$ and $\beta$ and finally calculate the PDF.

It would be very helpful if someone could give their insights on the above analysis, its validity, and correctness and their suggestions regarding how problems such as these should be dealt with.

The relationship between the axes of graph (1) and your variables $x$ and $y$ is not clear, so this generalized answer may be helpful or useless.

From graph (1) it appears that the correlation coefficient $\mathcal{C}$ of a quadratic fit of data set $\mathcal{S}$ would be much better.  Consider $y_1$ and $y_2$ approximations of $y$.

$$
\mathcal{C} (y_2, a, b, c, \mathcal{S}) > \mathcal{C} (y_1, a, b, \mathcal{S}) \\
y_2 = ax^2 + bx + c \\
y_1 = ax + b
$$

To achieve a more nearly uniform distribution, perform a least squares fit for $y_2$ against $y$ on $\mathcal{S}$ to obtain $(a, b, c)$.  Then find a mapping function that produces $y'$ and use it where the uniform distribution is desired.  A reasonable approximation is simply this.

$$y' = \frac{y}{y_2(x)}$$


Training a Yes/No NN for Image Classes I am fairly a newbie to Neural Networks.
I wanted to ask if it is possible to train a NN to identify only one type object? For instance, a table from a large set of images, where the NN should be able to identify if new images are tables.
If yes can you please guide me in the direction to get started?

EDIT:
I wanted to ask if it is possible to train a NN to identify only one type object?...
so my understanding as of now is that if i train an NN on one class and one class alone with a fairly large amount of data then i can get it trained in a small amount of time rather than training it on a huge amount of data and consuming a larger amount of data...
Objective is to train a large no of such smaller NNs to create an ensemble of NNs.

Yes, you can train a NN to detect only one type of object like a table. However, you probably will not want to train such a NN from scratch by showing some examples of tables and non-tables. You will need to use transfer learning on a model already trained on several image classes and teach it to also recognize your new class. This transfer learning requires a smaller set of desired images. You may need to give it some negative examples also. You should explore transfer learning with mobilenet, inception, and other pre-trained Tensorflow models if you are willing to use Python and Tensorflow


What functionality, does control look like in autonomous vehicles levels 4 and 5? We are doing a research design project on autonomous vehicles and have some questions on AV Levels 4/5; specifically on the roles, impacts and consequences of AV on society, government, users and other stakeholders.

We're currently stuck on this main question:

Q: What functionally, does control look like in AV levels 4 and 5?

For example, is the whole purpose of a level 4/5 that a user has no input into the control?

Could a driver in AV (level 5) stop in an emergency, or say they want to "take corners harder, speed up, slow down"?

Could I choose to change the equi-distance between my AV and the others around me because I like space?

We're wondering about what functionally, does AV level 4/5 offer a user; and what it looks like?

Context:

Our remit is within the world of design (design thinking), not specifically technology, or expert system functionality. We're looking at the issue from a design perspective; who does it impact, who are the stakeholders, what are the consequences and impacts. What role does a driver have an in level 5? Could an auto-manufacturer want to give drivers control in level 5? How do emergency services act in these situations? What are the touchpoints to society and whom does it impact and what does it say about the design of AV for the future of society.

Automation Levels

Most cars have some Level 1 automation, such as cruise control and various skid/flip probability reduction systems. Most high volume passenger vehicles have higher levels. Some military and private air, land, and sea equipment are already at Level 5.

Level 4 requires that driving be automated during normal driving conditions, with manual override.  However, to my knowledge, no one has published a mathematically terse and comprehensive distinction between normal and abnormal to aid in testing Levels 4 vehicles, so testing to a standard is probably not yet possible.

For legal and political reasons, Level 5 is essentially a statistical criteria. For fully automatic to be viable as a market product for general public use, the safety data for passengers and pedestrians must indicate at least the level of safety for manually driven vehicles. Although this will likely suffice from a law and public relations purpose, it is inadequate as a quality standard for automation engineering and testing.  The ambiguities are numerous.

Statistical criteria required to pass the test (i.e. sample size, duration, randomization, and double or single blindness)
Mathematically terse and comprehensive scenarios for the test
Allowable proportion of level 2, 3, or 4 vehicles in the control group.
Probably others

There will be no vehicle driver in what they are calling Level 5 — only passengers.  The idea is to give no power to the occupants of the AV other than destination or change in destination.

This has been the safety norm in other sub-sectors of transportation for a century.  Passengers cannot talk to the pilot of a jet or the engineer of a train.  In the majority of cases, safety is compromised whenever a person that has not undergone the discipline of intense safety training has control over any aspect of the vehicle's operation.

That is the primary impetus behind AV from the forward thinkers in government and academia.

Specificity and Insight

It is of paramount importance that researchers define system criteria more specifically and scientifically.  Systems architecture, software engineering, safety evaluation, and quality control policies and procedures of the automated system driving the AV requires such.

With a billion plus lives at stake, the design should progress with the diligence and care as if designing a human-occupied drone aircraft or a civilian Mars lander from the ground up, even if your first phase is to only achieve what is being called Level 4 for basic passenger cars.

Target Reliability and Safety

Humans eat food, have emotional conversations, text each other, ingest mind altering substances, and fall asleep while driving.  Judging the safety statistics of an AV by comparing with those of humans driving may sound practical, but it is absurd.  It will become clear just how absurd the popularized Level 5 criteria is as the parameters of design are enumerated.

Design should instead minimize the possibility of any accident ever.  The goal should be zero mistakes, both at point of sale (the dealership) and at later points in the product life-cycle as the AV learns.

Defining a Mistake

A mistake should be defined as follows:

  Any less than optimal state indicated by the correction signal used to direct reinforced learning in any of the system's re-entrant or coincident training mechanisms

The adaptive (i.e. machine learning) portion of the system must permit re-entrant or coincident training (reinforced learning) because there is no possible way to predict the common routes of the buyer at the vehicle's point of sale.

To comprehend the complexity of the problem space for AVs and begin to simplify it, consider the dimension of conditions, controls, and priorities (embodied in feedback signals) related to driving on roads for any vehicles that use roads.

Control Channels

Starters (there are two in the case of most hybrids)
Engine stop
Breaking controls (there are two in the case of regenerative breaking)
Steering shaft or hydraulic control position
Break control position (three or four depending on emergency/parking break design)
Transmission planetary gear clutch state, traditional automatic transmission control state, or clutch and traditional manual transmission control state
External lighting switch positions (Widely variable, but at least six for headlight, high beam, break light, left signal, right signal, and tail light)
The content of any messages broadcasted, multicasted, or sent specifically to any other vehicles with compatible reception (if birds do it, so can electronic systems) either via light, sound, or RF (this will require the development of layers of inter-vehicle communications protocols)
Horn
Probably others

Data Acquisition Channels

Wheel positions (there may be 2 or 4 positions to read with an encoder since one cannot assume perfect alignment)
Break torques (4 of them, which can be read by 16 redundant strain gauges)
Break metal temperatures (4)
Torques and temperatures for any independent emergency breaks
Accelerometers (two devices x three dimensions per device to detect acceleration/deceleration, centripetal force, and, with some math, tire skid velocity for all four tires in two dimensions)
Tachometers (one before the transmission and one on each wheel)
Engine and coolant temperature detectors
Cameras (must be high resolution to recognize animals, humans, shopping carts, curbs, road signs, train signals, speed bumps/humps, and hazards, which can be IR or visible and the more angles covered the better)
Wind turbulence resistant external microphones to detect horns and sirens (at least four to detect likely orientation of audio source)
Suspension strain gauges (to detect vertical road force on each tire)
The content of any incoming messages from compatible systems
Battery voltages and currents (two batteries for regenerative breaking or hybrid startup and motive assist, and possibly several currents)
Fluid levels, pressures, viscosity, and transparency (fuel, oil, steering, transmission, break hydraulic, and possibly others)
Probably others

A system can be operational and possibly reach what most would consider Level 5 with fewer channels of acquisition and control than above, but it would be poor technology planning to start designing systems with unnecessary limitations.  Such limitations will also very likely increase the cost of engineering and the effectiveness of training while saving nothing.

Why Vehicles Have Very Little Instrumentation Today

A human cannot make use of all of all the information above.  Nor could a human control all the channels listed above without a high frequency of mistakes.  A properly designed electromechanical learning system can.

It would be lazy systems architecture not to capitalize on the positive impact the additional instrumentation would have on safety, total cost of ownership, and other quality criteria for the AV buyer who can afford the extra sensors and computing power.  Furthermore, after a few years of manufacturing for a mass market, the cost of the additional components will become small in comparison with the cost of metal and plastic.

Operational Criteria and the Formalization of a Mistake

The problem space contains at least nineteen dependent variables (output channels) and forty-six independent variables (input channels).  Some are binary, some are floating point, some are streamed data, some are streamed audio, and some are streamed video.

Together they form a space in sixty-five dimensions.  That is what must be optimized according to some predetermined and possibly re-programmable formalization of what is optimal.

Let's consider this idea of optimum safety, thrift, and comfort as quality control criteria.  Real time quality control should follow TQM ideals, continuously ensuring quality in multiple dimensions and at multiple points in the total system.

Maximal distance from other vehicles
Maximal distance from stationary objects (bridge abutments, buildings)
Maximal distance from pedestrians on foot, bike, wheelchair, ...
Maximal distance from edges of pavement
Minimal lane switching
Minimal loads and torques on wheels
Minimal fuel consumption
Within operation parameters for tires, breaks, engine RPM, and dozens of other parts and subsystems
Shortest distance for route to destination
Shortest time distribution mean for route to destination
Safest route to destination
Least stops on route to destination
Probably others

Most of this must be balanced, so optimization criteria must be aggregated.  Such aggregation must go beyond the simplicity of a loss function.  Summing squares will not work at all, so give up aggregating in such a simplistic way.  A multivariate extension of the Pythagorean Theorem is find for calculating distance in linear space, but driving cars is very non-linear.  This kind of robotics system will require more thought in the formulation of balances, priorities, and the concept of emergency.

Further expanding on the above definition of a mistake, any real time control that does not optimize the sixty-five-plus dimensional surface is faulty.  Now what is optimal must be defined.  Consider the following quality control criteria, roughly in correct order.

Pedestrian safety
Passenger safety
Mechanical system integrity
Fuel conservation
Vehicle external coating integrity
Mechanical system wear
Passenger comfort
Time conservation in reaching destinations
Probably others

Applying Optimization in This Context

Aggregation of the incoming signals acquired are not only based on multiple criteria but the prioritization is not always constant, implying the need for a vector of correction signals rather than a single floating point number.

A single dimension of signaling to feed the disparity between ideal operation and the current system behavior (called a loss function in gradient descent) will not suffice.  There will, out of necessity, be a need for training and reinforcement with a complexity that involves the idea of preemption.  Evolution has declared preemption the design of choice for nervous systems with brains.

For instance, the pedestrian safety feedback signaling must always preempt the fuel conservation feedback no matter how much fuel would be consumed in staying clear from pedestrians, in planning a route where pedestrian density is lower, steering the vehicle clear of people, choosing speed, and applying breaking.

All biological systems have these preemption mechanisms — even bacteria.  A turtle doesn't balance the transportation aspect with safety when it retracts under its shell.  The behavioral interest in the turtle's destination is shelved (stored and temporarily forgotten) until the preemptive system that detected danger indicates the danger has passed.

Humans Should be Passengers on the Streets of the Future

The reason humans are generally driving in a continuous state of mistaken control is because the priorities that maximize the parameters of transportation for society (above) are inconsistently followed by humans.  Birds fly smarter than humans drive.  The priorities of an emotional being that wants to get somewhere fast and while talking, texting, eating, and possibly getting high will often be mistaken.

Future people may look back at the period between the advent of Model-T market penetration and the complete transition to AV as a period of strange inequality.  Stepping back, the worldwide interest in domestic security, airline safety, train and subway safety, and building code contrasts strongly against the cultural insistence of every household have instant access to drive anywhere, any time, and in any mental condition.


So if I understood correctly:
You have data from 2 sensors in time: Ar flow and BackGas Flow (SCCM, what is that?)
You have that data for multiple products.
1 - Since it is relatively low dimensional, you may try using raw data with K-Means or Self Organizing Maps.
2 - If you searching for anomalies in time, you might try using feature engineering with things like flow change:

i) Take 2 points in time and "derivate" (Point A - Point B)/(Time A - Time B)
ii) Take the result for A and repeat the process as a second derivative, try that for many levels of "derivative"
iii) Machine learning application for sensor failure detection in polymerization process. In: Simpósio Brasileiro de Telecomunicações e Processamento de Sinais.

3 - You might want to check other Time-Series related research, even on regression and classification, since they may give you ideas on relevant features or approaches:

i) An enseble approach to time dependent classification (10.1109/ICMLA.2018.00164)
ii) Multivariate Time Series for Data-driven Endpoint Prediction in the Blast Oxygen Furnace (DOI: 10.1109/ICMLA.2018.00231)

4 - Note that anomalies are outliers, so analysing data distant from the mean of any features you might engineer from the time. An model based on RBF kernels might pick up some information.
5 - Since you don't know the amount of clusters you can try hierarchical clustering.
6 - Don't forget to talk to people in your field about what to expect from this.
If you can elaborate on the type of anomalies you're looking for I may come with better ideas. This is pretty general tips.
I may have misunderstood what you wanted to cluster:
7 - If you want to detect anomalies in:

i) products behavior tips above were mostly for it and you might use all 23 features in short time intervals and cluster like this:

    For each product in dataset {

      samples = {} # empty set

      For each time interval in product {

        features in interval add to samples

      }

      perform clustering method using samples

    }

This algorithm would help cluster periods of time where the product behavied weirdly

ii) sensors behavior then for something like that you might try tip 8

iii) products itself create a feature vector with all avaiable information for each product, then cluster that. Since it is composed of variable time interval and 23 features per product (plus time) you might want to use a bit of dimensionality reduction (PCA should work fine, since it gives you the direction of greater variance and you're look for that). Using a regressor to model products behavior and cluster that could all be usefull (create a regressor in time for each product to model its behavior, then cluster the weights of the regressors as the product representation)

8 - Let's say you want to identify sensors behavior, it might be helpful to model the sensors output as a function f(time, extravariables) and do regression over it (try linear, then try non-linear). Points in time that the regressor had bad performance predicting the answer might indicate that that behavior is anomalous


How to design a classifier while the patterns of positive data are changing rapidly? In some situation, like risk detection and spam detection. The pattern of Good User is stable, while the patterns of Attackers are changing rapidly. How can I make a model for that? Or which classifier/method should I use?

The phenomenon where the prediction targets (in your case, behaviour) change over time is referred to as "concept drift".

If you search for that term, you'll find that there have been many publications attempting to tackle that over multiple decades, way too many papers to all summarize here in a single answer. It's still a difficult problem though, by no means a "solved" problem.

Two different, broad directions for ideas are:

Frequently re-training (offline) static models on the most recent training data
Using online learning approaches that can continuously be updated from a data stream, online as new labelled data becomes available.

This github page contains a large list of papers on credit card fraud detection, where the problem you describe occurs because fraudsters change their behaviour in an attempt to evade detection. Most of those papers discuss variants of the first approach. Basically, many of those papers use an ensemble of multiple Random Forests. Every day, new labelled data becomes available. They often then remove the oldest of multiple Random Forests, and add a new Random Forest trained on the most recent data made available that day.

There are also some variants where they don't always train new models at a fixed schedule (e.g., every day), but try to detect when the statistical properties of the data have changed using statistical tests, and only train new models when it is "necessary" (due to such changes).

For the second idea, you'll often be thinking of approaches that use Stochastic Gradient Descent-like approaches for learning; with a non-decreasing learning rate / step size, such techniques will naturally, slowly "forget" what they have learned from old data, and focus more on the latter data.

If you have some method to obtain accurate labels for certain instances relatively quickly, you could consider an approach like the one proposed in this paper (disclaimer: I'm an author on this paper). For example, in that paper the assumption is that human experts can relatively quickly investigate and obtain accurate labels for a small selection of transactions, and this can be exploited to quickly learn in an online manner.


What, exactly, does the REINFORCE update equation mean? I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
$$
Where  is usually the discounted future reward and  (|) is the probability of taken the action that the agent took at time . (Tell me if something is wrong here)
But I don’t understand how that is passed into a neural network for back propagation.
I have this pseudocode
probs = policy.feedforward(state)
This returns the probabilities if taking each actions, like: [0.6,0.4]
action = choose_action_from(probs) this will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.
Then later when it is time to update, is it:
gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?
If you need more explanation, I have this question on SO.

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.


Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem? I trained a simple CNN on the MNIST database of handwritten digits to 99% accuracy. I'm feeding in a bunch of handwritten digits, and non-digits from a document.

I want the CNN to report errors, so I set a threshold of 90% certainty below which my algorithm assumes that what it's looking at is not a digit.

My problem is that the CNN is 100% certain of many incorrect guesses. In the example below, the CNN reports 100% certainty that it's a 0. How do I make it report failure?

My thoughts on this:
Maybe the CNN is not really 100% certain that this is a zero. Maybe it just thinks that it can't be anything else, and it's being forced to choose (because of normalisation on the output vector). Is there any way I can get insight into what the CNN "thought" before I forced it to choose?

PS: I'm using Keras on Tensorflow with Python.

Edit

Because someone asked. Here is the context of my problem:

This came from me applying a heuristic algorithm for segmentation of sequences of connected digits. In the image above, the left part is actually a 4, and the right is the curve bit of a 2 without the base. The algorithm is supposed to step through segment cuts, and when it finds a confident match, remove that cut and continue moving along the sequence. It works really well for some cases, but of course it's totally reliant on being able to tell if what it's looking at is not a good match for a digit. Here's an example of where it kind of did okay.

My next best option is to do inference on all permutations and maximise combined score. That's more expensive.

The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of the model.

This is generally not possible with simple neural networks as they simply do not have this property, for this you need a Bayesian Neural Network (BNN). This kind of network learns a distribution of weights instead of scalar or point-wise weights, which then allow to encode model uncertainty, as then the distribution of the output is calibrated and has the properties you want.

This problem is also called out of distribution (OOD) detection, and again it can be done with BNNs, but unfortunately training a full BNN is untractable, so we use approximations.

As a reference, one of these approximations is Deep Ensembles, which train several instances of a model in the same dataset and then average the softmax probabilities, and has good out of distribution detection properties. Check the paper here, in particular section 3.5 which shows results for OOD based on entropy of the ensemble probabilities.


What is the Control Problem? The Wikipedia page describes AI control problem in very intricated way.

Therefore I would like to better understand it based on some simple explanation, what's going on.
Basically I don't want any copy & pastes from wiki, because the articles there are written in neutral point of view, in very general way where articles are evolving very slowly, so the definition from there doesn't suit me.

I believe this is what is discussed nowadays by government and it's important aspects of AI technology where it leds to.
I believe this could be a big problem in the near future, so I'm expecting to hear about this from people from much better and more up-to-date point of view.

So what is exactly the AI Control Problem?

The Control Problem is, in short, the idea that AI will eventually be much better decision-makers than humans. If we don't set things up correctly beforehand, we won't get a chance to fix it afterwards, because AI will have effective control.

There are three main areas of discussion with regards to the Control Problem:

Whether or not the problem is urgent. Many AI experts, cognizant of the difficulty of getting simple systems to work effectively today, think that AI able to take control is not urgent, and as detail-minded engineers, they think it will be profoundly difficult to do any useful work today. (Andrew Ng, for example, famously called these sorts of worries like worrying about overpopulation on Mars.) Given radical uncertainty among AI experts as to when this will become an issue, however, this means we can't rule out rapid AI timescales, and should do at least some work in anticipation of those timescales.
Whether or not the problem is hard. Many people give short, simple, and wrong solutions to the control problem. Probably the most famous is the idea that intelligence and morality are inherently interlinked, and thus a more intelligent machine, by definition, will be more moral. The Orthogonality Thesis is the claim of the opposite, that intelligence and morality (or, more specifically, goal alignment) are unrelated things.
What foundations we can lay now. There are a bunch of open problems (see, for example, MIRI's technical agenda) that deal with mathematical logic of the sort that would be useful for ensuring robust value alignment, or on how to effectively do value learning (without giving an incentive to distort values), or on how to build value functions and goals such that they are fixable if they turn out to be mistaken. Those look like problems that we can do useful work on now, even without knowing what the actual structure of a future AI will look like.


Why is no activation function needed for the output layer of a neural network for regression? I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back to a nice number between 0 and 1.
But in this beginner example on the TensorFlow website, the output layer has no activation function at all? Is this allowed? Wouldn't the result be a crazy number that's all over the place? Or maybe TensorFlow has a hidden default activation?
This code is from the example where you predict miles per gallon based on horsepower of a car.
// input layer
model.add(tf.layers.dense({inputShape: [1], units: 1}));

// hidden layer
model.add(tf.layers.dense({units: 50, activation: 'sigmoid'}));

// output layer - no activation needed ???
model.add(tf.layers.dense({units: 1}));

In regression, the goal is to approximate a function $f: \mathcal{I} \rightarrow \mathbb{R}$, so $f(x) \in \mathbb{R}$. In other words, in regression, you want to learn a function whose outputs can be any number, so not necessarily just a number in the range $[0, 1]$.
You use the sigmoid as the activation function of the output layer of a neural network, for example, when you want to interpret it as a probability. This is typically done when you are using the binary cross-entropy loss function, i.e. you are solving a binary classification problem (i.e. the output can either be one of two classes/labels).
By default, tf.keras.layers.Dense does not use any activation function, which means that the output of your neural network is indeed just a linear combination of the inputs from the previous layer. This should be fine for a regression problem.


OpenAI Gym interface when reward calculation is delayed? (continuous control with considerable reaction time) I'm about to create an OpenAI Gym environment for a flight simulator. I'm wondering, how to cope with the fact, that the result and reward for some action needs a considerable time to advance through the system due to the inherent time constants.

In the easy example Gym-environments (e. g. cartpole, or some games) the step can anstantly be executed and the resulting reard can be calculated.

In my continuous control system (aka flight simulator), There is some reaction time needed, until I can calculate the result from my action. E. g. When I pull the stick, it takes some time, until the aircraft lifts it's nose. So there is a considerable delay (maybe in the seconds ballpark) from commanding the action to the environment, and the earliest observation of that result and it's corresponding reward.

How can I cope with that. As far as I understood, the env.step(action) function blocks till it comes back with a new observation and a corresponding reward.

How can I cope with long lasting reward caclulations?
Is it possible to have overlapping actions somehow? E. g. command a new action every 100ms, but get the reward for that action only 1 second later. In this case there would be always 10 rewards pending.

I hope I made my point clear. Don't hesitate to ask for further details in the comments.

Any hints are welcome. Is there anything to read out in the wild dealing with a similar issue?

Cheers,

Felix

It's not on your end, as a creator of flight simulator, to worry about what action should get the credit for the reward that happened some time after the action was taken. You should return the reward when the actual event happens not when the action that caused it happened. It's the job of the reinforcement learning agent to figure that out. For example if you want to give a reward when airplane nose is at 45 degrees from the horizontal axis, you should return the reward when that event actually happens, RL agent should figure out that the crucial action happened some time ago. This may be difficult for the agent but its up to the user to use a proper algorithm and proper exploration strategy to solve the problem.


Why does the training time of SVMs dramatically decrease after applying dimensionality reduction to the features? Training an SVM with an RBF kernel model with c = 5.5 and gamma = 1.06, for a 5-class classification problem on the NSL-KDD train data-set with 122 features using one vs rest strategy takes $2162$ seconds. Also, considering binary classification (c = 10, gamma = 4), it takes $520.56$ seconds.
After dimensionality reduction, from 122 to 30, using a sparse auto-encoder, the training time falls dramatically, from $2162$ to $240$ and $520$ to $170$, while using the same hyperparameters for the RBF-kernel.
What is the reason for that? Is it not true that using kernel neutralized the effect of high dimensions?

SVM complexity is $O(\max(n,d)\min(n,d)^2)$ according to Chapelle, Olivier. "Training a support vector machine in the primal." Neural Computation 19.5 (2007): 1155-1178.
$n$ is the number of instances and $d$ is the number of dimensions. I'm assuming that you have more instances than dimensions giving a complexity of $O(nd^2)$. Hopefully this explains fully why reducing the number of dimensions will reduce the training time.


Mathematical foundations of the ability to learn I am an undergraduate student in applied mathematics with an interest in artificial intelligence. I am currently exploring topics where I could do research. Coming from a mathematical background I am interested in the question: Can we mathematically establish that a certain AI system has the ability to learn a task given some examples of how it should be done?
I would like to know what research has been done on this topic and also what mathematical tools could be helpful in answering such questions.

Computational learning theory (CLT or COLT) is devoted to the mathematical and computational analysis of machine learning algorithms, so it is concerned with the learnability (i.e. generalization, bounds, efficiency, etc.) of certain tasks, given a learner (or a learning algorithm), a hypothesis space, data, etc.
CLT can be divided into (at least) two subfields: statistical learning theory (SLT) and algorithmic learning theory (ALT). The most famous CLT frameworks are PAC learning and the VC theory (which extends PAC learning to infinite-dimensional hypothesis spaces).
The following are some good resources you can use to get started with CLT (but you can find more here).

The book Machine Learning (1997) by Mitchell
The book Foundations of Machine Learning (2012) by Mohri et al.
The book Understanding Machine Learning: From Theory to Algorithms (2014) by Shalev-Shwartz et al.
The paper An overview of statistical learning theory (1999) by Vapnik (who is one of the main contributors to SLT)
The paper Introduction to Statistical Learning Theory (2014) by Bousquet et al.

Here's a related question on this site: What sort of mathematical problems are there in AI that people are working on?.


Which deep reinforcement learning algorithm is appropriate for my problem? My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games (sparse rewards, etc.) or robotics (continuous action spaces, etc.). Since my problem is an optimization issue, I wonder which algorithm is appropriate for my setting:

limited number of discrete actions (like 20)
high-dimensional states (like 250 values)
instant reward after every single action (not only at the end of an episode)
a single action can affect the state quite a lot

There's no "goal" like in a video game, an episode ends after a certain number of actions. I'm not quite sure which algorithm is appropriate for my use case.

Theoretically video games and robotics problems are also about optimization(getting maximum reward). So, just like other reinforcement learning problems, I would expect PPO to be the most efficient in your case too. I don't think a "goal" is necessary for rl, all you need is the rewards.


Which models accept numerical parameters and produce a numerical output? I need a model that will take in a few numerical parameters, and give back a numerical answer (Context: predicting a slope based on environmental factors without having to actually take measurements to find the slope).

However, I am lost, given that I have not found (on the web) any model that is able to solve this task. If someone can suggest a type of model that would be able to solve this type of problem, I would greatly appreciate it. Resources and libraries would also be useful if you care to suggest any. I'm using Python.

You're probably looking for regression, either linear or non-linear, which usually refers to a set of methods that can be used to predict a continuous (or numerical) value (the value of the so-called dependent variable), given one or more possibly numerical values (the values of the independent variables). (The other common task is called classification, where the outcome variable is discrete.)

The dependent and independent variables can also be vectors. In that case, it is called multivariate linear or non-linear regression. If there is more than one independent variable (or, also called, predictor), then it is called multivariable (or multiple) linear or non-linear regression. See, for example, Multivariate or Multivariable Regression? (2013), by Bertha Hidalgo and Melody Goodman.

In linear regression, the dependent variable is assumed to be a linear combination of the independent variables, while, in non-linear regression, it is assumed to be a non-linear combination (which is any combination or relationship that is not linear).

The independent variables (or predictors) do not actually need to be independent of each other and, of course, the dependent variable does not need to be independent of the predictors (otherwise you would not be able to predict the value of the dependent variable, given the predictors), so the expression predictor is possibly more appropriate than independent variable. See, for example, In Regression Analysis, why do we call independent variables "independent"?, by Frank Harrell. However, the dependent variable is assumed to be dependent on the independent variables, so you could think of the independent variables as the variables that are not the dependent variables (if this helps you to memorize the concept).

There are more synonyms for the dependent and independent variables, depending on the context or area. For example, the predictors (or independent variables) can also be called covariates, regressors or explanatory variables. The dependent variable can also be called regressand, outcome variable, or explainable variable.

There are other types of regression models. In particular, there is the generalized regression model (GLM), which is (as the name suggests) a generalization of several regression models. The famous logistic regression (where the outcome is actually binomial or binary, so discrete) is an application of the generalized regression model, where the link function (which is a concept that is used in the context of GLMs) is the logit function. See, for example, Why is logistic regression a linear model?. Note that linear regression is also an application of a GLM. See also this article 1.1. Generalized Linear Models, which gives you an overview of different GLMs.

Have also a look at this article Choosing the Correct Type of Regression Analysis, which provides a few guidelines to help you choose the most appropriate regression model for your task.

In Python, you could use, for example, sklearn.linear_model.LinearRegression. The Python library sklearn provides several other regressors. For example, sklearn.ensemble.AdaBoostRegressor or sklearn.ensemble.RandomForestRegressor. (Of course, you will need a training dataset to train these regressors, when calling their method fit).


Are most things generally discovered because they work empirically and later justified mathematically, or vice-versa? In the original GloVe paper, the authors discuss group theory when coming up with equation (4). Is it possible that the authors came up with this model, figured out it was good, and then later found out various group theory justifications that justified it? Or was it discovered sequentially as it is described in the paper?

More specifically:

  Question. In AI research, are most things generally discovered because they work empirically and later justified mathematically, or is it the other way around?

There are definitely approaches that are theory driven (like SVMs), and others where the theory comes after the practice (like a lot of deep neural networks). I think it would be difficult to argue strongly that either direction is more common "in general" within AI, or indeed, within any other branch of science.

The approach that is currently in favor will tend to change over time as well. If we develop a new method that we don't understand, it will tend to attract interest from theorists. If they are able to understand it well, that understanding will likely lead to natural next steps that improve the model. As with other areas of science, the empirical cycle is at work.


How do you pass the image from one convolutional layer to another in a CNN? I am currently trying to write a CNN from scratch, but I don't understand how to feed the information from a max-pooling layer to the next convolutional layer. Specifically, I don't know what to do with the 6 filtered and pooled images from the first convolutional and max-pooling layers. How do I feed those images into the next convolutional layer?

The application of 1 kernel (aka filter) to an input (with a 2d convolution) is a matrix (a 2d array), which is often known as a feature map (aka activation map). The application of $k$ kernels to the same input is a 3d array (sometimes called tensor, though this may not be exactly correct, or 3d volume) with depth $k$, i.e. you have $k$ concatenated feature maps. This 3d array is the input to the next convolutional layer, which needs to have 3d kernels of depth $k$: this is the main requirement in the case of 2d convolutions (in the case of 3d convolutions, the kernels can have a different depth than the input 3d volume, but you can ignore this for now!).
Another thing that you need to take care of is the padding that you can add around this 3d volume. This, along with the stride (i.e. the step with which you move the kernel), determines the height and width of the output of the convolutional layer.
So, to recapitulate, what you have to do is set the depth of the kernels in the convolutional layer $l+1$ to be equal to the number of kernels that you use in the convolutional layer $l$. So, let's say that the number of kernels is a hyper-parameter (note that this is usually the case!), and that the user sets the number of kernels $k=8$ in the convolutional layer $l$, and, for simplicity, assume that all kernels are $3 \times 3$, then, in layer $l$, you will perform $8$ convolutions, one for each kernel. These $8$ convolutions produce a 3d array of dimensions $w \times h \times 8$ (where $w$ and $h$ depend on the dimensions of the input to layer $l$, the stride, and padding). Consequently, in the convolutional layer $l+1$, the kernels need to be $3 \times 3 \times 8$.
If it's the pooling layer that produces the 3d array to be passed to the next convolutional layer, you should still set the depth of the kernels in the convolutional layer $l+1$ to the number of kernels in the convolutional layer $l$ (provided that, in the pooling layer, you only redimension the width and height of the 3d array, which is usually the case).


What kind of word embedding is used in the original transformer? I am currently trying to understand transformers.
To start, I read Attention Is All You Need and also this tutorial.
What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?
In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms.  Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.
The embeddings are trained from scratch.


What is the difference between a non-stationary policy and a state that stores time? This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary policy and a state that includes time.

My understanding is that, in general, a non-stationary policy is a policy that doesn't change. My first (probably incorrect) interpretation of that was that it meant that the state shouldn't contain time. For example, in the case of game, we could encode time as the current turn, which increases every time the agent takes an action. However, I think even if we include the turn in the state, the policy is still non-stationary so long as sending the same state (including turn) to the policy produces the same action (in case of a deterministic policy) or the same probability distribution (stochastic policy).

I believe the notion of stationarity assumes an additional implicit background state that counts the number of times we have evaluated the policy, so a more precise way to think about a policy (I'll use a deterministic policy for simplicity) would be:

$$ \pi : \mathbb{N} \times S \rightarrow \mathbb{N} \times A $$
$$ \pi : (i, s_t) \rightarrow (i + 1, s_{t+1}) $$

instead of $\pi : S \rightarrow A$.

So, here is the question: Is it true that a stationary policy must satisfy this condition?

$$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$

In other words, the policy must output the same result no matter when we evaluate it (either the ith or jth time). Even if the state $S$ contains a counter of the turn, the policy would still be non-stationary because for the same state (including turn), no matter how many times you evaluate it, it will return the same thing. Correct?

As a final note, I want to contrast the difference between a state that includes time, with the background state I called $i$ in my definition of $\pi$. For example, when we run an episode of 3 steps, the state $S$ will contain 0, 1, 2, and the background counter of number of the policy $i$ will also be set to 2. Once we reset the environment to evaluate the policy again, the turn, which we store in the state, will go back to 0, but the background number of evaluations won't reset and it will be 3. My understanding is that in this reset is when we could see the non-stationarity of the policy in action. If we get a different result here it's a non-stationary policy, and if we get the same result it's a stationary policy, and such property is independent of whether or not we include the turn in the state. Correct?

So, here's is the question: Is it true that a non-stationary policy must satisfy this condition?

  $$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$

With your custom notation (which certainly isn't common, but seems reasonable)... I assume you meant to say that a stationary policy must satisfy that condition, rather than that a non-stationary policy must satisfy that condition. In that case, yeah, that seems correct to me. A stationary policy would satisfy that condition, and a non-stationary one wouldn't.

Wrapping back to the more usual notation, where $\pi(S, A)$ denotes a probability of selecting $A$ in $S$ (which also still covers the case of a deterministic policy, which would simply assign a probability of $1$ to a single action, and $0$ to all others)... I think it's still interesting to consider the case where we decide to "bake" a time-step counter into the state representation.

With this notation, for two different time steps $t$ and $t'$, such that $t \neq t'$, I'd say that a policy $\pi$ is stationary if and only if:

$$\pi(S_t, A_t) = \pi(S_{t'}, A_{t'}) \text{ if } S_t = S_{t'} \wedge A_t = A_{t'}.$$

Note that if we decide to include $t$ in the state-representation, the case that $S_t = S_{t'}$ with $t \neq t'$ will actually never hold within the same episode, states at different time steps will always automatically be different from each other if the time step is one of the "features" encoded in the state. So, within a single episode a policy with a "time-aware" state representation will always automatically be stationary because there cannot be any repeating states. Of course, if you start looking across multiple different episodes, this can change; this is what you're doing when you write:

  Once we reset the environment to evaluate the policy again, the turn, which we store in the state, will go back to 0, but the background number of evaluations won't reset and it will be 3.

If you chose to also embed that "episode counter" into the state representation, you would also no longer have any state repetitions at all anymore across different episodes (I don't think doing this would ever be a good idea though).


Is my understanding of back-propogation correct? I am trying to learn backpropagation and this is what I know so far.
To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss function using the chain rule. List all of these partial derivatives in a column vector and you have your gradient vector of your current parameter's on the loss function. Then by taking the negative of the gradient vector to descend the loss function and multiplying it by the learning rate (step size) and adding it to your original gradient vector, you have your new weights.
Is my understanding correct? Also, how can this be done in iterations over training examples?

Your understanding seems to be correct (although your explanation isn't completely precise), apart from "adding it to your original gradient vector". You add the gradient vector to the parameters/weights vector.
(Note that back-propagation is just the algorithm that computes the gradient vector. The update of the parameters with the gradient of the loss function is the gradient descent/ascent step, even though it's true that some people, at least in the context of deep learning, refer to the combination of gradient descent and the computation of the gradients as the back-propagation algorithm, but this is just a terminology issue, which you should not dwell too much on.)

Also, how can this be done in iterations over training examples?

If I understand this question correctly, you want to know how we would update the parameters when there's more than one training example. In that case, you compute a gradient vector for each training example. Then the average of the gradient vectors (which is also a vector) is the actual gradient vector that you use to update the parameters. See this answer for more info.


How does PCA work when we reduce the original space to 2 or higher-dimensional space? How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case.
$$\begin{array}{ll} \text{maximize} & \mathrm{Tr}\left( \mathbf{w}^T\mathbf{X}\mathbf{X}^T\mathbf{w} \right)\\ \text{subject to} & \mathbf{w}^T\mathbf{w} = 1\end{array}$$

You might want to have a look at the wikipedia article of PCA, where it says:
"The $k$th component can be found by subtracting the first $k − 1$ principal components from $\mathbf{X}$:"
$$\hat{\mathbf{X}}_k = \mathbf{X} - \sum_{s=1}^{k-1}\mathbf{X}\mathbf{w}_s\mathbf{w}_s^T$$
Then you repeat the process to find the next component:
$$\mathbf{w}_k = \arg\max \mathbf{w}^T\mathbf{\hat{X}}^T_k\mathbf{\hat{X}}_k\mathbf{w}$$
$$\text{s.t. } \mathbf{w}_k^T\mathbf{w}_k = 1$$


Is there a programming training dataset for robot program itself Where can I find training datasets like the ones provided for linguistic training, but to train a program to program itself.

I want to input this training dataset to a programming script and it should use it to program itself.

What do I need to consider in this kind of artificial intelligence?

On self improving AI

The way it's been done in the past is one AI turns a picture into text an the other turns text into a picture.

The AI generated material is added to known good material. The AI's job then is to guess what is real an what is fake (generated by AI).

At the point when the AI cannot tell in a false positive way both AI are said to be at their limit. If the limit is good enough you're done otherwise go back and code (more nodes or better function or more real data)

The thing to remember is you still need real data. The act of putting the AI together is good only to create more data then you could ever provide.

If that data is bad then you are going to rely on the cost function more then the data. But this allows you to label data real and AI so if the function rewards identifying the AI the AI's find each other's flaws and the better at avoiding flaws the better the AI get. (You still need real data! just less)

For example if two people cannot play ping pong if they play each other they learn slowly and only the cost function enforces their learning.
But have them play a pro (lots of good data) and they get better much faster.


How to measure sample efficiency of a reinforcement learning algorithm? I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample efficient but how does one reach this conclusion when comparing reinforcement learning algorithms?

First of all, let's recall some definitions.
A sample in the context of (deep) RL is a tuple $(s_t, a_t, r_t, s_{t+1})$ representing the information associated with a single interaction with the environment.
As for sample efficiency, it is defined as follows [1]:

Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance.

So, how you measure it is closely related to the way it is defined.
For example, one way to do it would be as shown in the figure below:

On the y-axis you have the performance of your RL algorithm (e.g. in terms of average return over episodes as done in [2], or mean total episode reward across different environment runs as done in [3])

On the x-axis you have the number of samples that you took.

The dashed line corresponds to your performance baseline (e.g. the performance at which a certain game or any other RL environment is considered solved).

So, you can measure sample efficiency at the intersection, where you get the number of samples needed to reach the performance baseline. So, an algorithm that requires less samples would be more sample-efficient.

Another way to do it would be the other way around, i.e. the RL agent is provided with a limited budget for the number of samples it could take. As a result, you can measure sample efficiency my measuring the area under the curve, as illustrated below. So, that would be how much performance you got just by using those samples in the budget. An algorithm that achieves a higher performance than another with the same amount of samples would then be more sample efficient.

I am not aware if there exist RL libraries that would provide you with this measure out-of-the-box. However, if you're using Python for example, I believe that using libraries like scipy or scikit-learn along with matplotlib could do the job.
NB: Image credits go to the following presentation: DLRLSS 2019 - Sample Efficient RL - Harm Van Seijen


How do I restrict the neural network structure to be acyclic in NEAT? I want my neural network structure to not have a circular/looping structure something similar like a directed acyclic graph (DAG). How do I do that?

The naive way is to generate connections randomly as you would for a cyclic graph, but then perform a test to reject any connections that form a cycle. This is the current approach in SharpNEAT and there has been some effort directed at improving the performance of the cycle test in the work-in-progress refactor branch.

One alternative would be to track the depth of all nodes, store a list of node IDs sorted by depth, and sample the connection endpoint nodes in such a way that the target node depth is always higher than the source node. Now I think about it that's probably the better method.


What is convergence analysis, and why is it needed in reinforcement learning? While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis needed in reinforcement learning?

Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves that theoretically the algorithm achieves the desired function. Without convergence, we have no guarantees that the value function will be accurate or the policy will be any good, so in other words the proposed RL algorithm can completely fail at serving its purpose even in simple cases.


Why do neural networks based AI players still require extensive search techniques? Today we have neural network based AI players that are comparable or better than humans in games that require extensive pattern matching and "intuition". AlphaGo is a prime example.

But these AI players usually have both neural networks and search algorithms in place. Humans, on the other hand, rely just on the pattern matching and "intuition" (even the best chess players can see just a handful of moves ahead).

So, why do AI players still require extensive search while humans don't? How would AIs like AlphaGo perform if we take the search part out?

It is not very accurate to say that AI players requires extensive search while humans don't. Rather, it is a question of degree.

AIs do a lot more calculating, because that's what computers are good at.

Human intuition is much more powerful than a neural network can currently hope to match, because it is much more integrated into a world of knowledge about the game, it uses orders of magnitudes more neurons and it is not a static thing that just provides a move.

But if a human player stops calculating ahead his playing strength will drop very significantly. This can be seen by assessing the performance in games with short time controls: The less time you have the less calculation is happening, your intuition however is fast.

If you take out the search component of AlphaGo it would still play quite strongly, probably at a low dan level. Of course that is also far below its strength.

So, search is always an important component of playing strength, just more so for machines.


Wind speed forecasting using ARIMA model in Python3 Recently, I started working on time-series models and would mention that I am very new to python and ML as a whole.
I tried to implement a time-series model on wind speed data. Being a newbie, I followed the steps given in this article: https://kanoki.org/2020/04/30/time-series-analysis-and-forecasting-with-arima-python/ and it's a great one to start with but somehow I am unable to forecast my data (or I would say the forecasted data is constant around the 5.88 value!)

# FORECAST
n=39
forecast,err,ci = results.forecast(steps=n, alpha= 0.05)
windspeed_forecast = pd.DataFrame({'forecast':forecast},index=pd.date_range(start='22/8/2020 01:00:00', periods=n, freq='MS'))
windspeed_forecast

 #plot for forecast
ax = windspeed[19:].SPEED.plot(label='observed', figsize=(20, 15))
windspeed_forecast.plot(ax=ax,label='Forecast',color='r')
ax.fill_between(windspeed_forecast.index, ci[:,0], ci[:,1], color='b', alpha=.005)
ax.set_xlabel('DATE')
ax.set_ylabel('SPEED')

plt.legend()
plt.show()

-What I think is the high AIC value of 700 might be the problem!

-Also, I am unable to figure out how can I create the column of Date-time for the forecasted values same as that of the original data(i.e. hourly based data of a specific date) [As shown in the ss number 1 and as shown in ss below - I need a column starting from 22/8/2020 with hourly gaps and so on].

Also, PFA the ss of my data in jupyter notebook (out of 191 total data, 152 used as train data and rest as test data)

Any suggestion/help regarding the same will be appreciated :)

First things first, time series is probably the most difficult of all things to predict. If it wasn't I'd be sitting on nice warm beach or on yacht cruising the Mediterranean.
Besides that here's a few hints. First, always visualise your data first. Tables of numbers are fine for computers by the MK1 eyeball is still your best weapon for time series. Is there any obvious trends or seasonal behaviour? If not then ARIMA is unlikely to be of much utility. Secondly, visualise your data after forecasting. Don't just rely on error measures like AIC, use your eyes. Most time series forecasts are useful only in the short term. So you may see the earlier part of your forecast make sense while the latter doesn't.
Finally ARIMA is paramterised by by p, d, and q. The value of d is the minimum number of differencing needed to make the series stationary. If the time series is already stationary, then d = 0. ‘p’ is the order of the ‘Auto Regressive’ (AR) term. It refers to the number of lags of Y to be used as predictors. And ‘q’ is the order of the ‘Moving Average’ (MA) term. It refers to the number of lagged forecast errors that should go into the ARIMA Model.
You have only shown results for p,d,q = 1,0,1. Much of ML is done by trial and error, trying different parameters. So I'd suggest you try varying your values of p, d and q. AIC can then be used to compare models, with smaller values indicating better models.


Could we update the policy network with previous trajectories using supervised learning? I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories.

Doesn't the following intuition from the OpenAI Vanilla Policy Gradient description indeed propose that learning from prior experience should still be possible?

  The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.

The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one.

I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we update the policy network with previous trajectories using supervised learning? The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.

Why is that not used/possible? What am I missing?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning). The only way to possibly know that is to make labels based on the return that you got from that action but the return is based on an old trajectory with the old policy. The return for that specific action depends on actions that happened after that action in the trajectory and return for those actions change with time.

To make things clearer, consider a simple case. Let's say you take action $a_1$ and you end up in state $s_1$ with reward $0$. Then you have two possibilities, you take action $a_2$ and end up in terminal state $s_2$ with reward $-10$ or you take action $a_2'$ and end up in terminal state $s_2'$ with reward $10$. Let's say you use trajectory $a_1 \rightarrow s_1 \rightarrow a_2 \rightarrow s_2$ with return $-10$ to learn about action $a_1$. Then your label for that action would probably be that that action is bad, but it actually isn't, if you took action $a_2'$ after $a_1$ your return for action $a_1$ would be $10$. So you learned that your action is bad even though it might not be. Now, if later you learn that taking action $a_2'$ is good to take after $a_1$ then you would also learn that $a_1$ might be good but if you keep using that old data with return $-10$ you will keep learning that $a_1$ is bad.

You can only use data gathered from the current policy to learn about it because older data might be outdated.


Any papers regarding different/inconsistent action space in Reinforcement Learning? This question is regarding Reinforcement Learning and different/inconsistent action space for every/some states.
What do I mean by different/inconsistent action space?
Let say you have an MDP where the number of actions varies between states (for example like in Figure 1 or Figure 2). We can express "inconsistent action space" formally as $$\forall s \in S: \exists s' \in S: A(s) \neq A(s') \wedge s \neq s'$$
That is, for every state, there exists some other state which do not have the same action set. In the figures (1, 2) there's a relatively small amount of actions per state. Instead imagine states $s \in S$ with $m_s$ number of actions, where $1 \leq m_s \leq n$ and $n$ is a really large integer.

Environment
To get a better grasp of the question, here's an environment example. Take Figure 1 and let it explode into a really large directed acyclic graph with a source node, huge action space and a target node. The goal is to traverse a path, starting at any start node, such that we'll maximize the reward which we'll only receive at the target node. At every state, we can call a function $M : s \rightarrow A'$ that takes a state as input and returns a valid number of actions.
Approches

A naive approach to this problem (discussed here and here) is to define the action set equally for every state, return a negative reward whenever the performed action $a \notin A(s)$ and move the agent into the same state, thus letting the agent "learn" what actions are valid in each state. This approach has two obvious drawbacks:

Learning $A$ takes time, especially when the Q-values are not updated until either termination or some statement is fulfilled (like in experience replay)

We know $A$, why learn it?

Another approach (first answer here, also very much alike proposals from papers such as Deep Reinforcement Learning in Large Discrete Action Spaces and Discrete Sequential Prediction of continuous action for Deep RL) is to instead predict some scalar in continuous space and, by some method, map it into a valid action. The papers are discussing how to deal with large discrete action spaces and the proposed models seem to be a somewhat solution for this problem as well.

Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action regarding that perticular state with $n$ valid actions. In other words, the performed action of a state $s$ with 3 number of actions will be predicted by $\underset{a}{\text{argmax}} \ f_{\theta_3}(s, a)$.

None of the approaches (1, 2 or 3) are found in papers, just pure speculations. I've searched a lot, but I cannot find papers directly regarding this matter.
Does anyone know any papers, or having other approaches, regarding different/inconsistent action space?

Does anyone know any paper regarding this subject?

I'm not familiar with any off the top of my head... I do know that the vast majority of Reinforcement Learning literature focuses on settings with a fixed action space (like robotics where your actions determine how you attempt to move / rotate a particular part of the robot, or simple games where you always have the same set of actions to move and maybe ''shoot'' or ''use'' etc.). Another common class of settings is where the action space can easily be treated as if it always were the same (by enumerating all actions that every could be legal in some state), and filtering out illegal actions in some sort of post-processing steps (e.g. RL work in board games).

So... there might be something out there, but it's certainly not common. Most RL people like to involve as little domain knowledge as possible, and I suppose that a function that generates a legal set of actions given a particular state can very much be considered to be domain knowledge.

  Is the terminology wrong? "Inconsistant", "Irregular", "Different"... ?

I wouldn't use inconsistent, because that word can be interpreted as implying that something would be "wrong" or "ill-defined". I would say that you have a variable action set (the action set varies per state). When I search for that, there aren't a lot of results either though... but I think that term would be more promising.

  Anyone having another approach worth digging into?

The problem you describe is mostly a problem in Reinforcement Learning with function approximation, in particular function approximation using Neural Networks. If you can get away with using tabular RL algorithms, the problem instantly disappears. For example, a table of $Q(s, a)$ values as commonly used in the tabular, value-based algorithms does not need to contain entries for all possible $(s, a)$ pairs; it's fine if it only contains entries for $(s, a)$ pairs such that $a$ is legal in $s$.

Variable action spaces primarily turn into a problem in Deep RL approaches, because we normally work with a fixed neural network architecture. A DQN-style algorithm involves neural networks that take feature vectors describing states $s$ as inputs, and provide $Q(s, a)$ estimates as outputs. This immediately implies that we need one output node for every action, which means you have to enumerate all the actions... which is where your problem comes in. Similarly, policy gradient methods traditionally also require one output node per action, which again means you have to be able to enumerate all the actions in advance (when determining the network architecture).

If you still want to use Neural Networks (or other kinds of function approximators with similar kinds of inputs and outputs), the key to addressing your problem (if none of the suggestions you've already listed in the question are to your liking) is to realize that you'll have to find a different way to formulate your inputs and outputs, such that you are no longer required to enumerate all actions in advance.

The only way I can think of doing that really is if you are able to compute meaningful input features for complete state-action pairs $(s, a)$. If you can do that, then you could, for example, build neural networks which:

Take a feature vector $x(s, a)$ as input, which describes (hopefully in some meaningful way) the full pair of the state $s$ and the action $a$
Produce a single $\hat{Q}(s, a)$ estimate as output, for the specific pair of state + action given as input, rather than producing multiple outputs.

If you can do that, then in any given state $s$ you can simply loop through all the legal actions $A(s)$, compute $\hat{Q}(s, a)$ estimates for them all (note: we now require $\lvert A(s) \rvert$ passes through the network rather than just a single pass as would normally be required in DQN-style algorithms), and otherwise proceed similarly to standard DQN-style algorithms.

Obviously the requirement of having good input features for actions is not always going to be satisfied... but I doubt there are many good ways to get around that. It's very similar to the situation with states really. In tabular RL, we enumerate all states (and all actions). With function approximation, we usually still enumerate all actions, but avoid the enumeration of all states by replacing them with meaningful feature vectors (which enables generalization across states). If you want to avoid enumerating actions, you'll in a very similar way have to have some way of generalizing across actions, which again means you need features to describe actions.


Why can't we train neural networks in a peer-to-peer manner? I have recently been exposed to the concept of decentralized applications,
I know that neural networks require a lot of parallel computing infra for training.
What are the technical difficulties one may face for training neural networks in a p2p manner?

Data management and bandwidth are key issues for interconnecting multiple GPUs.  These are such big issues that it is hard to think about other challenges like neural network architecture, metrics, etc.  The key to success for interconnecting multiple GPUs on a single computer is NVIDIA's NVLink:
NVLink is a wire-based communications protocol for near-range semiconductor communications developed by Nvidia that can be used for data and control code transfers in processor systems between CPUs and GPUs and solely between GPUs. NVLink specifies a point-to-point connection with data rates of 20 and 25 Gbit/s (v1.0/v2.0) per differential pair.
Compare 25 Gbit/s to a typical peer to peer connection over the web of 100Mbps.  NVLINK provides a 250x advantage assuming everything else is equal which it is not.  This means that, considering bandwidth only, a neural network which takes one day to train on a computer with two GPUs connected with NVLINK could take 250 days over the internet using two computers with the same GPU!


Bellman Expectation Equation leading to results where value iteration would not converge to the optimal policy When applying the bellman expectation equation:
$$v(s)=\mathbb{E}\left[R_{t+1}+\gamma v\left(S_{t+1}\right) \mid S_{t}=s\right]$$
to the MRP below, states further away from the terminal state will have the same value ${v(s)}$ as states closer to the terminal states. Even though it is clear that the expected total reward from states further away is lower. If the discount factor $\gamma$ would be even lower states further away would get a higher value. If we now make this an MDP where the agent can decide to go either direction from all states (with the first state having an action leading to itself), the agent would then choose to go further away from the terminal. Getting less reward over the whole episode. So, this seems to be an example where policy/value iteration would not converge to an optimal policy. I know there is something wrong with the reasoning here. I just cannot seem to figure out what.
What am I missing here?

EDIT: So, the problem actually was that I didn't take into account that the terminal state has to get a value of 0. If you put it at 0 at all times this will converge as expected because all the other states will get lower and lower values while, assuming a greedy policy, the one-to-last state will retain a value of -1. After a bit over 10 iterations (if gamma is close to 1) it will converge because the states further away will get a value less than -1.

What am I missing here?

You are not missing anything mathematically.
Potentially what you are missing is that the discount factor $\gamma$, is part of the problem definition. In reinforcement learning (RL), you do not always solve problems to obtain the highest total sum of rewards. Instead you solve problems to obtain the highest expected return on any action. If you use a discount factor, then it affects the return calculation, which in turn may affect which behaviours are optimal.
If you choose $gamma$ such that an agent would move away from a terminal state to receive the highest expected return, then you have defined the problem such that this behaviour is optimal.
This can be an issue when solving RL problems and you have taken a free choice for reward functions and definition of return (this is quite common, often it is the researcher's problem to set these values up to define the agent's goals). Often you can use a discount factor to make a problem more numerically stable, because it will keep sums of rewards over many time steps within bounds. Also, it can be used as a way to search for faster solutions - i.e. in less time steps to reach episode end - because the agent will prioritise reaching higher values sooner if it can. Your example MDP deliberately puts those two effects in opposition to each other.
Assuming you want the agent to find the terminal state by getting over the larger negative reward just before it: Probably you should not use a discount factor for your MDP. Alternatively, if the reward signal is more of a free choice for that problem, you could offset it (by $+0.1$) and then most values of discount factor will still resolve to reaching the terminal state as optimal behaviour.
There are a few common ways to mitigate or avoid the issue of changing problem definition by selecting $\gamma$:

Use a value of $\gamma$ derived from the problem. In some cases, the value of $\gamma$ is a realistic physical part of a problem. For instance in financial problems, cash now is often preferable to cash later.

Set $\gamma$ to relavtively high value, such as $0.99$ or $0.999$, depending on episode length and relative sizes of rewards. This can work well enough for a wide range of problems. It is a common solution when using DQN. Effectively this approximates the next approach (average rewards) for a low cost.

Use average reward settings. There are separate Bellman equations, TD updates etc based on maximising expected average reward per time step as opposed to maximising expected discounted return.