What is the fundamental difference between an ML model and a function? A model can be roughly defined as any design that is able to solve an ML task. Examples of models are the neural network, decision tree, Markov network, etc.
A function can be defined as a set of ordered pairs with one-to-many property from domain to co-domain/range.
What is the fundamental difference between them in formal terms?

A model as a set of functions
In some cases in machine learning, a model can be thought of as a set of functions, so here's the first difference.
For example, a neural network with an arbitrary vector of parameters $\theta \in \mathbb{R}^m$ is often denoted as a model, then a specific combination of these parameters represents a specific function. More specifically, suppose that we have a neural network with 2 inputs, 1 hidden neuron (with a ReLU activation function, denoted as $\phi$, that follows a linear combination of the inputs), and 1 output neuron (with a sigmoid activation function, $\sigma$). The inputs are connected to the only hidden unit and these connections have a real-valued weight. If we ignore biases, then there are 3 parameters, which can be grouped in the parameter vector $\theta = [\theta_1, \theta_2, \theta_3] \in \mathbb{R}^3 $. The set of functions that this neural network represents is defined as follows
$$
f(x_1, x_2) = \sigma (\theta_3 \phi(x_1 \theta_1 + x_2 \theta_2)) \tag{1}\label{1},
$$
In this case, the equation \ref{1} represents the model, given the parameter space $\Theta = \mathbb{R}^3$. For any specific values that $\theta_1, \theta_2,$ and $\theta_3$ can take, we have a specific (deterministic) function $f: \mathbb{R} \rightarrow [0, 1]$.
For instance, $\theta = [0.2, 10, 0.4]$ represents some specific function, namely
$$
f(x_1, x_2) = \sigma (0.4 \phi(x_1 0.2 + x_2 10.0)) \tag{2}\label{2}
$$
You can plot this function (with Matplotlib) for some values of the inputs to see how it looks. Note that $x_1$ and $x_2$ can be arbitrary (because those are just the inputs, which I assumed to be real numbers).
This interpretation of a model is roughly equivalent to the definition of a hypothesis class (or space) in computational learning theory, which is essentially a set of functions. So, this definition of a model is useful to understand the universal approximation theorems for neural networks, which state that you can find a specific set of parameters such that you can approximately compute some given function arbitrarily well, given that some conditions are met.
This interpretation can also be applied to decision trees, HMM, RNNs, and all these ML models.
A model in reinforcement learning
The term model is also sometimes used to refer to a probability distribution, for example, in the context of reinforcement learning, where $p(s', r \mid s, a)$ is a probability distribution over the next state $s'$ and reward $r$ given the current state $s$ and action $a$ taken in that state $s$. Check this question for more details. A probability distribution could also be thought of as a (possibly infinitely large) set of functions, but it is not just a set of functions, because you can also sample from a probability distribution (i.e. there's some stochasticity associated with a probability distribution). So, a probability distribution can be considered a statistical model or can be used to represent it. Check this answer.
A function as a model
A specific function (e.g. the function in \ref{2}) can also be a model, in the sense that it models (or approximates) another function. In other words, a person may use the term model to refer to a function that attempts to approximate another function that you want to model/approximate/compute.


Analogy between reinforcement learning and web servers I've recently come across the client-server model. From my understanding, the client requests the server, to which the server responds with a response. In this case, both the request and responses are vectors.

In reinforcement learning, the agent communicates with the environment via an "action", to which the environment sends a scalar reward signal. The "goal" is to maximize this scalar reward signal in long run.

Is there a more formal way to frame this problem such the agent searches the internet for obtaining more information about certain topic?

To paraphrase: "Is there an analogy between client/server in web development and agent/environment in reinforcement learning?"

The answer is "not really". There is no useful analogy here that allows any insight into RL from web server knowledge or vice versa.

However, you could set up an agent where the goal was to collect information, and the available actions were to make web requests. Clearly in order to to this, you would need to make use of the client/server model for web servers, with the agent having control over the client web requests, and the environment being the network and servers of the world wide web.

There are some very hard challenges to construct an open-ended "web assistant" agent. Here are a couple that I can think of:

How to describe actions? Starting with raw web requests composed as strings would likely be very frustrating. Probably you would simplify and have a first action be a call to a search engine with some variation of the topic description, and then decisions about which links to follow, or perhaps whether to refine the search to better fetch sites related to the topic as it is being built.
How to create a model of reward for collecting information? The first major stumbling block would be to measure the amount of useful information that the agent had found on any request.

I think with current levels of Natural Language Processing, setting an agent free to discover information according to some goal from a text topic description is a very hard task, beyond cutting edge research. It would definitely be unreasonable to expect any such agent to end up with any resemblance to "understanding" subject matter from text. The agent would have very little ability to tell the difference between accurate facts or lies, or just grammatically correct gibberish.

One interesting idea for agents trying to learn unsupervised from exploring an environment is creating a reward signal from data compression. An agent will have learned something useful about its environment, if on processing new information, it is able to compress its existing world model more efficiently. This is basic concept behind ideas of general learning agents from Jürgen Schmidhuber, Marcus Hutter and others. Research into this idea could be a driver towards creating more generic AI learning systems - however, it is one idea amongst many in AI, and so far is research-only, it has not yet led to anything as practical as an AI web-searching assistant.


How do you build a language model to predict the contextual similarity between two documents? How do you build a language model to predict the contextual similarity between two documents?

You can use an autoencoder for text. For example, you can refer to this example here: https://machinelearningmastery.com/lstm-autoencoders/

For comparing the contextual similarity, you can compare the encoded vectors for the distance maybe through a formula like mean squared error.

This works as the autoencoder compresses the input data into a vector of numbers, forcing the encoder and the decoder to learn specific features about text. The features cannot be understood by humans but have meanings.

Another approach will be using some word or document embedding like word2vec or GloVe. IT may also work well depending on your dataset size. You need to experiment through different methods to find out which is the best.

A supervised method can also be used if you have labels. For example, you can use a LSTM and train it like an Siamese network for LSTM using triplet loss. Here is an example: https://medium.com/@gautam.karmakar/manhattan-lstm-model-for-text-similarity-2351f80d72f1 Implementation: https://github.com/GKarmakar/deep-siamese-text-similarity

Hope this can help you and have a nice day


What are some concrete steps to deal with the vanishing gradient problem? I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer.

The problem I am facing is that the output being produced by the 3 output nodes are so close to 1 (for the first few iterations at least, and so I am assuming the problem propagates to future outputs as well) the weights are not being updated (or hardly updated) due to overflow (about $10^{-11}$). I can fix the overflow problem (but I don't think it is the culprit). I think such low values of error is the main culprit, and I cannot figure what is causing such low values of error.

What will cause the network to behave more responsively, that is, how will I be actually able to grasp the weight updates and not something in the order of $10^{-11}$?

The data set contain values in the order of $10$'s, and the weights randomly initialized are in the order of $0 < w < 1$. I have tried feature normalization but it is not that effective.

There is not single answer to the vanishing gradient problem. However, there a few things that can help.

As mentioned in the comments, use of Rectified Linear Units (ReLU) as your activation function can help, since the it does not get saturated for large neuron inputs.

Next, careful choice of weight initialization can help avoid saturation, as well. See Andre Ng's Coursera video for details.

Finally, if you are concerned that the scale of your training input is causing issues with training, you can normalize the training examples by subtracting the mean and dividing by the standard deviation. There is a normality assumption here, but this can often help avoid problems where one feature is far out of scale with another. However, this usually causes issues with optimization bouncing between the walls of long, thin trough, which complicates convergence, but does not cause vanishing gradient.


Are there any known models/techniques to determine whether a person in a store is a customer or a store representative? Are there any known models/techniques to determine whether a person in a store is a customer or a store representative?
For example, customer representatives can wear uniforms and then one possible way to identify customer representatives is by the color of their uniform, texture, etc. On the other hand, a customer can also wear the same color clothes as that of a customer representative. Likewise, a customer representative could be wearing "normal clothes." So the main problems that may occur could be:

A customer becomes misclassified as a customer representative.
A customer representative representative becomes misclassified as a customer

So using clothing as the only proxy to classify people as customers or customer representatives seems to be flaky. Any other known ideas?

Access to the back of the store, basically if you can track them entering and leaving an employees only area.
But this requires a really good tracking algorithm and if you lose track you're going to have to fall back on some other properties.


How to create an AI to solve a word search? This at first sounds ridiculous. Of course there is an easy way to write a program to solve a wordsearch.

But what I would like to do is write a program that solves a word-search like a human.

That is, use or invent different strategies. e.g. search randomly for the starting letter; go line-by-line;

Probably the AI will eventually find out that going line by line looking for a given starting letter of a word is a good strategy.

Any idea how you would write such a strategy-finding AI?

I think the main "moves" would be things like "move right one letter in the grid", "store this word in memory", "compare this letter with first letter in memory" and a few more.

This sounds like a problem that might be solvable with a LSTM-DQN approach, as described in Language Understanding for Text-based Games using Deep
Reinforcement Learning by Narasimhan et al., 2015, and then extended to a domain very similar to your problem in Deep Reinforcement Learning for Syntactic Error Repair in Student Programs by Gupta et al., 2019.

The basic idea is to treat the array of letters each of which can be 'circled' or not, plus the position of a cursor as a state. An action is to move the cursor or toggle the state of a letter. You then model the problem as a reinforcement learning problem, with rewards given every time an entire new word becomes 'circled', probably with a caveat about not circling any invalid letters (otherwise, it'll just learn to circle everything).

This is just a guess, but it seems like it's very closely related to your problem, so it's likely to be effective.


Preparing data set for the YOLO algorithm Hi I am working on a project which requires the You Only Look Once algorithm in order to classify and localise objects within images. I have to prepare my dataset (which has 2 classes, and predicts 6 objects per grid cell, and the 448 * 448 image is split into a 7*7 grid). What would be a viable approach to do that? I found  this code, found in this  article. However I do not understand why he has done what he has done, e.g why is he specifically checking the 24th element of the “box”, and so what element of the box would I have to check? Is there any tutorial running through that? Would it be possible for someone to explain or even adapt his approach to fit my dataset?
FYI: I am coding the YOLO algorithm from scratch

Ok, let go step by step.
What you are working on is YOLOv1, in this version of the YOLO algorithm, the maximum bounding boxes that the model can return is 7x7 = 49 boxes as 49 cells since the output shape is 7x7x30.
For each box, the depth of output is 30 because the number of labels of PASCAL VOCS 2012 is 20 (the author of YOLOv1 trained on this dataset) so from index 0 to index 19 will represent the label of that bounding box. From 20 to 23 are the position and size of that box.
The 24th represents 2 things, first is the confidence of that box, since this is ground truth so the confidence should be 1. Second, you know that YOLOv1 can only return 49 boxes at maximum (actually, you can edit the number of boxes by yourself) so the ground truth should only handle one per cell, hence the 24th is the binary value to make sure there are no duplicates bounding boxes in one cell. (25 to 29 is because the author predict 2 bounding boxes per cell)
In your case, the output should be 7x7x(2 + 5 x 2) = 7x7x12 with 2 classes and 2 boxes per cell.


Which features of a data set can be used for market campaigning using propensity scores? A dataset contains so many fields in which there is both relevant and irrelevant field. If we want to do a market campaigning using propensity scoring, which fields of the data set are relevant?
How can we find which data field should be selected and can drive to the desired propensity score?

The problem you are examining is called feature selection. There are many different techniques, but they fall broadly into three categories:

Filter approaches determine which features have high information content. A common approach is to score them based on their information gain.
Wrapper approaches score subsets of the features, using similar measurements. This is slower (since there are many subsets), but may yield better performance.
Embedded approaches use machine learning algorithms that can dynamically select features. For example, building a C4.5 decision tree learner on a dataset, and then selecting just the features that actually appear in the tree, yields a set of features similar to those found by information gain.


What career paths should be avoided with the growth in AI? Consider high school graduates entering higher education or the workforce, each making a decision about where to commit their efforts.  History tells us an important story about choosing in a changing economy.

Entering farming, not as a mechanized farm owner but as a family farmer, resulted in a challenging life for high school graduates in 1990.  John Steinbeck's Grapes of Wrath, a Pulitzer Prize winning novel provides a dismal view of what happened as a result of not predicting what is now termed technological unemployment.

Graduates in 1975 that entered manufacturing felt a similar reduction in options and wages in the last quarter of the 20th century as jobs went to emerging countries that would work for a tenth the wages.

Graphic artists were perhaps the first white collar workers to experience a reduction in demand because of the usability Adobe products.

What is the list of careers to avoid today, and on what evidence are those educational and career paths likely to narrow in the next twenty years?

Please don't use media hype as your evidence.  Please provide a rational argument why the actual trend of products and services points toward the elimination of some jobs because of technological shifts.  We should not base information passed to high school graduates on the sensational (and sometimes wild) guesses of media figures and CEOs who may (or may not) bias their predictions in such a way as to boost the value of their company's stock offering.

This has been studied pretty well academically. The most common study instrument is a survey of the opinions of AI researchers. The other mechanism is subjective assessment by business analysts about which sectors are most likely to be automated, using two criteria: ease of automation, and potential profit from automation (based on the size of the workforce). I have a fairly detailed discussion of some sources below.

The upshot is: any job that requires little training is probably a bad idea. Additionally, any job that requires directly caring for or educating another person, or involves the production of a fine-arts product is probably a good idea. In between, jobs like sales or administrative assistants are likely to become more "winner takes all": low-level sales jobs are likely to disappear, but high level ones that involve a lot of human contact, strong social skills, negotiation, and persuasion are likely to stay. In general, jobs that are plentiful, routine, and high salary are likely to be fully or partially automated first.

Sources:

This was discussed informally at AAAI a few years ago (2015?). One researcher's daughter was considering different career paths. The conclusion that was reached (half-jokingly) was that AI-researcher was the best bet. Everything else we could name, someone was working on automating.

More concretely, there are a number of recent papers and reports. A good starting place is the 2017 PWC report. Section 4.2 covers the different sectors at risk, and 4.3 provides a rationale based on what employees typically do in each sector. The report suggests that men without a high school diploma will be most readily affected, with about 3-4 times the rate of automation as compared to college graduates. The specific sectors at greatest risk are wholesale & retail trade (essentially: sales people), manufacturing, administrative support, transportation & storage, public administration and defense, financial & insurance, water, sewage & waste management, and electricity and gas supply.  All these sectors were estimated at being 30% or more automatable. The safest areas were education, and domestic assistants (e.g. live-in caretakers).

The PWC report is also mostly in line with the 2013 paper by Frey & Osborne. This paper proposed a 9 factor model (summarized in Table III), and then evaluated hundreds of occupations by their probability of automation. Jobs requiring a fine-arts component were very unlikely to be automated. Additionally, jobs that did not take place within cramped spaces were unlikely to be automated. Having a large component of caring for others, persuading others, negotiating with others, or social perception, reduced the risk of automation. Originality requirements also quickly reduced risk. Requirements for manual dexterity did not substantially reduce the risk, and actually increased it on average. I'm pretty sure there's a more recent followup paper, but I can't find it just now.

The 2016 Obama Administration also released a report summarizing the scope of the threat of automation. The main conclusions were that those with limited education, and jobs requiring a low amount of skill, were at greatest risk. They also pointed out that around 6% of US jobs are lost each quarter already though (and simply replaced by new jobs), so the proposed rate of job loss (up to ~50% over ~15 years) represents about a 15% increase in the rate of churn, which seems adaptable.


Reinforcement Learning in Commercial Strategy Games I'm a professional game developer investigating the potential for using reinforcement learning to build strategy game AI opponents that have more creative behavior compared to traditional techniques like behavior trees. I have a few questions I've bolded below, any thoughts would be helpful, and could save me from pursuing dead ends.

I created a very boring and tiny game as a test case. Two players each control a fleet of ships, each ship has health and can fire on one other ship each turn dealing some damage. The player and his opponent assigns orders to their respective ships, telling them which target to attack, and then the turn is resolved. Ships with 0 health are removed from the board. The player that loses all their ships first loses the game.

Assuming I was using TensorFlow, at a very high level I need to:

A) Create a training program that outputs a trained graph to a file. The training program will need to map gamestates into tensors, feed the tensors through the graph to produce actions, execute actions on the gamestate to generate a new state, and evaluate the reward function for the new state. Repeat a bunch.

B) Take the graph created in #1, load it at the game runtime, and use the graph to generate intelligent actions from real gamestates during the Player vs AI match.

As soon as I started digging into TensorFlow, questions immediately came up, and now I'm not quite sure if there is a more appropriate library to do this.

I) TensorFlow has a High Level python API, and a Low Level C++ API. Most games are built in C++, and thus using a C++ or C API is preferable, it makes integration with the game much simpler. In principle we could use pybind or some other scheme for sending state from C++ to Python and back again, but that's not ideal.
Question 1: How much do I lose by using the low level API specifically for reinforcement learning, compared to the high level API?

II) Platforms. 99.9% of the time, PC/Console games are developed in Windows environments, and so having Tensorflow work in Windows is critical. From my googling, Tensorflow just barely supports building in Windows using CMake, though it requires some finagling. More worryingly are other platforms:
Question 2: What hope is there of running the TensorFlow library on consoles like XboxOne, Playstation 4, or the Switch? I imagine this would require manually porting the entire source :(

III) TensorFlow is big, and it seems you need to basically link all of it to ship with the game.
Question 3: Is there any way to get a slimmed down "Runtime TensorFlow" library that is only capable of loading a graph and transforming states into actions?
It seems like if the answer was yes, it might also be easier to port this smaller runtime version to more platforms.

Question 4: Should I even be using TensorFlow for this? Is there perhaps something more suitable?

Thanks again if you read all that, I'm eager to start tinkering, but would like to set off in the right direction.

Investigating reinforcement as a way of producing more interesting behavior than behavior trees for AI based commercial strategy game development is a good idea. The simple test game given can be described briefly.

Human pitted against AI
Each player gets a fleet of ships
Each ship begins with a positive health level
One ship can be ordered to fire one shot at one ship per turn
Each hit produces a decrease in ship health
Ships with non-positive health are removed
Player with no remaining ships are removed
The remaining player wins

The question is correct in that game states must be represented as tensors. Ideally, they should be terse representations, with minimal redundancy. The actions must be represented as tensors too. The goal to train a network to play a game is generally feasible, but the simple game may not provide sufficient complexity to lead to learning. It is probably too simple. What is there to learn? In terms of the expectation regarding the result of the game, what advantage is there to firing on one ship over another?

It may be useful to understand that although machine learning can involve networks that can be represented as directed graphs, it is not the graphs of the network cells that are saved or loaded to persist learned behavior. The networks are parameterized, and those parameters are the representation of what is learned. Where this is most important is that, provided the form of the representation is standardized, the learning hardware and software environment need not match that of the environment that later leverages what is learned. The comment, "Build a custom runtime that can run the net [using the previously learned parameters]," is the correct method if this approach is taken.

In reinforcement learning, the representation is less like a graph and more like a structure involving a flexible behavioral policy that is affected by a changing distribution of probabilities. The expectations of future game states resulting from actions are part of the evaluation that must occur before each move. One theoretical approach to automated game play, for which a few varieties exist, is Q-learning. There are others using Markov trees and recurrent networks.

If the approach is to learn something in the lab that can be used in an xBox or Playstation, then supervised learning is indicated, and a high level language may assist in rapid prototyping. The result of training would be saved in language independent form, therefore readable by the portable c/c++. If the approach is to continuously change, then the reinforcement technology must be running on those platforms and most therefore be contained in statically or dynamically linked portable c/c++.

Both of these development approaches can be used in conjunction. For this basic investigative work and probably the first year or two of R&D, the use of Python for rapid prototyping would not require the use of Pybind. For the most common operations, the interoperability between Python and the mature C code ported from FORTRAN for tensor processing years ago is already present. The Python should never run on the game platform. It is not compiled and runs orders of magnitude less efficiently than C/C++ or Java. Determining whether the C++ API of tensorflow ports well to the desired target game environments is part of the architecture process. The Tensorflow underlying C++ API that Python ML newbies aren't generally concerned with is worth review by those who develop commercial games.

If 99.9% of the time, games are still developed on a Microsoft OS, game developers may be excessively patient. Fedora LINUX is much more productive for C, C++, Java, and Python development for both low level and high level programming tasks.

None of this has addressed the mention of a more creative opponent in the question.

The classic training goal in AI game play is to produce winning behavior, but games don't sell when the human always loses. Setting reward and loss functions to optimize on the basis of victory will not create sustainable sales of commercial game products. What a game developer might more wisely chose as a basis for optimization is engagement. Sacrifice victory in favor of sensation. You want gamers to blog to each other and talk in school and work about some crazy and unexpected thing the game did. Consequently, the goals of the learning must be adjusted in terms of training to maximize game value based on these product market realities.


Is pooling a kind of dropout? If I got well the idea of dropout, it allows improving the sparsity of the information that comes from one layer to another by setting some weights to zero.
On the other hand, pooling, let's say max-pooling, takes the maximum value in a neighborhood, reducing as well to zero, the influence of values apart from this maximum.
Without considering the shape transformation due to the pooling layer, can we say that pooling is a kind of a dropout step?
Would the addition of a dropout (or DropConnect) layer, after a pooling layer, make sense in a CNN? And does it help the training process and generalization property?

Dropout and Max-pooling are performed for different reasons.

Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.

Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).

As for your last question, yes. dropout can be used after a pooling layer.


What is the difference between Kaldi and DeepSpeech speech recognition systems in their approach? I would like to know how do Kaldi and DeepSpeech speech recognition systems differ algorithmically? Which one would be more accurate for continuous speech in time?

Both of them using the end-to-end approach for speech recognition. However, because of the code complexity in DeepSpeech, you can't tune the model for your work. Kaldi could be configured in a different manner and you have access to the details of the models and indeed it is a modular tool. I think Kaldi could be a better tool academically and also commercially. But, Deepspeech is a BlackBox and could be a proper tool if your work is near to the work of DeepSpeech.

Moreover, if you are working in a language that there is not enough data for learning, you have a dozen of tools such as grapheme-to-phoneme to establish your dataset to start the learning process. Also, you can start with other models of Kaldi to work on which do not depend on the big data to be learned. However, DeepSpeech requires many hours of samples that is not a common asset for everyone in every language.

Also, you can know more about the traditional technique of speech processing vs the contemporary technique (deep learning) in this medium post.


Feeding YOLOv4 image data into LSTM layer? How would one extract the feature vector from a given input image using YOLOv4 and pass that data into an LSTM to generate captions for the image?
I am trying to make an image captioning software in PyTorch using YOLO as the base object classifier and an LSTM as the caption generator.
Can anyone help me figure out what part of the code I would need to call and how I would achieve this?
Any help is much appreciated.

You can save the extracted features coming out of YOLOv4 and save it to pickle file and use them for later.
You can also find related information in this project made by Jason Brownlee
https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/


Why don't we use trigonometric functions for the output neurons? Why don't we use a trigonometric function, such as $\tan(x)$, where $x$ is an element of the interval $[0,pi/2)$, instead of the sigmoid function for the output neurons (in the case of classification)?

The main reason why the sigmoid function is used is because it 'does not blow up' since it is between 0 and 1 always. As for the relu it is used because it is computationally cheap and even resolves the problem of vanishing gradient(and hence used more often than sigmoid).
So a function like tan(x) will blow up for certain values of x. This can cause a problem of exploding gradients. So, I believe for this reason tan(x) cannot be a good non-linearity to be used.
As for any other function, it is more because of the results that we have gotten over the years and sigmoid and relu have been promising.


What is the actual learning algorithm: back-propagation or gradient descent? What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?

I am reading through chapter 8 of Parallel Distributed Processing hand book and the title of the chapter is "Learning internal representation by error propagation" by PDP research Group.
https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf

If there are no hidden units, no learning happens.
If there are hidden units, they learn internal representation by propagating error back.
Does this mean back propagation[delta rule] is the learning rule and gradient descent is an optimization algorithm used to optimize cost function?

You can run gradient descent without back propagation, in some cases:

Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost function value.
In "black box" gradient-based learning algorithms where you don't know how (or don't want to) calculate gradient analytically, so you choose to measure it. This is a slow process, similar to gradient checking of neural networks, i.e. adjust each parameter in turn by a small $+\epsilon$, measure your loss function, and divide difference by $\epsilon$ to get gradient estimate for that parameter.

In those cases, you can calculate the gradient, and then use the optimisation function of your choice - e.g. SGD, Adam, Nesterov Momentum - to make changes to the learned parameters. The model will be changed and should learn its task based on the cost function.

You can also run back propagation without gradient descent. In that case, the model will not change and nothing is learned.

That makes the different flavours of gradient descent technically the "core" learning algorithm in my opinion, and back propagation the preferred method for calculating gradients in neural networks. These gradients are most often estimates of an unknown "true gradient" due to training in batches of small samples of training data.

In practice, the two terms back propagation and gradient descent are rarely separated when discussing neural network training. So a lot of people will say that they train a network "using back propagation" or that they are using a gradient descent method without mentioning how they are calculating the gradients using back propagation (because that's pretty much a given). Or they might just say they are "using the Adam optimiser" which in the library they are using automatically runs an NN forward on a mini-batch of data, performs back propagation*, and then changes weight parameters based on the gradients, but heavily adjusted by previous gradient measurements.

* With auto-differentiating frameworks such as TensorFlow, PyTorch etc, it is not really clear that the algorithms are performing "classic back propagation" - although they work using the same chain rule, they may well not perform calculations in the exact same groups that you learned when you studied neural networks and had to write the code for calculating gradients by hand. Instead of back propagating gradients layer-by-layer, they are propagated through an abstract computation graph that includes the learnable parameters.


Are there Python packages for random search hyper-parameter optimisation? Which Python packages do you recommend for random search hyperparameter optimization to use? Is there any recent and good one (better than the one in scikit-learn)?

Welcome to AI.SE Enes.

I think by random search, you are referring to so-called "black-box optimization". Random search is sometimes used as a name for this, but BBO is a more common name, and might be easier to search for.

There are many BBO techniques. 'random search' is usually used to refer to a hill-climbing algorithm where you start at a random location, sample from 'adjacent' points in the input space, and then pick the best one to move to.

Scikit-optimize is fairly easy to use package, and implements some of the most reasonable algorithms (gaussian process is particularly strong on most problems). The package's website includes many good examples showing how to use the methods the package provides.

If you're not happy with those methods, then you are likely entering 'unsupported land'. There are lots of more complex, esoteric, methods, but I don't believe they are well consolidated into a single package (or even a single language). There's a nice collection on the Sahindis lab's website, but those are in a mixture of C and Fortran. There are also a huge number of algorithms submitted to recurring competitions, like this one at GECCO each year, but the implementations of these algorithms may not be available at all, or might be available only through the websites of their respective authors.


How does Keras BatchNormalization work? I have read some articles and watched some videos by Andrew Ng stating that it makes more sense to use batch normalization before applying the activation function.
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(BatchNormalization())
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(BatchNormalization())
model.add(Dense(no_classes, activation='softmax'))

We add layers to the sequential model, apply the activation function to the layer itself, and then add a batch normalization layer. So, is it similar to applying the activation function before normalizing the values? If so, is there a way to apply batch normalization before applying the activation function?

Check out this answer, and other answers too. They explained the what the ideal order is and why so. Also, you can just add layer of activation function.
from tensorflow.keras import layers
from tensorflow.keras import activations

model.add(layers.Activation(activations.relu))

They are actually first things when you search on google, so I kindly recommend you to search before asking


Are there other mathematical frameworks of artificial general intelligence apart from AIXI? AIXI is a mathematical framework for artificial general intelligence developed by Marcus Hutter since the year 2000. It's based on many concepts, such as reinforcement learning, Bayesian statistics, Occam's razor, or Solomonoff induction. The blog post What is AIXI? — An Introduction to General Reinforcement Learning provides an accessible overview of the topic for those of you not familiar with it.

Are there any other mathematical frameworks of artificial general intelligence apart from AIXI?

I am aware of projects such as OpenCog, but that's not really a mathematical framework, but more a cognitive science framework.

Gödel machine
There is another mathematical framework for AGI: the Gödel machine, which was proposed by Jürgen Schmidhuber (who also worked with Marcus Hutter). In the paper Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements (2003), Schmidhuber describes the Gödel machine as follows

They are universal problem solving systems that interact with some (partially observable) environment and can in principle modify themselves without essential limits apart from the limits of computability. Their initial algorithm is not hardwired; it can completely rewrite itself, but only if a proof searcher embedded within the initial algorithm can first prove that the rewrite is useful, given a formalized utility function reflecting computation time and expected future success (e.g., rewards). We will see that self-rewrites due to this approach are actually globally optimal (Theorem 4.1, Section 4), relative to Gödel's well-known fundamental restrictions of provability. These restrictions should not worry us; if there is no proof of some self-rewrite's utility, then humans cannot do much either.
The initial proof searcher is $O$()-optimal (has an optimal order of complexity) in the sense of Theorem 5.1, Section 5. Unlike hardwired systems such as Hutter's and Levin's (Section 6.4), however, a Gödel machine can in principle speed up any part of its initial software, including its proof searcher, to meet arbitrary formalizable notions of optimality beyond those expressible in the $O$()-notation. Our approach yields the first theoretically sound, fully self-referential, optimal, general problem solvers.

Space-time embedded agents
This work combines Russell's bounded rationality with Legg's and Hutter's definition of universal intelligence to provide a framework for AGI that takes into account resource (space and time) constraints and other issues (such as that the environment can modify or even destroy the agent).


How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control? Here's the approximated value using weighted importance sampling

$$
V_{n} \doteq \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}, \quad n \geq 2
$$

Here's the incremental update rule for the approximated value

$$V_{n+1} \doteq V_{n}+\frac{W_{n}}{C_{n}}\left[G_{n}-V_{n}\right], \quad n \geq 1$$

How is the second equation derived from the first?

These are used for the weighted importance sampling method of off-policy Monte Carlo control.

By definition of $V_{n+1}$, we have:

$V_{n+1} = \frac{\sum_{k=1}^{n} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{1}$

Then, taking the $n^{th}$ term out of the sum in the numerator, we have:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; \sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{2}$

Then, from the definition of $V_n$, $V_{n} = \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}$, we have:

$\sum_{k=1}^{n-1} W_{k} G_{k} = V_{n}*\sum_{k=1}^{n-1} W_{k} \; \tag{3}$

Then, substituting $(3)$ in the numerator of $(2)$, we get:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{4}$

Then, adding and subtracting $V_{n}W_{n}$ in the numerator of $(4)$, we obtain:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k} \; + \; V_n W_n \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{5}$

We factor $V_n$ in the numerator of $(5)$:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}(W_n \; + \; \sum_{k=1}^{n-1} W_{k}) \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{6}$

We simplify, taking into account that the denominator $\sum_{k=1}^{n} W_{k} = W_n + \sum_{k=1}^{n-1} W_{k}$, we get:

$V_{n+1} = V_n + \frac{W_n G_n - W_n V_n}{\sum_{k=1}^{n} W_{k}} \; \tag{7} $

Further rearrangements of the terms give us:
$V_{n+1} = V_n + \frac{W_n}{\sum_{k=1}^{n} W_{k}}[G_n - V_n] \; \tag{8}$

Finally, by definition of $C_n$ as the cumulative sum of the weights up to time $n$, we get the desired incremental update equation:
$V_{n+1} = V_n + \frac{W_n}{C_n}[G_n - V_n] \; \tag{9}$


Validation Loss Fluctuates then Decrease alongside Validation Accuracy Increases I was working on CNN. I modified the training procedure on runtime.

As we can see from the validation loss and validation accuracy, the yellow curve does not fluctuate much. The green curve and red curve fluctuate suddenly to higher validation loss and lower validation accuracy, then goes to the lower validation loss and the higher validation accuracy, especially for the green curve.
Is it happening because of overfitting or something else?
I am asking it because, after fluctuation, the loss decreases to the lowest point, and also the accuracy increases to the highest point.
Can anyone tell me why is it happening?

This question has been asked a year ago when I faced this problem I searched and it hasn't any answers on it, I tried different ways and finally, data augmentation helped me.
I used data augmentation and a very small learning rate. If the fluctuations are big, the batch size should be increased, and the learning rate should be decreased. After all, using more epochs helps you to have an almost smooth plot.


Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation? GLIE+MC control Algorithm:

My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks:

If we have multiple terminal states then we will only reach one (per Policy Iteration step PE+PI).
It is highly unlikely that we will visit all the states (during training), and a popular scheduling algorithm for exploration constant $\epsilon = 1/k$ where $k$ is apparently the episode number, ensures that exploration decays very very rapidly. This ensures that we may never visit a state during our entire training.

So why this algorithm uses single MC episode and why not multiple episodes in a single Policy Iteration step so that the agent gets a better feel of the environment?

I feel the general answer is that we want to be as efficient as possible in learning from experience.

Policy improvement here always produces an equivalent or better policy, so delaying the improvement step to gather more episodes will only slow down learning.

I would note too that often a different kind of Monte Carlo learning is used. Instead the speed of the update is typically controlled with a new hyper parameter $\alpha$, instead of keeping track of the visit counts. The Q estimate is then updated something like:

$$
Q \leftarrow Q + \alpha \left (G - Q \right)
$$

The value of $\alpha$ then lets you tune how much evaluation vs improvement happens. This is called constant alpha Monte Carlo. Often this is used as a stepping stone to introduce TD methods, e.g., in 6.1 of the Sutton and Barto book.


Can a neural network with linear activation functions produce a connection of linear functions? From a lecture in machine learning, I know that a linear activation function can only produce a linear function, but I don't know if it can produce a connected linear function, like the one in the image below. This function consists of multiple concatenated lines.

A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.
If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input, where the next layer's neurons would also calculate a weighted sum on that input, and, in turn, it fires based on another linear activation function.
No matter how many layers and neurons there are, if all are linear in nature, the final activation function of the last layer is also a linear function of the input of the first layer.
That means that any or all of these layers can be replaced by one layer. This completely loses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation, given that a combination of linear functions in a linear manner is still another linear function.
A good way to visualize this messing around with Tensorflow playground, which has a spiral data set similar to your data.
In contrast to the failure of linear activation functions for this data in the previous link, check out this much smaller network using a Tanh activation function which can separate the data within 100 or so iterations.
For further reference on activation functions, checkout this blog post.


Intuitively, why can the training of a neural network be formulated as a probability estimation problem? Neural network training problems are oftentimes formulated as probability estimation problems (such as autoregressive models).
How does one intuitively understand this idea?

Consider the case of binary classification, i.e. you want to classify each input $x$ into one of two classes: $y_1$ or $y_2$. For example, in the context of object classification, $y_1$ could be "cat" and $y_2$ could be "dog", and $x$ is an image that contains one main object.
In certain cases, $x$ cannot be easily classified. For example, in object classification, if $x$ is a blurred image where there's some uncertainty about the object in the image, what should the output of the neural network be? Should it be $y_1$, $y_2$, or maybe it should be an uncertainty value (i.e. a probability) that lies between $y_1$ and $y_2$? The last option is probably the most reasonable, but also the most general one (in the sense that it can also be used in the case there's little or no uncertainty about what the object is).
That's the reason why we can model or formulate this (or other) supervised learning problem(s) as the estimation of a probability value (or probability distribution).
To be more concrete, you can formulate this binary classification problem as the estimation of the following probability
\begin{align}
P(y_1 \mid x, \theta_i) \in [0, 1] \label{1}\tag{1}
\end{align}
where $y_1$ is the first class (or label), $(x, y) \in \mathcal{D}$ is a labeled training example, where $y$ is the ground-truth label for the input $x$, $\theta_i$ are the parameters of the neural network at iteration $i$, so, intuitively, $P(y_1 \mid x, \theta_i) $ is a probability that represents how likely the neural network thinks that $x$ belongs to the class $y_1$ given the current estimate of the parameters. The probability that $x$ belongs to the other class is just $1 - P(y_1 \mid x, \theta_i) = P(y_2 \mid x, \theta_i)$.  In this specific case, I have added a subscript to $\theta$ to indicate that this probability depends on the $i$th estimate of the parameters of the neural network.
Once you have $P(y_1 \mid x, \theta_i)$, if you want to perform classification, you will actually need to choose a threshold value $t$, such that, if $P(y_1 \mid x, \theta_i) > t$, then $x$ is classified as $y_1$, else it is classified as $y_2$. This threshold value $t$ can be $0.5$, but it can also not be.
Note that, in the case above, $P(y_1 \mid x, \theta_i)$ is a number and not a probability distribution. However, in certain cases, you can also formulate your supervised learning problem so that the output is a probability distribution (rather just a probability). There are also other problems where you don't estimate a conditional probability but maybe a joint probability, but the case above is probably the simplest one that should give you the intuition behind the idea of formulating machine learning problems as the estimation of probabilities or probability distributions.


What is the need for so many filters in a CNN? Consider the following coding line related to CNNS
Conv2D(64, (3,3), strides=(2, 2), padding='same')

It is a convolution layer with filter size $3 \times 3$ and step size of $2\times 2$.
I am confused about the need for $64$ filters.
Are they doing the same task? Obviously, it is no. (one is enough in this case)
Then how do each filter differ by? Is it in hovering over the input matrix? Or is it in the values contained by filter itself? Or differs in both hovering and content?
I am finding difficulty in visualizing it.

Then how do each filter differ by? Is it in hovering over the input matrix? Or is it in the values contained by filter itself? Or differs in both hovering and content?

The filters (aka kernels) are the learnable parameters of the CNN, in the same way that the weights of the connections between the neurons (or nodes) are the learnable parameters of a multi-layer perceptron (or feed-forward neural network).
So, the value of these filters is not fixed or pre-determined, but will depend on how you train the CNN, i.e. the learning algorithm, the objective function and the data. If you use gradient descent as the learning algorithm, you will be minimizing a loss (aka cost or error) function (e.g. the cross-entropy, in the case of classification problems). To do that, you need to find the gradient of the loss function with respect to the filters. You then apply a step of gradient descent (i.e. you add a scaled version of the gradient of the loss function with respect to the parameters to the parameters), so that this loss decreases.
To answer your question more directly, the only thing that usually changes is just the value of the filters. The convolution (or cross-correlation) operation is the same for all filters.
Why do you use more than one filter? The usual explanation is that each filter, when convolved with the input, will extract different features from it, and the specific features that they will extract will depend on the specific values of the filters, which, in turn, depend on the data, so we can say that CNNs are data-driven feature extractors. If you are familiar with image processing techniques, then you know that different filters, when convolved with the same image, can have different effects (e.g. blurring or de-noising).


If certain moves are compulsory, will there still be a need for a quiescence search? Certain games, like checkers, have compulsory moves. In checkers, for instance, if there's a jump available a player must take it over any non-jumping move.

If jumps are compulsory, will there still be a need for a quiescence search?

My thinking is that I can develop an implementation of a quiescence search that first checks whether jumps are available. If there are then it can skip all non-jumping moves. If there's only one jumping move available, then I won't need to run a search at all.

Therefore, I will only use a quiescence search if I initially don't have to make a jump on my first move. I will only active quiescence search in my alpha-beta pruning becomes active. (The alpha-beta will only be active if my first algorithm which first checks if there are jumps available returns a 0, which means there  are no jumps available.)

Is my thinking of implementing a quiescence search correct?

My options are slim when it comes to optimizations due to serious memory constraints, hence I won't be using PVS or other algorithm like that as they require additional memory.

I understand your question to be:

  If some moves are compulsory, and my agent has no choice about which move to make next, do I need to perform a search, or can I just return the compulsory move?

The answer depends on what your goal is.

If your goal is to make an interactive agent that will play the game against you, then you are correct: there's no need to perform a search. Just return the compulsory move, and then run the search next time your agent has a choice about what to do.

If your goal is to determine the optimal way to play a game, or the expected payoff from a certain game position (another common use of search techniques), then you should run the search as normal, since the forced move won't necessarily lead to a particular end state.

Tangentially, if you're interested in ways to speed up search for checkers, check out the Chinook papers. There's a popularized account here, and more technical ones here and here by Schaeffer et al.


What is the variant of the minimax tree with 3 types of nodes called? I have a task on my class to find all the nodes, calculate their values and choose the best way for the player on the given game graph:

Everything is fine, but I have no idea what these dots are. Is this a third player, or just a 'split' for player1 move? Some kind of heuristics?

The triangles pointing up are Max' nodes. We assume it starts. Then follows a random choice of moves at the circles, for instance, with a die. The triangles pointing down are from Min. This variant is called Expectiminimax, see https://en.wikipedia.org/wiki/Expectiminimax_tree.

At that circles you have to multiply the possibilities on the edges below that nodes to your current value and sum all products up. The circles in your picture mean that Min dices.


Are artificial intelligence learnings or trainings transferable from one agent to the other? One disadvantage or weakness of Artificial Intelligence today the slow nature of learning or training success. For instance, an AI agent might require a 100,000 samples or more to reach an appreciable level of performance with a specific task. But this is unlike humans who are able to learn very quickly with a minimum number of samples. Humans are also able to teach one another, or in other words, transfer knowledge acquired.

My question is this: are Artificial Intelligence learnings or trainings transferable from one agent to the other? If yes, how? If no, why?

The simplest case is copying the software. That is instant duplication of the learnings. In a similar, but less trivial way you can adjust pretrained neural network classifiers to new datasets by simply re-initializing the last layer.

It gets more interesting when you have multiple agents and you want to combine their knowledge. This can be done with many techniques; the simplest is averaging the weights of identical neural network architectures. One example of this combined learning happened is Google Has a Room Full of Robot Arms Learning Hand-Eye Coordination. The relevant paper is Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection.

The most complicated case, where you want to transfer just the gist and prevent forgetting is -- up to my knowledge, and I'm relatively certain it's outdated in that area -- unsolved. The problem is post-hoc explanations. Humans decide to do something, observe the outcome and then explain why they did something / why it was good to do it. So we automatically generate hypotheses about the world. And sometimes we are able to formulate them in a way that others can understand them. That is not possible with current learning machines. We don't know yet how to abstract automatically from arbitrary problems.


How to improve a trained model over time (i.e. with more predictions)? I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network.  I trained it and saved the model and weights together on a .h5 file.
Recently, I have been reading about backpropagation. From what I understand, it's basically a way to tell the neural network whether if it's identified the correct output and that it is applied during training data only.
So, I was wondering if there is a way for the model to 'improve' over time as it makes more and more predictions. Or is that not how it would work with Neural Networks?

That is exactly a neural network works like.
Suppose you have a 1000 examples. How you train a network is: First, you divide these 1000 into maybe 100 batches (10 each). After that's done, you feed a batch to the network get its output and compare it with the ground truth, whatever is the error gets backpropagated. Then, for the next batch and then another. Once all these batches are done, you say an epoch is over. So, the number of epoch is effectively the number of times the network has seen the whole data.
This is how a neural network gets better.


Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems? Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems?
Monte Carlo Tree Search may use the above-mentioned methods in its selection step, but why do value-based and policy gradient methods not use these techniques?

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for the details.
However, compared to $\epsilon$-greedy (widely used in RL), UCB1 is more computationally expensive, given that, for each action, you need to recompute this upper confidence bound for every time step (or, equivalently, action taken during learning).
To see why, let's take a look at the UCB1 formula
$$
\underbrace{\bar{x}_{j}}_{\text{value estimate}}+\underbrace{\sqrt{\frac{2 \ln n}{n_{j}}}}_{\text{UCB}},
$$
where

$\bar{x}_{j}$ is the value estimate for action $j$
$n_{j}$ is the number of times action $j$ has been taken
$n$ is the total number of actions taken so far

So, at each time step (or new action taken), we need to recompute that square root for each action, which depends on other factors that evolve during learning.
So, the higher time complexity than $\epsilon$-greedy is probably the first reason why UCB1 is not so much used in RL, where interaction with the environment can be the bottleneck. You could argue that this recomputation (for each action) also needs to be done in bandits. Yes, it's true, but, in the RL problem, you have multiple states, so you need to compute value estimates for each action in all states (i.e. the full RL problem is more complex than bandits or contextual bandits).
Moreover, $\epsilon$-greedy is so conceptually simple that everyone can easily implement it in less than $5$ minutes (though this is not really a problem, given that both are simple to implement).
I am currently not familiar with Thompson sampling, but I guess (from some implementations I have seen) it's also not as cheap as $\epsilon$-greedy, where you just need to perform an argmax (can be done in constant time if you keep track of the highest value) or sample a random integer (it's also relatively cheap). There's a tutorial on Thompson sampling here, which also includes a section dedicated to RL, so you may want to read it.


How should I penalize the model proportionally to the error? I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols $3$ and $4$ do not differ in a way that captures quantity in any intuitive manner.

However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras.

In order for your loss function to work, it will need to be differentiable and  smoothly changing as predictions get better. That rules out using any form of argmax for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the expected prediction, e.g. if $d_i$ is the numerical digit for example $i$ and $y_{i,j}$ is the ground truth expressed as a one-hot vector, where $i$ is the example and $j$ is the digit class, then $\hat{y}_{i,j}$ is the probability predicted by your model. You could use $\hat{d}_i = \sum_{j=0}^9 j\hat{y}_{i,j}$ for the expected value and MSE loss of $\mathcal{L}(d_i,\hat{d}_i) = \frac{1}{2}(\hat{d}_i - d_i)^2$

You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model.

Note this solution makes $0$ close to $1$ but far away from $9$. If you want the digits to be considered close on a cycle (e.g. $8$ is closer to $1$ than it is to $4$) the you will need something more creative.

Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.


What to do with a GAN that trained well but got worse over time? I am training a WGAN-GP network based on the following paper, though I am using a different dataset. Now, for the first ~ 60-70 epochs, my network trained really well, which I could see in the loss going down, but I also made sure to regularly check the quality of the images.
Unfortunately, what I am seeing now (for the last $20$ epochs) is that the generator is getting worse and worse, the images don't look that good anymore. I save checkpoints every epochs, so in principle, I could stop training and get myself a state of the network from where it was still performing quite okay.
However, my question would be: How can I improve the training of the GAN? Would you decrease the learning rate?
I use a batch size of 124 and a learning rate of 1e-3. Maybe I could/should continue training (with a checkpoint that was still quite okay) with a learning rate of 5e-4?
Any other hints would be appreciated!

This is from my own experience with (Vanilla) GANs, so it might not translate exactly to your application, but maybe it gives some orientation.

your learning rate seems quite high. I've quite frequently found that 1e-5 is a good value for me. The training might take longer but will probably be more stable.
have you tried using dropout? It's a good regularisation mechanism that can prevent from overfitting and also make training more stable. I've made good experience with a dropout rate of ~0.4 and keeping it in both training and testing stages.
what is your discriminator vs generator update ratio? In my experiments, ~5 discriminator updates per generator update has often shown to improve things.
how many training samples do you have? If they are limited, it might be helpful to add a bit of noise on them. This manipulates the data a bit, but might make training more stable.
I have often not seen a major change when varying the batch size.
what are your network architectures? I have the feeling that small networks are often sufficient...

This is entirely based on my own experience, but maybe there are some interesting directions for you to explore. Happy to hear back from you about the success of these tests.


Identify unnecessary inputs of NN Lately I've been wondering.
Is there's a way to locate redundant/unnecessary/misleading inputs by analysis of weights in the first layer?

An interesting way to check for redundancy not only in the first layer but in any layer is to see the weights of you neuron. Let's consider the input layer: The nodes that are connected with the weights that have high values are the most important for your NN performance and in other words your input to these nodes in what mostly makes sense for your task.

If you are in Keras, you can use the get_weights method(). You may have a look here.


How do recommendation systems work? How do recommendation systems (e.g. on Youtube) work? Apparently, every user gets different recommendations depending on his location, his past liked videos, etc. So it would seem like a training model is applied to every single user, but that can't be possible, so how are these recommendations user-specific without applying a unique training model to every single user?

Let me try to explain how recommender systems work in production, as intuitively as possible:

Let's say we want to build a rec sys. for a restaurant discovery product, where users can rate restaurants, add reviews, photos, etc and also order food from there.

So, the user's feed will have a list of restaurants in his/her area. But, as I gain money from restaurants in the $-per-click model, I need to maximize the number of times a user clicks on a restaurant.

Also, it is obvious that a user is more likely to click on a restaurant if he tends to like it the most. [Restaurant features being Cost-for-two, cuisines, ratings, etc]

So here, a user is a data point and so is the restaurant. So, let's say the distance between the user vectors and the restaurant is the "likeability" of the user for the restaurant.

Let's say the vector is in the form [Japanese Spanish Mexican Chinese Indian Thai Turkish Lebanese]
Let's say a restaurant's vector is: A = [0 0 0 1 1 0 0 0]

and the user's is: B = [2 23 4 53 43 21 2 45]

Each number being a particular cuisine. For restaurant, it is yes or no (1 or 0), a.k.a whether the cuisine is served or not.

For user, it can be the number of clicks he/she did on restaurants with that cuisine. (I am over-simplifying here. But, this can be as complex as a weighted score for clicks+transactions+content_generated like review, ratings, etc)

Now, the the cosine similarity measure between the 2 vectors in an 8-dimensional space (length of the vector) is the likeability score.

Now, the system uses these scores while doing it's feed ranking for each user.

This can be near-realtime to being updated hourly/daily, depending on the servers the company can afford.


What are ontologies in AI? What exactly are ontologies in AI? How should I write them and why are they important?

An ontology at its most abstract is a model of the world. It describes concepts that exist in the world and how those concepts are related.
Ontologies are similar to taxonomies. A taxonomy is a tree-like hierarchy that organizes concepts in increasing levels of specificity. What an ontology adds is a second type of link between those concepts that explains how they are connected. So a taxonomy could say $isA(DOG, ANIMAL)$ ("a dog is an animal"). An ontology may also describe something like: $chase(DOG, CAT)$ ("dogs chase cats").
Why use an ontology? It can be used to reason about the world. I subscribe to a very particular kind of language-centric ontology that isn't worth addressing here. Instead, the Semantic Web project (Tim Berners-Lee) is probably closer to what would interest you. Semantic Web uses a type of description logic, which is outside my realm of expertise. But there are tools for processing these kinds of DL and gaining "understanding" from them. To work with this kind of ontology you'll want to be familiar with the idea of Resource Description Framework triples.


How is direction of weight change determined by Gradient Descent algorithm The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for weight change, for the purpose of reducing the Loss function in a Linear Regression Model, determined by this algorithm?

First thing is that what does gradient descent do? Gradient Descent is a tool of calculus which we use to determine the parameters (here weights) used in a Machine Learning algorithm or a Neural Network, by running the Gradient Descent algorithm iteratively.

What does the vector obtained from one iteration of gradient descent tell us? It tells us the direction of weight change (when weights are treated as a vector) for the maximum reduction in the value outputted by a loss function. The intuition behind why Gradient Descent gives such a direction can be found here:

Why the gradient is the direction of steepest ascent - Khan Academy
Why is gradient the direction of steepest ascent?
Why does gradient descent work?

In general cases the cost/loss function is an n-dimensional paraboloid (we design function in such a way such that its convex).

A 2-D paraboloid with x and y as independent variables.

Now why do we reduce the weights in the direction of negative gradient only? Why not some other direction?

Since we want to reduce the cost in Linear Regression for better predictions we choose only the negative Gradient Descent direction, as it is the direction of steepest descent even though we could have chosen some other possible vector direction to reduce cost, but the negative gradient descent direction ensures that:

Cost is always decreased (if the step size is correct and in general only in cases of convex functions e.g paraboloid).
Cost is decreased by the maximal amount if we move in that direction, so we don't have to worry whether cost will decrease or not if we move in that direction.

Also we use learning rate alpha to scale by how much amount we want to decrease the weights.

EDIT: As pointed by @pasaba the error function may not be a paraboloid but in general a good cost function looks like a paraboloid with skewed axis.


Can Machine Learning make economic decisions of human quality or better? Basically, economic decision making is not restricted to mundane finance, the managing of money, but any decision that involves expected utility (some result with some degree of optimality.)

Can Machine Learning algorithms make economic decisions as well as or better than humans?

"Like humans" means understanding classes of objects and their interactions, including agents such as other humans.

At a fundamental level, there must be some physical representation of an object, leading to usage of an object, leading to management of resources that the objects constitute.

This may include ability to effectively handle semantic data (NLP) because mcuh of the relevant information is communicated in human languages.

Consider managing a memory structure as an economic function.  (Where to put, and how to manage, the resources constituted by data.)  This is something computers can do better and faster than any human.  The reason is that the system in which the economic decisions are being made is fully defined.

Routing of packages is a similar, economic function that computers do much better than humans.

These functions haven't been handled by Machine Learning in the past, but, soon after the AlphaGo milestone, Google found an economic application for Machine Learning.  Google's DeepMind trains AI to cut its energy bills by 40% (Wired)

So it's entirely context dependent.

As the model increases in complexity and nuanced, utility will be reduced. (In the former case it's a time and space issue related to computational complexity, and in the latter case, often a function of incomplete information or inability to define parameters.)

But as the sophistication of the machine learning algorithms increases, and the models continue to be refined, the algorithms will get better and better at managing intractability and incomplete information.


Advantages of CNN vs. LSTM for sequence data like text or log-files When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what does it depend on? Are there any papers or statistics that confirm this?
I'm thinking of data like Linux log entries or short sentence of length of less than 20 words/tokens.
Personally i would almost always use LSTM but I'm curious if CNN wouldn't be better in some cases, if its possible to implement them in a meaningful way. On short sentence there isn't much buffer to use CNN if i'm not mistaken.

The networks with one-dimensional convolution over the temporal dimensions are called Temporal Convolutional Networks. The authors of this study claim that TCNs are comparable to RNNs in performance on common tasks associated with RNNs.


How much power might the creator of the first Artificial General Intelligence obtain? If Artificial General Intelligence is possible, whoever creates it is likely to have a lot of power.

What might this look like in practical terms, and what might be limiters?

Good answers will include aspects such as historical precedents related to technology, and cover subjects such as potential societal and economic impacts, and limiters such as regulations and physics.

"Artificial Intelligence" has existed for many, many years.  Decades even.  Go back to Arthur Samuel and his Checkers playing program (if not before).  That's AI.

The thing is, as soon as computers can do something, people want to quit treating it as AI.  AI is almost, by definition, something that can't really exist, because it has to be about something humans can do, that computers can't.   So all progress made in the field is reclassified as "not AI" as soon as it does anything useful.

A more salient question might be "who will be the first to announce that Artificial General Intelligence" has been created.   AGI will, if it ever happens, be fundamentally different in terms of being able to learn to do anything, as opposed to, say, a Checkers program which can only play Checkers, or a Go program that can only play Go, etc.

And the answer to that question is -- nobody knows.  Nobody even know for sure that AGI will ever happen (though a lot of us presume that it will).  Anyway, there isn't a ton of research on AGI going on, relative to the amount of research in Machine Learning / Narrow AI.  But somebody working in their garage could make a huge breakthrough tomorrow for all we know.  I think a lot of people assume that creating this will take Nation-State level resources, but that hasn't really been proven.  So it might be some kid in a garage in Omaha tomorrow, or it might be the Chinese government in 2074.  There's just no way to really know.


What is the mean in the variational auto-encoder? Here's a diagram of a variational auto-encoder.

There are 2 nodes before the sample (encoding vector). One is the mean, one is the standard deviation. The mean one is confusing.

Is it the mean of values or is it the mean deviation?

$$\text{mean} = \dfrac{X_i+..+X_n}{N}$$

$$\text{mean deviation} = \dfrac{[X_i|+..+|X_n|}{N}$$

Kind of neither, although leaning towards the first definition of the Mean as a simple average of values.

It's a distribution parameter of the Gaussian, so it's the expected average of samples as the number of samples approaches infinity.

The distrinction is that you could draw three samples, -2, 0 and -1, from a Standard Normal - the mean of samples would be -1, but the distribution mean is still 0.

From a network architecture PoV, it's a learnable transformation applied on top of samples from a Standard Normal, so you only need to learn the transformation and get sampling for 'free'.


Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster? I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after some time). So, it is getting difficult for me to continue training in this environment.
The continuous state-space is $\mathcal{S}  \subseteq \mathbb{R}^{10}$ and the continuous action-space $\mathcal{A} \subseteq \mathbb{R}^{2}$.
What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
Or I should go with the behavior cloning technique to train the actor-network only, so that it converges rapidly?
I just want to get the work done first and then I can think of exploring the environment.

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?

You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still "correct", regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a "policy").
There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).

Or I should go with the behavior cloning technique to train the actor network only, so that it converges rapidly?

I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.


How is AI helping humanity? There was a lot of Negative news on Artificial Intelligence. Most people were first exposed to the idea of artificial intelligence from Hollywood movies, long before they ever started seeing it in their day-to-day lives. This means that many people misunderstand the technology. When they think about common examples that they’ve seen in movies or television shows, they may not realize that the killer robots they’ve seen were created to sell emotional storylines and drive the entertainment industry, rather than to reflect the actual state of AI technology.

There are few questions on our SE on how AI impacts/harms humankind. For example, How could artificial intelligence harm us? and Could artificial general intelligence harm humanity?
However, now, I'm looking for the positive impacts of AI on humans. How could AI help humankind?

For good or bad, AI is the next step in automation. The impact which is already visible, and trends show will continue in the future, is the eradication of repetitive and body-straining labor.
Hopefully, the transformation will be gradual enough for the global labor market to re-adjust, otherwise, we'll face a problem of growing unemployment. It seems to me that we've become aware enough to foresee bad outcomes of our inventions, hence in almost every dimension affected by AI, a plausible and either positive or negative future can be presented, depending on the sentiment of the storyteller.
Regardless of what different experts and sci-fi writers tell us about the future, actually predicting it is a futile endeavour. Considering that predictions made for a dynamic system, even when we have a lot of data and good models (like about the weather), become unreliable just for a few weeks ahead.


Why do we get a three-dimensional output after a convolutional layer? In a convolutional neural network, when we apply the convolution on a $5 \times 5$ image with $3 \times 3$ kernel, with stride $1$, we should get only one $4 \times 4$ as output. In most of the CNN tutorials, we are having $4 \times 4 \times m$ as output. I don't know how we are getting a three-dimensional output and I don't know how we need to calculate $m$. How is $m$ determined? Why do we get a three-dimensional output after a convolutional layer?

If you have a $h_i \times w_i \times d_i$ input, where $h_i, w_i$ and $d_i$ respectively refer to the height, width and depth of the input, then we usually apply $m$ $h_k \times w_k \times d_i$ kernels (or filters) to this input (with the appropriate stride and padding), where $m$ is usually a hyper-parameter. So, after the application of $m$ kernels, you will obtain $m$ $h_o \times w_o \times 1$ so-called feature maps (also known as activation maps), which are usually concatenated along the depth dimension, hence your output will have a depth of $m$ (given that the application of a kernel to the input usually produces a two-dimensional output). For this reason, the output is usually referred to as output volume.

In the context of CNNs, the kernels are learned, so they are not constant (at least, during the learning process, but, after training, they usually remain constant, unless you perform continual lifelong learning). Each kernel will be different from any other kernel, so each kernel will be doing a different convolution with the input (with respect to the other kernels), therefore, each kernel will be responsible for filtering (or detecting) a specific and different (with respect to the other kernels) feature of the input, which can, for example, be the initial image or the output of another convolutional layer.


Is it possible to pre-train a CNN in a self-supervised way so that it can later be used to solve an instance segmentation task? I would like to use self-supervised learning (SSL) to learn features from images (the dataset consists of similar images with small differences), then use the resulting trained model to bootstrap an instance segmentation task.
I am thinking about using Faster R-CNN, Mask R-CNN, or ResNet for the instance segmentation task, which is pre-trained in an SSL way by solving a pretext task, with the aim that this will lead to higher accuracy and also teach the CNNs with fewer examples during the downstream task.
Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy?

Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy?

Yes, it's possible and this has already been done. I don't know the details (because I have not yet read those papers), but I will provide you with some links to some potentially useful papers (based on their titles and abstracts) and associated code.

Learning Instance Segmentation by Interaction (code, 2018)
Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction (code, 2019)
White Matter Tract Segmentation with Self-supervised Learning (code, 2020)
Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy (code, 2020)
Simulation of Brain Resection for Cavity Segmentation Using Self-supervised and Semi-supervised Learning (code, 2020)

You can probably find more relevant papers here, where I also found some of the just cited papers.
The pre-text tasks designed in these papers could be useful in your case, but it may also turn out that you need to develop other pre-text tasks or combine multiple of them.
Maybe you can start from some pre-trained faster R-CNN or some appropriate model for instance segmentation (that you can find on the web, for example, here), which has been pre-trained on some imagery data similar to yours (either with SSL or by other means), then try to fine-tune this model with your labeled dataset for instance segmentation, and see if you get better results than just training a faster R-CNN from scratch. Eventually, if this pre-trained model does not lead to higher performance, you could pre-train it yourself with some SSL technique that you can come up with or one that is described in the literature. Of course, you should probably use a pre-trained model that has been pre-trained with data that is relevant for your downstream task (i.e. the instance segmentation task). You didn't describe the details of your unlabelled and labeled data, so I cannot be more specific (and I wouldn't currently be able to, in any case, because I didn't fully read those papers, and my experience with SSL techniques is mostly theoretical).
For more info about SSL, take a look at this and this answers.


In this implementation of pix2pix, why are the weights for the discriminator and generator losses set to 1 and 100 respectively? I am working on a pix2pix GAN model that was inspired by the code in this Github repository. The original code is working and I have already customized most of the code for my needs. However, there is one part I am unable to understand.
The pix2pix GAN is a conditional GAN network that takes an image as a condition and outputs a modified image - such as blurry to clear, facades to buildings, filling up cut out part of an image, etc. The combined model thus takes as input a conditional image, the discriminator compares it with the dummy matrix named valid or fake, containing 0s or 1s according to validity (0 for generated samples, 1 for real samples). The generator loss is according to similarity with real sample + discriminator. The following code corresponds to what I told:
self.combined = Model(inputs=[img_A, img_B], outputs=[valid, fake_A])
self.combined.compile(loss=['mse', 'mae'],
                      loss_weights=[1, 100],
                      optimizer=optimizer)

The losses are thus set as MSE for discriminator output and MAE for generator. That seems to be OK, but I can not understand why the implementation uses 1 and 100 for the weights of the discriminator and generator losses, respectively, which seems to imply that the discriminator loss is 100 times lower than the loss of the generator. I couldn't find the reason in the original article. Are my understandings of the GAN incorrect?
Disclaimer: I have posted this question on Stats SE, but have no luck with answers. Maybe it is more suitable for AI.

After further research, I have found the answer. nbro was of course right, the weighting is not implementation dependant, it was already introduced in the paper (arXiv). However, there is minimal information about that, only it is mentioned within the optimization function:
$\arg \min_G \max_D \mathcal{L}_{cGAN}(G,D) + \lambda \mathcal{L}_{L1}(G)$
In fact, the parameter lambda stands for weight of $L_1$ loss. There should be a second parameter $\epsilon$ for $\mathcal{L}_{cGAN}(G,D)$ because in real implementation you can actually change that as well. In theory, however, only modulation of $L_1$ loss is needed. The paper does not state why the $\lambda$ parameter is needed, only mentions that setting it to 0 will lead to pure cGAN implementation. David Brownlee in his blog states:

The adversarial loss influences whether the generator model can output
images that are plausible in the target domain, whereas the L1 loss
regularizes the generator model to output images that are a plausible
translation of the source image. As such, the combination of the L1
loss to the adversarial loss is controlled by a new hyperparameter
lambda, which is set to 10, e.g. giving 10 times the importance of the
L1 loss than the adversarial loss to the generator during training.

The loss weights thus are hyperparameters that tell the network how much plausible translation of the source image do we need.


Top Frequent occurrence word effect in Model Efficiency? Assume that I have a Dataframe with the text column.
Problem: Classification / Prediction
    sms_text
0   Go until jurong point, crazy.. Available only ...
1   Ok lar... Joking wif u oni...
2   Free entry in 2 a wkly comp to win FA Cup fina...
3   U dun say so early hor... U c already then say...
4   Nah I don't think he goes to usf, he lives aro...

After preprocessing the text

From the above WordCloud, we can find the most frequent(occurred) words like
Free
Call
Text
Txt

As these are the most frequent words and adds less importance in prediction/classification as they appear a lot. (My Opinion)

My Question is
Removing top frequent(most occurred) words will improve the model score?
How does this impact on model performance?
Is it ok to remove the most occurred words?

As far as I know, there are few aspects that would probably improve the model score:

  Normalization
  Lemmatization
  Stopwords removal (as you asked here)

Based on your question, "is removing top frequent words (stopwords) will improve the model score?". The answer is, it depends on what kind of stopwords are you removing. The problem here is that if you do not remove stop words, the noise will increase in the dataset because of words like I, my, me, etc. Here is the comparison of those three aspects using SVM Classifier.

You may see that without stopwords removal the Train Set Accuracy decreased to 94.81% and the Test Set Accuracy decreased to 88.02%. But, you should be careful about what kind of stopwords you are removing.

If you are working with basic NLP techniques like BOW, Count Vectorizer or TF-IDF(Term Frequency and Inverse Document Frequency) then removing stopwords is a good idea because stopwords act like noise for these methods. If you working with LSTM’s or other models which capture the semantic meaning and the meaning of a word depends on the context of the previous text, then it becomes important not to remove stopwords.

So, what's the solution?

  You may want to create a Python package nlppreprocess which removes stops words that are not necessary. It also has some additional functionalities that can make cleaning of text fast. For example:

from nlppreprocess import NLP
import pandas as pd

nlp = NLP()
df = pd.read_csv('some_file.csv')
df['text'] = df['text'].apply(nlp.process)

Source:

https://github.com/miguelfzafra/Latest-News-Classifier
https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52


How is MuZero's second binary plane for chess defined? From the MuZero paper (Appendix E, page 13):

In chess, 8 planes are used to encode the action. The first one-hot plane encodes which position the piece was moved from. The next two planes encode which position the piece was moved to: a one-hot plane to encode the target position, if on the board, and a second binary plane to indicate whether the target was valid (on the board) or not. This is necessary because for simplicity our policy action space enumerates a superset of all possible actions, not all of which are legal, and we use the same action space for policy prediction and to encode the dynamics function input. The remaining five binary planes are used to indicate the type of promotion, if any (queen, knight, bishop, rook, none).

Is the second binary plane all zeros or all ones?  Or, something else?
How is it known if the move is off the board?  For my game, I know if it is a legal move on the board, but do not know if the move is off the board.

Is the second binary plane all zeros or all ones? Or, something else? How is it known if the move is off the board? For my game, I know if it is a legal move on the board, but do not know if the move is off the board.

The second binary plane is one-hot by definition, there is a single one and everything else is zero. If this definition is not met, it's no longer "one-hot".
The paper doesn't state how exactly to implement "off the board" implementation. Research paper wouldn't go into coding level. However, detecting "off the board" is not a challenging task.

https://webcache.googleusercontent.com/search?q=cache:djj-G4T_PwgJ:https://craftychess.com/hyatt/boardrep.html+&cd=2&hl=en&ct=clnk&gl=au

The next step in board representation evolution is to enclose the board inside a larger array, so that illegal squares are "off" the edge and are easily detectable.

A possibility is add borders to your board. Crafty did that. Extend the board to 10x10. Not 9x9 because you need to deal with knight jumping.
Exactly how you should be doing it is implementation defined. We don't know what Google did because AlphaZero is not open source. I'm here just giving you an example.