Are Modular Neural Networks more effective than large, monolithic networks at any tasks? Modular/Multiple Neural networks (MNNs) revolve around training smaller, independent networks that can feed into each other or another higher network.

In principle, the hierarchical organization could allow us to make sense of more complex problem spaces and reach a higher functionality, but it seems difficult to find examples of concrete research done in the past regarding this. I've found a few sources:

https://en.wikipedia.org/wiki/Modular_neural_network

https://www.teco.edu/~albrecht/neuro/html/node32.html

https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&isAllowed=y

A few concrete questions I have:

Has there been any recent research into the use of MNNs?
Are there any tasks where MNNs have shown better performance than large single nets?
Could MNNs be used for multimodal classification, i.e. train each net on a fundamentally different type of data, (text vs image) and feed forward to a higher level intermediary that operates on all the outputs?
From a software engineering perspective, aren't these more fault tolerant and easily isolatable on a distributed system?
Has there been any work into dynamically adapting the topologies of subnetworks using a process like Neural Architecture Search?
Generally, are MNNs practical in any way?

Apologies if these questions seem naive, I've just come into ML and more broadly CS from a biology/neuroscience background and am captivated by the potential interplay.

I really appreciate you taking the time and lending your insight!

A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. Modularization as systems grow in complexity and demands on performance and capability grow is as old as industrialization.

Our laboratory works with robotic control, thermodynamic instrumentation, and data analysis, artificial networks are components in these larger system contexts. we have no single MLPs or RNNs that by themselves perform any useful function.

Contrary to the conjecture about hierarchies decades ago, the topology approach that seems to work well in most cases follows the more common system module relationships that are seen in power plants, automated factories, aeronautics, enterprise information architectures, and other complex engineered creations. The connections are those of flow, and if those are designed well, oversight functions are minimal. Flow occurs between modules involving protocols for communications, and each module performs its function well, encapsulating the lower level of complexity and functional detail. It is not one network overseeing another that seems to emerge most effective in actual practice but balance and symbiosis. Identification of clear master-slave design in the human brain seems to be equally slippery. It may be naive to assume that some all knowing subsystem can be a single causal factor in the remarkable collaborative synchronization of a number of other subsystems.

The challenge is not finding the information paths that make up the system information topology. The information flow is often obvious upon problem analysis. The difficulty is in discovering the best of strategies to train these independent networks. Training dependencies are common and often critical, whereas in animals, the training occurs in situ or not at all. We're discovering conditions under which that kind of learning in our systems is practical and how to achieve it. Most of our research along these lines is intended to discover ways to achieve higher reliability and lower burden in terms of research hours to get it.

Higher functionality is not always of benefit. It often produces lower reliability and consumes additional development resources with little return. Find a way to marry higher level automation, resource thrift, and reliability into one development process, and you might win an award and honorable mention around the web.

Parallel systems that have the same objective is a good idea, but not a new one. In one aeronautics system, nine parallel systems have the same objective, in groups of three. Each group uses a different computing approach. If two of the systems using the same approach provide the same output and the third differs, the matching output is used and the difference in the third is reported as a system fault. If two of the different approaches provide similar results and the third differs substantially, a merge of the two similar results is used and the third is reported as a use case to further develop the dissenting approach.

The improved fault tolerance has a cost, eight more systems and associated computing resources and connectivity plus the comparators at the tail, but in systems that are a matter of life and death, the extra costs are paid and the reliability is maximized.

Dynamic topological adaptation is related to redundant systems and fault tolerance but in some ways is quite distinct. In that area of development, the technology to follow is neuromorphic computing, which is partly inspired by neuroplasticity.

One last distinction to consider is between process topology, data topology, and hardware topology. These three geometric frames can produce greater efficiency together if aligned in specific ways that produce more direct mappings between the relationships between flow, representation, and mechanics. They are, however, distinct topologies. The meaning of alignment may not be apparent without diving deeply into these concepts and the details that emerge for specific product or service objectives.

Footnotes

[1] Deep networks that are trained as a single unit and function without connectivity to other artificial networks are not necessarily monolithic. Most practical deep networks have heterogeneous sequence of layers in terms of their activation functions and often their cell types.


What is a deep neural network? What is the definition of a deep neural network? Why are they so popular or important?

A deep neural network (DNN) is nothing but a neural network which has multiple layers, where multiple can be subjective.

IMHO, any network which has 6 or 7 or more layers is considered deep. So, the above would form a very basic definition of a deep network.


How long has it taken for autonomous driving cars to be being sold and used on the roads today? I remember the first time hearing about google trying to make driverless cars.  That was YEARS ago!

These days, I'm beginning to learn about Neural Nets and other types of ML and I was wondering:

Does anybody know how many hours (or days, months, etch) is needed in training time to get the results that are now used in today's self-driving vehicles?

(I am ASSUMING they use Neural networks for this...)

There are a couple of things to consider with this question:

First, training time can be a deceptive measure. Neural network training is considered "trivially" parallelizable. This means that the more computers you have access to, the faster you can train (most of the time, one computation doesn't depend on another, so you can do them both at the same time). Since the start of the current phase of self-driving cars, GPU performance has increased by more than a factor of 5. Further, large companies like Google have migrated to AI-specific ASIC chips. These are faster still. This means any estimate of "time" is likely to be confusing or misleading. The amount of time required to train a network has dropped rapidly over time, even if the number of processing cycles required has stayed the same.

Second, it is very unlikely that self-driving car companies have spent that time training a single model. Instead, they are probably starting from a given model (say, last month's), and trying different approaches to improving it, with new data, alternative training methods, or "expert" domain knowledge. This makes it difficult to reason about what it means to "get the results that are used in today's self-driving vehicles." If we count all the aborted attempts, probably it's on the order of high hundreds or low thousands of person-years of researchers (6-7 years, and there aren't thousands of people working in research in this area). If you also count related development and engineering efforts, it's probably an order of magnitude or more than that.

That said, it might be more interesting to think about the amount of simulated driving time that is needed to train a network in this way. As discussed in this excellent blog post, "Deep Reinforcement Learning Doesn't Work Yet", the amount of training experience required to train a neural net for this kind of problem is extremely large. To learn to play simple Atari games better than human players required about 244 hours of exposure. These games are typically just a single screen, and most humans can pick them up in a couple of minutes or less. This site estimates average time for a human to learn to drive competently at just shy of 70 hours. Applying the same ratio, we can infer that a deep neural net would want something like 2 years of experience driving to achieve human level performance. This seems like the right ballpark, but probably a lot of that driving is done "offline" in a simulated environment, rather than operating the vehicle directly.

Of course, these are just ballpark estimates. The exact figures are likely to be proprietary. Further, there are some reports that modern systems are abandoning neural nets, and moving back to a more "rule-based" paradigm because of the difficulties in training them. I'm not sure how much credit to give those reports, but it again makes it difficult to pin down a training time.


What are some examples of functions that machine learning models compute? My simple understanding of AI is that it is based on a mathematical model of a problem. If I understood correctly, the model is a polynomial equation and its weights are calculated by training the model with data sets.
I am interested to see a few example polynomial equations (trained models) which are used in certain problem areas. I tried to search it, but so far could not find any simple answers.
Can anyone list a few examples here?

If I understood correctly, the model is a polynomial equation

No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a polynomial, but, for example, in a neural network, you can combine many sigmoids to build complicated functions that may not necessarily be polynomials.
We usually distinguish between linear (straight-lines) and non-linear functions (rather than polynomials and non-polynomials). In some cases, it is straightforward to visualize the function that your model computes: for example, in the case of linear regression, once you learned the coefficients (i.e. the slope and y-intercept), you can plot the learned straight-line function. In other cases, for example, in the case of neural networks, it is not fully clear how to visualize the function that your model computes, given that it is the composition of many non-linear functions (typically, ReLUs, sigmoids or hyperbolic tangents).
If you are interested in solving problems with polynomials, take a look at polynomial regression.

and its weights are calculated by training the model with data sets.

Yes, in machine learning, we want to find a function that "fits the given data", and the specific meaning of "fitting the data" depends on the specific machine learning technique.
For simplicity, let's focus on supervised learning, a machine learning technique where we are given a labelled dataset, i.e. a dataset of pairs $D = \{(x_1, y_1), \dots, (x_N, y_N)\}$, where we assume that $f(x_i) = y_i$, for some typically unknown function $f$, and $y_i$s are the labels (the outputs of $f$) and $x_i$ the inputs of $f$. The goal is to find function $g_{\theta}$ that approximates well $f$. I will soon describe what the subscript $\theta$ represents.
For simplicity, let's assume that $f$ is a linear function (i.e. a straight-line). So, we can define a linear model $g_{\theta}$ that we can use to find a function that approximates well $f$. Here is the linear model
$$g_{\theta}(x) = ax + b,$$
where

$g_{\theta}(x)$ is the output
$x$ is the input
$ax + b$ is the linear function (a straight-line)
$a$ is the slope (a parameter, aka weight)
$b$ is the $y$-intercept (another parameter)
$\theta = \{ a, b \}$ (the set of parameters of the linear model)

Why is this a model? I call this a model because, depending on the specific values of the parameters $\theta$, we have different specific functions. So, I am using the term "model" as a synonym for a set of functions, which, in this case, are limited by the definition $ax + b$ and the specific values that $a$ and $b$ (i.e. $\theta$) can take.
So, what do we do with this linear model? We want to find a specific set of parameters $\hat{\theta}$ (note that I use the $\hat{ }$ to emphasize that this is a specific configuration of the variable $\theta$) that corresponds to a linear function (a straight-line) that approximates $f$ well.  In other words, we need to find the parameters $\hat{\theta}$, such that $g_\hat{\theta} \approx f$, where $\approx$ means "approximately computes".
How do we do that? We typically don't know $f$, but we know (or assumed) that $f(x_i) = y_i$, so the labeled dataset $D$ contains information about our unknown function $f$. So, the idea is that we can use the dataset $D$ to find a specific set of parameters $\hat{\theta}$ that corresponds to some function that approximates $f$ according to the information in $D$.
This process of finding $\hat{\theta}$ based on $D$ is often denoted as "fitting the model to the data". There different ways of fitting the model to the data that differ in the way they compute some notion of distance between the information in $D$ and $g_{\hat{\theta}}$. I will not explain them here because this answer is already quite long. If you want to know more about it, you should take a book about the topic and read it.

What are some examples of functions that machine learning models compute?

I don't have specific examples, but you can easily try to fit a linear regression model to some labelled data, then plot the function that you found. You could use the Python library sklearn to do that.


Find anomalies from records of categorical data I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values.

Ultimately, I am looking for a way to find anomalies i.e. to identify rows for which the combination of values seems incorrect with respect to the data I saw so far.

So far, I was thinking about building a model to predict the value for each column and then build some metric to evaluate how different the actual row is from the predicted row.

I would greatly appreciate any help!

First of all, you mention that you have categorical data. I don't see how you can define similarity so that you can also define the distance between the predicted value and the ground truth (error). You can do that only if the data are ordinal.

If you want to just classify between normal and anomalous points (binary classification), without caring about further classification of the anomaly types themselves, one of the most common algorithms is the One-Class Support Vector Machine (OC-SVM).

Anomalies are unpredictable in nature and sometimes hard to replicate and record. Therefore, there is usually lack of anomalous data and supervised learning approaches suffer because if you sacrifice some "precious" anomalous points to train the algorithm, you cannot use them to test it.

The main advantage of OC-SVM is that it is semi-supervised learning, meaning that you train it only with normal data and then it can detect samples that deviate from the trained behaviour during testing and classify them as anomalous. Thus, you "save" all the rare anomalous points for testing purposes!

Take a look at this short Python example, it has all you need :)


BERT: After pretraining 880000 step, why fine-tune not work? I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples
Pretrain parameters:
 15:47:02,534: INFO tensorflow 140678508230464   init_checkpoint: bertbase3layer-extract-from-google
 15:47:02,534: INFO tensorflow 140678508230464   optimizer_type: lamb
 15:47:02,534: INFO tensorflow 140678508230464   max_seq_length: 64
 15:47:02,534: INFO tensorflow 140678508230464   max_predictions_per_seq: 5
 15:47:02,534: INFO tensorflow 140678508230464   do_train: True
 15:47:02,535: INFO tensorflow 140678508230464   do_eval: False
 15:47:02,535: INFO tensorflow 140678508230464   train_batch_size: 32
 15:47:02,535: INFO tensorflow 140678508230464   eval_batch_size: 8
 15:47:02,535: INFO tensorflow 140678508230464   learning_rate: 5e-05
 15:47:02,535: INFO tensorflow 140678508230464   num_train_steps: 10000000
 15:47:02,535: INFO tensorflow 140678508230464   num_warmup_steps: 10000
 15:47:02,535: INFO tensorflow 140678508230464   save_checkpoints_steps: 1000
 15:47:02,535: INFO tensorflow 140678508230464   display_loss_steps: 10
 15:47:02,535: INFO tensorflow 140678508230464   iterations_per_loop: 1000
 15:47:02,535: INFO tensorflow 140678508230464   max_eval_steps: 100
 15:47:02,535: INFO tensorflow 140678508230464   num_accumulation_steps: 1
 15:47:02,535: INFO tensorflow 140678508230464   allreduce_post_accumulation: False
 15:47:02,535: INFO tensorflow 140678508230464   verbose_logging: False
 15:47:02,535: INFO tensorflow 140678508230464   horovod: True
 15:47:02,536: INFO tensorflow 140678508230464   report_loss: True
 15:47:02,536: INFO tensorflow 140678508230464   manual_fp16: False
 15:47:02,536: INFO tensorflow 140678508230464   amp: False
 15:47:02,536: INFO tensorflow 140678508230464   use_xla: True
 15:47:02,536: INFO tensorflow 140678508230464   init_loss_scale: 4294967296
 15:47:02,536: INFO tensorflow 140678508230464   ?: False
 15:47:02,536: INFO tensorflow 140678508230464   help: False
 15:47:02,536: INFO tensorflow 140678508230464   helpshort: False
 15:47:02,536: INFO tensorflow 140678508230464   helpfull: False
 15:47:02,536: INFO tensorflow 140678508230464   helpxml: False
 15:47:02,536: INFO tensorflow 140678508230464 **************************

Pretrain loss: (I remove nsp_loss)
{'throughput_train': 1196.9646684552622, 'mlm_loss': 0.9837073683738708, 'nsp_loss': 0.0, 'total_loss': 0.9837073683738708, 'avg_loss_step': 1.200513333082199, 'learning_rate': '0.00038143058'}
{'throughput_train': 1230.5063662500734, 'mlm_loss': 1.3001925945281982, 'nsp_loss': 0.0, 'total_loss': 1.3001925945281982, 'avg_loss_step': 1.299936044216156, 'learning_rate': '0.00038143038'}
{'throughput_train': 1236.4348949169155, 'mlm_loss': 1.473339319229126, 'nsp_loss': 0.0, 'total_loss': 1.473339319229126, 'avg_loss_step': 1.2444063007831574, 'learning_rate': '0.00038143017'}
{'throughput_train': 1221.2668264552692, 'mlm_loss': 0.9924975633621216, 'nsp_loss': 0.0, 'total_loss': 0.9924975633621216, 'avg_loss_step': 1.1603020071983337, 'learning_rate': '0.00038142994'}

Fine-tune code:
self.train_op = tf.train.AdamOptimizer(0.00001).minimize(self.loss, global_step=self.global_step)

Fine-tune accuracy: (restore from my ckpt pretrained from https://github.com/NVIDIA/DeepLearningExamples)
epoch 1:
training step 895429, loss 4.98, acc 0.079
dev loss 4.853, acc 0.092

epoch 2:
training step 895429, loss 4.97, acc 0.080
dev loss 4.823, acc 0.092

epoch 3:
training step 895429, loss 4.96, acc 0.081
dev loss 4.849, acc 0.092

epoch 4:
training step 895429, loss 4.95, acc 0.082
dev loss 4.843, acc 0.092

Without restore the pretrained ckpt:
epoch 1:
training step 10429, loss 2.48, acc 0.606
dev loss 1.604, acc 0.8036

Restore the google's BERT-Base pretrained ckpt. Or restore from a pretrained ckpt pretrained from https://github.com/guotong1988/BERT-GPU
epoch 1:
training loss 1.89, acc 0.761
dev loss 1.351, acc 0.869

change
bert_output = bert_model.get_pooled_output()

to
bert_output = tf.reduce_mean(bert_model.get_sequence_output()[:,1:,:],1)


What kind of problems is DQN algorithm good and bad for? I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) that are well solvable with the family of DQN algorithms? What kind of problems are not well fit for DQNs?

I don't currently have much practical experience with DQN, but I can partially answer this question also based on my theoretical knowledge and other info that I found.
DQN is typically used for

discrete action spaces (although there have been attempts to apply it to continuous action spaces, such as this one)

discrete and continuous state spaces

problems where the optimal policy is deterministic (an example where the optimal policy is not deterministic is rock-paper-scissors)

off-policy learning (Q-learning is an off-policy algorithm, but the point is that, if you have a problem/application where data can only be or has been gathered by a policy that is unrelated to the policy you want to estimate, then DQN is suitable, though there are other off-policy algorithms, such as DDPG)

This guide also states that DQN is slower to train, but more sample efficient than other approaches, due to the use of the experience replay buffer.
Moreover, if you have a small state and action space, it is probably a good idea to just use tabular Q-learning (i.e. no function approximation), given that it is guaranteed to converge to the optimal value function.
See also this and this questions and this article (which compares DQN with policy gradients).


How does Monte Carlo Tree Search UCT exploitation value change based on perspective? In this blog toward the end, the author writes the following:

For the sake of my question, let’s assume that a terminal state gives a reward of +1 for a win and -1 for a loss.

When the author says “for any two consecutive nodes this perspective is opposite,” does that mean that if $Q_i$ is positive (for example, 4) for player A at a given node, the same node will have the negative of that value for player B (-4 in my hypothetical)?

Do I need to compute two statistics to store the node value (one for each player) or can I simply store statistics for one player and flip the sign at every consecutive node?

In UCT, the value of Q(vi) / N(vi) is bounded between 0 and 1. Normally when applying MCTS to 2-player games, what happens is the following:
N(vi) corresponds to the total number of games simulated in node vi.
Q(vi) corresponds to the total number of games simulated and won in node vi.
So in each simulation Q(vi) will add +1 to the winning player and +0 to the losing player.

In a tree representation of a 2-player game, each level will represent player A and player B alternately. So you will add 1 to every other node. I don't think it makes sense to add negative values to Qi.

So to answer your question:
You only need to store N and Q for each node, where N is the total number of times that node was in the path of a simulated node and Q is the number of times it won, out of those simulations.

I can give you the example of the way I implemented that part, in a board game I saved in the state class a variable which said who the next player was ( 1 or 2). So I would know that in that state (each node is a state) the player who made the last move was the other one (by the formula: player = 3 - next_player), and I would add 1 to the Qi of those nodes.


Owner Search for given Server SNO I am newbie to NLP.
I have a excel sheet with following columns:
    Server_SNo,
    Owner,
    Hosting Dept,
    Bus owner,
    Applications hosted,
    Functionality,
    comments

a. Except the Server_SNo, other columns may or may not have data.
b. For some records there is no data except Server_SNo which is the first column.
c. One business owner can own more than 1 Server.

So, out of 4000 records, about 50% of data contain direct mapping for a server with owner.  Remaining 50% of data have combination of other columns (Owner, Hosting Dept, Bus owner, Applications hosted, Functionality and comments)

Here is my problem, I need to find the owner for the given Server_Sno for 50% of data which have combination of other columns (Owner, Hosting Dept, Bus owner, Applications hosted, Functionality and comments).

I have just started to build the code using Python and NLTK.

Is this an NLP problem? Am I going in right direction using Python and NLTK for NLP?

Any insights is appreciated.

-Mani

I don't think this classify as an NLP problem, there is almost no semantic analysis needed, it is more like a classification problem using categorical features.

NLTK is surely valuable if you want to perform some text 'cleaning' or preprocessing before encoding the variables. The only NLP application that I think you could apply here is some sentiment analysis on the comments to extract extra features (like a number expressing the negativeness or positiveness of each comment). Nevertheless you might want to do that using some pre-trained models cause your dataset is pretty small.


Tuning of PPO metaparameters: a high level overview of what each parameter does I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results.

However, I notice that there are many metaparameters available to give to the PPO algorithm:

 # the tensorforce agent configuration ------------------------------------------
    network_spec = [
        dict(type='dense', size=256),
        dict(type='dense', size=256),
    ]

    agent = PPOAgent(
        states=environment.states,
        actions=environment.actions,
        network=network_spec,
        # Agent
        states_preprocessing=None,
        actions_exploration=None,
        reward_preprocessing=None,
        # MemoryModel
        update_mode=dict(
            unit='episodes',
            # 10 episodes per update
            batch_size=10,
            # Every 10 episodes
            frequency=10
        ),
        memory=dict(
            type='latest',
            include_next_states=False,
            capacity=200000
        ),
        # DistributionModel
        distributions=None,
        entropy_regularization=0.01,
        # PGModel
        baseline_mode='states',
        baseline=dict(
            type='mlp',
            sizes=[32, 32]
        ),
        baseline_optimizer=dict(
            type='multi_step',
            optimizer=dict(
                type='adam',
                learning_rate=1e-3
            ),
            num_steps=5
        ),
        gae_lambda=0.97,
        # PGLRModel
        likelihood_ratio_clipping=0.2,
        # PPOAgent
        step_optimizer=dict(
            type='adam',
            learning_rate=1e-3
        ),
        subsampling_fraction=0.2,
        optimization_steps=25,
        execution=dict(
            type='single',
            session_config=None,
            distributed_spec=None
        )
    )

So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?

So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?

effect of network_spec: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.
effect of the parameters in update_mode: this is how often the network updates are performed.

batch_size is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?
frequency is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?
unit: no idea what this does

memory: this is the replay memory buffer.

type: not sure what this does or how it works.
include_next_states: not sure what this does or how it works
capacity: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.

DistributionMode: guess this is the model for the distribution of the controls? No idea what the parameters there do.
PGModel: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.
PGLRModel: idem, no idea what all these parameters do / if they should be tweaked.
PPOAgend: idem, no idea what all these parameters do / if they should be tweaked.

Summary

So in summary, would be great to get some help about:

Which parameters should be tweaked
How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?

Some investigation about the memory dict: The current type is latest, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
Also, include_next_state means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to False, otherwise an error is raised.
Your interpretation of capacity looks OK.

About the update mode spec dict, your current settings mean that : every 10 (frequency) episodes (unit), you pull a batch of 10 (batch_size) episodes (unit) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the unit defines both the unit of the optimization frequency and the type of the object fetched from the memory.


How data augmentation like rotation affects the quality of detection? I'm using an object detection neural network and I employ data augmentation to increase a little my small dataset. More specifically I do rotation, translation, mirroring and rescaling.

I notice that rotating an image (and thus it's bounding box) changes its shape. This implies an erroneous box for elongated boxes, for instance on the augmented image (right image below) the box is not tightly packed around the left player as it was on the original image.

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

Thank you in advance!

(Obviously, it seems advisable to use small rotation angles)

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Yes, it is clear from your examples that the bounding boxes become wider. Generally, including large amounts of data like this in your training data will mean that your network will also have a tendency to learn slightly larger bounding boxes. Of course, if the majority of your training data still has tight boxes, it should stell tend towards learning those... but likely slightly wider ones than if the training data did not include these kinds of rotations.

  Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

  (Obviously, it seems advisable to use small rotation angles)

I do not personally work directly in the area of computer vision really, so I'm not sufficiently familiar with the literature to point you to any references on this particular issue. Based on my own intuition, I can recommend:

Using relatively small rotation angles, as you also already suggested yourself. The bounding boxes will become a little bit wider than in the original dataset, but not by too much.
Using rotation angles that are a multiple of $90^\circ$. Note that if you rotate a bounding box by a multiple of $90^\circ$, the rotated bounding boxes become axis-aligned and your problem disappears again, they'll become just as tight as the bounding boxes in the unrotated image. Of course, you can also combine this suggestion with the previous one, and use rotation angles in, for example, $[85^\circ, 95^\circ]$.
Apply larger rotations primarily in images that only have bounding boxes that are approximately "square". From looking at your image, I get the impression that the problem of bounding boxes becoming wider after rotations is much more severe when you have extremely wide or thin bounding boxes (with one dimension much greater than the other). When the original bounding box is square, there still will be some widening after rotation, but not nearly as much, so the problem may be more acceptable in such cases.


How do I select the relevant features of the data? Recently I was working on a problem to do some cost analysis of my expenditure for some particular resource.
I usually make some manual decisions from the analysis and plan accordingly.

I have a big data set in excel format and with hundreds of columns, defining the use of the resource in various time frames and types(other various detailed use).
I also have information about my previous 4 years of data and actual resource usage and cost incurred accordingly.

I was hoping to train a NN to predict my cost beforehand and plan even before I can manually do the cost analysis.

But the biggest problem I'm facing is the need to identify the features for such analysis. I was hoping there is some way to identify the features from the data set.

PS - I have idea about PCA and some other feature set reduction techniques, what I'm looking at is the way to identify them in the first place.

Since you have all your data in a table, a relatively simple thing to do is to consider each column independently, and then seeing if the output variable (cost incurred) has a correlation to that.

If the column has no (or very low correlation) with the output variable, then consider it to be not important.  The ones that make the cut are then considered further.

This is obviously not very different from how a decision tree algorithm would work (such as ID3).


How can we connect artificial intelligence with cognitive psychology? On page 62 of Artificial Intelligence and Cognitive Psychology Applications, Models (2010), Gabriella Daróczy writes

Psychology is one of the basic sciences of artificial intelligence (AI). The founder of the psychology is Wilhelm Wundt (1832-1920), who engaged in empirical methods, and was interested in the thinking processes during his scientific work.
Psychology had two main leading schools: behaviourism, and cognitivism.
Behaviourism: Refused the theory of the mental processes, and insisted to study the resulted action or the stimulus strictly objective. The representatives of this theory have been decreasing with time.
Cognitive psychology: The brain is an information processing device. Artificial Intelligence supports also this idea, and the first picture illustrates the information processing model of the brain

How can we connect artificial intelligence with cognitive psychology? Can we integrate psychology into AI?
For example, in the context of human-computer interaction, could we have an AI software/application that makes use of phycology-based ideas/theories?

AI is already connected with cognitive psychology - there are dozens of AIs right this minute attempting to predict things like which Facebook posts you will like, and which ads you are most likely to click on. In other words, they are trying to predict how you think.

For more detailed info on this AI/cognitive science connection, there is some suggested reading on AITopics.org, such as Paul Thagard's summary of cognitive science.


What are the jobs that artificial intelligence will create? Artificial intelligence is changing everything. AI is going to replace our jobs. So, what are the jobs that AI is going to create or replace?

A report from PwC states that - AI will displace around 7 million and will be creating around 7.2 million new jobs.

According to MIT researchers –  AI will replace tasks, not jobs.

In my opinion AI revolution is mainly going to effect non-technological jobs. So, the next generation—of employees should focus on creating career in new roles like AI researchers, Data scientist, Robotic Engineer.

(Image Source : theedgesearch)

These are the new jobs that AI is going to create:

AI engineer
Big Data scientist
Real-Time Data Engineer
Machine Linguist
Robot counselor
Telepresence Specialists
Vertical Farming Specialist
Limb Engineer
Sensor Scientist
Virtual Experience Specialist
Robotic Engineer
Virtual Reality Engineer
Genome Specialist
Gaming engineer
IOT Engineer


What effect does a negative output of a neuron have on neighbouring neurons? Artificial neural networks are composed of multiple neurons that are connected to each other. When the output of an artificial neuron is zero, it does not have any effect on neighboring neurons. When the output is positive, the neuron has an effect on neighboring neurons.

What does it mean when the output of a neuron is negative (which can e.g. occur when the activation function of a neuron is the hyperbolic tangent)? What effect would this output have on neighboring neurons?

Do biological neural networks also have this property?

In the case of artificial neural networks, your question can be (partially) answered by looking at the definition of the operation that an artificial neuron performs. An artificial neuron is usually defined as a linear combination of its inputs, followed by the application of a non-linear activation function (e.g. the hyperbolic tangent or ReLU). More formally, a neuron $i$ in layer $l$ performs the following operation

\begin{align}
o_i^l = \sigma \left(\sum_{j=1}^N w_j o_j^{l-1} \right) \tag{1}\label{1},
\end{align}

where $o_j^{l-1}$ is the output from neuron $j$ in layer $l-1$ (the previous layer), $w_j$ the corresponding weight, $\sigma$ an activation function and $N$ the number of neurons from layer $l-1$ connected to neuron $i$ in layer $l$.

Let's assume that $\sigma$ is the ReLU, which is defined as follows

$$
\sigma(x)=\max(0, x)
$$

which means that all negative numbers become $0$ and all non-negative numbers become themselves.

In equation \ref{1}, if $w_j$ and $o_j^{l-1}$ have the same sign, then the product $w_j o_j^{l-1}$ is non-negative (positive or zero), else it is negative (or zero). Therefore, the sign of the output of neuron $j$ in layer $l-1$ alone does not fully determine the effect on $o_i^l$, but the sign of the $w_j$ is also required.

Let's suppose that the product $w_j o_j^{l-1}$ is negative, then, of course, this will negatively contribute to the sum in equation \ref{1}. In any case, even if the sum $\sum_{j=1}^N w_j o_j^{l-1}$ is negative, if $\sigma$ is the ReLU, no matter the magnitude of the negative number, $o_i^l$ will always be zero. However, if the activation function is hyperbolic tangent, the magnitude of a negative $\sum_{j=1}^N w_j o_j^{l-1}$ affects the magnitude of $o_i^l$. More precisely, the more negative the sum is, the closest $o_i^l$ is to $-1$.

To conclude, in general, the effect of the sign of an output of an artificial neuron on neighboring neurons depends on the activation function and the learned weights, which depend on the error the neural network is making (assuming the neural network is trained with gradient descent combined with back-propagation), which in turn depends on the training dataset, the loss function, the architecture of the neural network, etc.

Biological neurons and synapses are more complex than artificial ones. Nevertheless, biological synapses are usually classified as either excitatory or inhibitory, so they can have an excitatory or inhibitory effect on connected neurons.


Would a general-purpose AI need to collaborate? Human beings are more productive in groups than individually, possibly due to the fact that there is a limit to how much one human brain can improve itself in terms of speed of computation and areas of expertise.

By contrast, if a machine with general-purpose artificial intelligence is created and then assigned a task, would it be possible that the machine will be able to better accomplish its task by continuously improving its own computational power and mastery of various skills, as opposed to collaborating with other agents (whether copies of itself, other AI's, or even humans)?

In other words, would an AGI ever need to collaborate, or would it always be able to achieve its goals alone?

In my answer, I have often switched between AGI and ASI for reference. This is fine as an AGI will reach ASI as it is optimizing itself and learning.
I think it is not only important by necessary that AGI and ASI are of collaborative nature. Nick Bostrom, in his book Superintelligence: Paths, Dangers, Strategies in Chapter 10 described three ways in which an ASI might function:

  As an oracle, which answers nearly any question posed to it with accuracy, including complex questions that humans cannot easily answer—i.e. How can I manufacture a more efficient car engine? Google is a primitive type of oracle.
      As a genie, which executes any high-level command it’s given—Use a molecular assembler to build a new and more efficient kind of car engine—and then awaits its next command.
      As a sovereign, which is assigned a broad and open-ended pursuit and allowed to operate in the world freely, making its own decisions about how best to proceed—Invent a faster, cheaper, and safer way than cars for humans to privately transport themselves.

This is when and if AGI and ASI are of controlled manner and their output are as expected. By controlled I mean they don't start seeing human as a threat and start eliminating the human race. More on this here. As you can see, all the above ASI are of collaborative nature. Either they are collaborating with humans or they need to collaborate with other systems.

Now to your answer:
First, collaborative nature is of great use in terms of efficiency and performance. This is the reason why Distributed Systems are being made. We even have distributed OS now. Also, Modular approach in coding/ developments and huge success of Object Oriented Model are proofs of advantages of using collaboration among different entities.
If you think about it even the AGI is collaborating and using resources from other places in some way. As your AGI is learning, it is gaining information from the internet. It reads the information and tries to structure is accordingly (This will depend on its neural schema) and create knowledge (or something valuable) for itself. It is collaborating using network protocols with the outer world (other systems). If it doesn't collaborate then the firewall of the system might not allow it to use the service. Different services on the internet require a different set of protocols to be followed. So, if AGI wants to communicate for information it needs to follow those protocols. This way the AGI will learn to collaborate with different entities even at an early stage.
From AGI point of view communicating a web server and communicating another AGI machine is very similar. As we, humans, don't store all the information in our brain; similarly AGI won't find it efficient to store all the information within. Not all information is needed all the time. The memory hierarchy is proof of that. Even if AGI is made to store all the information within, with time it will figure out how inefficient it is and will re-program itself to only hold very vital information and use the internet for less frequently used information.
I will like to add one more thing to this. Let us begin with the human analogy first. We, as humans, collaborate with other humans. But we also collaborate with our different body parts. Like, what about the collaboration within us. Collaboration (or coordination) between our legs, hands, our body, and mind, etc. This leads us to the question of who as a person we are, our whole body, or just our brain. If I remove my hand and replace it with an organic implant, would I be still me?
Similarly, what is AGI? Is it the whole structure, or just the code. Does the RAM, ROM, hard disks also a part of AGI? If you think the hardware is not the part of AGI then your AGI is co-ordinating with these devices too, using certain protocols. To some level, this is collaboration too.


Is the traditional meaning of "strong AI" outmoded? Traditionally, "strong AI" refers to Artificial General Intelligence, the human mind understood as an algorithm (Searle, Chinese Room) and Artificial Consciousness.

But recent advances in Artificial Intelligence, most notably breakthroughs involving Machine Learning and Neural Networks, have led to strong AI being used in the context of an algorithmic intelligence that can outperform the top humans in specific tasks.  (This is sometimes qualified as "strong narrow AI", but the qualifier seems less used time goes on.)

Because the traditional usage of strong AI are based on theoretical concepts, where the current usage is based on practical, real-world results, does the new usage of "strong AI" supercede the traditional usage?

If this is the case, at what point does the Wiki need to be updated?  Is a major paper focused on re-coining the term necessary to formally reference the new usage of the term, or are numerous, informal uses sufficient?

Related: "Strong" has a precise, analytic, formal usage in the context of solved games.  This game theoretic usage is similar in spirit to new usage of strong AI in that it connotes results that can be validated.

The meaning of strong AI has changed as you correctly indicated, and the term "strong narrow AI" is more appropriate as now people are shifting towards the practical uses.
This also bring to light the number of breakthroughs that AI is having and literature and other various related resources will often(unlike in other fields) need to be changed constantly.


How can one find / collect data for, and come up with ideas for, using Deep Learning / AI to improve one's everyday life? I can see a lot of tutorials and examples about using TensorFlow and other free, open-source AI/ML/DL frameworks on enterprise level where enough data was collected for such AI solutions.

How can one can collect enough data in normal everyday life to practically and effectively make use of such freely available AI/ML/DL technologies to improve one's life and security?

Speaking to the "collecting data" part of the question, I'll say this:

Keep in mind that not everything requires massive amounts of data. Consider also that large amounts of data about all sorts of things are available in the OpenData / LinkedData realms. Governments in particular are sources of massive amounts of data.  See, for example, data.gov or Google around and see if your (state|city|county|whatever) has an "open data" portal. You might be surprised.

Finally consider that inexpensive sensor systems and distributed sensor networks can be constructed fairly easily / inexpensively using things like Arduino / Raspberry Pi and communications protocols and software often associated with the term Internet of Things.  You may find that you can actually collect your own data, especially if you crowd-source the effort and get a bunch of other people involved.

See also:

http://opendata.reddit.com

http://datasets.reddit.com

http://www.wikidata.org

http://www.dbpedia.org

etc.


What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions? I find the terms cost, loss, error, fitness, utility, objective, criterion functions to be interchangeable, but any kind of minor difference explained is appreciated.

What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?

We can first consider what kind of elements are in this list.  They are all quantifications that guide a process of adaptation, learning, or innovation — controls a process that optimizes or attempts to optimize the development of another process.  Quantification allows for comparison and the determination of first and second-degree derivatives to use in approximating steps in improvement.
On the basis of this generalization, the list may be expanded to include other potential quantifications that can control.

Pain
Pleasure
Disparity
Value
Advantage
Expense
Profit
Gain
Return
Distance
Burden
Optimality

In English, there is quite a bit of nuance in terms of normal contextual use.  In academic papers on machine learning topics there are some conventions forming there, producing jargon, meanings beyond the normal English use.  We can first divide them symmetrically around the most basic qualifier.  Is the increase in the number what is intended or something to be avoided?
Decreases are Desirable

Cost
Loss
Error
Distance
Pain
Disparity
Expense
Burden

Increases are Desirable

Fitness
Utility
Objective
Criterion
Pleasure
Value
Advantage
Profit
Gain
Optimality
Complete

Other Features
Some of these are relative, such as Advantage being a relative measure between two Values associated with two Action options.  Some are direct inversions, as in that the quotient of Gain over Loss equals -1 in the usual contexts of those words.  Some appear to be opposite but in fact are not, as in the case of pain and pleasure occurring concurrently and via different neural mechanisms in biological neural systems.
Some are discrete and have only two possible values, such as Criterion and Complete.  Those can be well represented by Boolean values rather than integers, fixed-point, or floating-point numbers.
Some are conditionally synonymous, as in the case of Distance and Error if the output dimensionality is one or the output is multidimensional but the error aggregation method used is the square root of the sum of squares.
Some seem better in financial contexts ... others in robotic contexts.
Words like Disparity and Optimality are generalizations.
One, in particular, is a quotients based on investment &mdash Return — which could be applied to energy return on energy invested (as Hall suggests is the fitness function for genetic evolution).
What is missing from this list is learning based on multiple degrees of freedom, not used in mainstream ML yet, and probably will emerge in papers and then libraries and then VLSI designs in time.  This likely direction would introduce the plurals of many of these quantifications because it will be vectors of corrective signaling back-propagating to perform curved modifications on network parameters.  It is likely that brain signaling (pulses, energy balances, neurochemical secretions and uptake, and genetic expression) will not be well simulated until this happens.
In all cases, it is the definition of these quantifications that drive adaptation, learning, and innovation in biological and artificial systems.  It is thus likely that the application of ethics and risk analysis might best be applied to AI in the formulation of the calculation of these quantities in adaptive systems.


What type of network for a repeated experiment good day

I have a problem where i have 9 data points that are collected every minute for 40 minutes, and by the 40th minute the solution would be either end up being black or white.

I would like to set up a neural network; which would take the live input of every minute; and I was hoping within the 25-30 minute mark to predict the outcome of what the results would be at 40 minute; which is a classification.

I have over 3000 historical runs of this experiment; each containing 40 rows of 9 columns data per experiment.

What network would I need to set up; so that it can learn from each run at every minute mark per experiment with the results; and then set it up for live input, when the experiment is running again.

i feel like I might need more than one system to accomplish this; any help in pointing me towards the right path would be greatly appreciated

I am using python (keras) to try to solve this problem.

So it seems any time series modeling would do the trick. If you new to neural nets and seems you want to play with keras, maybe start by throwing it into a simple LSTM.


How to detect patterns in a data set of given IP addresses using a neural network? How to detect patterns in a data set of given IP addresses using a neural network?

The data set is actually a list of all the vulnerable devices on a network. I want to use a neural network that detctes any patters in the occurrences of these vulnerabilities with reference to their IPs and ports.

It seems that you want to detect ranges of IP addresses that are vulnerable/dangerous/etc, right? Such ranges are essentially numeric intervals, and so my suggestion is to look at decision tree learning instead of neural networks, because you are essentially doing a classification task where you want to test both categorical data and splits over numerical attributes.

The result will be a tree-like function (nested conditionals) of the form

IF ...> address > ...
   THEN [vulnerable]
   ELSE IF port=...
              THEN [not vulnerable]
              ELSE [vulnerable]

where a huge benefit is that it is also more human-readable than a neural net.

The most prominent algorithms for decision trees are ID 3 and its successor C4.5.


What is the relation between Q-learning and policy gradients methods? As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain state, policy gradients directly predict the action itself.

However, both approaches appear identical to me, i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Is the difference in the way the loss is back-propagated?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).

Both methods are theoretically driven by the Markov Decision Process construct, and as a result use similar notation and concepts. In addition, in simple solvable environments you should expect both methods to result in the same - or at least equivalent - optimal policies.

However, they are actually different internally. The most fundamental differences between the approaches is in how they approach action selection, both whilst learning, and as the output (the learned policy). In Q-learning, the goal is to learn a single deterministic action from a discrete set of actions by finding the maximum value. With policy gradients, and other direct policy searches, the goal is to learn a map from state to action, which can be stochastic, and works in continuous action spaces.

As a result, policy gradient methods can solve problems that value-based methods cannot:

Large and continuous action space. However, with value-based methods, this can still be approximated with discretisation - and this is not a bad choice, since the mapping function in policy gradient has to be some kind of approximator in practice.
Stochastic policies. A value-based method cannot solve an environment where the optimal policy is stochastic requiring specific probabilities, such as Scissor/Paper/Stone. That is because there are no trainable parameters in Q-learning that control probabilities of action, the problem formulation in TD learning assumes that a deterministic agent can be optimal.

However, value-based methods like Q-learning have some advantages too:

Simplicity. You can implement Q functions as simple discrete tables, and this gives some guarantees of convergence. There are no tabular versions of policy gradient, because you need a mapping function $p(a \mid s, \theta)$ which also must have a smooth gradient with respect to $\theta$.
Speed. TD learning methods that bootstrap are often much faster to learn a policy than methods which must purely sample from the environment in order to evaluate progress.

There are other reasons why you might care to use one or other approach:

You may want to know the predicted return whilst the process is running, to help other planning processes associated with the agent.
The state representation of the problem lends itself more easily to either a value function or a policy function. A value function may turn out to have very simple relationship to the state and the policy function very complex and hard to learn, or vice-versa.

Some state-of-the-art RL solvers actually use both approaches together, such as Actor-Critic. This combines strengths of value and policy gradient methods.


Is it better to adjust the natural lighting (while recording the video) or to subsequently apply filters on the original video? For the purpose of object detection, is it better to adjust the natural lighting (while recording the video) or to apply filters (e.g. brightness filters, etc.) on the original video to make it brighter?

My intuition is that it shouldn't matter when you adjust the natural lighting or do it after with video filters.

Personally, I'd say as long as the object is visible don't do either. If the model has been well built and if lighting changes would help, the convolution operation weights would learn an operation similar to contrast or brightness changes.

On the other hand if the object visibility is an issue, then natural lighting changes would be better, due to the lack of potential artefacts a filter would create.

So overall, I'd say natural lighting changes should be more helpful (Assuming model is built well) and brightness filters would not be very helpful as the convolution operations would learn them if they were useful, also there would be  artefacts in the input which can lead to the model learning irrelevant details.

Hope this helped!


Using NVIDIA VOLTA V with Mac or Windows externally I have a Macbook Air and Asus ROG 702 with a built in GeForce 1070 GPU. I'm considering getting an NVIDIA Volta for AI applications (tensorflow / pytorch) to run on either the mac or windows. Would they even be compatible? And if so, what box would I need to house the GPU? Would it be system agnostic? (I was thinking Aikito Node).

Thank you! All info is appreciated by this novice :D

The NVidia Volta architecture is capable of delivering an astounding 110 teraflops of computing power. This coupled with the NVidia CUDA Neural Network library (cuDNN). Which can be used to speed up DNN's on learning frameworks such as  Caffe 2, Microsoft Cognitive Toolkit, TensorFlow, Theano, and PyTorch. Makes GPU's based on the Volta architecture such as the Titan V specifically suited to accelerate AI workloads.

Regarding your MacBook Air. Unfortunately, it comes with an integrated GPU that can't be replaced because it is soldered to the logic board. So, unfortunately, we have to rule out upgrading it because there isn't any other port to connect a GPU. You can read more about this on Apple's official discussion forum https://discussions.apple.com/thread/5206086 .

Having said that, you possibly can upgrade your GPU within the same series which is the 10 series. Other powerful GPU's within the 10 series include The Titan X, Titan Xp, GeForce GTX 1070 Ti and the GeForce GTX 1080 Ti. Unfortunately, the Geforce 10 series is based on the Pascal architecture meaning your laptop might not by default accept a Volta based GPU such as the Tesla V100 or the Tesla Titan V. Consequently it could involve BIOS modification, chassis modification or heatsink assembly modification to make it work. This kind of upgrade will be diving into uncharted territory since it will void your warranty.

My recommendation is that a Volta based GPU might be unsuitable for your Asus laptop. I'd recommend a GPU that uses an architecture that is similar to your current GPU. Meaning a Pascal-based GPU such as the Titan X or GeForce GTX 1080 would be better suited to satisfy your demands.


How to understand the average l2 loss? In the snippet below, the highlighted part is the average norm, but since $1/|p_i|$ is outside the summation, it is very confusing to understand.

is $|p_i|$ l2-norm(as per wolfram) or l1-norm or absolute value as per wiki.
Should the $i$ inside summation be considered for $1/|p_i|$, which is outside the summation?

I agree that this notation is unclear. I would interpret it as follows:

Given that the expression is supposed to denote the average norm $|p_i|$ is likely the cardinality of the set $\{p_i\}$.

In that case the expression would just be the sum over all norms divided by the number of norms, resulting in the average norm. The authors likely use this notation because they didn't want to introduce the number $n$ of $p_{i < n}$. $|\{p_i\}|$ would be clearer, but maybe uglier to typeset.


What is the purpose of the actor in actor-critic algorithms? For discrete action spaces, what is the purpose of the actor in actor-critic algorithms?

My current understanding is that the critic estimates the future reward given an action, so why not just take the action that maximizes the estimated return?

My initial guess at the answer is the exploration-exploitation problem, but are there other, more important/deeper reasons? Or am I underestimating the importance of exploration vs. exploitation as an issue?

It just seems to me that if you can accurately estimate the value function, then you have solved the RL challenge.

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms?

In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used to derive a policy, and in "vanilla" Actor-Critic cannot be used in this way at all unless you have the full distribution model of the MDP.

  It just seems to me that if you can accurately estimate the value function, then you have solved the RL challenge.

Often this can be the case, and this is how e.g. Q-learning works, where the value function is more precisely the action value function, $q(s,a)$.

Continuous or very large action spaces can cause a problem here, in that maximising over them is impractical. If you have a problem like that to solve, it is often an indicator that you should use a policy gradient method such as Actor-Critic. However, you have explicitly asked about "discrete action spaces" here.

The main issue with your statement "if you can accurately estimate the value function" is that you have implicitly assumed the the learning is complete. You are looking at the final output of the algorithm, after it has converged, and asking why not use just half of its output. Whilst choosing between RL algorithms is more often related to how they learn, how efficiently they learn, and how they behave during the learning process.

  My current understanding is that the critic estimates the future reward given an action, so why not just take the action that maximizes the estimated return?

The critic is $v_{\pi}(s)$ so does not take actions into account. You can of course learn $q_{\pi}(s,a)$ instead or as well - and some policy gradient methods also estimate the action value, such as Advantage Actor-Critic.

The main difference is in how data gathered from experience is applied to changing the policy. Recall at any step during reinforcement learning, that all estimates are based around a best guess at values of a current target policy. Policy gradient methods directly improve the policy, whilst value-based methods have an implied policy based on the same learned value functions. The $\text{argmax}_a q(s,a)$ approach used in Q-learning or SARSA gives a simple but crude mapping from values to policy, and means that the trajectory through policy space taken whilst learning is different when comparing policy gradient methods and value-based methods.

The following are notable differences affecting performance between policy-based methods and value-based methods:

Policy-gradient methods tend to make small changes to policy on each step, the policy changes smoothly, adjusting probabilities between choice of different actions by small amounts. This in turn means that the targets for estimated value functions change slowly. In comparison, when a value based method finds a new maximising action, it can make a big change to the policy, making the updates required less smooth.
Policy-gradient methods can learn to balance stochastic policies, whilst value-based methods assume deterministic policies. So if you have a situation like paper, scissor stone game where the ideal policy is random choice between actions, you should use policy gradients. This is a bit niche, but is worth noting.
Function approximators (typically neural networks) are easier to train on simpler target functions. The policy function $\pi(a|s)$ and action value function $q(s,a)$ are quite different views of the same problem, and it can be the case that one has a simple form whilst the other is more complex. This can make a big practical difference on which algorithms learn fastest.

For smaller, discrete action spaces, there is not always a clear choice between policy gradient and value-based methods. You could use either and it may be worth doing an experiment to find the most efficient algorithm for certain classes of problem.


How is the expected value in the loss function of DQN approximated? In Deep Q Learning the parametrized Q-functions $Q_i$ are optimised by performing gradient descent on the series of loss functions

$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$ , where

$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$.

In the actual algorithm, however, the expected value is never computed. Also, I think it cannot be computed since the transition probabilities of the underlying MDP remain hidden from the agent. Instead of the expected value, we compute $y_i = r_i + \gamma \max_a Q(\phi_{i+1},a;\theta)$. I assume some sort of stochastic approximation is taking place here. Can someone explain the details?

Just as the paper says
$$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$$
where
$$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$$
Then in the Background section of the paper, it says

Differentiating the loss function with respect to the weights we arrive at the following gradient:

$$\nabla_{\theta_i} L_i(\theta_i)\\= E_{(s,a)\sim p,s'\sim\mathcal{E}}\left[\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)\right]\tag{1}$$

Rather than computing the full expectations in the above gradient, it is often computationally expedient to optimize the loss function by stochastic gradient descent.
...
and the expectations are replaced by single samples from the behavior distribution $ρ$ and the emulator $\mathcal{E}$ respectively.

If you're familiar with SGD and Stochastic Optimization then you know what happens here:
The expression inside the expectation of (1), i.e. $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ , is an unbiased estimation of the real gradient $\nabla_{\theta_i}L_i(\theta_i)$ - its expectation is the real gradient. In other words,
$$\widehat{\nabla L}=\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i).$$
Then by theory of Stochastic Optimization we can optimize $L$ by $\theta\leftarrow \theta - \alpha\widehat{\nabla L}$ , which is how SGD works.
The unbiased estimation $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ can be sampled and calculated directly - you can run the emulator $\mathcal{E}$ and current behavior policy to collect $r, s', a'$ and calculate the gradient of your Q network $\nabla Q$ using TensorFlow. So the gradient to $L$ can be approximated.
The $y_j$ in the pseudocode  of the paper is also an estimation. It'll be more proper to denote it as $\hat{y_j}$.
(I'm not an native English speaker so forgive my poor expression.)


Solution to classify product names I have a bunch of training data for classifying product names, around 30,000 samples. The task is to classify these product names into types of product, around 100 classes (single words).

For example:

dutch lady sweetened uht milk => milk
samsung galaxy note 10        => electronics
cocacola zero                 => softdrink
...

All words in inputs are indexed to numbers, and so classes. I've tried to use tf.estimator.DNNClassifier to classify them but no good results. The outcome is just an accuracy of 4% which is no meaning.

Should it be I'm in the case that classes (Y values) are distributed kinda randomly and too hard to do multi-time linear separation?

Are there any existing solutions to classify a list of names? like my product names?

If you're looking for an existing solution, the best approach I found was using a TF-IDF model, check out the links below which have similar examples which should be easily adapted for your dataset.

https://www.kaggle.com/selener/multi-class-text-classification-tfidf#targetText=Text%20classification%20(multiclass)&targetText=With%20the%20aim%20to%20classify,one%20of%20the%20product%20categories).

https://github.com/susanli2016/Machine-Learning-with-Python/blob/master/Consumer_complaints.ipynb

However if you specifically want to go for a DNN approach, there are a few options you can take for a multi-class text classification. Try looking into a simple CNN classifier, which is a relatively lightweight approach, computationally speaking, yet showing pretty good results:

https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f

Alternatively, you can use a word2vec or a doc2vec model to map your sentences to unique vectors, and then put them through a regression algorithm.

https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568


How much knowledge of the world is learnt through words? We know a lot of common sense about the world. Things like "to buy something you need money".

I wonder how much of this common sense comes about through actual someone explicitly telling you the instructions "You need money to buy things". Which we store in our brains as a sort of rule. As opposed to just intutively understanding things and picking it up.

I am imagining children playing at shop-keeping and saying things like "I give you this and you give me that". And other children not quite understanding the concept of buying things until being told by a teacher.

If so, giving a computer a list of common sense rules likes these is no different to teaching a child. So I am wondering why this area of AI research (semantic webs etc.) has been frowned upon in the last decade in favour of trying to learn everything through experience like deep neural networks?

If I understand correctly, what you are looking for is called "common sense reasoning" in NLP research.

Research in this field revolves around benchmark data sets, where good performance indicates some ability to do common sense reasoning. Here is a nice collection of data sets and research by Sebastian Ruder:

http://nlpprogress.com/english/common_sense.html

In the end, the main question is not

  How much knowledge of the world is learnt through words?

since it is virtually unanswerable if asked in this form. A question that is answered in NLP common sense reasoning research is

  Out of 100 specific decisions that my model needs to take, for how many does the model show the ability to reason correctly?


How can I use autoencoders to analyze patterns and classify them? I generated a bunch of simulation data from a complex physical simulation that spits out patterns. I am trying to apply unsupervised learning to analyze the patterns and ideally classify them into whatever categories the learning technique identifies. Using PCA or manifold techniques such as t-SNE for this problem is rather straightforward, but applying neural networks (autoencoders, specifically) becomes non-trivial, as I am not sure splitting my dataset into test and training data is the right way.

Naively, I was thinking of the following approaches:

Train an autoencoder with all the data as training data and train it for a large number of epochs (overfitting is not a problem in this case perse I would think)
Keras offers a model.predict option which enables me to just construct the encoder section of the autoencoder and obtain the bottleneck values
Carry out some data augmentation and split the data as one might into training and test data and carry out the workflow as normal (This approach makes me a little uncomfortable as I am not attempting to generalize a neural network or should I be?)

I would appreciate any guidance on how to proceed or if my understanding of the application of autoencoders is flawed in this context.

When using an autoencoder, I believe the data u feed in has to be correlated in one way or another. For example, If i want to learn a latent representation of an image of a cat, The training data that I feed into the autoencoder should constitute only cat images.

Similar to other neural networks, you feed the autoencoder with a set of training data and hope that the network learns a set of weights that is able to output from the latent representation the exact image. To see whether the weights learnt by the autoencoder is able to generalise to other unseen cat images, you would have to use a test set for this.

Here is a paper about autoencoding. https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf

I hope this helps you somewhat in deciding whether you should use an autoencoder.


Tips for keeping the distribution of weights normal I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.

Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor $(qf)$.
the final quantized weights will be integer$(value * qf)$.

If my weights are very sparse and have a very bad distribution, I lose more precision.

For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher.

Does anybody have any other suggestions? Has anyone tackled this problem before?

Quantizing of weights is used very often in implementing neural networks with the tensorflow library on mobile devices. The reason is, the such microcontrollers doesn't have a floating point unit and the developer is trying to increase the overall performance. Quantizing means, to map float values to integer values:

Integer Float
0      -5.0
128    0.0
256    5.0

The problem isn't trivial, because the float point numbers can have a certain minimum, maximum and precision. Maintaining the original distribution is also an issue.

To avoid such kind of problems, the best idea is to implement a floating point library from scratch. In the stackbased Forth language this can be realized in under 400 bytes. It was described in the paper “Willi Stricker: Forth Floating Point Word-Set without Floating Point Stack, 2012”


Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs? Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this with an example?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of the input sequence.
What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one $x$ we have a sequence of related, i.e. not i.i.d. data such as time series data, data $\{x_i\}_{i=1}^n$. Assuming we are given some initial $h_0$ (a hidden state) then an RNN will take as input $x_1$ and $h_0$ and output a prediction $y_1$ and a new hidden state $h_1$. In general an RNN will take as input $x_n$ and $h_{n-1}$ and output $y_n$ and $h_n$ where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the $x$'s and $h$'s will all be the same, i.e. $x_i \in \mathbb{R}^d$ and $h_i \in \mathbb{R}^c$ for all $i$, where $d$ does not necessarily have to be equal to $c$ (and in my experience rarely is).
Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder $\{e_i\}_{i=1}^n$. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder $e_n$ as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in this paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.
Now, with this data $\{x_i\}_{i=1}^n$ you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found here.


Are there any other machine learning models apart from Reinforcement Learning and Q Learning to play video games? OpenAI's Universe utilises RL algorithms and I have heard of some game-training projects using Q learning, but are there any others which are used to master/win games? Can genetic algorithms be used to win at a game?

As I see it, it all comes down to game theory, which can be said to form the foundation of successful decision making, and is particularly useful in a context, such as computing, where all parameters can be defined.  (Where it runs into trouble is with the aggregate complexity of the parameters per the combinatorial explosion, although Machine Learning has recently been validated as a method of managing intractability specifically in the context of games.)
You might want to check out Playing Games with Genetic Algorithms and Evolutionary game theory.


Do we need the transition probability function when calculating the importance sampling ratio? I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused.

There is one thing I don't particularly understand. Why do we need the state-transition probability function when calculating the importance sampling ratio for off-policy prediction?

I understood that one of the main benefits of MC over Dynamic Programming (DP) is that one does not need to have a model of the state-transition probability for a system. Or is this only the case for on-policy MC?

There is one thing I don't particularly understand. Why do we need the state-transition probability function when calculating the importance sampling ratio for off-policy prediction?

It is not needed for calculation. It must be included in the theory, to compare the correct probability of each trajectory (on-policy vs off-policy). However, the state transition probabilities are then also shown to cancel out exactly, so there is no requirement to know what the values are.

State transition probabilities are irrelevant to probability ratios between identical trajectories where the policy varies but the environment does not. Which is the case for off-policy learning.

This is all explained on the relevant pages of the book, but replicating here for completeness. If your trajectory is from steps $m$ to $n$ and called $\tau = (s_m, a_m, r_{m+1}, s_{m+1}, a_{m+1}, r_{m+2} . . . r_{n}, s_n)$, then the probability of seeing that trajectory under two different policies $\pi(a|s)$ and $b(a|s)$ is:

$$p(\tau|\pi) = (\prod_{i=m}^{n-1} \pi(a_i|s_i))(\prod_{j=m+1}^{n} p(s_j,
r_j|s_{j-1},a_{j-1}))$$

$$p(\tau|b) = (\prod_{i=m}^{n-1} b(a_i|s_i))(\prod_{j=m+1}^{n} p(s_j, r_j|s_{j-1},a_{j-1}))$$

You can clearly see that the second product in both cases is the same, and cancels out in the ratio:

$$\frac{p(\tau|\pi)}{p(\tau|b)} =\frac{\prod_{i=m}^{n-1} \pi(a_i|s_i)}{\prod_{i=m}^{n-1} b(a_i|s_i)}$$

So it doesn't matter what $p(s', r|s, a)$ actually is, just that it exists and is not allowed to change in theory between the on-policy and off-policy cases. Or in other words, that the environment is a consistent MDP.


Which library would you recommend to begin with deep learning? Which library (TensorFlow or Keras) would you recommend for a first approach to deep learning?

I'm a neuroscience student trying for the first time computational approaches, if that matters.

Keras is a simple and high-level neural networks library, written in Python, that works as a wrapper for Tensorflow and Theano. It's easy to learn and use. Using Keras is like working with Lego blocks. It was built so that people can do quick experiments and proofs-of-concept before launching into a full-scale build process.

With that in mind, it was made to be highly modular and extensible. Now, it can be used for a lot more than just experiments. It can help with RNN, CNN, and combinations of both.

If you want to begin and make a prototype ready solution, then I will recommend you start with Keras. To know the details under the hood, then learn TensorFlow. It has huge active community and also very good resources are available, for example, this Youtube series.

See also https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html.


Are there RL techniques to deal with incremental action spaces? Let's say we have a problem that can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), and we have already well trained an offline DQN model. Later, we have to add more actions for some reasons (and the number of action is now $n_2$, where $n_2 > n_1$).
Are there some solutions to update the value function or policy (or the neural network) with only minor changes?

Is there some solutions to update the model with only minor changes?

In general, assuming the new action choices are meaningful - in at least some states, the expected return from taking one of the new actions is higher than the current optimal policy using just the old action selection - then the answer here is "no".

At the very least you will need to re-train your agent such that it explores the new action choices, and learns the new value function and policy. You can of course start this re-training using data and internal representations learned from the earlier environment, and that may help if the new actions have not changed things too radically.

There are a couple of things that might help improve performance on this re-training:

If the actions are not entirely discrete, but have some features that could be generalised from, you could base your value function or policy function estimators on those features instead of discrete actions. So for example in DQN your input to the neural network would be concatenated state, action feature vectors, and output a single value. Then it may generalise to the new actions quickly, in some cases perhaps even getting close to the correct value estimates from the start.
If you were training using DynaQ+, this includes an exploration term (which is added to planning assessments of immediate reward) that will prioritise exploring new state/action pairs when they appear. Other planning algorithms may have similar adjustments, although I am not aware of specific ones that could be dropped straight into a DQN agent.

If you know in advance which states the new actions are likely to be most useful in, you may be able to insert that knowledge into some initial action selection or value estimate helpers to avoid the need to train from scratch.


Mnist CNN Architecture In this tutorial from Jeremy Howard: What is torch.nn really? he has an example towards the end where he creates a CNN for mnist. In nn.Conv2d he makes the inchannels and outchannels: (1,16), (16,16), (16,10). I get that the last one has to be 10 because there are 10 classes and we want 'probabilities' of each class. But why go up to 16 first? How do you choose this value? And why not just go from 1 to 10, 10 to 10, and 10 to 10? Does this have to do with the kernel_size and stride?

All of the images are 28x28 so I can't see any correlation between these values and 16 either.

class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))
```

I understand your question as: "How did the author select the number of neurons in their hidden layer?"

The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more intricate decision barriers. However, the more complex function is harder to optimize and will lead to lower performance scores. The goal here is to find the right tradeoff to maximize your performance. You can tune the number of hidden neurons as a hyper-parameter using cross-validation.

There isn't any formula to determine the number of neurons you will need, however you can get an intuition based on the number of inputs and outputs you will have. Generally, you want more hidden neurons than input and output neurons. Since most people are programmers who are writing neural networks, we are used to working with units in $2^n$. Thus, 16 is chosen over 10, and 32 would be chosen over 28.


Is color information only extracted in the first input layer of a convolutional neural network? In a convolutional neural network (CNN), since the RGB values get multiplied in the first convolutional layer, does this mean that color is essentially only extracted in the very first layer?

Snippets from Stanford CS231n Chapter on CNN:

  [...] One dangerous pitfall that can be easily noticed with this
  visualization is that some activation maps may be all zero for many
  different inputs, which can indicate dead filters, and can be a
  symptom of high learning rates [...] Typical-looking activations on
  the first CONV layer (left), and the 5th CONV layer (right) of a
  trained AlexNet looking at a picture of a cat. Every box shows an
  activation map corresponding to some filter. Notice that the
  activations are sparse (most values are zero, in this visualization
  shown in black) and mostly local.

Neural networks are all about taking raw input data (RGB values and pixel location) and learning useful features that are relevant to some downstream task. This process of aggregating raw inputs into higher-level features can start at the first layer past the inputs.

So yes, only the first layer of the network is using the actual raw color information from the image. Beyond that, the network has already started to put together nearby pixels and disparate color channels in order to find more complex patterns. Deeper layers in the neural network typically do further aggregation on features learned in earlier layers, rather than taking raw color information as input.


Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data? Consider linear regression. The mean squared error (MSE) is 120.5 for the training dataset. We've reached the minimum for the training data.
Is it possible that by applying Lasso (L1 regularization) we would get a lower MSE for the training data? Would it get lower for the test data? Would this also hold for ridge regression (L2 regularization)?

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation.
Mean square error for training data
Given some training data $\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the sum of squares, i.e. they are the minimisers given by
$$ \mathrm{arg\,min}_{a, b} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
This gives the same coefficients as minimising the mean square error
$$ \mathrm{MSE}\left((x_1, y_1), \dots, (x_n, y_n)\right) = \frac{1}{n} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
So, by definition, the coefficients $(a, b)$ minimise the MSE on the training data. Any regularisation will only increase the MSE on the training data.
Generalisation performance
The main point of regularisation is to prevent overfitting on the data and improve the generalisation performance (i.e. on the test set).
With an appropriate parameter for regularisation, you may obtain a smaller MSE on the test set. This depends on your dataset and the parameters you choose: strong regularisation may lead to underfitting, whereas weak regularisation might not make much difference to the coefficients that you fit.


Should binary feature be in one or two columns in deep neural networks? Let's assume I have a simple feedforward neural network whose input contains binary 0/1 features and output is also binary two classes.
Is it better, worse, or maybe totally indifferent, for every such binary feature to be in just one column or maybe it would be better to split one feature into two columns in a way that the second column will have the opposite value, like that:
feature_x (one column scenario)

[0]

[1]

[0]

feature_x (two columns scenario)

[0, 1]

[1, 0]

[0, 1]

I know this might seem a bit weird and probably it is not necessary, but I have a feeling like there might be a difference for a network especially for its inner workings and how neurons in the next layers see such data. Has anyone ever researched that aspect?

You're simply adding a redundant feature by having it as two: $X_2 = 1 - X_1$. It would be equally useful to duplicate the first column. At best this will not improve your model, at worst it will decrease accuracy.


What are some books or state of the art papers about the development of a strong-AI? I am looking for books or to state of the art papers about current the development trends for a strong-AI.

Please, do not include opinions about the books, just refer the book with a brief description. To emphasize, I am not looking for books on applied AI (e.g. neural networks or the book by Norvig). Furthermore, do not consider AGI proceedings, which contains papers that focus on very concrete aspects. The related Wikipedia describes some active investigation lines about AGI (cognitive, neuroscience, etc.) but can not be considered an educational/introductory resource. Finally, I am not interested in philosophical questions related to AI safety or risks or its morality if they are not related to its development. Development doesn't exclude mathematical foundation about it.

By example, if I look by example at this list "https://bigthink.com/mike-colagrossi/the-10-best-books-on-ai", the final candidates list became empty.

There is actually a book called Artificial General Intelligence by  Ben Goertzel and Cassio Pennachin. It's a bit out of date (from 2008), and published as a Springer-Verlag monograph (which tends to have fairly low editorial standards). This one is also an anthology, with each chapter written by a different author. It's probably not suitable as an undergraduate level book, but it does seem to contain something like the information that's wanted.


Can a crossover result in a node with no outgoing connections? I'm currently implementing the original NEAT algorithm in Swift.

Looking at figure 4 in Stanley's original paper, it seems to me there is a chance that node 5 will have no (enabled) outgoing connection if parent 1 is assumed the fittest parent and the connection is randomly picked from parent 2.

Is my understanding of the crossover function correct and can it indeed result in a node with no outgoing connections?

The disabled/enabled bit in a connection gene indicates whether or not it should be expressed in the calculation of the network.

Here's an example:

This is a neural network and its corresponding connection genes which represent the layout of the network among other things.
The top connection gene going from 1 -> 3 that has a weight of 0.1 is expressed in the calculation of the network.
The bottom connection gene going from 2 -> 3 that has a weight of 0.4 is not expressed in the calculation of the network.

Calculating the network:

  1633 * 0.1 = 163.3

  Given the weights in the example the output of this network is 163.3

Hypothetically, if both connection genes had their enabled bit set to true (which can happen in the future, to quote the paper: "The disabled genes may become enabled again in future generations") then the output of the network would be:

  (1633 * 0.1) + (10 * 0.4) = 167.3

To answer your question, the connection between two nodes still exists in the network regardless of whether the bit in the gene is enabled or disabled, but, if the bit in the gene is disabled it is not used when the output of the network is being calculated as I've shown above.


How can I constraint the actions with dependent coordinates? I am working on a customized RL environment where each action is represented as a tuple $a = (a_1,a_2,\cdots,a_n)$ such that certain condition must be satisfied for entries of $a$ (for instance, $a_1+a_2+\cdots+a_n \leq \text{constant}$).

I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions.  Is there any work done in this direction?

For the constraint $a_1+a_2+\cdots+a_n \leq \text{constant}$, I was thinking about generating $n+1$ uniform random variables $U_1,U_2,\cdots,U_n, U$, and set $a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.

At first glance, I thought this was similar to "continuous-discrete" action selection (https://arxiv.org/pdf/1810.06394.pdf). However, I think your problem is different.

I am assuming that each $a_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a_1,a_2,\dotso,a_n)$ and not an individual $a_i$. Then you could treat it like a hierarchical problem. If you want $a_1 + a_2 < 2$ for example, then you could sample $a_1 \sim U(0,2)$ and $a_2 | a_1 \sim U(0, 2-a_1)$ and have $p(a) = p(a_2 | a_1)p(a_1)$. The specifics of how you do this depends more finely on how your problem is set up.

Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.


What ensemble methods are used in the state-of-the-art models? What ensemble methods are used in the state-of-the-art models?
When I surveyed the state-of-the-art methods of classification and detection, e.g. ImageNet, COCO, etc., I noticed that are few or even no references to the use of ensemble methods like bagging or boosting.
Is it a bad idea to use them?
However, I observed that many use ensemble in Kaggle competitions.
What makes it so different between the two groups of researchers?

In my opinion, it is not because ensemble methods are not good, just the state-of-the-art and Kaggle competitions are two different fields.
Kaggle competitions can be understood as an industry project where the target (accuracy, distance value, etc) is the most important, and they can select some computationally expensive way such as ensemble methods to reach it.
The state-of-the-art models in other ways belong to the research area, where the most important is the contribution for science, you can not just combine a lot of models then call it is the research (and so unfair with some small researcher groups). If you want to contribute something depend on ensemble idea, it should be like this paper.