DeepBase: Deep Inspection of Neural Networks
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Sellam, Thibault
Lin, Kevin
Huang, Ian
Yang, Michelle
Vondrick, Carl
Wu, Eugene
Although deep learning models perform remarkably well across a range of tasks such as language translation and object recognition, it remains unclear what high-level logic, if any, they follow. Understanding this logic may lead to more transparency, better model design, and faster experimentation. Recent machine learning research has leveraged statistical methods to identify hidden units that behave (e.g., activate) similarly to human understandable logic, but those analyses require considerable manual effort. Our insight is that many of those studies follow a common analysis pattern, and therefore there is opportunity to provide a declarative abstraction to easily express, execute and optimize them. This paper describes DeepBase, a system to inspect neural network behaviors through a unified interface. We model logic with user-provided hypothesis functions that annotate the data with high-level labels (e.g., part-of-speech tags, image captions). DeepBase lets users quickly identify individual or groups of units that have strong statistical dependencies with desired hypotheses. We discuss how DeepBase can express existing analyses, propose a set of simple and effective optimizations to speed up a standard Python implementation by up to 72x, and reproduce recent studies from the NLP literature.
 My name is Thibault and this is DeepBase, Deep Inspection of Neural Networks. This is joint work with Kevin Lin, Ian Huong, Michel Young, Carl von Drick and Eugene Wu, and all the work was done at Columbia University. So it's pretty clear that nowadays neural nets are popular and people use them to solve all kinds of crazy new tasks, like playing competitive video games or like driving cars. But of course not everything is so easy with neural nets. Neural nets are notoriously hard to debug. And indeed, if you look inside a neural net, you will find this. The decision making process of the system, so the internal logic, is embedded into thousands, sometimes millions, sometimes hundreds of millions of weights, which are virtually impossible for a human to understand. This is why it became almost a cliché to compare neural nets to black boxes and black boxes are almost impossible to debug. So another problem that motivates this talk is that sometimes neural nets do the right thing, but for the wrong reason. So to explain this, let me show you an example. This here is a screenshot of VQA, visual question answering, and it's a system from George Atteck. So the idea is that you show a picture to a model, you ask a question, and the model will give you an answer. So in this case, I showed the picture of those two bears, asked what animal is that, and the system tells me that it's a bear, so the blue bar here tells me that I have 9% confidence. It's pretty cool. So to illustrate the point, let's poke the system a little bit. The questions that are interesting here are counting questions. How many? So we can ask how many bears are there in the picture. And then the model will reply, two. We see that we have a high confidence, 66.9%. I think it's pretty awesome that we went that far, and it's actually pretty spectacular. So you can say, okay, it seems that the model knows how to count. So let's continue and probe the model a little bit further. We can ask how many rocks are there in the picture. So this is a slightly harder question because there's those two rocks here in the back, and you can say that is it one big rock, is it two separate rocks? It's kind of hard to say. And indeed, the model replies, well, it's somewhere between three and four, perhaps two. So I don't think it's ridiculous, and it's pretty close to what the humans would actually answer. So now let's continue and push the model even further. How many Strubevapher's are there in the picture? So in case you wondered, a Strubevapher is a typical Dutch cookie. It's delicious. You probably never heard of it, but don't worry because neither did the model. And here the answer is two. So at first glance, this result seems absurd, but actually it makes perfect sense. It makes perfect sense because two is the majority class. It turns out that most questions that start with the keywords how many have the answer to. So the model was actually very smart. He did not get the question. He completely missed the point, but he thought, maybe I'm going to try my luck with the majority class. The problem here is that we lose confidence in the predictions of the model. If they pull this kind of trick, how can we trust it? Let's go back to the first question here for a second. How many bears are there in the picture? Do you think the model replied two because it counted two bears? Or did it reply two because it tried to trick us? And this is a real screenshot. So the problem that I would like to make here is that neural nets are really sneaky, and they will pick up whatever correlation they can to maximize their scores. Sometimes the correlation makes no sense at all. Sometimes the correlation should never be picked up. So because those problems are so important, there's been a huge push in the machine learning community to try to better understand what happens inside those models. So this is a very fast growing field of research, and many approaches are being developed concurrently. But we identified a recipe, kind of a very popular pattern, a very popular class of analysis. And in fact, it is so popular that we decided to give it a name. There is deep-dough inspection. So all of those papers here are deep-dough inspection papers. So the idea behind deep-dough inspection is to map a subset of the model. So for instance, a hidden unit or a whole layer, or a selection of hidden units to a function that we know and that we understand. We want to make statements like, oh, it seems that those two hidden units here are in charge of recognizing the body of the bear. And we're going to do this with a statistical method, as I'm going to share in a minute. So let me give you a few examples. The most popular example of deep-dough inspections is perhaps Andre Coppati's blog post on regular neural nets. So I think anybody who's working with a neural net has seen this before. So the idea is that Andre Coppati trained a model to predict the next, so to generate realistic C code. And what he realized is that there is one hidden unit that can keep track of all the open and closed parenthesis. It's the parenthesis neuron. So of course, this was a long time ago, 2015, so in machine learning speed, it's like, poof, it's really antiquated, and there's been many other things in between. So just two months ago, at ICLEI 2019 in New Orleans, so a team from MIT and QCRI studied neural machine translation models, and they identified some hidden units that are particularly sensitive to very specific class of grammatical structures, like verbs in English. It's a hidden unit 1902, or like articles in Germans. In the computer vision literature, same thing. So at the same conference, another team from MIT looked at GANs, so generative adversarial networks, so those are models that can produce images, and they identified hidden units that are in charge of specific objects. So here it's the tree and bushes hidden unit. If you ablate the hidden unit, all the trees and the bushes will disappear from the picture. If you force its output, you will see some little bush and some little forest popping up now and there. It's very cute. So to be clear, this is still an essential field of research, and there are many open questions. In fact, it's quite controversial. What does the discovery actually mean? How do they generalize? How can we use them? What happens if you change the model? And so on and so forth. Nevertheless, what is beyond doubt is that this kind of approach is very popular. In fact, I can cite at least 10 papers that were published by the time our paper got accepted at CIGMODE, I think eight or nine of them are actually here at the bottom of the slide. So we decided to go through the code base of those approaches, and we made three discoveries. So first discovery, so those approaches vary in complexity. It goes from a few hundred lines of code to thousands of lines of code, but very often they do the same thing over and over again. Second discovery is few of those code bases are optimized. Some are, and they run very fast, but very often they will take hours and hours to run because the first it isn't here is correctness. It's experimental code. It's not made to be used by production pipelines. Finally, third finding, all of those code bases are completely application dependent, so they will not generalize. Therefore, we thought that there would be space for a declarative system, and this is D-base. So D-base is our attempt to make deep non-inspection faster and easier. So before I describe the system, let me take a step back and tell you a few words about neural nets and neural nets interpretation. So let's start at the basics. Neural net 101, a neural net is basically a function that takes something as input and produces an output. So in this case, this is a character level language model, which takes a character of C code as input and brings the next character. So the little white bubbles here are the hidden units. You can think of them as subroutines, typically, so on the slide there's four, but in practice there's thousands of them. And our aim is to understand why they do. So the most simple way to do this is to run the model through a test set and keep track, so log, the output of those hidden units. We obtain a time series that looks like this one. So in this case, we can look at it and make guesses. It seems here that this hidden unit outputs low value for everything that's between parenthesis, high value of URL, so perhaps we found the parenthesis neuron. It's actually exactly what Carpathian did in his blog. Of course, things get more interesting when you look at net one, but several neurons at once. So in this case, it's a bit harder to do things by just looking at the time series, so we need something more systematic. So what is quite popular in the machine learning literature is to use correlations. You would generate an artificial time series that would give you one, every time you're between parenthesis and zero everywhere else. It's called one-hot encoding. It's a type of binary encoding. And then you look for correlations. So you can compute the correlation between this time series here and the output of all of the hidden units, and it will give you a kind of a score that will give you to what extent the hidden unit seems to replicate the logic of this parenthesis function. So to be clear, it's just a correlation. There is no causation, but it's a starting point for analysis. So another type of analysis that is popular in the literature is to look at groups of hidden units that work together to encode a signal, and the way we detect those is to use a classifier. So suppose I have those two time series here at the top, and I wonder if together they can encode the parenthesis signal. The way I can do this is to fit an SVM, or logistic regression classifier, and see to what extent I can predict the value of this parenthesis signal here. If they do, then it means that those two hidden units together encode this signal. So it turns out that everything I've shown you so far are instances of the same recipe, which is deep-no-inspection. So now let's look at the general form. It's basically based on three steps. First of all, you take a model, you take a dataset, you run the model on the dataset, and you log some behaviors. So in everything we've seen so far, we log the output of the hidden units, but you can look at other things, like the gradient or the activation of some gates if you like LSTMs. Second step, you formulate some hypothesis about what you think the model is doing, and you materialize the dataset with the time series here. Final step, use a scoring method, so most likely it's going to be some measure of statistical dependency, to create a mapping between the hidden units of the model at the bottom, here in gray, and the hypothesis here in red. So deep-base is a tool to make this process systematic and hopefully easier. So to see how it works, let's look at an example. So this is an ACL 2017 paper in which Bellingcoff and his team, so from Harvard and MIT, I think, asked themselves the following question. How much grammar does the translation model know? More exactly, they were interested in part of speech tags, and they wanted to know if the model was particularly sensitive to say nouns or adjectives or adverbs or verbs and so on and so forth. So they ran the following analysis. They took six models for six different pair of languages, they ran them on a corpus called the TED corpus that has 2.3 million words, and they recorded everything, is the time series which are here in the middle. Then they run a part of speech tagger on the data, and they checked what extent they could reconstitute the part of speech tags by looking at only the activation of the hidden units. So the part of speech tags are the red time series here. So the first one will activate, you have one for every verb and zero everywhere else. So they found that indeed it's possible to a large extent to predict the part of speech tags by looking at the activation of the hidden units. Nevertheless, this varies based on which language you use and which part of speech tags, which indicates that indeed the model does have some sensitivity to specific types of grammatical constructs. So suppose that I wanted to reproduce the analysis with a debate, how would I do this? So it's quite simple. First of all, we load a data set, then we load the models. So in this case, I load two models, English to French and English to Arabic. Then I load the part of speech taggers. So this is an array and every tag in N, that VB and tag each are Python functions. So in D-Base, we call those hypothesis functions because they encode the hypothesis that we want to verify about the model. They can be any Python function as long as the dimension of the output is compatible with the dimension of the layer that we inspect. So finally, I specified the score that I want. So I want this logistic regression score. So I call D-Base.inspect and the system is going to work a little bit and produce a table that will give me the different scores for different hypotheses and different models. Then of course, it's possible to export this score to our favorite plotting tool. So what's nice here is that the analysis runs in five lines of codes. Okay, so there are some helper functions. If I include the code there, there may be a few dozen lines of codes. So we are pretty far away from the thousand-ish line of code from the initial code base. So in everything we've seen so far, we just use text models. But of course, it's possible to do the same type analysis for images. So this is a paper from CVPR 2017 in which the authors asked the following question. How do object detection neural nets work? So they run the following experiments. They took 65 convolutional neural nets. They run them on a bank of 63,000 images called the Broden dataset, and they recorded everything. So now it's three-dimensional surfaces. It's not time-series anymore because we look at convolutional layers. So you can think of the X and the Y axis as pixels in the image, and the Z axis as a response of a convolutional filter. It's kind of like the same thing, but we add one dimension. It's not in time domain anymore. It's in space now. So the author asked crowd workers to annotate several objects and textures and materials, for instance, tables and chairs and wood and so on and so forth. And they checked to what extent, excuse me, and they checked to what extent, so the visual concepts, so the mask actually overlap with the activations. And they did this with something called the intersection of a union, which is kind of like the jacquard distance. They found that indeed, there are hidden units in the model which are responsible for specific object classes. So here again, if I wanted to do this analysis with Debase, I could do it with only a few lines of code. Also, as a sanity check, we downloaded the code base of the system and we compared its output to our model. So the results are here. So every bubble on this chart represents a visual concept, so an object or a texture. On the X axis, we have the score by an X axis. On the Y axis, we have the Debase score. And we find that there is a very strong correlation. So there is a little spread here. We explain it by the fact that the score is actually non-deterministic. The equation approximation is based on sampling. So optimizations. So a nice feature of Debase is that it optimizes the computations for you. So to see how it does this, let's first review the neighbor approach. First of all, you would materialize all of the activations from the model, so it's a big time series. Then you would materialize all of the output of the hypothesis functions, and then you would compute all the correlations. The problem here is that if you do this, you're going to incur a lot of pressure on memory. So we can do some very simple back-and-enveloped calculation. So if I wanted to run VGGNet, which is a very popular image analysis model, and record the output of all of its layers within 900 megabytes per image, I think in the database, we have 65,000 images. So that's a lot of data. If this is not a data management problem, I don't know what is. So one of the strict that we can do to alleviate this is to partition the data. We basically partition the data in block, and for each block, we update the matrix. If we have a GPU, we can do this very fast in parallel. Another trick that we can use is early stopping. So the idea is that we compute those correlations and we maintain some confidence bounds around them, and we stop the processing when we reach a certain target confidence, like 95%. So to check how well those optimization work, we created a benchmark. So the starting point for the benchmark is the parent-less neural experiment. We basically replicated this experiment. We created our own version, but we applied it on hundreds of different grammar rules. So this was inspired by an EMA NLP 2016 paper. So the models that we inspect are SQL auto-completion models. So these models can take a window of 30 characters as input and create the next character. The hypothesis that we use in the benchmark are the grammar rules of SQL, but binary encoded. So for instance, we have one hypothesis function that will give you one for every table name and zero everywhere else. One hypothesis function that will give you one for every quality predicate and zero everywhere else, and so on and so forth. So the next thing with this setup is that we can arbitrarily scale the number of hypothesis functions that we use to probe the model. So by default, we have 142 of those. We run the model on 29,000 records, so that's about 900,000 characters. And the model has 512 hidden units. So here on the left side, I show the runtime of debates as we vary the number of hypothesis functions. In the middle, we vary the number of records. And on the right side, we vary the number of units. So the blue curve here, so on top, shows the runtime of debates unoptimized. So it's basically vanilla, Python, and MPI, and it's very close in spirit to the scripts that you will find online. So we see that in almost all cases, the analysis takes over one hour, which makes sense because there are no optimizations. The red line here shows the impact of early stopping. So the idea is that we materialize the full-time series, but we stop computing the correlation when we have 95% confidence. And we see that in most cases, the analysis runs in less than 10 minutes. And finally, the green line here in the middle shows debates, so with all of the optimizations enabled. And we see that we are basically insensitive to the number of records in the database. So of course, this was just a glimpse in our experiments, and I invite you to build the full paper for more results. So to conclude, we presented deep-no-inspection, which is a broad class of machine learning diagnostic method, and we presented debates, which is a declarative system to make this easier and faster. So for debates, this is just the beginning of the story. Currently, we're trying to make it work for bigger models and for bigger collections of models. And indeed, everybody talks about using smaller, more interpretable models, but it does not seem that models are getting any smaller. In particular, I'm thinking about the natural language processing community. Currently, there is almost a war raging for who is going to build the biggest language model. So those models ingest a lot of data, but also produce a lot of data. To understand what they do internally, we're going to need a lot of math, but also a lot of systems work. So I think there's a lot of very exciting research challenges here for the people in this room. Thank you very much. Thank you.
 My name is Thibault and this is DeepBase, Deep Inspection of Neural Networks.  This is joint work with Kevin Lin, Ian Huong, Michel Young, Carl von Drick and Eugene Wu,  and all the work was done at Columbia University.  So it's pretty clear that nowadays neural nets are popular and people use them to solve all kinds of crazy new tasks,  like playing competitive video games or like driving cars.  But of course not everything is so easy with neural nets.  Neural nets are notoriously hard to debug.  And indeed, if you look inside a neural net, you will find this.  The decision making process of the system, so the internal logic, is embedded into thousands,  sometimes millions, sometimes hundreds of millions of weights,  which are virtually impossible for a human to understand.  This is why it became almost a cliché to compare neural nets to black boxes  and black boxes are almost impossible to debug.  So another problem that motivates this talk is that sometimes neural nets do the right thing,  but for the wrong reason.  So to explain this, let me show you an example.  This here is a screenshot of VQA, visual question answering, and it's a system from George Atteck.  So the idea is that you show a picture to a model, you ask a question, and the model will give you an answer.  So in this case, I showed the picture of those two bears, asked what animal is that,  and the system tells me that it's a bear, so the blue bar here tells me that I have 9% confidence.  It's pretty cool.  So to illustrate the point, let's poke the system a little bit.  The questions that are interesting here are counting questions. How many?  So we can ask how many bears are there in the picture.  And then the model will reply, two.  We see that we have a high confidence, 66.9%.  I think it's pretty awesome that we went that far, and it's actually pretty spectacular.  So you can say, okay, it seems that the model knows how to count.  So let's continue and probe the model a little bit further.  We can ask how many rocks are there in the picture.  So this is a slightly harder question because there's those two rocks here in the back,  and you can say that is it one big rock, is it two separate rocks?  It's kind of hard to say.  And indeed, the model replies, well, it's somewhere between three and four, perhaps two.  So I don't think it's ridiculous, and it's pretty close to what the humans would actually answer.  So now let's continue and push the model even further.  How many Strubevapher's are there in the picture?  So in case you wondered, a Strubevapher is a typical Dutch cookie.  It's delicious.  You probably never heard of it, but don't worry because neither did the model.  And here the answer is two.  So at first glance, this result seems absurd, but actually it makes perfect sense.  It makes perfect sense because two is the majority class.  It turns out that most questions that start with the keywords how many have the answer to.  So the model was actually very smart.  He did not get the question.  He completely missed the point, but he thought,  maybe I'm going to try my luck with the majority class.  The problem here is that we lose confidence in the predictions of the model.  If they pull this kind of trick, how can we trust it?  Let's go back to the first question here for a second.  How many bears are there in the picture?  Do you think the model replied two because it counted two bears?  Or did it reply two because it tried to trick us?  And this is a real screenshot.  So the problem that I would like to make here is that neural nets are really sneaky,  and they will pick up whatever correlation they can to maximize their scores.  Sometimes the correlation makes no sense at all.  Sometimes the correlation should never be picked up.  So because those problems are so important,  there's been a huge push in the machine learning community  to try to better understand what happens inside those models.  So this is a very fast growing field of research,  and many approaches are being developed concurrently.  But we identified a recipe, kind of a very popular pattern,  a very popular class of analysis.  And in fact, it is so popular that we decided to give it a name.  There is deep-dough inspection.  So all of those papers here are deep-dough inspection papers.  So the idea behind deep-dough inspection is to map a subset of the model.  So for instance, a hidden unit or a whole layer,  or a selection of hidden units to a function that we know and that we understand.  We want to make statements like,  oh, it seems that those two hidden units here are in charge of recognizing the body of the bear.  And we're going to do this with a statistical method,  as I'm going to share in a minute.  So let me give you a few examples.  The most popular example of deep-dough inspections is perhaps  Andre Coppati's blog post on regular neural nets.  So I think anybody who's working with a neural net has seen this before.  So the idea is that Andre Coppati trained a model to predict the next,  so to generate realistic C code.  And what he realized is that there is one hidden unit  that can keep track of all the open and closed parenthesis.  It's the parenthesis neuron.  So of course, this was a long time ago, 2015, so in machine learning speed,  it's like, poof, it's really antiquated,  and there's been many other things in between.  So just two months ago, at ICLEI 2019 in New Orleans,  so a team from MIT and QCRI studied neural machine translation models,  and they identified some hidden units that are particularly sensitive  to very specific class of grammatical structures,  like verbs in English.  It's a hidden unit 1902, or like articles in Germans.  In the computer vision literature, same thing.  So at the same conference, another team from MIT looked at GANs,  so generative adversarial networks,  so those are models that can produce images,  and they identified hidden units that are in charge of specific objects.  So here it's the tree and bushes hidden unit.  If you ablate the hidden unit, all the trees and the bushes will disappear from the picture.  If you force its output, you will see some little bush  and some little forest popping up now and there.  It's very cute.  So to be clear, this is still an essential field of research,  and there are many open questions.  In fact, it's quite controversial.  What does the discovery actually mean?  How do they generalize?  How can we use them?  What happens if you change the model?  And so on and so forth.  Nevertheless, what is beyond doubt is that this kind of approach is very popular.  In fact, I can cite at least 10 papers that were published  by the time our paper got accepted at CIGMODE,  I think eight or nine of them are actually here at the bottom of the slide.  So we decided to go through the code base of those approaches,  and we made three discoveries.  So first discovery, so those approaches vary in complexity.  It goes from a few hundred lines of code to thousands of lines of code,  but very often they do the same thing over and over again.  Second discovery is few of those code bases are optimized.  Some are, and they run very fast, but very often they will take hours and hours to run  because the first it isn't here is correctness.  It's experimental code.  It's not made to be used by production pipelines.  Finally, third finding, all of those code bases are completely application dependent,  so they will not generalize.  Therefore, we thought that there would be space for a declarative system,  and this is D-base.  So D-base is our attempt to make deep non-inspection faster and easier.  So before I describe the system, let me take a step back  and tell you a few words about neural nets and neural nets interpretation.  So let's start at the basics.  Neural net 101, a neural net is basically a function that takes something as input  and produces an output.  So in this case, this is a character level language model,  which takes a character of C code as input and brings the next character.  So the little white bubbles here are the hidden units.  You can think of them as subroutines, typically, so on the slide there's four,  but in practice there's thousands of them.  And our aim is to understand why they do.  So the most simple way to do this is to run the model through a test set  and keep track, so log, the output of those hidden units.  We obtain a time series that looks like this one.  So in this case, we can look at it and make guesses.  It seems here that this hidden unit outputs low value for everything that's between parenthesis,  high value of URL, so perhaps we found the parenthesis neuron.  It's actually exactly what Carpathian did in his blog.  Of course, things get more interesting when you look at net one, but several neurons at once.  So in this case, it's a bit harder to do things by just looking at the time series,  so we need something more systematic.  So what is quite popular in the machine learning literature is to use correlations.  You would generate an artificial time series that would give you one,  every time you're between parenthesis and zero everywhere else.  It's called one-hot encoding. It's a type of binary encoding.  And then you look for correlations.  So you can compute the correlation between this time series here  and the output of all of the hidden units, and it will give you a kind of a score  that will give you to what extent the hidden unit seems to replicate the logic of this parenthesis function.  So to be clear, it's just a correlation. There is no causation,  but it's a starting point for analysis.  So another type of analysis that is popular in the literature is to look at groups of hidden units  that work together to encode a signal, and the way we detect those is to use a classifier.  So suppose I have those two time series here at the top,  and I wonder if together they can encode the parenthesis signal.  The way I can do this is to fit an SVM, or logistic regression classifier,  and see to what extent I can predict the value of this parenthesis signal here.  If they do, then it means that those two hidden units together encode this signal.  So it turns out that everything I've shown you so far are instances of the same recipe,  which is deep-no-inspection. So now let's look at the general form.  It's basically based on three steps.  First of all, you take a model, you take a dataset, you run the model on the dataset,  and you log some behaviors.  So in everything we've seen so far, we log the output of the hidden units,  but you can look at other things, like the gradient or the activation of some gates if you like LSTMs.  Second step, you formulate some hypothesis about what you think the model is doing,  and you materialize the dataset with the time series here.  Final step, use a scoring method, so most likely it's going to be some measure of statistical dependency,  to create a mapping between the hidden units of the model at the bottom,  here in gray, and the hypothesis here in red.  So deep-base is a tool to make this process systematic and hopefully easier.  So to see how it works, let's look at an example.  So this is an ACL 2017 paper in which Bellingcoff and his team,  so from Harvard and MIT, I think, asked themselves the following question.  How much grammar does the translation model know?  More exactly, they were interested in part of speech tags,  and they wanted to know if the model was particularly sensitive to say nouns or adjectives or adverbs or verbs and so on and so forth.  So they ran the following analysis.  They took six models for six different pair of languages,  they ran them on a corpus called the TED corpus that has 2.3 million words,  and they recorded everything, is the time series which are here in the middle.  Then they run a part of speech tagger on the data,  and they checked what extent they could reconstitute the part of speech tags  by looking at only the activation of the hidden units.  So the part of speech tags are the red time series here.  So the first one will activate, you have one for every verb and zero everywhere else.  So they found that indeed it's possible to a large extent to predict the part of speech tags  by looking at the activation of the hidden units.  Nevertheless, this varies based on which language you use and which part of speech tags,  which indicates that indeed the model does have some sensitivity to specific types of grammatical constructs.  So suppose that I wanted to reproduce the analysis with a debate, how would I do this?  So it's quite simple.  First of all, we load a data set, then we load the models.  So in this case, I load two models, English to French and English to Arabic.  Then I load the part of speech taggers.  So this is an array and every tag in N, that VB and tag each are Python functions.  So in D-Base, we call those hypothesis functions because they encode the hypothesis  that we want to verify about the model.  They can be any Python function as long as the dimension of the output  is compatible with the dimension of the layer that we inspect.  So finally, I specified the score that I want.  So I want this logistic regression score.  So I call D-Base.inspect and the system is going to work a little bit and produce a table  that will give me the different scores for different hypotheses and different models.  Then of course, it's possible to export this score to our favorite plotting tool.  So what's nice here is that the analysis runs in five lines of codes.  Okay, so there are some helper functions.  If I include the code there, there may be a few dozen lines of codes.  So we are pretty far away from the thousand-ish line of code from the initial code base.  So in everything we've seen so far, we just use text models.  But of course, it's possible to do the same type analysis for images.  So this is a paper from CVPR 2017 in which the authors asked the following question.  How do object detection neural nets work?  So they run the following experiments.  They took 65 convolutional neural nets.  They run them on a bank of 63,000 images called the Broden dataset,  and they recorded everything.  So now it's three-dimensional surfaces.  It's not time-series anymore because we look at convolutional layers.  So you can think of the X and the Y axis as pixels in the image,  and the Z axis as a response of a convolutional filter.  It's kind of like the same thing, but we add one dimension.  It's not in time domain anymore. It's in space now.  So the author asked crowd workers to annotate several objects and textures and materials,  for instance, tables and chairs and wood and so on and so forth.  And they checked to what extent, excuse me,  and they checked to what extent, so the visual concepts,  so the mask actually overlap with the activations.  And they did this with something called the intersection of a union,  which is kind of like the jacquard distance.  They found that indeed, there are hidden units in the model  which are responsible for specific object classes.  So here again, if I wanted to do this analysis with Debase,  I could do it with only a few lines of code.  Also, as a sanity check, we downloaded the code base of the system  and we compared its output to our model.  So the results are here.  So every bubble on this chart represents a visual concept,  so an object or a texture.  On the X axis, we have the score by an X axis.  On the Y axis, we have the Debase score.  And we find that there is a very strong correlation.  So there is a little spread here.  We explain it by the fact that the score is actually non-deterministic.  The equation approximation is based on sampling.  So optimizations.  So a nice feature of Debase is that it optimizes the computations for you.  So to see how it does this, let's first review the neighbor approach.  First of all, you would materialize all of the activations from the model,  so it's a big time series.  Then you would materialize all of the output of the hypothesis functions,  and then you would compute all the correlations.  The problem here is that if you do this,  you're going to incur a lot of pressure on memory.  So we can do some very simple back-and-enveloped calculation.  So if I wanted to run VGGNet, which is a very popular image analysis model,  and record the output of all of its layers within 900 megabytes per image,  I think in the database, we have 65,000 images.  So that's a lot of data.  If this is not a data management problem, I don't know what is.  So one of the strict that we can do to alleviate this is to partition the data.  We basically partition the data in block, and for each block, we update the matrix.  If we have a GPU, we can do this very fast in parallel.  Another trick that we can use is early stopping.  So the idea is that we compute those correlations  and we maintain some confidence bounds around them,  and we stop the processing when we reach a certain target confidence, like 95%.  So to check how well those optimization work, we created a benchmark.  So the starting point for the benchmark is the parent-less neural experiment.  We basically replicated this experiment.  We created our own version, but we applied it on hundreds of different grammar rules.  So this was inspired by an EMA NLP 2016 paper.  So the models that we inspect are SQL auto-completion models.  So these models can take a window of 30 characters as input and create the next character.  The hypothesis that we use in the benchmark are the grammar rules of SQL, but binary encoded.  So for instance, we have one hypothesis function that will give you one for every table name and zero everywhere else.  One hypothesis function that will give you one for every quality predicate and zero everywhere else, and so on and so forth.  So the next thing with this setup is that we can arbitrarily scale the number of hypothesis functions that we use to probe the model.  So by default, we have 142 of those.  We run the model on 29,000 records, so that's about 900,000 characters.  And the model has 512 hidden units.  So here on the left side, I show the runtime of debates as we vary the number of hypothesis functions.  In the middle, we vary the number of records.  And on the right side, we vary the number of units.  So the blue curve here, so on top, shows the runtime of debates unoptimized.  So it's basically vanilla, Python, and MPI, and it's very close in spirit to the scripts that you will find online.  So we see that in almost all cases, the analysis takes over one hour,  which makes sense because there are no optimizations.  The red line here shows the impact of early stopping.  So the idea is that we materialize the full-time series, but we stop computing the correlation when we have 95% confidence.  And we see that in most cases, the analysis runs in less than 10 minutes.  And finally, the green line here in the middle shows debates, so with all of the optimizations enabled.  And we see that we are basically insensitive to the number of records in the database.  So of course, this was just a glimpse in our experiments, and I invite you to build the full paper for more results.  So to conclude, we presented deep-no-inspection, which is a broad class of machine learning diagnostic method,  and we presented debates, which is a declarative system to make this easier and faster.  So for debates, this is just the beginning of the story.  Currently, we're trying to make it work for bigger models and for bigger collections of models.  And indeed, everybody talks about using smaller, more interpretable models,  but it does not seem that models are getting any smaller.  In particular, I'm thinking about the natural language processing community.  Currently, there is almost a war raging for who is going to build the biggest language model.  So those models ingest a lot of data, but also produce a lot of data.  To understand what they do internally, we're going to need a lot of math, but also a lot of systems work.  So I think there's a lot of very exciting research challenges here for the people in this room.  Thank you very much.  Thank you.
Sellam_Lin_Huang_Yang_Vondrick_Wu_0_155.png Sellam_Lin_Huang_Yang_Vondrick_Wu_156_263.png Sellam_Lin_Huang_Yang_Vondrick_Wu_264_311.png Sellam_Lin_Huang_Yang_Vondrick_Wu_312_371.png Sellam_Lin_Huang_Yang_Vondrick_Wu_372_443.png Sellam_Lin_Huang_Yang_Vondrick_Wu_444_515.png Sellam_Lin_Huang_Yang_Vondrick_Wu_516_563.png Sellam_Lin_Huang_Yang_Vondrick_Wu_564_815.png Sellam_Lin_Huang_Yang_Vondrick_Wu_816_875.png Sellam_Lin_Huang_Yang_Vondrick_Wu_876_923.png Sellam_Lin_Huang_Yang_Vondrick_Wu_924_1055.png Sellam_Lin_Huang_Yang_Vondrick_Wu_1056_1187.png Sellam_Lin_Huang_Yang_Vondrick_Wu_1188_1391.png Sellam_Lin_Huang_Yang_Vondrick_Wu_1392_1511.png Sellam_Lin_Huang_Yang_Vondrick_Wu_1512_3347.png Sellam_Lin_Huang_Yang_Vondrick_Wu_3348_3467.png Sellam_Lin_Huang_Yang_Vondrick_Wu_3468_4643.png Sellam_Lin_Huang_Yang_Vondrick_Wu_4644_4835.png Sellam_Lin_Huang_Yang_Vondrick_Wu_4836_7619.png Sellam_Lin_Huang_Yang_Vondrick_Wu_7620_26123.png Sellam_Lin_Huang_Yang_Vondrick_Wu_26124_26471.png Sellam_Lin_Huang_Yang_Vondrick_Wu_26472_26519.png Sellam_Lin_Huang_Yang_Vondrick_Wu_26520_26638.png