Flagsifier----------This challenge consisted in finding the correct input for a Deep Neural Networkthat classifies images (size: `1064x28`) and has 40 output neurons.
There were some example images, composed of 38 letters from what looked like the EMNIST dataset.All of them activated the fourth neuron, therefore being classified as the fourth class.
Some quick tests by moving around random letters and removing some others (plus, the structureof the network) hinted us that there was a softmax and the classes were represented as one-hotencoding. Therefore, the network classifies images into 40 classes. Time to discover what they are!
So, at first we transcribed the sample images and used the combinationof tile + corresponding text as dataset.
```pythondataname=["RUNNEISOSTRICHESOWNINGMUSSEDPURIMSCIVI",        "MOLDERINGIINTELSDEDUCINGCOYNESSDEFIECT",        "AMADOFIXESINSTIIIINGGREEDIIVDISIOCATIN",        "HAMIETSENSITIZINGNARRATIVERECAPTURINGU",        "ELECTROENCEPHALOGRAMSPALATECONDOIESPEN",        "SCHWINNUFAMANAGEABLECORKSSEMICIRCLESSH",        "BENEDICTTURGIDITYPSYCHESPHANTASMAGORIA",        "TRUINGAIKALOIDSQUEILRETROFITBIEARIESTW",        "KINGFISHERCOMMONERSUERIFIESHORNETAUSTI",        "LIQUORHEMSTITCHESRESPITEACORNSGOALREDI"]```(Finding typos in this transcription is left as an exercise to the reader:smile: )
After that we divided all the sample images in 38 28x28-tiles (one tile perletter).We have done that using this script:
```pythondataset=[]datalet=[]datax={}
for i in range(0,8):    img = Image.open('./sample_%d.png'%i)    for j in range(0,38):        x=img.crop((j*28,0,(j+1)*28,28))        dataset.append(x)        datalet.append(dataname[i][j])        let=dataname[i][j]        if let not in datax:            datax[let] = []        datax[let].append(len(dataset)-1)```
* `dataset` contains the images.* `datalet[i]` contains the corresponding text of `dataset[i]`.* `datax` contains the  mapping between letters and array of samples. Basicallyit answers to questions like: "which dataset entries correspond to a particularletter?"
Then, we experimented as follows: for each letter, starting from a black image,put the letter in position 0...38, and classify these images. We saved all thepredictions, and then averaged them to see the most likely class for each letter.
We discovered that neurons 14...40 clasified letters: neuron 14 activated for A,neuron 15 for B, up to neuron 40 for Z.
We then need to discover what the neurons 1...14 classify, as some of them probablyclassify the flag.To do that, we need to try to find inputs that maximize the activation of these, one ata time. Another thing that we can leverage is that the flag likely starts by `OOO`.
So, what would one usually do here, with a "real" network? Decide which neuron (e.g.,the first) to try to activate, then create random 38-letters inputs, and then use thelog-likelihood of that neuron for that input as the fitness function for his favouriteoptimization algorithm (e.g., this problem looked perfect for genetic algorithms).
But before throwing cannons to the problem, let's try something simpler (and if it fails, move to more advanced but computationally intensive stuff).The suspect here is that the network is trained on a small dataset, and is stronglyoverfitting the flag on some of the "low" neurons. This could maybe mean thatthe function we need to optimize is not crazily non-linear and with tons of local optimathat require complex optimization algorithms to escape from. Therefore we tried witha simple greedy strategy: for each of the 38 positions, pick the letter that maximizes theoutput of the target neuron. And it worked!
Trying `OOO` as a test string showed activation of neurons 2 and 3 - let's focus on them.
### Results
Neuron 2 has been our first guess, which gave us these (highly noisy) strings, with the greedy strategy:```OOOOTHISISBYKCOZMEKKAGETONASTEYOUATIMWOOOOTHISISBYKCOZMKYKAGZTONBSTWVOUATIWMOOOOTNISISBDKCOZMKSGBGETONMSTXVOUWTIRROOOOTOISISOYECOIUEYSOGETONOSTNVOUOTIWW```
We tried (failing) to submit some flags like* `OOOOTHISISBYKCOZMESSAGETOHASTEYOLATINE`* `OOOOTHISISBYKCOZMESSAGETOHASTEYOLATINW`* ...
After realizing that neuron 2 was just a test-neuron, we changed output neuronfrom 2nd to 3rd and we got sentences like:
```OGOSOMEAUTHTNTICINTEIXIGXNCCISVEGUIWEGOOOSOMRAUTHGNTICINTGIIIGGNGGISMRGUIWEGOOOSOMXAUTHENTICINTEKXIGXNCRISRRRRIREROOOSOYEOLTUTNTICINTEIIIGCNCEIIETOLIRTIRROSOMEAUTHTNTICINTEIXIGXNCCISVEGUIWEG```
We obtained `OOOSOMEAUTHENTICINTELLIGENCEIS........` by averaging (and manuallycorrecting) them and after few tries of guessing ( :bowtie: ) the last word weobtained the correct flag: `OOOSOMEAUTHENTICINTELLIGENCEISREQUIRED`.
### Python scriptYou can find here the full python script we have used (keras + tensorflow-gpu):
```python#!/usr/bin/env python
import numpy as npfrom keras.models import load_modelfrom keras.preprocessing import imagefrom keras.datasets import mnistfrom keras.applications.resnet50 import preprocess_input, decode_predictions
from PIL import Image, ImageDraw, ImageFontimport string, random, sys
dataset=[]datalet=[]datax={}dataname=["RUNNEISOSTRICHESOWNINGMUSSEDPURIMSCIVI",        "MOLDERINGIINTELSDEDUCINGCOYNESSDEFIECT",        "AMADOFIXESINSTIIIINGGREEDIIVDISIOCATIN",        "HAMIETSENSITIZINGNARRATIVERECAPTURINGU",        "ELECTROENCEPHALOGRAMSPALATECONDOIESPEN",        "SCHWINNUFAMANAGEABLECORKSSEMICIRCLESSH",        "BENEDICTTURGIDITYPSYCHESPHANTASMAGORIA",        "TRUINGAIKALOIDSQUEILRETROFITBIEARIESTW",        "KINGFISHERCOMMONERSUERIFIESHORNETAUSTI",        "LIQUORHEMSTITCHESRESPITEACORNSGOALREDI"]
for i in range(0,8):    img = Image.open('./sample_%d.png'%i)    for j in range(0,38):        x=img.crop((j*28,0,(j+1)*28,28))        dataset.append(x)        datalet.append(dataname[i][j])        let=dataname[i][j]        if let not in datax:            datax[let] = []        datax[let].append(len(dataset)-1)
def genImg(n):    img = Image.new('1', (1064,28), color='black')    #for i in range(max(0,len(n)-1),len(n)): # only this letter and everything else black    for i in range(0,len(n)):        img.paste(dataset[n[i]], (i*28,0))    return img
model = load_model('model.h5')model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
def eeval2(a, op):    img = genImg(a)    x = image.img_to_array(img)    x = np.expand_dims(x, axis=0)    classes = model.predict(x)    score = float(classes[0][op])    return score
for oo in range(2,40): # start from 2, this one seems correct#   out=[datax[x][0] for x in 'OOOSOMEAUTHENTICINTELLIGENCEIS'] #do not start from zero    out=[]    for i in range(len(out),38):        maxv=([], -99999)        for j in datax:            for k in datax[j]:                out.append(k)                score = eeval2(out, oo)                if score > maxv[1]:                    maxv = (0, score, j)                out.pop()
        sys.stdout.write("[%d] %38s : %.10lf      \r" % (oo, ''.join([datalet[x] for x in out]), maxv[1]))        sys.stdout.flush()        out.append(datax[maxv[2]][0])
    print("")    print("--Neuron %d: %s" % (oo, ''.join([datalet[x] for x in out])))```