# Code Golf## Google CTF 2019
## Index* [Acknowledgements](#Acknowledgements)* [The problem](#The-problem)  * [Problem statement](#Problem-statement)  * [Some background](#Some-background)  * [Lessons](#Lessons)* [The solution](#The-solution)  * [A first pass](#A-first-pass)  * [The first correct code](#The-first-correct-code)  * [More compact code](#More-compact-code)  * [The first accepted code](#The-first-accepted-code)* [Golfing transformations](#Golf-you-a-Haskell)* [NP-Completeness](#NP-Completeness)
## Acknowledgements
This writeup is done by Cole Kurashige.
I'd like to thank Kye Shi for his help designing a faster algorithm, and Giselle Serate for algorithm verification and actually running the code. Also forshowing me the CTF.
# The problem
As a foreward/warning, this is a lengthy writeup. If you want the TL;DR highlights , first read the [problem statement](#Problem-statement) if you aren't familiar with the problem.I think the most interesting highlights are the[golfing tips](#Golf-you-a-Haskell), specifically [this one](#Finding-the-possible-shifts), andthe [NP-Completeness proof](#NP-Completeness).
Or just skim the titles and skim/read what is interesting. A lot of the length comes from headers,newlines, and code blocks - as far as text goes I've tried to edit things so they're to-the-point.
## Problem statementThe problem could be found [here](https://capturetheflag.withgoogle.com/#challenges/)as of 6/27/19. In case it gets moved or taken down, I've described it below.
Given a list of strings with gaps in them (denoted by a space), you're asked to combine them intoa single string. Imagine all of the strings stacked on top of each other. You want to shift the stringsto the so that only one or zero characters are in each position. You also want to minimize thelength of the resulting string (ignoring trailing and leading gaps). The tie-breaker for multiplesolutions is lexicographic length. Your solution is a function `g :: [String] -> String`.
An example given is the strings `"ha  m"` and `"ck e"`:
If you overlay them:
    "ha  m"    "ck e"
Shift them by an appropriate offset:
    "ha  m"      "ck e"
You get the resulting string `"hackme"`.
The catch to all of this is that it must be done in Haskell in 181 bytes or fewer.
Oh, also, it is [NP-Complete](#NP-Completeness).
## Some background
I spent a _long_ time on this problem. Its theme for me was"comfort's a killer." I really like Haskell, and have been using it recently.I really like code golf, and have golfed code in the past. I probably should've spent more time thinking through an algorithm, but I dove inheadfirst. I knew there was a tips for golfing in Haskell on the code golf stack exchange, but I neglected to use it.
And so, I spent an entire weekend on one problem. Welcome to my hell.
## Lessons
I learned three important lessons from this endeavor.
### Lesson #1
Code golf skills don't just magically manifest themselves in another language.
I primarily code golf in [J](https://www.jsoftware.com/indexno.html) and [><>](https://esolangs.org/wiki/Fish). Haskell is neither of these. I used prettymuch none of my existing code golf knowledge in golfing this challenge.
### Lesson #2
Imports are useful.
I should've used more imported functions, especially those from `Data.List` more. Our final solution used only two imports: `join` from `Control.Monad` and`(\\)` from `Data.List`.It can probably be reduced to just using `Prelude`, while still mantaining an acceptablebytecount.
### Lesson #3
Think before you write code.
My first algorithm was far from perfect, and even had a flaw in its logic. Whenthis was fixed and it was golfed, we learned much to my chagrin that it used toomuch memory and was silently failing. Soooooo ... we had to come up with a new onefrom scratch. And then golf it again. Did I mention that I spent a lot of timeon this problem?
# The solution
## A first pass
The first thing I did was write a half-golfed program. 
The algorithm was essentially to find every possible way to shift each of the strings and overlay them. Some of these shifts would be invalid, so I discarded those.
I ignored the possibility of strings having trailing spaces, since that wouldjust be cruel. Later solutions also ignored the possibility of strings having leadingspaces. This turend out to be an OK assumption to make (though I wish it were toldto us in the prompt).
There is a mistake in this code: I am only taking the lexicographically minimalsolution, not the minimal length solution. This is fixed in the golfed version,at the cost of many bytes.
```haskellimport Data.Listimport Data.Maybeimport Control.Monad
-- | the solution functiong :: [String] -> Stringg xs = minimum . catMaybes $ [sequence . dropWhile (==Nothing)                              . map collapseW . transpose $ s | s <- shifts l xs]  where    l = sum $ map length xs
-- | find all possible ways to shift a string 's' with at most 'l'-- characters of padding.wordShifts :: Int -> String -> [String]wordShifts l s = [space <> s | space <- spaces]  where    spaces = inits $ replicate l ' '
-- | find every possible way to shift a list of strings, where-- each string can be shifted at most 'l' characters to the leftshifts :: Int -> [String] -> [[String]]shifts l [] = [[]]shifts l (w:ws) = do  shifted <- wordShifts l w  rest <- shifts l ws  return $ shifted : rest
-- | try to collapse a string into a single character (think of this as reducing-- a column to 1 character, this fails if there are more than 2 non-space characters-- or no non-space characters)collapseW :: String -> Maybe CharcollapseW s = do  [y] <- foldMap convert s  pure y  where    convert ' ' = Nothing    convert c   = Just [c]```
I golfed this down to 178 bytes.
```haskellf=foldMaph ' '=[]h c=[c]k l s=[f<>s|f<-inits l]z l[]=[[]]z l(x:y)=[a:b|a<-k l x,b<-z l y]g x=minimum[concat y|s<-transpose<$>z(f(>>" ")x)x,let y=f h<$>s,all((<=1).length)y]```
## The first correct code
At precisely 181 bytes, this was the first code that picked the correct solution,sorting the possible ones by length, then lexicographically.
```haskellh=lengthf[]=" "f l=lz l[]=[[]]z l(x:y)=[a:b|a<-[i<>x|i<-inits l],b<-z l y]g x=snd$minimum[(h a,a)|s<-z((>>" ")=<<x)x,let y=concat.words<$>transpose s,all((<=1).h)y,let a=f=<<y]
```
The problem now is that the code was failing silently! Usually the server would tellyou if you had an error (compilation, runtime, byte length), but it didn't respond andinstead failed after about 25-30 seconds.
We tested with code that ran infinitely, which would get cut off at exactly 1 minute,so with some more testing we concluded that the code was using too much memory andgetting killed.
(we tested the memory usage with `foldl (+) 0 [1..]`, which timed out after 20 or so seconds, and confirmed it was a problem with memory that caused silent failurewith `foldl' (+) 0 [1..]`, which only timed out after a minute. Ahhh Haskell, where one character makes a huge difference)
## More compact code
I further golfed the first correct solution to 151 bytes. I don't remember why I did this.It was golfed during the phase where we were trying to figure out why the serverwouldn't accept our solution. Maybe I was trying to fit space stripping into thecode, but didn't get around to it.
I added some spaces to the last function `g` so it doesn't wrap so hard, but theyaren't part of the bytecount.
```haskellh=lengthf""=" "f l=lg x=snd$minimum[(h a,a)|s<-forM[(>>" ")=<<x|_<-x]inits,  let y=concat.words<$>transpose(zipWith(<>)s x),all((<=1).h)y,let a=f=<<y]```
## The first accepted code
The first correct code took somewhere between 2 to 4 hours of effort to reach.By midday Saturday, I had something that was morally right, but didn't pass the tests.That evening, Giselle and I tracked down the root of the problem to space issues.
I complained to Kye about my solution not being good enough, despite beingshort enough, and he devised an algorithm that was better (at least memory-wise) thanmy like `O(n^n)` space algorithm. This was maybe around midnight on Saturday (whichwas 3 AM his time...).
He, however, did not golf it, so I still had to reduce his ~500 bytes to the below.
I spent an hour or two golfing his solution after I woke up on Sundayand brought it down to exactly 181 bytes.
```haskellm(a:b)(x:y)=max a x:m b ym x y=x<>yf s[]=[s]f s r=join[f(m s x)$r\\[x]|x<-r,and$zipWith(\x y->x==' '||y==' ')s x]<>[h:y|(h:t)<-[s],y<-f t r]g y=snd$minimum[(length x,x)|x<-f""y]```
Yup, this code is _a lot_ different. It ended up being a lot more inefficient thanhis original code, too, since I cut out all of his optimizations when I golfed it.
Kye ended up writing an even _more_ efficient version, which thankfully I didn't needto golf.
I just wish that we knew earlier that the "make it snappy" flavortext didn'tmean to make the code short, but instead meant "efficient enough." I'm used to seeingtime/space restrictions given upfront on the Code Golf Stack Exchange, where therecan be answers that are right by observation but too inefficient to run anything butthe simplest of test cases.
# Golf you a HaskellHere were some of the more inventive or useful golfing transformations I used. SinceI foolishly neglected to use any resources other than the documentation, these were all found by myself.
## Infix your code!Haskell has a lot of infix operators (operators like `+` or `-` that go betweentheir arguments). It's made fun of in this [article](http://uncyclopedia.wikia.com/wiki/Haskell)(warning: somewhat NSFW language), which includes the following code that produces[an infinite list of powers of 2](https://stackoverflow.com/questions/12659951/how-does-this-piece-of-obfuscated-haskell-code-work/12660526#12660526). 
```haskellimport Data.Function (fix)-- [1, 2, 4, 8, 16, ..]fix$(<$>)<$>(:)<*>((<$>((:[])<$>))(=<<)<$>(*)<$>(*2))$1```
The next few points are on how you can use these operators.
### `$`A simple transformation that I often use in code read by people other than myselfis `$`. `$` is a function that just applies its left argument to its right argument.
```haskellf $ x = f x```
It has really low priority, though, so it can save you parentheses like in
```haskell-- these are the samegcd (1+2)  (3*4)gcd (1+2) $ 3*4```
### `map``<$>` and `map` are the same for lists, but the former has lower priority, which letsyou reap some of the benefits of `$`, while also not needing a space between itsarguments (unlike `map`).
### `concatMap`I found myself often using `concatMap`. I first reduced this to `foldMap`, and then to the infix `=<<`. All of these have the same definition for lists.
### `return`I used `return` to convert an element `x` to a singleton list. This becaume `pure` and then finally `[x]` once I realized I was being a moron.
## Filling holesA tricky part of this problem was combining two strings with holes (spaces) in themto produce one where the holes were filled. As it turns out, the only printable ASCIIless than space (ASCII 32) is whitespace, so I figured these wouldn't show up in thestrings. Thus, given two chars in the same column, we can take their maximum to findthe non-space char (if it exists).
## Cheeky pattern matchingI had code that I wanted to give a list if `x` pattern matched one thing and the emptylist if it did not. Something that looked like
```haskellcase x of  (h:t) -> foo h t  [] -> []```
I converted this to 
```haskell[foo h t|(h:t)<-[x]]```
This abuses the fact that when the pattern match `(h:t)` fails in the context of alist comprehension, an empty list is returned instead of an error. This is a specialcase of how pattern matching is desugared inside of a `do` block.
N.B. `foo` has a different type between these examples.
## Finding the possible shiftsGiven `strings :: [String]`, I wanted to find all of the ways these strings could beshifted. I eventually boiled this down to finding `paddings ::[[String]]`, where each`padding :: [String]` in the list was the same length as `strings` and had a variednumber of spaces in each element. So each `padding`, when combined element-wise with`strings` would give a different shift.
A way of doing this would be
```haskellimport Data.List (inits)
cartesianProduct :: [[a]] -> [[a]]cartesianProduct []       = [[]]cartesianProduct (xs:xss) = [x : ys | x <- xs, ys <- cartesianProduct xss]
paddings :: [String] -> [[String]]paddings strings = cartesianProduct $ replicate (inits maxPadding) (length strings)  where    maxPadding = replicate (sum $ map length strings) ' '```
Let's take a look at
```haskellmaxPadding strings = replicate (sum $ map length strings) ' '```
In order to make sure that I was shifting enough, I found a maximum padding thatwas equal to the sum of the lengths of the strings.
We can reframe this as converting each element in `strings` to a string that is the samelength but only consisting of spaces, then concatenating all of these elements together.
```haskellmaxPadding strings = concatMap (\str -> replicate (length str) ' ') strings```
`replicate . length` is way too long, so let's replace it with `(>> " ")`.
```haskellmaxPadding strings = concatMap (\str -> str >> " ") strings```
Eta reduce and obfuscate `concatMap` to give
```haskellmaxPadding strings = (>> " ") =<< strings```
Much better.
This is only the max padding for a single element though. I wanted to find `paddings`.`inits :: [a] -> [[a]]` will get us part of the way there, since it will give allpossible prepended spaces, from 0 to `maxPadding`.
```haskellinits [1,2,3] = [[],[1],[1,2],[1,2,3]]inits (maxPadding ["a","bc","d"]) = ["", " ", "  ", "   ", "    "]```
We then want the cartesian product of `inits maxPadding` repeated `length strings` times. `cartesianProduct` is a long definition, so why don't we use the list Monad some more?
```haskellpaddings strings = sequence (replicate (inits maxPadding) (length strings))```
`sequence` is the same as `\x -> forM x id` or `\x -> mapM id x`, so we can convert to 
```haskellpaddings strings = forM (replicate (inits maxPadding) (length strings)) id```
We want to be applying `inits` to every element anyway, so we can pull it out.
```haskellpaddings strings = forM (replicate maxPadding (length strings)) inits```
Then get rid of this `replicate` nonsense by using a list comprehension that ignoresall of the values of `strings`.
```haskellpaddings strings = forM [maxPadding | _ <- strings ] inits```
Substitute the definition of `maxPadding` and we're done.
```haskellpaddings strings = forM[ (>> " ") =<< strings | _ <- strings] inits```
354 bytes of (reasonably) readable code down to 67 bytes of nonalphanumeric soup.
Don't you love code golfing?
# NP Completeness
On Saturday evening, I was banging my head against a wall trying to optimize thespace and time complexity of my algorithm. But every time I tried to think througha faster algorithm, something felt wrong. I had a feeling that the problem was [NP Complete](https://en.wikipedia.org/wiki/NP-complete), and so the only thing I could get optimal would be space. I eventually sat down and proved it NP-Complete.
## ProofThe proof is by reduction from the [Bin Packing Problem](https://en.wikipedia.org/wiki/Bin_packing_problem) (BPP). This isa pretty rigorous proof, but I tried to make it understandable. I think the [Reduction, visualized](#Reduction-visualized)section does a pretty good job of building intuition for how the proof works, but if youhaven't seen a reduction before this all might seem kind of obtuse.
### Bin Packing Problem (decision version)The BPP asks, given `m` bins of size `V` and a set `S` of elements with an associatedcost function `f : S -> N` (where `N` is the natural numbers), can `S` be partitionedinto at most `m` subsets, each of whose total cost is less than or equal to `V`? Inplain English, given `m` bins, each with capacity `V`, can we fill each bin to atmost capacity with all of the elements in `S`?
### Crypto Problem (decision version)First, I will state the decision variant of the Crypto Problem (CP). Given a "set" `T`of strings that have holes in them represented by spaces and a target length `l`, theCrypto Problem asks whether all elements of `T` can be overlaid such that there is atmost one non-hole per column and the length of the overall result (after removingtrailing and leading spaces) is `l` or less.
(this "set" may have duplicates - I don't want to be as rigorous as in the BPP) 
### ReductionWe can construct a CP from an arbitrary BPP as follows.
We first will construct theset `T`. Create `m` strings, each of length `3V + 2`. The first and last `V+1` characters of these strings are `#`, and the rest (the center) are spaces (holes).Character choice doesn't matter here, so I pick an arbitrary one.These strings will represent our bins, and I will refer to them as "bin strings".
For each element `s` in the set `S` from the BPP, we want to make an analagous elementfor the CP. This element will be `f(s)` long, and consist only of the character `*`.Again, character choice doesn't matter here. I'll refer to these as "`*` strings".
Now, we pick a length `l`. This CP will have a maximum length of `(3V + 2) * m`.
This reduction clearly takes polynomial time as it involves iterating as many timesas there elements of `S` and bins.
We claim that the given BPP is solvable if and only if this constructed CP is solvable.
### Reduction, visualized
Let's consider a BPP with `V = 3`, `m = 3`, and elements of size `1`, `1`, `2`, and `2`.
The bins:```# #  # #  # ## #  # #  # ## #  # #  # ####  ###  ###```
The elements:```*  *  *  *      *  *```
A valid placement of these is putting a `1` in its own bin, a `2` in its own bin, anda `1` and `2` in the last bin. Another valid placement would be tofill two bins entirely and leave one empty.
```# #  # #  #*## #  #*#  #*##*#  #*#  #*####  ###  ###```
The equivalent problem in CP is the following strings:
```####   #### (x3)* (x2)** (x2)```
And an equivalent solution is the following:
```####   ####    *                 ####   ####               **                      ####   ####                          ***```
which, when flattened looks like
```####*  ########** ########***####```
Note how the length is `(3V + 2) * m` = 33.
### Reduction proof (forward direction)
Given a solution to the BPP, we can make a solution to the CP. Suppose in the BPPsolution that some arbitrary bin `B` was filled to a volume `V' <= V` using theelements in the set `S'`. This implies that
```sum(f(s') for s' in S') = V' <= V```
Since we created a bijection from elements of `S` in the BPP to elements of `T` in theCP, find the corresponding elements from `S'` and put them in the set `T'`. We claimthat the elements of the set `T'` can be made to fit into a single bin string.
Why is this? Well, the sum of the lengths of these strings is less than `V`, which isthe amount of space in the middle of each "bin" string. Therefore, we can justconcatenate all of these strings together and place them in the center of the binstring.
Since we can do this for an arbitrary bin, and there is a bijection from `S` to `T`,i.e. all bins in the solution of the BPP span all of `S`, we can fit all of the elementsin `T` corresponding to elements in `S` inside of the "bin" strings. This means thatthere exists a solution where we place all of the `*` strings inside of the binstrings. Therefore, the overall length of this solution is just the length of all thebin strings combined, which is `(3V + 2) * m` as desired.
### Reduction proof (backwards direction)
We'll prove the contrapositve of the backwards direction. If there doesn't exista solution to the BPP, we cannot make a solution to the CP.
First, it should be reasonably obvious from the previous direction that the strategyof placing the `*` strings in the bin strings won't work, since the BPP is unsolvable.If we could place them in the bins, then that would imply that the BPP was solvable,which contradicts the premise.
So we would have to find a solution to the CP that doesn't involve _just_ putting the`*` strings inside of the bin strings. Since our target length is `(3V + 2) * m`, weneed to fill all of the holes in the bins. If we aren't _just_ putting `*` stringsinside of bin strings, this means we would have to have bin strings intersect. This isimpossible, as on either end of the `V` holes in the center there are `V + 1` holeson the side.
Visually, for `V` = 3, we get alignments that look like this.
```top bin    | ####   ####  | ####   ####   | ####   ####    | ####   ####     | ...bottom bin |  ####   #### |   ####   #### |    ####   #### |     ####   #### | ...```
The bins all have to intersect in _some_ column, which is not allowed in an answer.
Therefore, it is impossible to have a solution to the CP.
### Conclusion
Since we showed a reduction existed from the BPP to the CP that took polynomial time,and that the constructed CP was solvable if and only if the corresponding BPP was,we conclude that the Crypto Problem is NP-Complete.