Choice 1 requests an `m` and plugs it into `(rh+m)%p`, where `r` is random and `h` and `p` are preset. Choice 2 gives a ciphertext and asks us to find `m`. 
We can use the following diagram to find `p`. All we need is a sufficiently small `m1` and a sufficiently large `m2` that is less than `p`. We find that a good input for `m1` is `A` and a good input for `m2` is 50 `A`'s. These turn into the numbers `65` and `658220557159408071873077436000767732672277947191762856211736698726047433063116382944811560640035133503269435993053282625`. We run the service a few times until the second output wraps around the modulus,![](https://i.imgur.com/0TP3ChI.png)
This gives us `p=2129236650498506197214865121017813676962270980934541379925587741818174020229784960110052122450619093813474017151250421361`.
Now that we have `p`, we can now find `h`. We collect values of `rh` and take their gcd's to get that `h=5652773385368202689409947437764703572562115505999698456043486537028395595811789626496940450622715417097798491386547`.
Now that we have `p` and `h`, we can attempt choice 2. We get a ciphertext, plug it into Mathematica (or wolfram cloud) and get an answer in a few seconds. Then we copy-paste the answer into the service and we get the flag: `AceBear{r4nd0m_is_fun_in_my_g4m3}`.
![](https://i.imgur.com/vtdStnR.png)