Boys and girls, I got news : we're being probed.

Saturday, 16 March, Year 5 d.Tr. | Author: Mircea Popescu

Mar 15 23:21:41 mircea_popescu could not connect to remote server. Am I alone ?
Mar 15 23:22:21 Namworld Yes. I hope it's not too lonely out there.
Mar 15 23:23:21 mircea_popescu "Hmm, isn't loading right now. The computers that run are having some trouble. Usually this is just a temporary problem, so you might want to try again in a few minutes." dude it's not even resolving.

Mar 15 23:23:25 jborkl Butterfly Labs getting ddos.
Mar 15 23:25:51 Namworld Would be funny that BFL never delivers...
Mar 15 23:26:21 ThickAsThieves Error 21 (net :: ERR_ NETWORK_ CHANGED): Unknown error.
Mar 15 23:26:25 mircea_popescu Namworld It's mostly not loading. I think they're getting da DDoS.

A number of other major services have been hit with DDoS attacks in the past week or two : MPEx (and once before in January but that didn't do much at all), as pretty good examples. Bitcointalk was also down intermittently and sluggish generally, but then again from what I'm told that's par for the course for that site. MtGox was clocking half hour delays all through the past week, but then again that may be just the result of trading spurts.

Back in February it was, although this is older and may be related to their involvement in the NovaCoin scam thing. In January it was sealswithclubs (although this is even older and atypically received a ransom note - unless, of course, Micon is making shit up). Walletbit and BitPay were both offline back in September (the former also receiving a ransom), Bitstamp was hit in October.

To sum up, pretty much every major piece of Bitcoin infrastructure was DDoS'd in the past six months or so. I can't speak for the others, but as far as MPEx goes what we've seen is not exactly standard fare. Some characteristic points to consider :

  • Very high spikes. While baseline remained for the entire duration in the 5-10 Gbit range, I've seen spikes as high as 100Gbit and I'm not even sure I've actually measured the highest ones.
  • Very good quality DDoS. IPs that hit twice in the same hour are extremely rare - and quite possibly false positives. The traffic mix and other characteristics speak, to me, of professionalism.
  • Extremely extended periods. It is trivial to set up a few Gbit DDoS to hit a website for a few seconds or half a minute. Pretty much any kid 15 or older should be able to do it if they're neither lazy nor stupid. On the other hand I've had billions of requests, Petabytes worth of traffic. This is not exactly common, or for that matter cheap to produce.
  • Reasonably good management. The January attack could be considered a a case of fire-and-forget brute force DDoS. The March attack could not - if only for the WhiteHouse episode which produced a reaction within 30 seconds or less, it was clear that technically competent individuals were supervising the attack non stop, 24/7, and I do mean 7.

Originally, in the heat of events, my lead theory was that this is probably a heavy customer doing his own version of testing MPEx' squishiness. As that heat subsided and I've had time to look around, my lead theory is that we're being probed, systematically, by some sort of an organisation with deep pockets and serious technical competence on staff. To put it in other words : if all this isn't part of some secret NSA/ASAi project to map out the Bitcoin ecosystem and document its weaknesses, possible points of failure scl etc I would be very much surprised.

I would very much like for the other people with direct knowledge of these matters - such as for instance sysadmins or service operators involved - to weigh in on this subject. I'd also appreciate any corrections or additions to the list presented from anyone who knows better. Thanks.

  1. Alphabety Soup Agency []
Category: Bitcoin

26 Responses

  1. ThickAsJoseCanseco`s avatar
    Saturday, 16 March 2013

    I do believe it's more likely BTC-e was hit by the same attackers. Makes more sense than angry altcoin lovers taking aim...

  2. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    There was a lot of talk on the forum from angry people, but... yeah. Who knows.

  3. On the contrary, botnets of almost arbitrary size are cheap and readily available to virtually anyone.

    The MPEx DDOS was not the work of a government. DDOS always was, and will be, an amateur's game. Why DDOS when you can perforate skulls?

    Even a very low-budget government organization would rather glue several kilos of plastique, under the right people's cars. After all, "death is faster, cheaper, easier, and more persuasive than persuasion." Or than DDOS, for that matter.

  4. And if you want to know who was behind the DDOS: look to see who profited. That is, who retained access to the sites under attack (including yours) and made a profit from having continued access, or at the very least from knowing the exact moment of unavailability.

  5. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    @Stanislav Datskovskiy Well, you make some assumptions in your first. They may be correct or not, in the case at hand. In general they are wrong, however. I could point to something like stuxnet as proof of this theoretical observation.

    As best I can see nobody profited. 90% or more of the trade volume happening during the DDoS, while MPEx was only accessible through the irc bots and private proxies consisted of open option positions covering into MPOE bot's book. I can't envisage a scenario in which whoever was trading would have significantly benefited from a DDoS in that context.

    Possibly worth the mention that trade is not quite as brisk as you seem to think, MPEx does something like 1-200k BTCs' worth a month but the average trade is something like 15BTC. If you do some very rough math that comes to about 10k trades every 2.6 mn seconds, and so unsurprisingly there's whole hour intervals where the ticker is silent.

  6. Add to that list.

  7. Confirmed. Then again how is BFL part of the Bitcoin infrastructure?

  8. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    Since Bitcoin is backed by drama BFL is roughly as important as Coinlab : make wild claims, fail to deliver, baww and butthurt ensues, price goes up.

  9. Conspiracy.

  10. Minor nitpick: Stuxnet was not a botnet in the usual sense.

    Government organizations can and do release "targeted" malware (the American empire is safe for as long as its enemies are dumb enough to use Microsoft products) - but DDOS is another matter. Why DDOS when you control the Internet backbones? And, for that matter, just about everything else?

  11. Re: NSA vs. Bitcoin: Ft. Meade has a working semiconductor plant on campus.

    How many mining ASICs do you suppose it could produce on a month's notice without the slightest risk of cutting into the agency's cocaine and hookers budget? And, for that matter, how many "terahashes" worth of FPGA clusters are on site, waiting to be used?

    Bitcoin lives because they let it live. For now.

  12. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    @Stanislav Datskovskiy Certainly wasn't a botnet in the usual sense, but certainly wasn't a bomb either. For that matter, it is almost never the case that the government applies sexy, efficient or clean solutions to any problems it encounters. The cool guys in Casino might bomb each other's cars, but the government does no such thing.

    Just leisurely visit a CattleServices office anywhere in the US (you know, those places where they dispense drugs, food stamps or whatever else to inner city poor) and see exactly what the sorts of solutions government employs are. Compare the seating there to the seating you favour, admire the lasting cheapness of the flooring and wallpaper... Pretty much the only place where a government bombs some dude's car is in a movie.

    The US does not control the backbone, who told you that ?!

    Re the producing ASICs at Ft Meade ; there is a protection baked into the sloppy Bitcoin code to wipe out attacking ASICs by slighlty modifying the calculation they need to make (something ASICs can't cope with - have to be reburned). Consequently that vector is not economically feasible now (and will not be economically feasible in the future because the ASIC window is rapidly closing).

    The entire stuff with "they let it live" reads to me like spillout from bad fiction. Back here on planet Earth government (any, all) is unwarrantedly loud and fundamentally powerless.

  13. Hanging around to see how people take it is irresistible for most, so who has been watching and paying attention following these incidents?

    Not just BFL. Coinabul and others were affected. Whole datacenter basically.

  15. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    @Chett The blessing of the Internet is that you never know who lurks moar.

    @Anon I wonder if this will be followed up with a hot summer of hacks.

  16. Re: ASICS: the turnaround time of the NSA plant is three weeks (from design documents to packaged chips) - at least according to official documents. If true, this would be quite enough for multiple rounds of attack, ad infinitum. And consider the fact that TOR (and who knows how many clandestine variations on the theme) exist. An NSA ASIC cluster can be made entirely indistinguishable from an arbitrary number of ordinary people mining with graphics cards.

    Re: CattleServices: the notion of the U.S. having a single, publicly-identifiable government is quite mistaken. There are at least two. One is responsible for "CattleServices", the postal service, etc.; the other for the Federal Reserve and all the related shadow apparatus that enables an obviously bankrupt empire to trade green toilet paper for most of the planet's production of various goodies.

    Compare a Soviet apartment block or cafeteria to a Soviet missile base. Having two governments, with entirely separate funding, managerial philosophy, and quality of results is not a new thing at all. It is the 20th century norm. Just as a farmer has entirely separate tools for dealing with cattle and for dealing with rustlers. This should surprise no one.

  17. killerstorm`s avatar
    Saturday, 16 March 2013

    (mostly unrelated) Boys and girls, and this is why we want a decentralized stock exchange based on colored bitcoins, so we would depend only on Bitcoin alone, not Bitcoin+some web site.

  18. Mircea Popescu`s avatar
    Mircea Popescu 
    Saturday, 16 March 2013

    Stanislav Datskovskiy Well... takes three weeks to make the asics, takes nutsos like Luke about two hours to notice it and apparently a fix can be deployed in another half hour to hour and a half. The cost to attack is the cost of the run, the cost to defend is not really worth the mention. Economically this isn't very feasible (not to say it won't be done, that's the point of governments in the end, to do economically unfeasible stuff).

    The latter point, re tor/user impersonation is valid. I have no idea what the defense would be.

    The dual govt point is certainly on the ground. Taken.

  19. Mircea Popescu`s avatar
    Mircea Popescu 
    Sunday, 17 March 2013

    @killerstorm The problems with colored coins are discussed here. To add to that, imagine what the pretend-devs would have to say about something moving satoshis around, considering the nonsense they've been spewing about S.DICE.

    Currently Bitcoin financials exchanging is the only problem actually solved, with regard to these threats (as detailed here). Much more serious concerns are, which is by its nature very vulnerable to ddos-ing but also very much needed (and yet somehow people centralise unwarrantedly, blockinfo is dead pretty much) and things like the fiat exchanges.

  20. Bitcoin Megastore`s avatar
    Bitcoin Megastore 
    Sunday, 17 March 2013

    Lately, Coinotron pool was under heavy DDoS for a long time and on few occasions as well. Many other Bitcoin related websites, from small to big, have been going down for up to few hours as well.

    Some IS probing the Bitcoin related services and websites, that is for sure!

  21. Mircea Popescu`s avatar
    Mircea Popescu 
    Monday, 18 March 2013

    Hey, are you running Coinotron thing ?

  22. Bitcoin Megastore`s avatar
    Bitcoin Megastore 
    Monday, 18 March 2013

    No, I just used to mine there. After many DDoS I gave up and moved to Bitparking. Here is a link to Coinotron thread on bitcointalk forum, you can PM owner there:

    Or check for contact info at pool website:

  23. Mircea Popescu`s avatar
    Mircea Popescu 
    Monday, 18 March 2013

    Aha, cool.

  1. [...] Mircea Popescu Aha, cool. [...]

  2. [...] Boys and girls, I got news : we’re being probed. [...]

  3. [...] B may be the addresses of an unrelated third party. Like you know, back when I was being DDoS'd by "someone" and then I pointed the DNS to the White House ? Yeah, exactly like that. If "we know where you [...]

Add your cents! »