
Yandex.Mail's successful migration from Oracle to Postgres [pdf] (pgcon.org)
415 points by postila on Sept 13, 2016 | hide | past | web | favorite | 210 comments

My take aways:

* Reasons to migrate were: unresponsive support, inconvenient deployment, closed source, huge cost.

* It took just 10 man-years to rewrite whole project from Oracle to Postgres (e.g. 10 developers, 1 year), which is amazing.

* They benefited from Postgres data versioning, arrays and composite types.

* They liked writing logic on PL/pgSQL more than on Oracle PL/SQL: reduced code size, increased test coverage.

* Easier deployment of changes due no library cache locks.

Great thing for community is that Yandex now commited to Postgres, meaning it will get more testing and bug-fixes faster.

I have no love for Oracle (at all, actually), but let's not kick the donkey when it's down. Having been through tech refreshes myself I'll bet most of those benefits would've been realized, regardless of data provider. Too much opportunity for a competent team to reduce or eliminate technical debt.

People are still buying Oracle licenses, we have to kick it until it's dead.


That remains to be seen. There's a fair amount of huge postgres users that don't even bother to report bugs they find...

But you're right – a lot of such companies exist unfortunately.

If Uber DBAs came to IRC or any public place (see http://Postgres.chat) where Postgres experts are present and didn't hesitate asking questions, they would still be running Postgres.

Yandex has a pretty good track record for contributing back to open source projects they use (and open-sourcing their own technologies).

It's already seen during couple of years – google author's name on postgresql.org (maillists archives)

Yandex.Mail provides custom domain E-Mails for free, the only large company to do so and not to mention how good it is.

Good tip. I recently did some research on email with free custom domain (well, really just googling and visit the first results) and did not know about them.

So for the moment I'm using zoho.com.

zoho offers free custom domain email hosting. It's also surprisingly good, the UI/integration between the pieces is very well done.

Yandex offers unlimited users (up to 1000, then you have to contact them and I guess they will allow more) and claims unlimited storage, zoho free tier is 25 users and 5 GB storage/user.

Any idea if they support email 'distribution lists' or similar? I want something like test@example.com > delivered to list of users with a mix of email addresses at other domains. Gandi.net does this now (they call it a forwarding only address) but their included mailbox size is 1GB. I've got a non-profit domain that needs more email storage.

I tried Zoho last year after hearing multiple recommendations and it was nothing but problems with spam coming in and my outbound emails being incorrectly marked as spam. I even had SPF + DKIM tested and working from the start! Not sure I would try them again after that experience.


Doesn't seem to support addresses from other domains though.

Zoho is OK, but I wouldn't call it great. Huge delays happen from time to time.

It's can be also used as DDNS with simple API.

Russian email hosting. What could go wrong?

American email hosting, what could go wrong?

Yeah, I always find it amusing when people find russia and china on a lower moral/ethical ground when everything on american soil is equally watched and processed. if not by nsa, it's for ads.

Please take a look at the Human Rights Watch reports[0] for USA, China and Russia. I honestly wonder how you can reach such a conclusion about freedom of expression and human rights in those countries.

[0] https://www.hrw.org/world-report/2016

Mmm, I'd agree but the parent is specifically talking about surveillance, not about all human rights, and it's true - the US likes to condemn cyber-espionage and surveillance while simultaneously spying on the entirety of the planet and attacking/infiltrating the planet as it sees fit.

Human Rights Watch being an American organisation, led by a member of the US power elite (e.g. Goldman Sachs, Soros' OSF) might have something to do with it?

Nah, perish the though!

HRW funding and staff indeed looks surprisingly US-biased. But Amnesty and PEN also score the US a couple notches above Russia and China.

As someone who resides in the USA, I'd rather China or Russia have access to my data, regardless of any abuse of human rights. Unlike the US, they're extremely limited in how they can use my data to harm me.

The problem is that they can sell your data to someone more interested in targeting you specifically. If that happens within the US there is a means to litigate the guilty parties. Outside, you're SOL.

You don't need to look there, at all. Just look at the US aggressive wars and subversive activities in this century, e.g. Iraq, Libya and it becomes clear that the value of human rights abroad from the US point of view is exactly 0.

Iraq was a disgrace but bringing up Libya means you are either ignorant or being disingenuous. Gaddafi had imminent plans to commit another cleansing of the opposition. It was always going to be a lose-lose situation for human rights. Doing nothing is not always the right answer.

It's better to state the reality not the myth sold by USA : Ghadafi wasn't about to commit anything else than what US allies have already done.

Now Libya is a failed state thanks to US and HRW lies.

So how do you rate the situation in Libya before and after Gaddafi?

Bad with him there. Much worse with him gone.

By calling Iraq a disgrace you are effectively calling the 9/11 a micro-disgrace. That's how these two differ in the order of numbers of lives lost.

Human Rights Watch. By Americans for Americans.


i hope you're kidding or trolling.

the current situation with incarcerations in america sounds horrible, yes, but its a far cry from systematically eradicating millions of people per year...

Really? I don't see much difference. Slavery is slavery. Torture is torture. Rape is rape. The first two are legal in prison and all three are widely practiced as punishment. Yes, you have a better chance of surviving our prisons, but by that point, your life is ruined and pretty much over anyway. There are much worse things than death.

Exactly!, I mean for sure slavery is not so bad too if you treat the slaves nicely right? Systematically enslaving them is also a far cry from eradicating them right?

Asking seriously, but isn't Russia fundamentally closer to a kleptocracy? Dishonesty is apparently more pervasive and built in to their culture. From my understanding, bribe culture and cheating are an accepted norm. Not to say these characteristics do not exist in America, just not anywhere to the same extent?

I've lived in SPB for a few years now (american expat), and there's...some truth to it. The idea of cops just grifting you non-stop is overstated; more likely than not they just wanna not have to do work. Even when it comes to checking documents for folks from the southern countries (uzbekistan, tajikistan, etc) most of the time the police try to speed stuff along so they can get back to smoking, having tea, or stopping off at a kebab place.

Even though everyone I know here calls the process "bribing", it's greasing the wheels - in some ways, it's bad that it happens. At the same time though, everyone is allowed to participate in the bribing and whoever takes the bribes can and will settle for what they can get.

I wouldn't say dishonesty is more pervasive - quite the opposite, most people are brutally honest, sometimes to a fault. When it comes to official things, it's true, you can grease the wheels and get preferential treatment. Certain problems can go away with a few well placed notes, and so on. Schools are a bit different too since the collapse of the Soviet Union. From what I'm told, there's a lot of pressure from the state to ensure that students pass, so students will repeat through classes as long as necessary to pass them, and often professors will just give out the lowest possible passing grade to get rid of bad students. A little bit of cash will net you a better grade, which will get you access to better universities.

Just as an outsider looking in and watching, I see it more as the same bad system that exists elsewhere but at least the system is open to everyone.


"Greasing the wheels" is a popular euphemism for bribery, not a distinct different act which contrasts with bribery.

No, I think you're conflating when I'm specifically talking about just getting better service.

See there's another slight disconnect with service in Russia versus service in the US. You ever have someone go out of their way to make something easier or better for you as part of their job? Go the extra mile? That's not really the status quo in Russia. You goof up your registration form at the post office (one mistake on the 4th page of a 10 page document)? Guess who's filling it out again, even if the office worker has white out handy. Only have 30 minutes to take care of something at the nearby government office but it happens to be 1:45 pm and time for the official's tea break? Guess who's not getting their document done.

When people are talking about greasing the wheel, it's getting access to systems they otherwise don't have. It's people going the extra mile on demand, getting the inside scoop, making sure a document gets processed today, taking care of problems for you internally. Do the big bribes happen? Yeah, I'm not denying that. My point more is that the "bribes" that are talked about so much aren't really what we normally think of as bribes.


That's pretty much my experience in most government offices in the US, too.


Paying privately to the official for access for service from public officials that is not generally made available is "bribery". Yes, "greasing the wheel" is a euphemism for bribery that is often used particularly for small bribes for small favors (though "small" is relative to who is speaking.)


They are exactly what most people with experience in domains where there is the kind of lack of accountability which makes pervasive bribery a thing expect as the most common kind of bribery. (When there is some accountability, the perceived risk/reward of smaller transactions becomes less favorable more quickly than is the case for larger transactions -- if as a public employee you are going to get fired and be unemployable in the public sector, and maybe prosecuted, if you are caught taking any bribe, and there is even a modest risk of detection in even small bribes, its no longer worth it to take small bribes in any case.)

The implied difference is that in bribery you're asking the official to do something that they must not do, to have him act against his employer.

To use a restaurant analogy, tipping the bartender to get quicker service is "greasing the wheels" but tipping the bartender to get drinks "on the house" or to get an exclusive event without paying the bar owner - that would be bribery.


That's not the usual definition of "bribery" (though generally it is true of all bribery, including the type you are trying to distinguish, since taking extra personal pay for service is usually formally prohibited, even in places where it isn't effectively enforced, so even what you try to distinguish as "greasing the wheels" as distinct from "bribery" is asking the official to do something that they must not do, and acting against the employer.)

Which part of the city you are in, btw? :)

Russian culture and the current situation in the country are quite different. Honesty and sincerity are highly valued traits, to the point that Westerners are still labeled as disingenuous for smiling too much - Russians view this as fake and therefore less trustworthy. This of course is changing, as the younger generations are more exposed to western behaviors.

The real problem is the economic situation and people that focus on personal enrichment by any means instead of building something sustainable. Basically, in an environment where you cannot be sure of what next year is going to look like, it's brutally rational to try and extract as much value now, as opposed to invest for the future.

As the result, you get everything from poor infrastructure investment and lack of true small business growth, all the way to bribing because the official taking the bribe isn't sure he or she will have this access tomorrow, so better to enrich now and as fast as possible.

This is not something that is endemic to Russian culture, but rather to the survival situation. Unfortunately, Russia seems to teeter in this state for centuries at a time.

The US just uses different words, such as "campaign financing", or "creative accounting".

Go on and try offering "campaign financing", or "creative accounting" to a policeman stopped you for speeding somewhere in California. Don't forget to record it on your smartphone.

In Russia, literally ANY road policeman not only takes bribes, but has a bag of tricks how to get more from you.

Unfortunately it's well-known to me, from my own experience. (10+ years of driving on Russian roads, 4+ in California)

In the US the policeman can legally just take your car and your cash. They don't need to mess around with tricks. (Look up civil assert forfeiture.)

And I understand what you're saying. Similar things are happening when your child goes to school/college/etc. You pay for smth and she/he gets preferences. As I see, it's common both in US and Russia.

But between bribing and things you mentioned there is one essential difference, that changes everything. It's this question:

- Are rules known in advance, are they written and available in public?

I believe this question helps to distinguish black and white.

+ lobbying

I don't think you can lump Russia and China together. China is truly "on lower moral/ethical grounds". I can't trust any country that has such pervasive censorship policy and such blatantly protectionist policies as China. Frankly I trust the Russian government as much as I trust the US government or as much as I trust my government (the Mexican government). I don't think I would feel safe living in China! I wouldn't even travel to China but that's just me.

What about a country which has one forth of the China population which has institutional rules about the police being able to seize the money you carry, your money on cards you carry and your car without even charging you with an offense, where 6% of the black population is in jail, and 1% of the total population, and where you can be routinely shot dead if you are mentally disabled or if you play poker at home or accused of treason of you publish information about some of the most impacting activities of the govt? I've already been in Russia, but I'll never go to USA.

Routinely shot dead if you are mentally disabled or playing poker ?

You completely invalidate any point you had when you resort to ridiculous hyperbole.






> seen by their willingness to bet and lose large amounts of money to take down sports bookies






I can't find the highly graphic one about the one who got shot, like, 18 times, while was unarmed in front of a shop. Anyway poker shooting is a reality: It's illegal gambling, therefore it's considered high-level criminality.

Not to judge who's right or wrong, who's "supposed to know" and who's at risk - I'm only saying the risk of being shot dead by the USA police is immensely higher than in any other country in the world, including Russia and China.

Looks like there are few errors in your description of Russia.

Forgot to mention the mandatory dashcams to fight fraudulent insurance claims.

I've been in China three times in the past year. I agree that it is creepy not to be able to access Gmail, Facebook, Twitter, Youtube. But in terms of personal safety and freedom to walk around and see things, I felt very free. I feel more nervous walking around most North American cities' downtown area (LA, NYC, SF, Chicago). And also Mexico city.

Nervousness with regard to both police and local citizenry.

Russia is on a much lower ground ethically. China hasn't recently annexed territories from its neighbours whom they promised to protect and support just a few years ago.

In this aspect there is really no differences between these 3 countries, China hasn't started a war like US, but using it's power to push boundaries in both west and east.

Or helped shoot down a passenger jet...

Vladimir Putin is not our friend.

It isn't exactly a moral point but the recognition of two items: 1) the large number of people who are employed at cyber crime in certain parts of the world 2) the recognition that certain countries have a vested interest in reading the emails of the people of certain other countries

Certain people in various countries could make a similar argument about US hosted emails. This is how the world works.


If your mail is in Russia, FBI/DOJ will have an impossible time getting a warrant. Compare that to the US/EU mail hosters who willfully give your mail away.

Exactly I use it often

Mail delivery delays can get huge, that's what's wrong with Yandex. Mail routinely takes 10-20 minutes to arrive.

Please, send us headers for one of such messages to support@mail.yandex.com. You can get them this way https://yandex.com/support/mail/faq.xml#mail-titles.

Thanks, will do.

If you use PGP it doesn't matter where you host, right? If you're concerned about local officials having access to your data I don't think it's much worse than Google scanning my email to better target ads or "just to remind me" that I'm going at X or Y place in Z days.

From a purely technical and tinfoil-hat point of view, they'd have meta-data about your emailing habits. They wouldn't be able to read the mail for your encrypted traffic, but lets be honest - most people don't. So that mail could be read too.

compared to what?

A place that wasn't a police state of sorts with increasingly draconian legislation. Opposite end would possibly be Swiss or Icelanfic jurisdiction. I'd never use an email service in Russia. I have an untrusted, Gmail account for availability purposes. Confidential mail still encrypted and preferrably sent through Swiss account.

Thinking about switching untrusted one over to Fastmail. Still surveillance state but less surveillance and supposedly good uptime.

Fastmail has been my go-to for almost as long as they have existed--almost 15 years. There is NOTHING better in my estimation and I've used them all before--free and paid.

Rob and the Fastmail team are most likely the best in the world at what they do, and as an IT guy with decades of experience, I don't say that lightly. They have more than earned my shekels every year. Give them a go... you will be very pleasantly surprised.

I recall a story about Swiss company, Crypto AG: http://mediafilter.org/caq/cryptogate/

I've told the story many times. It can happen anywhere. It was an unscrupulous group of people in it solely for the money primarily selling to people like NSA. The target wasn't popular with anyone. Guy running it was offered some cash to sabotage the equipment of that target to help a major customer. "Happy to oblige!"

My endorsement of Switzerland is about legal jurisdiction, culture that respects privacy, and stability. Well-intentioned owners won't be forced to do the kinds of things we see in America or Russia. At worst, it will be something selective with a warrant. A nice foundation to build on.

Think about it this way. If you live in US and use gmaik, NSA spies on you and you have potential to be put away in terrorist military prison.

If you live in US but use yandex, Russian gov spies on you but you live in USA and therefore cannot be arrested as easily.

Russia's extensive use of online trolling means I'd be as worried about their government surveilling me.


I worked for Yandex. AMA. Will not violate NDA, obviously.

Are you able to say anything about the scale of their hardware? It says "3x hardware" but I'd love to hear how big that ends up being, at either the Oracle or Postgresql size.

Hardware for Oracle and PostgreSQL are different. We used to have big boxes for Oracle with lots of memory, SSD disks and rigid economy of CPU (small number of very fast cores because Oracle is licenced per CPU cores). And that hardware was really efficiently used in consuming nearly all system resources (CPU, I/O, disk space and even network bandwidth).

For PostgreSQL we use commodity servers and right now they are not really efficiently balanced (i.e. on some hosts we don't have enough disk space but other resources are utilized ~30%). So we are working on improving that. But I can't tell you exact numbers of hosts for Oracle or PostgreSQL (which is actually ~x3 from Oracle hosts number :)).

I actually like it, sure they read it but they can't get me and they won't allow usa snooping. I use it a lot

I feel the same way about Chinese IoT vendors - the cheap Xiaomi 'Little Ants' webcam that watches packages get dropped off on my front porch is routing notifications (and video, on demand - it stores stuff locally on an mSD) to servers in China, but it seems extremely unlikely that the PRC government has any interest in it, or use for it; and I doubt Xiaomi would bother giving it up to any other government.

Yandex is doing quite some interesting things, I think they don't get enough love from HN.

It's not a malicious bias, but a bias nonetheless. We tend to show more love towards services we know of, use, hear about, or have a friend work at.

That Yandex chiefly targets the Russian, other Cyrillic, and Turkish markets limits its exposure to people who aren't in those markets. And some of it has to do with the kitchen-sink nature of what Yandex does (a lot of tech and some media), in that it's easy to mistake them to be similar to other such companies (Yahoo, Verizon, AOL -- incidentally they're now all the same) that have a similar spread of services, and yet they're not generally complimented in their tech achievements.

Never heard that expression, kitchen-sink, what does it mean?

It technically means 'everything, whether necessary or not' [1][2], but the subtext and context I intended it in is that their range of services is forms an extremely wide spread, which could be mistaken for a lack of focus.

[1] https://en.wiktionary.org/wiki/everything_but_the_kitchen_si...

[2] http://dictionary.cambridge.org/us/dictionary/english/everyt...


Yeah ok but you could say the same thing about google and they're doing well. You have yandex mail, web analytics, ads etc. How is that different from what google is going? In fact, I would say that yandex is trying to google's roadmap on how to conquer the world and see if they can pull it off in russia with the advantage that they currently have there.

A few things:

- There are a lot of Googlers on HN, not so many Yandex-ers

- Google, in the general public's minds, is buoyed by positive mindshare that was generated a very, very long time ago (early-to-mid-2000s). Today's Google has appreciably changed from those days, and yet their likeability has mostly been unaffected.

- Google tried to make their lack of focus official by spinning off a corporate umbrella parent by the name 'Alphabet', like Philip Morris Companies the "tobacco" company renamed itself to 'Altria' in 2003 because at the time it owned 84% of Kraft Foods Inc., a large food and beverage company [1].

I maintain that to an American, Yandex is most analogous to Yahoo or AOL rather than Google, because Google is, as of writing, a company that makes money primarily from ads and is under transition to pursue higher revenue streams from business-to-business sources (like Google Cloud Services) more so than by brokering advertising. Whereas Yahoo or AOL are tech-media hybrids. I know Yahoo and AOL don't evoke such positive brand power anymore than, say, Google, but their structure I believe is more reflective of Yandex.

[1] https://en.wikipedia.org/wiki/Kraft_Foods_Inc#Philip_Morris_...

Rambler (rambler.ru) is more like Yahoo, not Yandex. Like Google, Yandex is more service-oriented, Rambler is more media-oriented. Rambler's numbers are also going down during many-many years, like Yahoo's. It alsotruggling from crazy rotations of C managers each few years during last ~15 years

Like for Google, the main revenue stream for Yandex is also ads – they have >50% in market share in Russia, context ads network and RTB. With good Partner Network accepting websites. From recent products, Yandex.Taxi is a huge success, also taking leadership on Russian market, beating Uber and Gett. So with strong leadership in search, ads and young taxi markets in Russia it doesn't even close to Yahoo in comparison, it's definitely "Russian Google".

I agree with you, and add that to a russian, the leader is yandex, not google. Now, I'm not saying that it's all relative, google is objectively be a better company in most respects. But in terms of the mindshare that you talk about, yandex beats google in russia.

Not really "trying Google roadmap", because in some areas (search, maps, mail) Yandex was first. They were not as successful, however, due to the focus on Russian market, management and technology specifics and business climate.

It's not really different, it's just that Google became well known when it only had Search.

it comes from the phrase "Everything but the Kitchen Sink." So "kitchen sink" becomes a shorthand reference for the above phrase, which itself means, "a whole bunch of random stuff all collected together"

Never seen any non-russian yandex user myself.

I don't see any other free providers of custom domain emails; so when a friend incorporated and wanted to have bob@bobscompany.com then I set him up with yandex - all "english market" services suddenly wanted a rather unreasonable business-class pricing for that simple thing.

as someone else mentioned: zoho. I think it's $2.5/month for unlimited domains connected to one mailbox.

Apparently, they are big in Turkey…

Tiny. 2%.

6, if you believe in comScore data.

No one believes comScore data... :)

Well, it's 8% if you believe in "independed measurement" :)

Hello. Nice to meet you.

I'm non-russian and I use Metrika, their analytics product.

Yandex doesn't seem like it would appeal to the kinds of people who comment here. If you think that having your data hosted in the USA is a privacy concern, imagine having it hosted in Moscow.

Let's talk Oracle vs Postgres rather than USA vs Russia

So - as a long time Oracle DBA (who also loves Postgres), I've been seeing a lot of companies do this sort of move.

There are a couple of downsides, but really - there's a lot more upsides to Postgres than downsides (IMO).

I'd love to hear more detail. We hear a lot from the PostgreSQL team and its fans (amongst whom i count myself!), but a lot less from people on the Oracle side.

Oh, as the benefits of Oracle vs Postgres?

So the biggest in my experience is not enterprise support. Folks generally expect it to be. The biggest is actually flexibility. Oracle DB is good at handling some edge-case scenarios, such as a RAC cluster with a failover cluster running an Advanced Dataguard Reporting instance in a whole different state.

Basically you are running a (super expensive) near real-time cluster that's doing compressed log-shipping to another cluster, and you have a standby on that secondary cluster that's open in a read-only mode. And none of it requires cron jobs or excessively complicated setup.

However... even though I've done stuff like this for a number of clients, there are SO MANY better ways to address these sorts of scenarios with creativity and elbow-grease that Postgres tends to be a better tool.

The caveat, of course, is when the person who sets it up leaves, and some other person has to take over. With Oracle's scenario, it's mostly just a slightly unusual config of a standard set of tools. The more creative Postgres solution might be quite a bit more of a learning curve.

This is just an example off the top of my head, but it is one that came up recently. A client had a shipping/receiving system that could not go down, so the 20k+ per node was trivial. To reduce the load for reporting, they just simply had us open a standby DB for read-only reporting on the failover cluster - and they liked that so much, we set up another one for them local to the offshore reporting team.

It's absolutely possible to do something like that with Postgres clustering, but there's a whole lot fewer folks doing it.

Actually, i was thinking more of the upsides to Postgres - apologies for not being clear.

I thought you were already a huge Postgres advocate? Or do you mean 'upsides to postgres that appeals to heavy oracle sites'? /edit Perhaps that wasn't you...

So #1 benefit of Postgres for Oracle shops is transition cost. It's cheaper than Oracle (by sometimes substantial amounts), and relatively painless to switch to. A lot of code can move unmodified, all the same platforms, and often even the same teams of folks in the same jobs. Postgres scales very well, at the very least it can handle 90% of a given company's Oracle load with no real issue. Things like clustering, backups, failovers, and reporting systems can be re-engineered to work in similar ways on Postgres, though some will require substantial hackery.

The #2 benefit is stability. Oracle has a very bad track record (especially lately) when it comes to patching and supporting older databases. If you just want a DB to sit in the corner and work for a year, with push-button easy security patches at most... Oracle isn't really the product anymore. Oracle CPU (critical patches) are cumulative now, and the bloat is getting out of hand. I recently just installed a set of patches that were almost an extra 5GB for the footprint... minor for a 20TB database, but ridiculous if you're thinking of thousands of small distributed hosts.

What's Oracle's patch deployment strategy now? The last time I had to deal with it was around 2008, when a security update was a tarball with a README file telling you which files to copy where. I've heard it's improved considerably since then.

Not a ton different, if you've used OPatch already. Basically it copies around files, runs a bunch of perl, runs a bunch of sql, and then you run the usual patch upgrade scripts at the end. The in-place patching is safer and smoother in 12c than it was in 11g, but the cumulative patching has really gotten out of hand.

For example - on a QA system we were running a couple of custom one-off patches delivered by Oracle Support. These blew up the entire patch process. I had to stop, back them out, re-patch, then re-apply the one-off patches.

Oh, and 12c is a seriously whiny bitch about XDB. Just for added fun.

You seem to think that having your data hosted in Moscow is a worse privacy concern than in USA? That's a joke. You realise that google just made location tracking mandatory for anyone using maps or app store, right? The USA is way worse in this respect. Or let me re-phrase that: we know for a FACT that the USA is very very bad in this respect, whereas Russian just has a bad reputation.

I absolutely think that data hosted in Moscow is a worse privacy concern, yes - because while I don't doubt the governments of both countries are spying on that data, I trust the US government far more than I trust the Russian government.

A useful series of tweets by @SwiftOnSecurity about why storing data in the US isn't a bad idea:


One word: jurisdiction. The GU might read my emails, but they don't have the authority to execute 4am raids on my property and kill me with impunity in the name of "officer safety" as an eventual consequence of something XKeyScore flagged.

At the end of the day, the people who have the guns and the authority to use them against you take a more prominent place in one's threat model than a foreign entity with no jurisdiction.

You trust the us gov? The same that has denied manning his/her rights, driven her to suicide and now hunger strike? The same that got sweden prosecutors to re-instate assange's rape accusations after they had been independently dismissed by the initial prosecutor, so they can extradite him to the us? Ok buddy, keep trusting the us gov.

I think you've been so thoroughly brain-washed by russian-hating american media that you can't even think straight any more. I realise that this isn't really an argument, but I already presented my argument (that with the usa gov you have facts, whereas the russian gov you have suspicions) and it didn't seem to make a difference, so i don't really care to continue this conversation.

Jesus Christ. No.

It ought to be obvious to anyone that applying black and white judgements in these situations is idiotic. Do I think the US government is beyond reproach? No. Do I think the US government is more open and accountable than the Russian government? Yes.


Yes, because the US government is open and accountable enough that this information is public. The same cannot be said for the Russian government.

Would Snowden agree with you?

That's actually a very interesting question. I don't know. And I doubt he is currently in a position to be very frank and honest about it.

There you go with your anti-russian bullshit again. Snowden talks shit about russia as well all the time. Why wouldn't he? Because putin comes and kill him, right?


Yes. In fact, I'd say that's the case even if your only concern is with the US government.

Virtually all of the controversy over US government actions with regard to local data is about violation of expectations (some of which are codified into law) about domestic surveillance by doing things which are uncontroversial in the context of foreign surveillance, and its not like the US lacks capacity or will for foreign surveillance.

That said, Russia doesn't even have the constraints (legal or mere expectations) on domestic surveillance that the US domestic actions raise concern for violating, so if your concern extends beyond US government surveillance, then that compounds the problem when hosting in Russia.

How many reporters and dissidents are murdered in the US versus Russia? Russia is way worse in civil rights and surveillance. Doesnt excuse America for the schemes they pull but it's definitely safer here unless you're in the hood.

Well, it's kind of safe for their citizens too, if they don't get involved in the things that get them killed, as those are not random things. But civil rights aside, online surveillance there is not yet at the level of reading everyone's emails and while they really want to do that they have a hard time implementing something like this for various reasons. It's definitely safer to use those online services, than the ones in your own country or allies.

How do you know that? Russia doesnt publish what Russian intelligence is doing. They also don't have a Snowden who leaked almost all their secrets. That we did is only reason you can make thag comparison. I'd love to see an equivalent dunp for Russia to do the comparison. ;)

It's not like that in Russia, they don't benefit from hiding surveillance, they do it openly. They had SORM for some time now, but only recently, like a month ago, created a law attempting to force local companies to implement backdoors and provide access to and store everything they have on users, etc.

The USA is probably bad, but hides it well. Russia makes no qualms about hiding it.

They moved most of their hosting to Amsterdam (and Stockholm?) 2-3 years ago, I heard. Since then the founder has been forced out by Putin, so I guess it might be back in Moscow now though. (Haven't really had any semi-inside information for the past year.)

now it's split between Russia (for russian audience) and Netherland and Finland (for others). Also this https://sputniknews.com/world/20151024/1029057531/finland-he... may be interesting for you.

My feeling is that data center services are much cheaper and more reliable in Europe, so they have probably moved most things that are not directly frontend-facing to CIS there.

(Everything just works and there are no insane bribes to pay to various mafias.)

Oh no, at least not now. Electricity and land so cheap today in Russia, what Yandex starts to build and sell their datacenter to provide "cheap AWS alternative for Russian companies".

But how about the bribes (to get e.g. electricity)?


Bribes for infrastructure is not how modern days high level corruption works in Russia. What happens is that for various preferences that a company receives from the state like tax rebates, cheaper electricity etc. one pays back to the official who authorized the deal. The payment (otkat or "rollback" in Russian) is typically indirect, like using a supplier or outsourcing to a company where the official or their relatives have a stake.

The direct bribes is really for low-level clerks like police officers or fire hazard inspector s.

Yandex is a public company with stocks on NASDAQ. Can you ever imagine how it is hard to hide bribes inside of public company budget? :)

Just give someone a big performance bonus with the understanding that part of it is to be used on bribes? Seems rather easy.

Nope, both salaries and bonuses (even on C level) is too low in Russia to hide any amount of bribe in them. Anyway, I never heard about bribes for electricity for last 5 years in Russia.


Let's say someone claims that their programmer genius Ivan saved a company ~1m USD/year with latest commit; and awards him $100k. And you see this unusual bonus, how are you going to know 99% of that doesn't go to someone other than Ivan?

Typical Ivan with good developer skills costs $30k/year in Moscow. But you are right, theoretically it is possible to hide some money if you really wants to. As far as I know there's no big non-gov companies who spend something on bribes right now. May be it's because I too believe in humankind.

I think you're an advanced version of a putinbot :).

He's an employee of Yandex, well-known in the Russian dev community.

Well-known? Thanks :)

Not very. International listings don't need to abide by normal US listing regulations for financial reporting.

Two words: "consultancy services"

You can check it on sec.gov. Yandex spend nothing on consultancy for 2014 and 2015 years

I can't be bothered to dig right now, but you're saying that a company with 5000+ employees did not use any consultants at all, zero, for 2014 and 2015? I really can't believe that.

Edit: there are many ways to hide bribes, of course. Typically it involves overpaying for something. It would be hard for even a non-corrupt tax authority to tell the difference between having bad negotiation skills and bribes.

My point is "big non-government companies in Russia doesn't pay any bribes". But you're right, it's definitely possible do hide some amount of bribe money if you really want to.

It was the Vkontakte founder, Pavel Durov, that was forced out. Yandex hasn't had any notable public conflicts with the Putin regime.

Except that it had to sell it's "golden" share to state-owned Sberbank.

Putin himself verbally attacked in 2014 [1]

[1] https://www.bloomberg.com/view/articles/2014-04-25/how-putin...

The founder has been driven out by Putin? Aren't you confusing with Parov, the founder of VK?

Yeah, I probably did mix them up.

I was possibly confused by this article just before posting that comment:


(It's hit #4 in Google for his name.)

> Parov, the founder of VK

It's Pavel Durov. https://twitter.com/durov

So having read through the slides, these are not bad slides, but I'm missing the 'real' wow-factor of why Postgres was a better choice in the end than Oracle. There's no doubt that the rewrite (which, because of PL/SQL -> PL/pgSQL, there was a rewrite) benefited them, but all I can glean from the slides is Postgres hasn't failed them yet and the rewrite worked.

So as much as I want to talk about Oracle vs. Postgres, these slides aren't giving me anything technical to debate.

1. Price if obvious reason of migration, and in their case (multiple datacenters and "-1 datacenter" policy) it was huge benefit.

2. The other concern was Oracle's slow bugfixing and reaction to feedback.

So, this is a good case showing how a company can stop wasting millions of $ for proprietary software, joining Open Source community, contributing to it and taking benefits from tech collaboration with core developers of the product.

So, they say 'shop.oracle.com' was their 'main reason' for switching but that reads as a half-joke, half-serious riff on Oracle's famous ability to extract money from people; the other salient points I can find are:

- PL/SQL was harder to deploy than PL/pgSQL (as stated later)

- Only synchronous interface in Oracle's OCCI

- Problems with Oracle's development environments

- Not very responsive support from Oracle.

But the dev environment and support for Postgres are not talked about.

Also I can't quite figure out if "3x more hardware" is supposed to be a good thing on their last slide. It's possible that it's a good thing (if it's, say, we can replicate to more servers), but to me it doesn't read like one.

Details were in the talk itself, watch the video https://www.youtube.com/watch?v=-SS4R1sFH3c. In particular, they mentioned how Tom Lane fixed a bug they found and did it just in 38 minutes.


Initial architecture with 2 servers was set up that way to save money on Oracle per-core licensing. They wanted to add more, but that was too costly.


Not having to deal with stuff like this while having the same features/performance/stability/etc may qualify as a 'wow':



I'm guessing it was a few million dollars worth of wow-factor.

Postgres is free

10 man-years are not though.

I'm a postgres fan and booster myself but I'd need a good reason to migrate a data backend on any project I'm managing.

I don't think you realize just how painfully expensive Oracle is. The reason they only went with an active/passive server layout vs. going active/active/active RAC is because they would've had to move to enterprise licensing. Then you're talking (at what I imagine their scale is) 7-8 figures a year.


So if you look at just enterprise licensing, it's $47k per "processor" - which in their world, a single intel core counts as 0.5 processors. So if you've got a 36 core machine (which I think they mentioned in the video) that's 18 "processors". Or $846,000 for one box. You want clustering (RAC) - that's another $315,000. Partitioning? $207,000. So now we're up to ~$1.4 million for a single server. YAY!

Obviously these are list prices, so you should be paying less than that, but it gets ugly FAST.

> 10 man-years are not though.

Judging by the salaries from job sites and my yandex interview, the cost would most certainly top up at $0.5 million (that's a very generous guess, probably half or two thirds that, realistically). Not sure how much Oracle costs at Yandex scale (a LOT more?).

Am I missing something? How do you figure 10 man years = $500K ? Or is Yandex super stingy? DBAs and Developers worth their salt (who could pull off a multi-TB migration/etc without significant outage), would easily each go for $140K/yr minimum (so, 180-200K fully loaded with benefits and such).

Still though: your point is valid in that, Oracle RAC cluster licensing and support is serious bucks.

Yes, you're missing the fact that it's Russia, not SV.

Couple of years ago ruble's exchange rates dropped significantly (it was ~33RUR for USD, now it's 64-65 rubles).

Monthly salaries for a good engineer in Moscow are usually 100-200k rubles, or 1.2-2.4M rubles/year.

It's only $18-36k per year. Looks insane from typical californian HR's point of view, isn't it?

Taxes take additional ~50% (in Russia, employer pays all the taxes and payments to the pension fund), so $500k was a pretty good estimate.

it wouldn't have been 10 man years if they had less PL/SQL. I don't understand why people keep falling into that trap of putting complex logic in stored procedures.

No, you are not right. In video (Q&A section in the end) you can hear that rewriting stored logic took a couple of weeks of one developer. Maybe a couple of month with good tests.

It took much more time to rewrite backends logic because of lots of legacy code in many applications without using abstraction libraries.

Oh, cool. Can you expand on that? What kind of logic had to change? I mean, SQL queries are SQL queries? Is that because you were using low level C Oracle drivers of some sort? Also if you could say a couple of words about the MySQL failure it would be rather helpful because I might have a friend who might be currently deciding on whether to migrate to MySQL or Postgres from Oracle.

We had several kinds of stored logic in Oracle: 1. for logical consistency of data (i.e. when you store new message you should increment counter for folder, thread, etc.). It seems correct to have such logic in the database rather than in the application. 2. for common code (i.e. moving a message from one folder to another may be done from web backend, mobile backend, imap backend or pop3 backend). This kind of logic is easier to have in one place than in all backends. 3. some business logic (i.e. finding thread for a new incoming message was done in PL/SQL).

First type is still in the database. Some part of type 2 was moved to our abstraction library (that is used by all applications). Logic of thrird type was moved to our applications and it greatly reduced stored code size.

About failed attempt of moving to MySQL. It was more than 10 years ago and I wasn't working at Yandex at that time but AFAIK the main problems were not around technical problems, they were organizational - not all members of the team agreed on moving to MySQL.


That's actually huge for companies that have invested in Oracle and feel trapped.

And the companies that buy Oracle, have a slight tendency to write all their core business logic inside the database.

Which is really the best place to put it in the end.

oh, it was a pretty funny message. Official one! With huge amount of grammar errors in Russian text :-)

Oracle licensing costs are ridiculous.

I am using Yandex for my FOSS project's domain. So far, I haven't encountered any big issues. The web interface is plain and simple, but occasionally has some glitches.

Their mobile app (Android) is quite pleasant and lightweight.

What might be a problem for non-Russian speaking people is the lack of documentation in English. Quite often a random link redirects to a page in Russian.

To stay on topic: Does anyone else have stories of moving massive datasets to postgres from other commercial databases?

We're looking to make the switch from SQL Server when 9.6 comes out (and proves to be more scalable).

SQL Server performs slightly better than PostgreSQL for analytics. This is mostly because of parallel queries which will still be more efficient on SQL Server when PostgreSQL 9.6 is released.

On the other hand, PostgreSQL can take advantage of ZFS compression which is great for archiving cold data.

SQL Server also comes with SSAS, if we're talking analytics. There's the very mature MDX-based Multidimensional mode, and a heavily optimized columnstore engine with the DAX-based Tabular mode.

For analytical workloads, SQL is often a poor choice.

Full disclosure, I work for an analytics and BI Microsoft partner. There are plenty of great alternative semantic layers beyond SSAS, but I am not as comfortable with each of those.

Minor note: ZFS compression is great for live data too -- lz4 compression has very low CPU overhead.

Yes and no. It depends on the use case, if you do a lot of updates, you do want a small recordsize(8kb) and if so, you do not benefit greatly from LZ4.

I'd also look into Citus (PG plugin for scaling)

> 3x more hardware # for Postgres

Tells you something about how insanely costly Oracle's licenses are. Whatever you say about Oracle though - Oracle DB is an incredibly well performing and reliable piece of software. (Attested by experience - we had several Oracle DBs running on HP-UX Itanium that only needed to be handled when OS and Oracle patches were needed. Massively used too - think 36 PA-RISC cores at peak, 24 IA64 ones. It also helps that there are lot of DBAs with great deal of Oracle experience.)

Nobody argues that Oracle is a very good database. But 3x more hardware and 10 man-years are an order of magnitude cheaper than Oracle licencies cost (even with maximum discount).

Yes that's the point I was trying to make. Licenses and support are an ongoing cost and from what I keep hearing oracle has increased them significantly in recent years.

Yandex.Mail is my favorite email provider and I would invite anyone reading this to give it a try.

Please cite examples as to why it's your favorite Email provider.

For starters it's not very crowded on usernames, so I got to use my 'namelastname' as username, also I don't think I've seen any ads so far. Another aspect of the service I really like, is that it has less magic than Gmail. I always get email labeled wrong in Gmail. At first I tried to label them properly and expected Google to get them right over time, now I don't bother. I would definitely pay for this service if this was an option. To be fair Russia along with the US, France & Japan is one of my favorite countries, so I might be very biased.


While I never tried yandex's email, I do share the impression that gmail is too magical and shouldn't be labelling my emails outside of the explicitly rules that i have set.


Haha same here! It makes me so sad the propaganda war that russia and the us are having. Those two countries are similar in so many ways. Russia was unfortunate that communism had a successful revolution, so they are currently some 30 years behind culturally :-/


That "unfortunate" revolution took Russia from the feudal age into the nuclear age in a single generation, so without it they'd presumably be about 100 years behind the USA.

This may explain why your negative opinion of the USSR/communism is not necessarily shared by Russians.

The most popular leaders of the 20th century, as voted by Russians, are Brezhnev, Lenin and Stalin, while the most unpopular are Gorbachev and Yeltsin, responsible for the collapse and break up of the USSR.

Only a rigged and unfair election prevented the Communist Party leader being elected president by the people of Russia in 1996, after they had experienced the joys of several years of capitalism.

In 2016, a quarter of a century after the breakup of the USSR (which happened against the express wishes of the Soviet people), the Communist Party is still the second most popular party in Russia, and most Russians (as demonstrated in repeated polls) would like to see the restoration of the USSR.

As for Yandex, the founder started in business in the USSR, and he is not Russian.


Brezhnev is more responsible for the collapse of the USSR than either Gorbachev or Yeltsin (or anyone else, except maybe Yuri Andropov.) Oh, sure, he was dead and the other two were around when it happened, but it was his intervention in Afghanistan that essentially sealed the fate of the USSR.

> Those two countries are similar in so many ways.

Like invading and annexing their neighbours?

Exactly. Although to be fair to the US, their military is quite a bit more powerful, so they've already pacified their neighbours and instead invade countries half way around the globe. Russia can't stretch itself as much. But yeah, same principle.

So, US invaded and annexed someone in past 50 years? How's the weather in your version of reality?

Not sure if trolling or just ignorant.


CTRL-F united states, 38 FUCKING results.

You can set up your own domain for free

Go on. They give you a domain? I never saw that... Link please?

You can link your domain to the mail. And have unlimited number of inboxes.


This is in Russian. Maybe there is English version


Just replace .ru with .com and everything else is the same, except the language.

I just registered and yandex mail really seems like a fine service.

не надо, спасибо огромный за сылку

No, they don't. But you can use Yandex.Mail with your own domain by configuring your DNS and setting up Yandex.Mail account. It's a great solution for small businesses and families.

I was reading that link. Do I have to use their DNS servers, or is it just enough to point the MX record to yandex?

I think they mean you can have email on a custom domain for free. Not that they give custom domains for free.

This seems good.

Not related with the migration itself but I tried to use yandex and its mailing service a couple of years ago when attempting to become google independent but it didn´t work well with spanish queries so I eventually went back to google. But I recall that yandex mail was really nice.

I look at the operations of a lot of middle market tech companies that are fairly old (10+ years in existence) and I'm starting to see more and more companies move off of Oracle and on to Postgres. I expect this trend to continue.

What version of Postgres did they end up using?

We started our first production systems with 9.3. When we started migration process and until it ended we used 9.4. Right now all our production clusters are 9.5.

I see that document is poorly formatted. Lacks sufficient indentation.


Personal attacks are not OK on Hacker News, regardless of what you're replying to.

Come on, it was funny...
