A better rule of thumb is to not exchange any cash at all, but use an ATM in the country of arrival to withdraw money, letting your home bank do the currency conversion
(so withdraw in HKD, not in EUR; the ATM itself will offer Direct Currency Conversion but that's usually a much worse rate for you).
It's also quite logical; HKD banknotes in the Euro zone have to be transported there, so they are more expensive.
That leaves the problem that you might have some HKD left when you leave the country; as long as it's not too much, I tend to keep it for the next visit.
Or you can buy something (souvenir, drinks) at the airport.
(I don't have any experience in whether it would be better to exchange it back to EUR in Hong Kong or in Europe, so I'll leave that part to another user.)
Anecdotal evidence: in some (Western/Northern European) countries you don't even need cash at all.
I recently visited Iceland and the only moment when I didn't have the option to pay by card was the offerings during the church service...
travel.stackexchange.com/users/28333/Glorfindel
There is no general rule.
At the moment Wikipedia lists 180 currencies being used throughout the world.
Do not expect banks (or even some small exchange offices) throughout the world to have all of them.
Actually, should I decide to travel tomorrow to some small country in Africa with substantial cash, I would probably have to change cash twice .
At home in the Czech republic from our crowns to euros or dollars (or something like that, internationally known) and then after arriving from euros or dollars to local currency.
Because there is no option to obtain that exotic currency here at home and after arrival I would have probably hard time trying to find someone willing to buy Czech crowns.
If you are lucky and can choose, my tip would be to choose the more recognized and respected currency.
If all goes well, it does not really matter what you have, but if there is some natural disaster (like volcano eruption) or something and your airplane gets diverged to an airport in some third country where you get stuck for days or even weeks, you definitely want to have locally recognized cash to get by and
not somethings locals have never seen not heard of.
travel.stackexchange.com/users/85354/JJZ
I concur with Glorfindel , the best option is usually just to withdraw at local ATMs and use your card directly for most payments.
A few caveats, though: You must of course have a card that can be used internationally and widely accepted (should be the case of all VISA, MasterCard and Maestro cards).
Check your card issuer's fees for withdrawals, payments and the exchange rate they use.
It can vary quite substantially.
Note that in most cases, it's cheaper to pay rather than withdraw.
Some banks may have an international presence or international partners where fees are lower or waived.
Make sure you inform your bank that you are travelling abroad.
Some have dedicated interfaces for that on their online banking service (you specify the country and travel dates), others will require you call them.
Check that the ATM you are planning to use does indeed take the right type of card.
There are stories of cards "swallowed" because they're the wrong type.
Take a bit of your own local money to exchange just in case
there's a problem.
Usually, refuse the offer for currency conversion by the ATM or payment terminals, though of course you should compare with the rate your bank offers.
In some countries, some of the ATMs may add their own fee when withdrawing.
This is becoming quite common in the US.
Make a note of your card details and emergency contact before you leave.
In most cases, fees (by your bank or by the ATM, other than currency exchange) are usually a fixed amount per transaction, so larger withdrawals/payments cost you less, but YMMV.
travel.stackexchange.com/users/30703/jcaron
In my experience, if I have to change currency (and cannot withdraw cash as the other answers suggest), usually the rates seemed better at the country of departure, probably because you can choose to do it outside of airports, so more choice means better rates.
However, the one exception I've encountered was Singapore: exchanging Euros there seemed much, much better than buying Singapore dollars in Europe.
But Singapore is a hub country full of traveling foreigners, which probably explains it.
If Hong Kong follows a similar pattern, then it could be better to sell euros there, instead of buying HKD in Europe.
travel.stackexchange.com/users/8078/anol
Not a big deal but
Just one point nobody mentioned...
Say you're in Local country and you have (exactly) 500 units (5x 100 notes).
When you go to Distant Country, you can give them the 500 and get in return in your hand the precise amount of local currency, to the cent (peso .. yen .
whatever) including small change.
If you change in Local country, they don't keep small change in Distant Currency.
You get annoying rounding issues.
In general for this reason, and the rate, you always change "there" not "here".
(If, for some reason, you archaically are dealing in cash.)
travel.stackexchange.com/users/19233/Fattie
As a general rule, can one say that it is better to exchange cash for a foreign trip in the country of departure or of arrival? My main concern here is a favorable exchange rate.  As a concrete case, I am planning a trip to Hong Kong and am wondering whether I should carry EUR or (substantial) HKD.
General rule for where to exchange cash: country of departure or destination?
travel.stackexchange.com
<money><exchange>
travel.stackexchange.com/questions/123831
travel.stackexchange.com/users/73394/Drux
You may be able to find a better exchange rate in your home country since you don't have to rely on the rate at the airport, but they may not have exact change available in that currency. The best option is often to use an ATM to withdraw cash, and to use cards for most payments. Though there is no general rule, some would prefer to exchange their  money at the destination as there are certain benefits to doing this such as; being able to get all of your money exchanged and potentially receiving better rates. For others they would rather withdraw money at a local ATM in the destination country.
A better rule of thumb would be to withdraw money from your distination via ATM. There is no general rule to where you should exchange currency. You do get certain perks by exchanging your money at your destination i.e you occassionally get better rates and you can get every bit of your money converted instead of it being rounded off.