In that order (my opinion, of course): Tires.
Bad tires suck your energy and are prone to flats.
Good tires can make your bike fly, sometimes even act as a suspension.
As an extra, tires aren't actually part if the bike, so if you have good tires, you can use it in other bike(s); Brakes and brake levers.
This is safety and comfort.
Bad brakes can make your hands sore, if you have to squeeze them too much to stop.
Good brakes give you much more control of the bike than anything else.
Good brakes are THE things that allow you to go really fast if you want/need. Saddle and grips.
These are contact points on the bike.
In the end, all the bike-rider interaction flows through them (and the shoes you use to pedal, but they are not part of the bike).
These parts alone can turn a very nice ride into a very painful one, and vice-versa.
I think if these are OK, it's possible to live with a relatively cheap bike, provided it fits and don't require too much mainteinance, which are also required for a nice biking experience.
bicycles.stackexchange.com/users/2355/heltonbiker
In a bike that costs, new, only 150GBP, you have to ask where the savings will have been made and the easy first answer is: everywhere.
The frame will likely not be massively stiff and will be fairly fragile in comparison with its more expensive peers.
Steel is fairly forgiving giving a comfortable ride, but you don't want too much give.
The wheels will probably need attention too, expect the tyres to be thin and offer little puncture protection, the rims will be weak and probably lose their trueness relatively quickly and the hubs and bearings will need TLC quite soon too.
Expect the chain to be thinner and stretchier than you'd like and the rear cog to lose its bite quickly.
The brake compound will be soft and disappear fast and the cables won't hold their tension for long.
Obviously these are all worse case scenarios -
but given that you could spend the purchase price on just a pair of tyres, a pair of inner tubes and two pairs of brake pads - a bike at this price isn't necessarily the saving you might hope for.
In order to get from A to B, this bike would work, but if A and B are too far apart
and you're making that journey fairly frequently, then it might not work for that long.
But to make a real difference, well, send it back and spend a little more up front.
In general all new bikes make some concessions to price, but they'll be more than the sum of their parts.
The manufacturers pay less than you'd be able to buy the components in the shop, so you'd be better off finding a better bike to start with.
As the old Irish saying goes, "if you want to get there, I wouldn't start from here."
bicycles.stackexchange.com/users/425/Unsliced
First thing to do is to get the bike completely tuned up -- wheels trued, bearings adjusted, shifters adjusted, brakes adjusted.
Changing out the brake pads (the ones supplied are often too hard or too soft) can often improve braking effectiveness.
Depending on the bike design, adjusting brake lever position may or may not be possible and may or may not improve riding comfort and brake effectiveness.
And, of course, adjust the seat and handlebar.
If the handlebar can't be adjusted upward enough or has the wrong "reach" (probably even more common with expensive bikes)
then replacement parts are available to fix this.
Good quality tires, appropriate to your riding style, are a fairly cheap investment.
The right seat (which is something of a personal choice) can make a big difference in comfort.
Your critical factors are comfort, stability, braking effectiveness, and ease of use (eg, shifters that work reliably, etc).
Plus basic bike "fit" to your body.
Many of the features of fancier bikes -- light weight, exotic materials, bladed spokes, et al -- are vastly overrated and will make little difference in your riding experience.
bicycles.stackexchange.com/users/1584/Daniel R Hicks
The cheapest thing to upgrade on a bicycle is the motor.
It costs as much as a bit of extra food and time.
Ride a lot.
Ride up big hills a lot.
Go as fast as you can.
Edited because some dumbass downvoted this most brilliant answer: None other than Eddy Mercx, (who won the Tour de France 5 times and once cycled 49.431 km in an hour), once said: Don't buy upgrades, ride up grades.
QED
bicycles.stackexchange.com/users/2824/Darth Egregious
Nice thing about cheap bikes is that they cost less than the sum of their components, and when you ride them, they get you from a-b just like the expensive bike.
The cheapest upgrade is to adjust and grease the bearings properly.
While doing this, the bearings with cages can be replaced with loose ball bearings giving more bearing contact points.
Adjust the wheels properly for true and spoke tension.
I you learn to do this yourself
it costs almost nothing, just need a spoke wrench, and a cheap bike is a good one to learn skills on.
I put chain adjusters on my fixie's rear axles to help align the wheel while setting the chain tension, there are some very inexpensive ones on the web.
Tools needed to work on your own bike are always a good investment.
bicycles.stackexchange.com/users/3849/bayc
I have to add another option to the answer.
The most cost effective way to improve a bicycle is sell it and buy a better bike.
This is almost always a better option for all bikes, from cheap to expensive, except custom builds.
As a rule a bicycle is worth a lot less than the cost of its parts.
Unless you can get parts for significantly less than normal prices, upgrading the entire bike is more cost effective.
If you have a particular desire to keep the bike and upgrade it, often the most cost effective upgrade is a donor bike, unfortunately this requires a large outlay up front.
The most expensive way to upgrade a bike is piece by piece as you can afford it.
bicycles.stackexchange.com/users/3924/mattnz
What are things to improve on a cheap bicycle that make a real difference? I'm looking at the bang for buck tradeoff.   The bike is an inexpensive Fixie/Track Bike (less than 150GBP).    11 kg steel frame  Flat bar  Front and rear V-Brakes
Best bang for the buck to improve a cheap fixie?
bicycles.stackexchange.com
<maintenance>
bicycles.stackexchange.com/questions/7992
bicycles.stackexchange.com/users/3068/will
Cheap bikes are often not worth it as putting them right will cost more than the bike itself. It is often better to invest in a more expensive bike. However, if you must fix up a cheap bike, there are a number of ways to improve it. The major things to look at are tires, brakes, brake pads, a good saddle, adjust brakes, bearings and shifters, adjust seat and handlebar. Buying a better quality bike may be cheaper than upgrading a cheap bike with expensive individual parts. Another cheaper option is to use parts from a donor bike. For the best value upgrades, good quality tyres are more durable, comfortable and efficient, and can be used on other bikes. Good brake levers and pads give you better control and safety. A comfortable saddle and grips will make a big difference. Proper tools will allow you to maintain your bike. Completely tune the bike and adjust the positioning of the seat, handlebars and brake levers if possible for the best fit.
Buying a better quality bike may be cheaper than upgrading a cheap bike with expensive individual parts. Another cheaper option is to use parts from a donor bike. Good quality tyres are more durable, comfortable and efficient, and can be used on other bikes. Good brake levers and pads give you better control and safety. A comfortable saddle and grips will make a big difference. Proper tools will allow you to maintain your bike. Completely tune the bike and adjust the positioning of the seat, handlebars and brake levers if possible for the best fit.