In the book to create a new house it typically has to be granted to you by a highlord or the king.
A knight winning favor may be then risen to a lord and given land.
As for your question whats to stop say a rich merchant to have land and set himself up, in Westeros it appears that all land is owned by the crown first, high lords second (Stark, Greyjoy, Lannister, Tyrell, others) then given out by highlords to the lesser lords.
Therefore as is typical in medieval Europe, especially Britain, there is no way to own your own land or to go into the woods and claim land to become a noble.
Now as a side note say you are a wealthy merchant, you may be able to buy yourself land from the king, highlord and receive a title for a price, this seems to follow Westeros style politics though I don't recall any instances of it happening as of yet.
Edit - also say you set yourself up as a lord without approval, your local lord or highlord will come and kill you for treason ASAP.
scifi.stackexchange.com/users/29220/Himarm
A run of the mill peasant can't declare himself lord of anything.
For the simple reason that he doesn't own the land he lives and works on .
A commoner rents the land from its owner, who is a lord.
Which is why being given lands is such a big deal in Westeros (and any feudal state).
A title is just an empty honorific without land.
There are countless knights running around in Westeros, desperately looking for employment because the vast majority of knights are not landed knights, i.e. the title didn't come with lands.
scifi.stackexchange.com/users/887/System Down
Westeros operates under a Feudal system .
In such a system, if anyone steps out of line, someone above them in the hierarchy will be there to... remind them of their place.
For example, a Lord might have a peasant executed for treason for declaring themselves a Lord.
In order for a new house to be established then, it would fall to someone with the proper authority to grant a family the titles.
We see this quite a bit over the course of the story.
For example, after the Battle of the Blackwater , many of the fighting men are granted honors, lands, titles, etc by the crown.
scifi.stackexchange.com/users/5798/Justin Ethier
In the Game of Thrones taking a house and giving yourself a name is more complicated than is described.
You have to find a area that no one owns.
You have to be able to protect it from other people and thieves.
And as Justin Either's answer says: a Lord might have a peasant executed for treason for declaring themselves a Lord.
You have to have the necessary forces.
You can see from here: On Dragonstone, Stannis declares he is the true heir of Robert Baratheon to the Iron Throne, as Joffrey has no true claim to the throne.
Since the king's death Stannis has been gathering what strength he can from the lords of the narrow sea and from Myrish and Lyseni sellswords, but his forces are too few to challenge the Lannisters in King's Landing You have to be able to defend the position you say you have.
Just anyone can't do that.
I mean Stannis barely did it.
In either position you have to have power.
I mean the series is called "Game of Thrones".
Its a game of power, you either fight for your position or you get dethroned.
scifi.stackexchange.com/users/28219/Pobrecita
There are actually two separate questions you ask: How does one found a House, and how does one become a lord.
There are many noble Houses that do not have lords.
Prominent examples at the start of ASoIaF are House Poole, the stewards of Winterfell, and House Clegane, whose head is known as Ser Gregor.
The general rule seems to be that only Houses that rule over lands and castles are called Lords, although the Cleganes prove there are exceptions.
Lordless Houses seem to be created primarily through knighthood.
Two characters, Davos and Rolly, obtain surnames of their own creation upon being knighted.
However, some knights seem to lack surnames (such as Ser Duncan the Tall or Gendry).
My guess would be that men knighted by high lords and kings are often also granted nobility and a surname, which would limit the number of nobility.
Typically, lordship and nobility is granted-- and can be stripped-- by the high lords of the Seven Kingdoms or the King himself.
Lordship can be conferred for pretty much any reason, but usually happens after another House has gone extinct or turned traitor.
Notably we only ever see nobles raised to lordship.
Examples include the Spicers (who are named the Lords of Castamere for their role in the Red Wedding) and the Royce cadet branch (granted lordship of the Gates of the Moon by Petyr Baelish to consolidate his rule).
The title Lord is also conferred on a select few in the realm who hold certain ranks.
The head of the Night's Watch or the Kingsguard is styled Lord Commander, and members of the small council are called Lords, even someone as lowborn as Varys.
scifi.stackexchange.com/users/30859/TenthJustice
What's to stop any old smallfolk from declaring himself Lord of (his dwelling), making a simple drawing, taking a last name, and sharing that name with all members of his family and their descendants? And even if that's not recognized, how would a highborn person do the same?
How does one found a house in Westeros?
scifi.stackexchange.com
<game-of-thrones><a-song-of-ice-and-fire>
scifi.stackexchange.com/questions/65038
scifi.stackexchange.com/users/31331/Entertainment
Typically, lordship and nobility is granted-- and can be stripped-- by the high lords of the Seven Kingdoms or the King himself. All land is owned by the Crown and the highlords and is distributed to the lesser Lords by both of these parties. Also, a house may be granted to a person upon being knighted and small folk are killed for treason for declaring themselves as Lords without prior approval. 
A new house in Westeros is typically granted to a person by highlords and kings. All land is owned by the Crown and the highlords and is distributed to the lesser Lords by both of these parties. A house may be granted to a person upon being knighted. Small folk are killed for treason for declaring themselves as Lords without prior approval. 