The anti-migrant and anti-Islam Alternative fuer Deutschland (AfD) could get more votes than Mrs Merkel's CDU in Mecklenburg-West Pomerania.
This would weaken Ms Merkel ahead of national elections next year.
However, all Germany's other parties have ruled out forming a governing coalition with the AfD.
So the party stands no chance of forming a government in the state.
'Rapefugee' message in Rostock as AfD challenges Merkel
Germany jolted by AfD success
Is Merkel really on the ropes?
Profile: AfD leader Frauke Petry
But it has seen huge gains in various regional elections over the last year.
Mecklenburg-West Pomerania, in the former East Germany, is where the chancellor's own constituency is located.
Polls show her Christian Democrats (CDU) polling behind the AfD, and Ms Merkel told voters in the state: "It's going to be a tight race - every vote counts."
One voting intentions poll showed the centre-left Social Democrats first with 28%, while the AfD edged ahead of the CDU with 23% to 20%.
The AfD is believed to be attracting voters away from the centre-right CDU.
Party candidate Leif-Erik Holm said: "We hope to become the strongest party in Mecklenburg-West Pomerania."
Germany is taking in large numbers of refugees and migrants - 1.1 million last year - and anti-immigrant feeling has increased.
The AfD, initially an anti-euro party, has become the outlet of choice for voters frustrated by the chancellor's policy of welcoming migrants.
Its slogan nationwide is: "Let's end asylum chaos."
But on Saturday Ms Merkel told Bild newspaper:  "We did not reduce benefits for anyone in Germany as a result of the aid for refugees. In fact, we actually saw social improvements in some areas.
"We took nothing away from people here. We are still achieving our big goal of maintaining and improving the quality of life in Germany."
Only 2% of migrants arriving in Germany have gone to live in Mecklenburg-West Pomerania.
Voters in a north-east German state are going to the polls in a vote seen as a test of Chancellor Angela Merkel's policies towards migrants and refugees.