The unidentified man, charged with falsely defecting so as to gather information, was jailed for four years.
He had spent a decade in China tracking down North Korean defectors before coming to the South, the court said.
South Korean media also said he had admitted trying to organise a hit-and-run accident targeting Kim Jong-nam.
The South Korean court said that the 50-year-old man - who has a son who still lives in the North - became a spy after he was threatened by North Korea's spy agency.
He defected to South Korea this year, citing poverty, but later told investigators that he was a spy.
Local media reports citing prosecutors say the man also admitted he had been told by North Korean authorities to attack Kim Jong-nam.
He reportedly went as far as hiring a taxi driver to run Mr Kim over in 2010, but the plot went no further.
Kim Jong-nam is thought to have fallen out of favour with Kim Jong-il in 2001 after he was caught trying to sneak into Japan using a false passport. He told officials that he was planning to visit Tokyo Disneyland.
Bypassed in favour of his youngest half-brother for succession, the eldest son of Kim Jong-il has maintained a low profile overseas. He was quoted by Japanese media in 2011 as saying he opposed ''dynastic succession''.
He was thought to have been living in Macau but media reports indicate he may have moved to Singapore.
The court said that it had taken the spy's co-operation into consideration. But it said "stern punishment" was required given the extent to which he could have "greatly compromised the country".
The court added that he "seriously violated human rights of North Korean defectors", as he was trying to get them to return to their country.
A South Korean court has jailed a North Korean spy reportedly ordered to attack the eldest son of late leader Kim Jong-il, officials say.