After fighting the longest war in its history, the US stands at the brink of defeat in Afghanistan. How could this be possible? How could the world’s sole superpower have battled continuously for more than 16 years – deploying more than 100,000 troops at the conflict’s peak, sacrificing the lives of nearly 2,300 soldiers, spending more than $1tn (£740bn) on its military operations, lavishing a record $100bn more on “nation-building”, helping fund and train an army of 350,000 Afghan allies – and still not be able to pacify one of the world’s most impoverished nations? So dismal is the prospect of stability in Afghanistan that, in 2016, the Obama White House cancelled a planned withdrawal of its forces, ordering more than 8,000 troops to remain in the country indefinitely.

In the American failure lies a paradox: Washington’s massive military juggernaut has been stopped in its steel tracks by a small pink flower – the opium poppy. Throughout its three decades in Afghanistan, Washington’s military operations have succeeded only when they fit reasonably comfortably into central Asia’s illicit traffic in opium – and suffered when they failed to complement it.

It was during the cold war that the US first intervened in Afghanistan, backing Muslim militants who were fighting to expel the Soviet Red Army. In December 1979, the Soviets occupied Kabul in order to shore up their failing client regime; Washington, still wounded by the fall of Saigon four years earlier, decided to give Moscow its “own Vietnam” by backing the Islamic resistance. For the next 10 years, the CIA would provide the mujahideen guerrillas with an estimated $3bn in arms. These funds, along with an expanding opium harvest, would sustain the Afghan resistance for the decade it would take to force a Soviet withdrawal. One reason the US strategy succeeded was that the surrogate war launched by the CIA did not disrupt the way its Afghan allies used the country’s swelling drug traffic to sustain their decade-long struggle.

Despite almost continuous combat since the invasion of October 2001, pacification efforts have failed to curtail the Taliban insurgency, largely because the US simply could not control the swelling surplus from the country’s heroin trade. Its opium production surged from around 180 tonnes in 2001 to more than 3,000 tonnes a year after the invasion, and to more than 8,000 by 2007. Every spring, the opium harvest fills the Taliban’s coffers once again, funding wages for a new crop of guerrilla fighters.

At each stage in its tragic, tumultuous history over the past 40 years – the covert war of the 1980s, the civil war of the 90s and its post-2001 occupation – opium has played a central role in shaping the country’s destiny. In one of history’s bitter ironies, Afghanistan’s unique ecology converged with American military technology to transform this remote, landlocked nation into the world’s first true narco-state – a country where illicit drugs dominate the economy, define political choices and determine the fate of foreign interventions.

During the 1980s, the CIA’s secret war against the Soviet occupation of Afghanistan helped transform the Afghani-Pakistani borderlands into a launchpad for the global heroin trade. “In the tribal area,” the US state department reported in 1986, “there is no police force. There are no courts. There is no taxation. No weapon is illegal … Hashish and opium are often on display.” By then, the process of guerrilla mobilisation to fight the Soviet occupation was long under way. Instead of forming its own coalition of resistance leaders, the CIA had relied on Pakistan’s powerful Inter-Services Intelligence agency (ISI) and its Afghan clients, who soon became key players in the burgeoning cross-border opium traffic.

The CIA looked the other way while Afghanistan’s opium production grew from about 100 tonnes annually in the 1970s to 2,000 tonnes by 1991. In 1979 and 1980, just as the CIA effort was beginning to ramp up, a network of heroin laboratories opened along the Afghan-Pakistan frontier. That region soon became the world’s largest heroin producer. By 1984, it supplied a staggering 60% of the US market and 80% of the European. Inside Pakistan, the number of heroin addicts surged from near zero (yes, zero) in 1979 to 5,000 in 1980, and 1.3 million by 1985 – a rate of addiction so high the UN termed it “particularly shocking”.

Facebook Twitter Pinterest A farmer sowing poppy seeds in Nangarhar province, Afghanistan. Photograph: Sean Smith/The Guardian

According to a 1986 state department report, opium “is an ideal crop in a war-torn country since it requires little capital investment, is fast growing and is easily transported and traded”. Moreover, Afghanistan’s climate was well suited to growing poppies. As relentless warfare between CIA and Soviet surrogates took its toll, Afghan farmers began to turn to opium “in desperation”, since it produced “high profits” that could cover rising food prices. At the same time, the state department reported that resistance elements took up opium production and trafficking “to provide staples for [the] population under their control and to fund weapons purchases”.

As the mujahideen guerrillas gained ground against the Soviet occupation and began to create liberated zones inside Afghanistan in the early 1980s, the resistance helped fund its operations by collecting taxes from peasants who grew the lucrative opium poppies, particularly in the fertile Helmand valley. Caravans carrying CIA arms into that region for the resistance often returned to Pakistan loaded down with opium – sometimes, reported the New York Times, “with the assent of Pakistani or American intelligence officers who supported the resistance”.

Charles Cogan, a former director of the CIA’s Afghan operation, later spoke frankly about the agency’s choices. “Our main mission was to do as much damage as possible to the Soviets,” he told an interviewer in 1995. “We didn’t really have the resources or the time to devote to an investigation of the drug trade. I don’t think that we need to apologise for this … There was fallout in term of drugs, yes. But the main objective was accomplished. The Soviets left Afghanistan.”

Over the longer term, the US intervention produced a black hole of geopolitical instability that would never again be sealed or healed. Afghanistan could not readily recover from the unprecedented devastation it suffered in the years of the first American intervention. As the Soviet-Afghan war wound down between 1989 and 1992, the Washington-led alliance essentially abandoned the country, failing either to sponsor a peace settlement or finance reconstruction.

While Washington turned away from Afghanistan to other foreign policy hotspots in Africa and the Persian Gulf, a vicious civil war broke out in a country that had already suffered, between 1979 and 1989, some 1.5 million dead, about 10% of the country’s population. During the years of civil strife among the many well-armed warlords the CIA had left primed to fight for power, Afghan farmers raised the only crop that ensured instant profits: the opium poppy. Having multiplied twentyfold during the covert-war era of the 1980s, the opium harvest would more than double again during the civil war of the 1990s.

In this period of turmoil, opium’s ascent is best understood as a response to severe damage from two decades of destructive warfare. With the return of some three million refugees to a war-ravaged land, the opium fields were an employment godsend, requiring nine times as many labourers to cultivate as wheat, the country’s traditional staple. In addition, only opium merchants were capable of accumulating capital rapidly enough to be able to provide poor poppy farmers with much-needed cash advances, which often provided more than half their annual income. That credit would prove critical to the survival of many impoverished villagers.

In the civil war’s first phase, from 1992 to 1994, ruthless local warlords combined arms and opium in a countrywide struggle for power. Later, Pakistan threw its backing behind a newly arisen Pashtun force, the Taliban. After seizing Kabul in 1996 and taking control of much of the country, the Taliban regime encouraged local opium cultivation, offering government protection to the export trade and collecting much-needed taxes on both the opium harvested and the heroin manufactured. UN opium surveys showed that, during the Taliban’s first three years in power, Afghanistan’s opium crop accounted for 75% of world production.

In July 2000, however, as a devastating drought entered its second year and hunger spread across Afghanistan, the Taliban government suddenly ordered a ban on all opium cultivation, in an apparent appeal for international acceptance. A subsequent UN crop survey of 10,030 villages found that this prohibition had reduced the harvest by 94%.

Three months later, in September 2000, the Taliban sent a delegation to UN headquarters in New York to trade upon the country’s continuing drug prohibition in a bid for diplomatic recognition. Instead, the UN imposed new sanctions on the regime for protecting Osama bin Laden. The US, on the other hand, actually rewarded the Taliban with $43m in humanitarian aid, even as it seconded UN criticism over Bin Laden. Announcing this aid in May 2001, secretary of state Colin Powell praised “the ban on poppy cultivation, a decision by the Taliban that we welcome”, but still urged the regime to end “their support for terrorism; their violation of internationally recognised human rights standards, especially their treatment of women and girls”.

After largely ignoring Afghanistan for a decade, Washington “rediscovered” the country in the aftermath of the 9/11 terrorist attacks. In October 2001, the US began bombing the country, and then, with the support of British forces, launched an invasion spearheaded by local warlords. The Taliban regime collapsed with a speed that surprised many government officials. In retrospect, it seems likely that its opium prohibition was a crucial factor.

To an extent not generally appreciated, Afghanistan had, for two full decades, devoted a growing share of its resources – capital, land, water and labour – to the production of opium and heroin. By the time the Taliban banned cultivation, its agriculture had become little more than an opium monocrop. The drug trade accounted for most of its tax revenues, much of its export income, and a significant share of its employment.

Facebook Twitter Pinterest British troops patrol Helmand province in southern Afghanistan. Photograph: Getty

The Taliban’s sudden opium eradication proved to be an act of economic suicide that brought an already weakened society to the brink of collapse. A 2001 UN survey found that the ban had “resulted in a severe loss of income for an estimated 3.3 million people”, about 15% of the population. In this context, it became, according to the UN, “easier for western military forces to persuade rural elites and the population to rebel against the regime”.

In little more than a month, the lethal US bombing campaign, combined with ground attacks by its warlord allies, smashed the Taliban’s weakened defenses. But the longer-term US strategy would plant the seeds, quite literally, for the Taliban’s surprising revival just four years later.

While the American bombing campaign raged throughout October 2001, the CIA shipped $70m in cash into the country to mobilise its old cold war coalition of tribal warlords for the fight against the Taliban, an expenditure President George W Bush would later hail as one of history’s biggest “bargains”. To capture Kabul and other key cities, the CIA put its money behind the leaders of the Northern Alliance, an ethnic Tajik force that had fought the Soviets in the 1980s and then resisted the Taliban government in the 1990s. They, in turn, had long dominated the drug traffic in the area of north-east Afghanistan that they controlled during the Taliban years. The CIA also turned to a group of rising Pashtun warlords along the Pakistan border who had been active as drug smugglers in the south-eastern part of the country. As a result, when the Taliban collapsed, the groundwork had already been laid for the resumption of opium cultivation and the drug trade on a major scale.

Once Kabul and the provincial capitals were taken, the CIA quickly ceded operational control to allied military forces and civilian officials. In the years to come, those forces’ inept drug-suppression programmes would cede the heroin traffic’s growing profits first to the warlords and, in later years, largely to Taliban guerrillas. In a development without historical precedent, illicit drugs would be responsible for 62% of the country’s 2003 gross domestic product (GDP).

But for the first few years of the occupation, according to a 2007 New York Times report, defense secretary Donald Rumsfeld reportedly “dismissed growing signs that drug money was being funnelled to the Taliban”, while the CIA and the military “turned a blind eye to drug-related activities by prominent warlords”.

In late 2004, after nearly two years of outsourcing opium control to its British allies and police training to the Germans, the White House was suddenly confronted with troubling CIA intelligence suggesting that the escalating drug trade was fuelling a revival of the Taliban. Backed by George W Bush, secretary of state Colin Powell then urged a forceful counter-narcotics strategy for parts of rural Afghanistan, including the same kind of aggressive aerial defoliation then being used against Colombia’s illicit coca crop. But the US ambassador to Afghanistan, Zalmay Khalilzad, resisted this approach, seconded by his local ally Ashraf Ghani, then the country’s finance minister (and, since 2014, its president), who warned that such an eradication program would mean “widespread impoverishment” in the country, without $20bn in foreign aid to create a “genuine alternative livelihood”. As a compromise, Washington came to rely on private contractors such as DynCorp to train Afghan teams to eradicate drugs. By 2005, however, that effort had, according to New York Times correspondent Carlotta Gall, already become “something of a joke”.

By 2007, the UN’s Afghanistan Opium Survey found that the country’s then-record opium harvest of approxinamtely 8,200 tonnes provided 93% of the world’s illicit heroin supply. Significantly, the UN stated that Taliban guerrillas have “started to extract from the drug economy resources for arms, logistics, and militia pay”. In 2008, the rebels reportedly collected $425m in “taxes” levied on the opium traffic, and with every harvest they made enough funds to recruit a new crop of young fighters from the villages. Each of those prospective guerrillas could count on monthly payments of $300 – far above the wages they would have made as agricultural laborers.

To contain the spreading insurgency, Washington decided to commit 40,000 more US combat troops to Afghanistan in mid-2008, raising allied forces to 70,000. Recognising the crucial role of opium revenues in Taliban recruitment, the allied command was also fielding specialist teams that used development aid to encourage drug-suppression efforts in poppy-rich provinces. It was a fortuitous moment. That record harvest in 2007 had created an opium surplus that depressed prices, while simultaneous food shortages made wheat a competitive crop. As farmers used foreign aid to plant food crops in key areas of Helmand and Nangarhar provinces, the country’s poppy cultivation slipped from a record 200,000 hectares (494,000 acres) in 2007 to just 123,000 two years later – still sufficient, however, to sustain the Taliban. Meanwhile, inept and heavy-handed attempts to suppress the drug trade only ended up increasing opposition to the US and its allies.

By 2009, the guerrillas were expanding so rapidly that the new Obama administration opted for a “surge” of US troop strength to 102,000 in a bid to cripple the Taliban. After months of rising deployments, President Obama’s big breakthrough strategy was officially launched in the darkness before dawn on 13 February 2010, at Marja, a remote market town in Helmand province. As waves of helicopters descended on its outskirts spitting up clouds of dust, hundreds of marines sprinted through fields of sprouting opium poppies toward the village’s mud-walled compounds. Though their targets were the local Taliban guerrillas, the marines were in fact occupying one of the capitals of the global heroin trade.

A week later, Gen Stanley McChrystal choppered into town with Karim Khalili, Afghanistan’s vice-president. They were there for the media rollout of new-look counter-insurgency tactics that were, the general told reporters, certain to pacify villages such as Marja. The local opium traders, however, had other ideas. “If they come with tractors,” one Afghan widow announced to a chorus of supportive shouts from her fellow farmers, “they will have to roll over me and kill me before they can kill my poppy”. Speaking by satellite telephone from the region’s opium fields, one US embassy official told me: “You can’t win this war without taking on drug production in Helmand province.”

Facebook Twitter Pinterest Men smoking heroin in Quetta, Pakistan, near the border with Afghanistan. Photograph: Reuters

By attacking the guerrillas but failing to eradicate the opium harvest that funded new insurgents every spring, Obama’s surge soon faltered. Amid the rapid drawdown of allied forces to meet Obama’s politically determined deadline of December 2014 for “ending” all combat operations, a marked reduction in air operations allowed the Taliban to launch mass-formation offensives, which killed record numbers of Afghan army troops and police.

At the time, John Sopko, the special inspector for Afghanistan, offered a telling explanation for the Taliban’s survival. Despite the expenditure of a staggering $7.6bn on “drug eradication” programs during the previous decade, he concluded that, “by every conceivable metric, we’ve failed. Production and cultivation are up, interdiction and eradication are down, financial support to the insurgency is up, and addiction and abuse are at unprecedented levels in Afghanistan”.

As the 2014 opium crop was harvested, fresh UN figures suggested that production levels were approaching the country’s 2007 high. In May 2015, having watched this flood of drugs enter the global market as US counter-narcotics spending climbed to $8.4bn, Sopko tried to translate these developments into a comprehensible all-American image. “Afghanistan,” he said, “has roughly 500,000 acres, or about 780 square miles, devoted to growing opium poppy. That’s equivalent to more than 400,000 US football fields – including the end zones”.

During Afghanistan’s 2015 fighting season, the Taliban decisively seized the combat initiative, and opium seemed ever more deeply embedded in its operations. In October 2015, the UN released a map showing that the Taliban had “high” or “extreme” control in more than half the country’s rural districts. Within a month, the Taliban unleashed offensives countrywide that were aimed at seizing and holding territory. Not surprisingly, the strongest attacks came in the poppy heartland of Helmand province, where half the country’s opium crop was then grown.

In 2016, 15 years after Afghanistan was “liberated”, and in a significant reversal of the Obama administration’s drawdown policies, Washington launched a mini-surge by “hundreds” of new US troops into Helmand province to deny insurgents the “economic prize” of the world’s most productive poppy fields. Despite support from US airpower and 700 special-operations troops, in February and March 2016 embattled Afghan government forces retreated from two more districts, leaving the Taliban largely in control of 10 of the province’s 14 districts.

With its forces demoralised and the Taliban fielding aggressive fighters equipped with night-vision and sophisticated weapons, US airstrikes became the Afghan government’s last, tenuous line of defence. And in a tacit admission of failure, the Obama administration ended its planned withdrawal in June 2016, allowing US forces to move beyond advising and rejoin actual combat, and announcing, a month later, that 8,400 troops would remain there for the foreseeable future.

In Helmand and other strategic provinces, the Afghan army seemed to be losing a war that was now driven – in ways that eluded most observers – by a battle for control of the country’s opium profits. In Helmand province, both Taliban rebels and provincial officials are locked in a struggle for control of the lucrative drug traffic. “Afghan government officials have become directly involved in the opium trade”, the New York Times reported in February 2016. In so doing, they expanded “their competition with the Taliban … into a struggle for control of the drug traffic”, while imposing “a tax on farmers practically identical to the one the Taliban uses”. In a process that implicated virtually the entire government, provincial officials then passed a portion of their illicit profits “up the chain, all the way to officials in Kabul … ensuring that the local authorities maintain support from higher-ups and keeping the opium growing”.

Simultaneously, a UN security council investigation found that the Taliban had systematically tapped “into the supply chain at each stage of the narcotics trade” – collecting a 10% tax on opium cultivation in Helmand, fighting for control of heroin laboratories and acting as “the major guarantors for the trafficking of raw opium and heroin out of Afghanistan”. No longer simply taxing the traffic, the Taliban was so deeply and directly involved that, according to the New York Times, it “has become difficult to distinguish the group from a dedicated drug cartel”.

These dismal trends persisted throughout 2017, as the Afghan opium harvest nearly doubled to 9,000 tonnes, well above the previous peak of 8,200 tonnes in 2007. Inside wartorn Helmand province, the poppy area increased by 79% to 144,000 hectares, representing 44% of the country’s total crop. In November, convinced that opium is providing 60% of the Taliban’s funds for wages and weapons, the US command – emboldened and expanded by Donald Trump’s decision to “win” the Afghan war – dispatched, for the first time ever, F-22 fighters and B-52 bombers to destroy 10 of the Taliban’s heroin laboratories in Helmand, a small share of the country’s 500 drug refineries.

For the foreseeable future, opium will likely remain entangled in the rural economy, the Taliban insurgency, and government corruption whose sum is the Afghan conundrum.

The failure of America’s intervention in Afghanistan offers broader insight into the limits to its global power. The persistence of both opium cultivation and the Taliban insurgency suggest the degree to which the policies that Washington has imposed upon Afghanistan since 2001 have reached a dead end. For most people worldwide, economic activity, the production and exchange of goods, is the prime point of contact with their government. When, however, a country’s most significant commodity is illegal, then political loyalties naturally shift to the economic networks that move that product safely and secretly from fields to foreign markets, providing protection, finance and employment at every stage. “The narcotics trade poisons the Afghan financial sector and fuels a growing illicit economy,” John Sopko explained in 2014. “This, in turn, undermines the Afghan state’s legitimacy by stoking corruption, nourishing criminal networks and providing significant financial support to the Taliban and other insurgent groups.”

The fortune-teller of Kabul | May Jeong Read more

After 16 years of continuous warfare, Washington is faced with the same choice it had back in 2010, when Obama’s generals airlifted those marines into Marja. Just as it has been over the past decade and a half, the US can remain trapped in the same endless cycle. As snow melts from the mountain slopes and poppy plants rise from the soil every spring, there will be a new batch of teenage recruits from impoverished villages ready to fight for the rebel cause.

Even for this troubled land and its dauntingly complex policy problem, however, there are alternatives. Investing even a small portion of all that misspent military funding in the country’s agriculture can produce more economic options for the millions of farmers who depend upon the opium crop for employment. Ruined orchards could be rebuilt, ravaged flocks repopulated, wasted seed stocks regrown and wrecked snow-melt irrigation systems – which once sustained a diverse agriculture before these decades of war – repaired. If the international community continues to nudge the country’s dependence on illicit opium downward through sustained rural development, then maybe Afghanistan will cease to be the planet’s leading narco-state – and just maybe the annual cycle of violence could at long last be broken.

Adapted from In the Shadows of the American Century by Alfred W McCoy, which will be published in the UK by Oneworld on 25 January

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.