The risks of isocyanates was brought to the world's attention with the 1984 Bhopal disaster, which caused the death of nearly 4000 people from the accidental release of methyl isocyanate.  In 2008, the same chemical was involved in an explosion at a pesticide manufacturing plant in West Virginia.
LD50s for isocyanates are typically several hundred milligrams per kilogram. Despite this low acute toxicity, an extremely low short-term exposure limit (STEL) of 0.07 mg/m is the legal limit for all isocyanates (except methyl isocyanate: 0.02 mg/m) in the United Kingdom. These limits are set to protect workers from chronic health effects such as occupational asthma, contact dermatitis, or irritation of the respiratory tract.
Since they are used in spraying applications, the properties of their aerosols have attracted attention. In the U.S., OSHA conducted a National Emphasis Program on isocyanates starting in 2013 to make employers and workers more aware of the health risks. 
Polyurethanes have variable curing times, and the presence of free isocyanates in foams vary accordingly.
Both the US National Toxicology Program (NTP) and International Agency for Research on Cancer (IARC) have evaluated TDI as a potential human carcinogen and Group 2B "possibly carcinogenic to humans". MDI appears to be relatively safer and is unlikely a human carcinogen. The IARC evaluates MDI as Group 3 "not classifiable as to its carcinogenicity in humans".
All major producers of MDI and TDI are members of the International Isocyanate Institute, which promotes the safe handling of MDI and TDI.