This AI turns #FoodPorn into recipes you can use

Take a photo of some food, give it to an AI and get recipe suggestions in seconds

20 Jul 2017

The system probably can't recognise meals with bugs in at the moment

Getty Images / Didier Baverel / Contributor

Open up your Instagram app and scroll through your feed of photos and videos. How long before you come across some #FoodPorn?

While pictures of food are everywhere on Instagram, the app doesn't allow links in posts so there's no easy way to find recipes. But that could be about to change. Pic2Recipe!, a website created by MIT electrical engineering and computer science student Nick Hynes, is a neural network that's been trained to recognise food from more than one million recipes on and AllRecipes.

"It can look at a photo of a dish and be able to predict the ingredients and even suggest similar recipes," Hynes says. "Its training allows it to find patterns and make connections between the food images and the corresponding ingredients and recipes".

The website lets anyone upload images that are then analysed by machine learning systems. When the system is able to determine food in photos it then searches against a database and suggests similar recipes.

For instance, the AI provides a variety of similar salad options – orange romaine salad, baby spinach salad – when it looks at an image with salad leaves and carrots in. Similarly, a loaf of bread... brings up recipes for baking bread.

Nick Hynes

The system, however, isn't perfect. We tried to upload a few Instagram pictures and couldn't get a direct match. Although, this may be down to the quality of our friends' cooking rather than the algorithm.

"It still has a lot of difficulty identifying certain food types, including blended and mixed foods like smoothies, soups and sushi rolls," Hynes says. "It makes a big difference if photos are taken from close up or far away, or if the photo is of a single item, multiples, or part of a complete dish". A cookie, for example, can look like a pancake.

Hynes isn't the first person to experiment with using machine learning and artificial intelligence to identify food in photos. In May, social network Pinterest revealed a new "dish recognition" feature that can broadly identify types of food in a photo. Hynes says his system is different as it attempts to pin foods to a recipe rather than just work out what they are.

And the technology can do more than just identify what's in your Instagram feed. "It could allow people to analyse their meals and determine their nutritional value, or even to manipulate an existing recipe to be healthier or to conform to certain dietary restrictions," Hynes says.

A system like this has already been developed by computer scientists from the Institute of Electrical and Electronics Engineers. In a paper published at the end of 2016, Stavroula Mougiakakou and colleagues showed how two photos of food can create a way to estimate the volume of food. This is being used to create a way for diabetics to monitor the amount of carbohydrates they consume.

"First you place a credit card-sized reference object next to the dish, and take a couple of photographs from different points of view," Mougiakakou says. "With these photos and the card we can build a 3D model of the food itself".

Mougiakakou continues to say she believes this process is the start of using smartphones to quickly and easily monitor food consumption. "We are currently working on expanding to the “big 4”, namely, calories, carbohydrates, fat and protein and, in some cases even to micronutrient content," she says.

"This way, it could be used for the entire spectrum of diet-related diseases such as obesity or chronic diseases such as kidney stones, in which managing nutrition is part of the treatment."