# Image Based Password Authentication System

Sanjida Akter Sharna1,2,3, Sheikh Ashraf Ali1,2,4
1Military Institute of Science and Technology, Dhaka-1000, Bangladesh
2Department of CSE,Military Institute of Science and Technology
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Preservation of information and computer security is broadly dependent on the
secured authentication system which is underpinned by password. Text based
password is a commonly used and available system for authentication. But it
bears many limitations like shoulder surfing, dictionary attack, Phishing,
guessing the password etc. In order to overwhelm these vulnerabilities of
ancient textual password, many graphical or image based password
authentication system has been introduced form last few years. But none of
this graphical system is considered as enough adventurous to keep pace with
these issues. Here we have proposed an image based password authentication
system which is more methodical and can cope up with every vulnerability of
recent password authentication system. To make our system hassle free and more
reliable, we will only take username from an user for registration purpose as
our system will generate a unique key number for that particular user and this
key will be used as password for later login procedure. The user name and key
both will be encrypted using a cryptography algorithm to prevent database
hacking. There will be a randomized clickable image grid in our system. By
clicking on this image grid, user will input the password key for login
purpose. Here we have developed another method namely ‘shoulder surfing
resistant password’. To prevent the attack of shoulder surfing, if any user
wishes to change our system provided password key then he/she is allowed to do
so by using this method. Besides this method allows user to change the
password every single time of login. A user doesn’t need to enter any textual
password for authentication in our recent module and hence combination of all
these features improve the security, usability and user friendliness of our
system.

###### Index Terms:

Image, Password, Authentication, Shoulder surfing, Graphical.

##  I Introduction

Authentication system is of great importance from the raise of information and
technology for the confidentiality of data, information, statistics and many
other stuffs of individuals or any organizations. From the ancient era,
textual based authentication system has been used for the security purpose.
But now with the proliferation of technology and advancement, hackers have
become smart enough to decrypt any type of textual password. They have found
many ways like password guessing, dictionary attack, hash guessing, rainbow
table, password sniffing, brute force attack and many other efficient ways to
crack any strong password[11]. Considering all these reasons, the information
and data under this type of faulty authentication system are at great risk of
disclosing[12]. To get rid of this problem, graphical authentication system
has been introduced a few years ago. There are many types and categories of
graphical authentication systems which have been launched to preserve
confidentiality of information. But none of those systems are enough
systematic to protect data fully. Now a days shoulder surfing is the main
obstacle to this graphical authentication system. If any observer observes the
graphical login system for some time then he could probably guess the pattern
or type of graphical password which leads the main failure of this
authentication system[8]. Recalling all these impediments, we have designed
such a system which is shoulder surfing registrant and not textual. During
registration process, our system will provide system generated a key number to
user which will be required for login as password. While login, user has to
provide the user name and the key number which will be taken using special
technique. That is the user has to click on the images in a 10x10 image grid
to input key number. The image grid is used to input pass key number from
user, so that anyone is spying on the user can not get the key number. The
user can also make his own key number by adding the current date to the
previous key number which is named as shoulder surfing registrant password.
Throughout the authentication system, the main objective is to develop such a
module which is not fully textual so that no hackers can easily crack the
password using the latest technique of password decryption. Another prime
objective is that to get rid of the failure (shoulder surfing) of graphical
authentication system[7].

##  II Literature review

###  II-A A Shoulder surfing resistant graphical password scheme-WIW

Recently a shoulder resistant scheme was introduced where they describe their
system as follows: In this system, they mainly used four factors. A user needs
to choose ‘h’ number of images to make his password. This variable is
dependent on each user. They can choose it according to their wish. While log-
in the content of this h images will change randomly. These images are
described as scenes here. For each scene, there will be ‘n’ objects which have
fix ranging from 250-300. Then k is defined as pass-object which is only
chosen by the user and it is the part of the password. Then another factor m
is defined as perturbation i,e the appearance of each pass object in each
scene. While log-in the user must be able to recognize those pass objects. One
image is stand for one letter, which is determined by the system. Now user
need to identify the appearances and locations of pass objects in each scene.
This combination spells a letter for each pass object. This way user needs to
identify all the combination which spells some letter. The the main task is to
identify all the letters for successful log-in. Each time of log-in, this pass
objects and non pass objects and their appearances will be randomized. For the
attacker, it is difficult to identify those letters and recognize the pattern
as each time it is randomized. The main problem with this system is that, if
the value of h, k and m is smaller than its quite easy for a user to memorize
those factors but then the attackers may recognize the pattern while studying
for some days. So it weakens the system. If the user chooses large value of
h,k and m, then for log-in the user needs to memorize a lots of factors like
all the pass objects form each scene accurately. In case, the user fails to
choose the pass objects correctly then log-in process will be unsuccessful. So
here the whole system is not at all user friendly or not usable for ordinary
users [1]. In our system the user has to face no difficulties to form his
password, he just needs to provide his user name which is comparatively easy
to memorize. For the user, our system will generate a key number and provide
him. As it is generated by the system,it will be highly secured. Then while
log-in the user just need to enter the key number from image grid.

###  II-B A shoulder surfing resistant image based authentication system with
temporal indirect image selection

In this system, it displays images in sequence like slide show and an user
needs to select slide show which includes his pass image among several slides
show. The system presents several images to user. user selects several pass
images p, which is altogether used as his password. And the user must remember
this pass images. In log-in phase the system present N slide shows and M is
the number of images in each slide show. Now the user needs to recognize in
which slide show the pass image is included, then he needs to select the exact
pass images from each slide show which changes with a fix interval of time 1/v
per image. By choosing the correct sequence, the log-in will be successful.
Now this system has almost 50 percentage threats of shoulder surfing. If any
hacker or attacker take a video clip of any user through CCTV camera or
somehow, then after studying for some days he can easily recognize the
pattern.1st the correct slide show is selected then the pass image form that
particular images sets of slide show is selected. then by observing patiently,
the attacker will be aware of the pass images. Here the switching interval
speed v is a great factor which is in millisecond. If the switching interval
is less or can not be controlled properly then the threat of revealing the
pass images is higher [2].
After studying this paper, we observe that our system uses no such slide shows
or here switching interval or speed is not a factor, we are randomizing our
image grid which it is free of threat of revealing pass images. In our system,
we are using another additional method of entering key which is described as
below. User has to add the decimal digits of the log-in date in such a way
that it will form a single digit (0-9). By repeating this single digit 4 times
it will form a 4-digit number. then user has to add this repeated 4-digit
number with the original 4 digit key. If the summation is greater than10 then
only last digit will be taken. So, our system is highly secured of shoulder
surfing problem.

###  II-C A review on two level authentication using image selection and voice
recognition

In this system image and voice is used for authentication. During registration
user has to select an image and enter a watermark. This image with the
watermark is one level of authentication. Then user needs to give the voice
summon which is the 2nd level of authentication. In this system, they used
voice input which is not very much user friendly. Different voice may occur
during recognition [3].
Our system is very much user friendly. We just need to remember a 4-digit code
to log in. In this system there is a high possibility of shoulder surfing
while selecting the image. User need to select just one image to log-in. So,
it is easy for attacker to know the required image by shoulder surfing. Our
system is highly preventable form shoulder surfing problem. We used a random
image grid and a shoulder surfing resistant key which is described previously
to prevent this problem.

###  II-D Image based authentication system

In this system, both password and image are used to overcome the problems.
Like other authentication system, it also consists of two phases that is
registration and log-in. In registration phase user needs to fulfill basic
personal requirements like name, DOB, e-mail address etc. Next user needs to
enter a password which must maintain their constraint like minimum 8-character
password, minimum one uppercase and one numerical number and one special
character. By full filling these criteria, user can choose their desired
password. After this stage user must select one image as his pass image
otherwise the registration will be unsuccessful. There is an image category
selection option, from where user needs to select his desired category like
natural scenarios, animals, flowers etc. Every time these categories will be
randomized by refreshing the page. Now if user selects 3 images, then he/she
needs to select the category, then from his chosen category a 3*3 image grid
of same type but discrete images will appear. From this grid user will choose
some of his pass images. Then he has to choose his 2nd category. By choosing
2nd category, again a 3*3 grid of this category will appear. Once again user
needs to choose pass image from this category. In the same manner user will
choose the 3rd category and pass images. User must memorize this pass images.
In login phase, user must provide username, password and pass images. Pass
images will be randomly displayed in login round. In each login round, there
may be all pass images or some images. It is also possible that none of pass
images appear in any round. User has to choose the pass images as the exact
sequence he chooses them through registration phase. By choosing the correct
sequence from login rounds, login will be successful [4].
In their system, they cannot prevent offline dictionary attack and the system
is slower comparing traditional textual authentication system for displaying
image grid several time. They do not use any encryption system to encrypt the
password. If any hacker hacks the database then he can easily get the password
and user name and it can make one step easier for the hacker to hack the
system. And in their system, shoulder surfing is prevented but not in a
wholesome manner.
Comparing with this system, our system almost prevents offline dictionary
attack and database hacking. As in our system the user name and password which
is a key number (generated by our system) will be encrypted through AES and
will be saved in database. So, if hacker somehow manages to hack the database,
he won’t get the exact password. As the hacker don’t know the key, he won’t be
able to decrypt it. Our image grid appears only once, and randomizes in every
click. For this reason, it is faster than the mentioned system. And because of
our advanced and randomized image grid, the possibilities of shoulder surfing
are almost prevented which is explained broadly below.

###  II-E A secure graphical password authentication system

This system allows a user to create a graphical password by selecting an image
from a collection of available pictures. From the selected images user has to
select one grid as the password. The selected image is watermarked with a
cover image using Generic Visible Watermark Embedding technique. The method is
based on the use of deterministic one-to-one compound mappings of image pixel
values for overlaying a variety of visible watermarks of arbitrary sizes on
cover images. During login, after entering the user details a QR Code is
generated in the computer. User has to scan the QR code using his mobile
phone. After scanning, a collection of images will be appeared in the screen
of the phone. User has to select the image. After choosing correct image, the
watermarked image will be appeared on the screen. User has to choose the
correct grid position that he has already registered in the watermarked image
[5].
This system is not user friendly at all. Normal user won’t feel comfortable to
use such kind of inconvenient system. So, this system will be failed initially
for lacking of user friendliness. On the other hand, our system is quite user
friendly.

##  III Our proposed system

###  III-A Registration system architecture

####  III-A1 Provide User Name

The initial phase of our authentication system is registration. In this phase
like other authentication system, user needs to provide his username which
must be alphanumeric. Here we are providing an extra benefit that is an user
does not require to provide or create any password by thyself. As the user
doesn’t need to remind any password, our system will be more comfortable and
user friendly.

####  III-A2 Key number

After providing user name, our system will generate a unique key number which
consists of 4 digit decimal number. User just has to memorize this key number
for login. In another word, we can say this key number can be regarded as his
password.

####  III-A3 AES (Advanced Encryption Standard) encryption

The generated key and user name will be encrypted using the cryptography
method AES-128[14]. After that the encrypted key and user name both will be
saved in database so that if any hacker hacks the database, he will not get
the actual user name and key number. As the hacker doesn’t know the AES key,
he cannot easily decrypt the user name and key number.

###  III-B Login system architecture

####  III-B1 Enter User Name

If a user wants to login, then the system will initially ask for his user
name. A window will appear where user has to enter the accurate user name. If
the user name is similar to his previously provided user name while-
registration phase, then the system will let him to proceed to the next step.
In next step a 10x10 image grid will appear.

####  III-B2 10x10 image grid

In this step, a 10x10 image grid[10] will appear. User must enter the
previously provided key number by clicking on this image grid[6]. Images of
this grid are clickable except the images of 1st row. The image grid contains
25 original images and 3 copies of each images in total 100 images which will
be arranged in 10 rows and 10 columns[9]. In the 1st row of the grid, 10
different images will be appeared randomly from the original 25 images. The
1st row represents a value from left to right according to its index number
which is numbered as 0 to 9. The images which will appear on 1st row, their
value will be set as index number. The duplicates of these 10 images will
represent the same value of original images in the 1st row. The rest of the 15
original images and its copies represent garbage value.

Figure 1: 10x10 Image Grid

For example if the key number is 6431, then the user needs to watch out the
6th index image of 1st row and then find the same image anywhere on the grid
and click. The system will take this as input digit 6. Following the same
method rest of the digits will be entered by clicking on the image grid. The
images and 1st row of the image grid will randomize on every login session.
When the user will click images to enter the 4 digit key number, by each click
the whole image grid including the 1st row will be shuffled randomly so that
any person or observer who is trying to assume the key number gets puzzled to
find out the image pattern. As the grid will shuffle in each click, the images
will change randomly every time. In case if the observer somehow possesses an
eidetic memory/photographic memory, it will be very tough for him to recall
every images correctly and he/she will be perplexed to find the relation of
key number and image pattern. Hence hacker will be failed to guess the key
number.

####  III-B3 Shoulder surfing resistant password

If a user wishes to enter his own key number except the generated one by the
system, then he/she can do so. To do this, a user must follow a few
instructions which are set by us. And we have named it as Shoulder surfing
[15] resistant password. The instructions are described as follows. A user
needs to add the decimal digits of the date of the current day in which he/she
is login to the system in such a way that it will form a single digit (0 to
9). Like if the current date is 27, then the sum of these two digits is
(2+5=7). By repeating this single digit 4 times (i,e 7777) it will form a
4-digit number. Then user has to add this repeated 4-digit number with the
original 4-digit key. If the summation is greater than 10 then only the least
significant digit will be taken. As example if the original key is 1241 then
after adding 7777 with it [Fig-2].

Figure 2: Generating Shoulder Surfing Resistant key

Here the summation of 4+7=11 so the least significant digit of summation which
is 1 will be taken only. Hence, the final key will be 8918. Then this will be
the new key number which can also be used for login purpose to avoid shoulder
surfing.

System decodes the original key by subtracting 4-digit repeated number which
is formed from current date following the same procedure during
encoding[Fig-3].

Figure 3: Extraction Of Original Key

If any digit of the new shoulder surfing resistant key (minuend) is less than
the 4-digit repeated number (subtrahend),the system will add 10 with the
minuend and then will subtract it. Here the 3rd digit 1 is less than 7 so
system adds 10 with 1(1+10= 11) and 11-7= 4 is taken. By doing this, system
generates the original key and this key is cross checked with the key saved in
database.

####  III-B4 Architecture of the whole System

Our system has two phases which interact between user and system. Mainly One
is registration phase another is log in phase. The whole architecture is
showing below with a Flow diagram.

Figure 4: Flow Diagram of System Architecture

##  IV Advantage and Limitation

###  IV-A Advantage

Our system will be able to prevent most of the chances of shoulder surfing
which is secure for any system. It also subdues most of the drawbacks of
textual password system.The Shoulder surfing resistant password mechanism is
unique. The randomization speed of image grid is good enough which makes our
system high performing and shoulder surfing resistant. Moreover, percentage of
database hacking is almost low. In case of hacking occur, the information (key
number and username) will be safe as these are encrypted and then saved in
database. User memorization skill is not a big concern here. User just need to
memorize the user name and key number and nothing else. So, all these features
altogether provide huge advantage which makes our system efficient.

###  IV-B Limitation

User name is textual which is a limitation here. To make shoulder surfing
registrant password, we are only using one factor (current date) to randomize
the key. No recovery system is included in our system yet which is a
considerable limitation. Use of Static key for AES encryption makes our system
a little insecure.

##  V Evaluation and analysis

By analyzing our developed system performance, we have reached to the
conclusion that our system is quite high in performance. All the modules are
designed in a user friendly and efficient way. The image grid randomization
speed is expectedly good and every time shuffle properly. In short ,the system
performance is according to our expectation.The below figure is showing the
successful registration and login phase of our system.

Figure 5: successful implementation of registration and login phase

##  VI Future Work

Using stenography process any text can be hidden in a image. This image is
called stego image. In future username will be hidden in image and will be
provide to user for login purpose. This will replace the criteria of providing
user name for log-in. If a user wish to login, then the system will initially
ask for that stego object from user in which the user name is hidden. A
portable storage device like pen drive can be used to store that stego image.

Initially we are encrypting the key number and user name using a static AES
key. Further we will use a randomize key which will enrich the security of the
system.

In future we will add an extra feature in our system which will prevent brute
force attack. That is by entering wrong key number 5 times the during login,
system will block that user for almost half an hour. This way hackers attempt
will go in vain if he tries brute force technique.

##  VII Conclusion

In this paper, we are proposing a new method of graphical authentication
system and shoulder surfing resistant password mechanism which will enable us
to overcome the shortcomings of password leakage. Moreover, the proposed
system is free from ancient textual based password. It will be an efficient
tool for protecting highly confidential data and information which will add a
new dimension to the security purpose.

## References

  * [1] Man, Shushuang & Hong, Dawei & Matthews, Manton. (2003). A Shoulder-Surfing Resistant Graphical Password Scheme - WIW.. Proc. International Conference on Security and Management. 3. 105-111.
  * [2] Yamamoto, Takumi & Kojima, Yuko & Nishigaki, Masakatsu. (2009). A Shoulder-Surfing-Resistant Image-Based Authentication System with Temporal Indirect Image Selection.. 188-194.
  * [3] Raut, P., Kolte, S., Pachare, S. and Shinde, A., 2017. A Review on Two Level Authentication Using Image Selection and Voice Recognition.
  * [4] Shah, Pintu. (2013). Image Based Authentication System.
  * [5] Prakash, M.S. and MK, M.S., 2016. A Secure Graphical Password Authentication System. International Journal of Computational Intelligence Research, 12(2), pp.197-201.
  * [6] Chiasson, S., Van Oorschot, P.C. and Biddle, R., 2007, September. Graphical password authentication using cued click points. In European Symposium on Research in Computer Security (pp. 359-374). Springer, Berlin, Heidelberg.
  * [7] Almulhem, Ahmad. ”A graphical password authentication system.” 2011 World Congress on Internet Security (WorldCIS-2011). IEEE, 2011.
  * [8] Almuairfi, Sadiq, Prakash Veeraraghavan, and Naveen Chilamkurti. ”A novel image-based implicit password authentication system (IPAS) for mobile and non-mobile devices.” Mathematical and Computer Modelling 58.1-2 (2013): 108-116.https://www.overleaf.com/project/5eef267981f888000168a15e
  * [9] Osborn, Steven L., et al. ”Methods and Systems for Graphical Image Authentication.” U.S. Patent Application No. 11/677,562.
  * [10] Tao, Hai. ”An arrangement and method of graphical password authentication.” U.S. Patent Application No. 11/163,115.
  * [11] Raza, Mudassar, et al. ”A survey of password attacks and comparative analysis on methods for secure authentication.” World Applied Sciences Journal 19.4 (2012): 439-444.
  * [12] Sonar, Aishwarya N., et al. ”Survey on Graphical Password Authentication Techniques.” (2017).
  * [13] Parmar, Himika, Nancy Nainan, and Sumaiya Thaseen. ”Generation of secure one-time password based on image Authentication.” Journal of Computer Science and Information Technology 7 (2012): 195-206.
  * [14] Nechvatal, James, et al. ”Report on the development of the Advanced Encryption Standard (AES).” Journal of Research of the National Institute of Standards and Technology 106.3 (2001): 511.
  * [15] Eiband, Malin, et al. ”Understanding shoulder surfing in the wild: Stories from users and observers.” Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017.


# RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal
Models

Pengkun Jiao1, Xinlan Wu1,
Bin Zhu2, Jingjing Chen1, Chong-Wah Ngo2, Yugang Jiang1

###### Abstract

Large Multi-modal Models (LMMs) have significantly advanced a variety of
vision-language tasks. The scalability and availability of high-quality
training data play a pivotal role in the success of LMMs. In the realm of
food, while comprehensive food datasets such as Recipe1M offer an abundance of
ingredient and recipe information, they often fall short of providing ample
data for nutritional analysis. The Recipe1M+ dataset, despite offering a
subset for nutritional evaluation, is limited in the scale and accuracy of
nutrition information. To bridge this gap, we introduce Uni-Food, a unified
food dataset that comprises over 100,000 images with various food labels,
including categories, ingredients, recipes, and ingredient-level nutritional
information. To mitigate the conflicts arising from multi-task supervision
during fine-tuning of LMMs, we introduce a novel Linear Rectification Mixture
of Diverse Experts (RoDE) approach. RoDE utilizes a diverse array of experts
to address tasks of varying complexity, thereby facilitating the coordination
of trainable parameters, i.e., it allocates more parameters for more complex
tasks and, conversely, fewer parameters for simpler tasks. RoDE implements
linear rectification union to refine the router’s functionality, thereby
enhancing the efficiency of sparse task allocation. These design choices endow
RoDE with features that ensure GPU memory efficiency and ease of optimization.
Our experimental results validate the effectiveness of our proposed approach
in addressing the inherent challenges of food-related multitasking.

Figure 1: RoDE’s Emphasis on Food-Related VQA Tasks. RoDE primarily targets
multi-task learning specific to food, i.e., food classification, ingredient
recognition, and nutrition estimation.  Figure 2: Label Constitution. The
annotation consists of category, ingredient list, recipe instructions, and
nutrition information.

## Introduction

Food occupies a central position in our daily lives, leading to the emergence
of various food-related tasks, e.g., ingredient recognition, recipe
generation, and nutritional estimation. These tasks have attracted
considerable research interests over the years (Chen and Ngo 2016a; Zhu et al.
2019; Pan et al. 2020; Chen et al. 2020; Min et al. 2023; Luo et al. 2023).
Building on the success of Large Language Models (LLMs) (Radford et al. 2018;
Touvron et al. 2023), Large Multi-modal Models (Liu et al. 2023a; Luo et al.
2023) have begun to make a significant impact in many specialized areas,
including the food domain (Yin et al. 2023).

The success of LLMs and LMMs can be largely attributed to the availability of
large-scale training data. In the food domain, while Recipe1M (Salvador et al.
2017) provides over a million web-crawled recipes and thousands of food
images, it lacks comprehensive nutritional information. While Recipe1M+ (Marin
et al. 2019) includes a subset containing nutritional information, it suffers
from limitations in data scale and annotation quality. Similarly, nutrition-
focused datasets such as Nutrition5k (Thames et al. 2021) are also limited in
scale and may suffer from domain shift issues, as the labeled data primarily
consists of lightly processed ingredients rather than fully cooked meals.
Moreover, previous studies (Zhou et al. 2022; Chen, Jie, and Ma 2024) have
indicated that integrating diverse sources of data for training can lead to
data conflicts. The performance of an LMM on a specific task also depends
heavily on the representation of that task’s data within the overall training
dataset (Chen, Jie, and Ma 2024). These challenges highlight the need for a
unified dataset covering all food-related tasks from the same source, aiming
at mitigating data conflicts and ensuring balanced representation across
tasks.

In order to overcome these obstacles, we introduce Uni-Food, a unified dataset
that encapsulates various food-related information, i.e., food category,
ingredients, recipes, and valuable ingredient-level nutritional information
for each food image. Specifically, we curate 100,000 samples from Recipe1M
(Salvador et al. 2017) and employ ChatGPT-4 (Achiam et al. 2023) to generate
nutritional information for each ingredient list. This information is then
aggregated to derive the nutritional data for the entire dish. Furthermore, in
order to ensure a high-quality gold standard set for testing, we utilize human
curation to isolate a superior subset specifically designated for testing
purposes.

Our proposed dataset facilitates large-scale training of LMMs for various
food-related tasks within the same dataset. Nevertheless, this endeavor also
presents new challenges associated with multi-task learning when fine-tuning
large models. Mixture of Experts (MoE) (Huang et al. 2023; Dou et al. 2023;
Liu et al. 2023b; Ponti et al. 2023) has been a common strategy in the field
of Natural Language Processing (NLP) to handle multi-task fine-tuning. This
method involves the utilization of multiple expert models, each specialized in
different task or segment of the data distribution. Recently, there has been a
surge in the application of MoE to LMMs. MoE-LLaVA (Lin et al. 2024), for
instance, integrates MoE into its Feed-Forward Network (FFN) layers to enhance
the model’s adaptability across tasks. On the other hand, LLaVA-MoLE (Chen,
Jie, and Ma 2024) incorporates the MoE paradigm into the Low-Rank Adaptation
(LoRA) (Hu et al. 2021) module, selecting the top-1 experts for each specific
task to guarantee the spare activation of experts. However, as MoE-LLaVA
incorporates experts throughout the FFN layer, the resulting increase in
training parameters can be prohibitive. Conversely, although LLaVA-MoLE
utilizes LoRA experts, which are more parameter-efficient, its strategy of
selecting only the top expert may constrain the model’s flexibility on skill
leverage. This is because certain tasks might share foundational skills. For
example, in the food domain, both ingredients recognition and recipe
generation are sensitive to the composition of ingredients in a dish.

To address the aforementioned issues, we propose Linear Rectified Mixture of
Diverse Experts (RoDE). Following LLaVA-MoLE (Chen, Jie, and Ma 2024), we
design experts from the LoRA perspective to ensure efficient use of trainable
parameters. Unlike LLaVA-MoLE, which dedicates each expert to a particular
task, LLaVA-RoDE conceptualizes experts as granular skill modules, allowing
multiple experts to participate in a single task. This design inspires us to
develop experts with diverse capabilities—that is, we allocate varying amounts
of trainable parameters to each expert in an effort to conserve GPU memory. To
integrate the experts, we then introduce a linear rectified router to assign
tasks to the appropriate LoRA experts. The router employs a Rectified Linear
Unit (ReLU) to refine its output, which confers two significant benefits: 1)
it enables sparse task-skill matching, which has been proved to surpass dense
activation methods in effectiveness (Zhou et al. 2022; Chen, Jie, and Ma
2024); 2) the simplicity of the ReLU function makes it inherently easy to
optimize. Consequently, this leads to a sparing activation of the diverse
expert set. RoDE is engineered to optimize the allocation and usage of the
model’s resources, significantly bolstering the capabilities of large-scale
models in multi-task learning scenarios. Our empirical findings underscore the
superior performance of RoDE in the realm of food multi-task learning
challenges, validating the strength of our proposed methodology.

Our contributions can be summarized as follows:

  * •

We introduce Uni-Food, a novel dataset encompassing a variety of food vision
tasks. Building upon the substantial ingredient and recipe dataset, we further
incorporate valuable ingredient-level nutritional information to facilitate
relevant dietary research.

  * •

We propose Linear Rectified Mixture of Diverse Experts (RoDE) approach
designed to effectively tackle food multi-tasks learning. RoDE approach
leverages a combination of LoRA experts with varying ranks to model tasks of
different complexities and employs linear rectified router to sparsely
allocate these experts to appropriate tasks.

  * •

Experimental results demonstrate the effectiveness of our proposed approach on
food multi-task learning. And ablation studies highlight the GPU memory
efficiency and the sparse allocation of experts intrinsic to our RoDE model.

## Related Work

Large Multimodal Models (LMMs). In recent years, Large Language Models (LLMs)
have showcased remarkable prowess in Natural Language Processing (NLP), paving
the way for breakthroughs in multimodal learning. LMMs typically integrate a
pre-trained vision encoder with a LLM architecture. Subsequently, visual
features undergo adaptation via a projection module, facilitating their
integration into the hidden space of LLMs for joint processing with textual
inputs. Through multimodal training, LMMs acquire the capability to generate
responses based on both visual and textual inputs. LLaVA (Liu et al. 2023a),
for instance, introduces a vision encoder to LLMs and demonstrates significant
enhancements across various vision-language tasks. Building upon LLaVA’s
advancements, LISA (Lai et al. 2023) further enriches multimodal models by
incorporating a segmentation module. This addition augments the model’s
capacity to discern fine-grained details within visual inputs, resulting in
more nuanced and contextually richer representations. GPT-4v (Achiam et al.
2023) stands out as one of the most powerful LMMs, capable of providing
instructional data across numerous research domains. Its potency extends
beyond NLP, contributing to advancements in various interdisciplinary fields.

Food Multi-task Learning. With food playing a significant part in human life,
there has been a lot of work carried out doing research in this domain. In the
early stage, food classification (Bossard, Guillaumin, and Van Gool 2014; Qiu
et al. 2022) happened to be the most popular task related, bringing up a surge
in the number of relevant datasets, like Food-101 (Bossard, Guillaumin, and
Van Gool 2014), UEC Food 256 (Kawano and Yanai 2014) and Food2K (Min et al.
2023). As the tasks in the food domain developed into greater diversity, from
ingredient recognition (Wang et al. 2022; Gao et al. 2023; Luo et al. 2023),
recipe generation (Salvador et al. 2019; Chhikara et al. 2024) to nutrition
estimation (Ando et al. 2019; Thames et al. 2021), food datasets have also
grown to be more diverse and large-scale. Vireo Food-172 (Chen and Ngo 2016b)
and 251 (Chen et al. 2021) are datasets containing not only category
information but also ingredient labels, Recipe1M (Salvador et al. 2017) and
Recipe1M+ (Marin et al. 2019) are datasets involving a wealth of recipe data,
and Nutrition5K (Thames et al. 2021) is a dataset specialized in high accuracy
nutritional content annotation. While all of the above methods have
demonstrated their performance on food, they were all limited to address one
or a few tasks, rather than integrating all tasks into a single multimodal
model. To accomplish this, Yin et al. came up with FoodLMM (Yin et al. 2023),
a versatile food assistant based on LMMs that could handle a variety of tasks
in the food domain. However, their method might lead to task conflict when
several tasks were fine-tuned at the same time, lacking the capability to
efficiently implement multi-task learning.

Dataset | category | ingredient | recipe | nutrition
---|---|---|---|---
Food-101 | 101k | 0 | 0 | 0
Food2K | 1,036,564 | 0 | 0 | 0
Vireo Food-251 | 169,673 | 169,673 | 0 | 0
Nutrition5k | 0 | 20k | 0 | 20k
Recipe1M | 887,706 | 887,706 | 887,706 | 0
Recipe1M+ | 13,735,679 | 13,735,679 | 887,536 | 42,713
Uni-Food | 100k | 100k | 100k | 100k

Table 1: Annotation Diversity in Food Datasets. The volume and variety of
annotations differ significantly across various food datasets. Uni-Food offers
a large-scale training dataset, accommodating a broad spectrum of food-related
tasks.

Mixture of Experts (MoE) dynamically combines the decisions of multiple
experts on the same input to improve overall performance. In LLMs, which
typically adopt transformer architecture, MoE is often implemented within MLP
layers. LoRAHub (Huang et al. 2023) initially trains a series of LoRA weights
on upstream tasks. To adapt these to a downstream task, it employs a gradient-
free method to search for the coefficients that will combine the pre-trained
LoRA set. MOELoRA (Liu et al. 2023b), on the other hand, utilizes a router
conditioned on a task identifier to dynamically merge multiple LoRA outputs.
In contrast, MoCLE (Gou et al. 2023) designs a router that is conditioned on
the clustering information of each individual input sample. LoRAMoE (Dou et
al. 2023) partitions the LoRA experts into two groups and purposefully
cultivates different capabilities within each group. All these mixture-of-LoRA
methods have pre-set hyperparameters that require meticulous selection, and
the LoRA experts are densely combined. (Zhou et al. 2022) conducted a
comparison between the dense and sparse mixtures of LoRA experts for large
language models, concluding that a dense mixture yields superior performance.
Some studies have migrated MoE designs into LMMs. MoE-LLaVA (Lin et al. 2024)
introduces multiple FFNs in stead of a single one and integrates a router
mechanism to sample predictions. LLaVA-MoLE (Chen, Jie, and Ma 2024) applies
multiple LoRAs and activates one specifically for a task. However, these
methods fall short in effectively handling the nuances of rational, fine-
grained task-skill allocation. Our paper aims to address this gap.

## Dataset Construction

In this paper, we construct a large-scale dataset called Uni-Food. Different
from other publicly available food datasets (Bossard, Guillaumin, and Van Gool
2014; Marin et al. 2019; Chen and Ngo 2016a; Thames et al. 2021), Uni-Food
contains various attributes used in food-related tasks, including food
category, ingredients, recipe and nutrition for each food image. To the best
of our knowledge, this is the largest dataset that provides all the attributes
in one dataset. Table 1 summarizes the tasks and sample sizes of existing
primary datasets and our Uni-Food dataset.

### Attribution

Our objective is to construct a unified and comprehensive dataset containing
rich information relevant to food, encompassing the following key attribution
for each image. Category: Classifying each food item into specific categories
to facilitate organization and categorization within the dataset. Ingredients
Information: Providing a thorough breakdown of the ingredients used in each
dish, including their names and quantities. Cooking Instructions: Offering
step-by-step instructions on how to prepare each dish, ensuring clarity and
completeness for easy reproduction. Nutrition Information: Incorporating
detailed nutritional data for each dish, such as macronutrient content (e.g.,
carbohydrates, proteins, fats), micronutrients, and total calories. An
intuitive sample demonstration of these attributions is shown in Figure 2. The
distribution across various categories is visualized in Figure 3

Figure 3: Dataset Statistics.The category distribution of the dataset. Figure
4: The illustration of our proposed Linear Rectified Mixture of Diverse
Experts (RoDE) approach. We use LLaVA (Liu et al. 2023a) as the foundational
LMM. RoDE module is incorporated into both the query projection layer and the
value projection layer of each transformer block.

### Nutrition Labeling

As the ingredient and recipe information can be easily collected from
Recipe1M+ (Marin et al. 2019), we proceed to annotate the nutrition content.
To acquire precise nutritional information, we feed both the food image and
the ingredients with their respective amounts into ChatGPT4-vision 111We
intentionally omit recipe information to enhance processing efficiency, as we
have observed that including recipes increases query time without considerably
improving precision.. The output text is meticulously processed to extract the
nutritional values. Leveraging the capabilities of ChatGPT4-vision, we collect
detailed nutrition information at the ingredient level, encompassing metrics
such as mass, fat content, protein content, carbohydrate content, and energy
content. Importantly, the ingredient list in recipes contains the quantity for
each ingredient, along with the food image, ChatGPT4-vision is capable of
generating high-quality ingredient-level nutrition information according to
our manual check based on USDA 222https://fdc.nal.usda.gov/. The ingredient-
level nutrition information is then aggregated to derive the comprehensive
nutrition profile of the entire dish. Through this process, we obtain a
holistic understanding of the nutritional composition of the dish, enabling
precise estimation and analysis of its nutritional value.

### Golden set selection

To ensure accurate evaluation, we construct a precise gold standard as the
test set. To accomplish this goal, we manually filter out inaccurate samples
by cross-referencing their nutritional information with the USDA database.
Additionally, we eliminate certain overrepresented categories to achieve a
balanced distribution of sample categories.

## Method

In this section, we first provide a preliminary overview of the problem
formulation for multi-task tuning of LMMs, as well as the commonly used
techniques, i.e., Low-Rank Adaptation (LoRA) (Hu et al. 2021) and Mixture of
Experts (MoE) (Dou et al. 2023) for addressing this problem. Subsequently, we
introduce our Linear Rectified Mixture of Diverse Experts approach in detail.

### Preliminary

#### Problem Formulation.

A Large Multi-modal Model (LMM) can be constituted by a vision encoder and a
Large Language Model (LLM), as shown in Figure 4. Normally, LMM is initially
pre-trained with tremendous amounts of data, and then fine-tuned to adapt to
downstream tasks. The utilization of multi-modal documents for Supervised
Fine-Tuning (SFT) (Radford et al. 2018) is a common practice in the fine-
tuning of LMMs.

Let us denote the set of multimodal documents as $D$, where
$D=\\{(I_{i},T_{i})\\}_{i=1}^{M}$, where $I_{i}$ signifies the image, and
$T_{i}$ represents the associated set of tasks with that image. $M$ stands for
the total number of documents. Each task set $T_{i}$ encompasses sequence-
specific tasks $\left\\{(q_{i}^{j},a_{i}^{j})\right\\}_{j=1}^{\mathcal{T}}$,
where $q_{i}^{j}$, $a_{i}^{j}$ is the question and answer for task $j$, and
$\mathcal{T}$ is the number of task types. The primary objective of SFT is to
using $D$ to fine-tune the LMM so that it can provide corresponding answers to
given questions based on an image. More specifically, for an image $I_{i}$ and
a question $q_{i}^{j}$, the training objective is to maximize the probability
$p(a_{i}^{j}|I_{i},q_{i}^{j},\theta)$, where $\theta$ represents the trainable
parameters of the large model.

#### Low-Rank Adaptation (LoRA).

Fully tuning large models can be resource-intensive. Parameter-Efficient Fine-
Tuning (PEFT) (Ding et al. 2023) introduces additional adapters to effectively
customize large models for downstream tasks with minimal resource overhead.
Let $W$ represent the weight of a linear layer $L$ in the large-scale model,
which is initially set as $W_{0}$ after pre-training. In PEFT, $W_{0}$ remains
frozen in order to preserve the acquired world knowledge. To facilitate
adaptation to downstream tasks, a learnable branch linear adapter, denoted as
$\Delta W$, is introduced to modify the initial frozen linear weights $W_{0}$.
Consequently, the output of the adapted linear layer $L$ can be expressed as
$W=W_{0}+\Delta W$. LoRA (Hu et al. 2021) further decomposes $\Delta W$ into
two matrices, $A$ and $B$, where the connection rank is considerably smaller
than that of $W_{0}$. LoRA allows the model to adapt to downstream tasks while
reducing the number of training parameters.

#### Mixture of Experts (MoE).

The Mixture of Experts (MoE) (Dou et al. 2023) paradigm of LoRA is proposed to
adapt large models to multiple downstream tasks by employing multiple LoRAs in
the MLP layers. Each LoRA module can be considered as an expert, wherein all
experts receive the same input and combine their outputs to improve overall
performance. Let $R=\left\\{A_{i},B_{i}\right\\}_{i=0}^{N}$ denote a series of
LoRA modules, where $N$ signifies the number of LoRAs. We use $h(\cdot)$ to
denote the linear router, which takes the dimension of the input feature $x$
and outputs the allocation vector of $R$. The output of layer $L$
incorporating the MoE module can be represented as:

$\displaystyle x^{\prime}$ $\displaystyle=$ $\displaystyle
W_{0}x+\sum_{i=0}^{N}\frac{\alpha}{r}\sigma(h_{i})B_{i}^{r}A_{i}^{r}x.$ (1)

Here, $\sigma$ stands for the softmax operation, $\alpha$ is a hyperparameter,
and $x^{\prime}$ is the output feature.

### Linear Rectified Mixture of Diverse Experts

Our proposed RoDE framework incorporates a variety of experts, each with
distinct capabilities, and a linear rectification router to integrate the
contributions of these experts. The overall structure of the framework is
depicted in Figure 4. A comprehensive illustration of the framework is
provided in the subsequent sections.

#### Diverse Capability Experts

Typically, within the Mixture of Experts (MoE) framework, all experts share
the same architecture. In the case of LoRA, for example, each expert is
assigned the same rank. However, this uniformity assumes equal capabilities
across all experts. This implies that for more complex tasks, each expert
might need to possess a large number of parameters to adapt effectively.
However, this could be prohibitively demanding in terms of GPU memory
requirements.

To mitigate the issue of GPU memory constraints, we conceptualize the experts
as fine-grained skill modules. The key idea is that a task may activate a
combination of these modules, and these modules can be shared across various
tasks. This modular design intuitively leads us to develop LoRA experts with
different capabilities tailored to tasks of varying complexity. Drawing
inspiration from Low-Rank Adaptation (Hu et al. 2021), which suggests that a
low-rank adapter may be sufficient for certain tasks, we create LoRAs with
varying ranks. Consequently, the resulting skill space comprises both high-
rank and low-rank LoRA experts, providing greater flexibility and efficiency
in addressing a diverse set of tasks.

To construct such skill space, we configure a heterogeneous set of experts,
represented as $R=\left\\{A_{i}^{r_{i}},B_{i}^{r_{i}}\right\\}_{i=0}^{N}$,
where the rank $r_{i}$ of $i-th$ LoRA module may vary from the others.
Therefore, we can establish a skill space composed of various LoRAs, each
tailored to accommodate tasks of different complexity levels.

Figure 5: Comparison between LLaVA-MoLE (Chen, Jie, and Ma 2024) and our
proposed RoDE. In our RoDE framework, experts are utilized across all tasks.
For individual tasks, expert activations are optimized to achieve sparsity.

#### Linear Rectified Router

The router consolidates the contributions of each expert based on the demands
of the specific task. Previous research (Zhou et al. 2022) has shown that the
use of sparse mixtures of LoRA experts outperforms dense experts in the
context of large language models. LLaVA-MoLE (Chen, Jie, and Ma 2024) presents
a top-1 selection strategy, which ensures the sparsity of expert selection in
LMM. While some methodologies in the Natural Language Processing (NLP) field
adopt a ’soft’ approach to combine expert outputs—for instance, (Dou et al.
2023) utilizes softmax and (Ponti et al. 2023) employs Gumbel softmax—these
techniques are not as sparse and can be challenging to optimize.

In contrast, our methodology adopts a novel approach by utilizing the
Rectified Linear Unit (ReLU) (Nair and Hinton 2010) to rectify the output of
the routers, thereby encouraging the sparse learning of LoRA expert
activations. A visual illustration of our routing strategy is shown in Figure
5. Leveraging the intrinsic properties of the Rectified Linear Unit (ReLU),
our approach benefits from a simplified optimization landscape and fosters
sparsity within the network, which enables our model to boost both efficiency
and efficacy.

Let $\gamma$ represent the linear rectification operation,
$\gamma(x)=\max(x,0)$. We utilize ReLU to rectify the linear router, resulting
in an adjusted linear output that can be expressed as follows:

$\displaystyle
x^{\prime}=W_{0}x+\sum_{i=0}^{N}\frac{\alpha}{r_{i}}\gamma(h_{i})B_{i}^{r_{i}}A_{i}^{r_{i}}x,$
(2)

where $(A_{i}^{r_{i}},B_{i}^{r_{i}})$ represents the operation performed by
the $i$-th LoRA expert. The summed output from the RoDE module is then added
to the frozen linear output and forwarded to the next module.

This linear rectification mechanism, enhances the sparsity of skill selection,
facilitating efficient utilization of expertise across a broad spectrum of
tasks. Overall, diverse LoRA experts and linear rectified arrangement yield a
fine-grained task-skill arrangement space, contributing to the optimized
allocation and utilization of resources within the model.

Method | Ingredient recognition | Recipe generation | Nutrition estimation (pMAE)
---|---|---|---
IoU $(\uparrow)$ | SacreBLEU $(\uparrow)$ | Rouge-L $(\uparrow)$ | Mass $(\downarrow)$ | Energy $(\downarrow)$ | Fat $(\downarrow)$ | Proteins $(\downarrow)$ | Carb $(\downarrow)$ | Avg.$(\downarrow)$
FoodLMM (Yin et al. 2023) | 22.59 | 7.59 | 36.44 | 48.49 | 47.72 | 61.49 | 56.61 | 51.25 | 53.11
FoodLMM+MoE | 24.54 | 8.7 | 38.5 | 49.23 | 48.24 | 60.59 | 58.37 | 51.87 | 53.66
RoDE (ours) | 26.86 | 9.66 | 40.11 | 48.18 | 46.96 | 60.96 | 58.16 | 48.65 | 52.58

Table 2: Comparative results for ingredient recognition, recipe generation and
nutrition estimation on the Uni-Food Dataset. The FoodLMM+MoE is derived by
integrating a Mixture of Experts (MoE) paradigm into the FoodLMM model.

### Optimization Objects and Inference

Following previous LMM-based methods (Liu et al. 2023a), the input images are
transformed into image tokens and concatenated with text tokens to send to
LLM. The training process adheres to the autoregressive model, predicting the
next token based on the input tokens. We employ the standard cross-entropy
loss as the optimization objective to train our model.

During inference, for each task, we input the image along with the
corresponding question into the model. The model’s output tokens are then
converted into words to formulate the answer.

## Experiment

In this section, we first present our experimental setup. Then we elaborate
experimental results of multiple food tasks based on multi-task learning.
Following this, we perform an ablation study to evaluate the impact and
effectiveness of our main components.

Nutrition5k | Mass $(\downarrow)$ | Cal $(\downarrow)$ | Fat $(\downarrow)$ | Proteins $(\downarrow)$ | Carb $(\downarrow)$ | Avg. $(\downarrow)$
---|---|---|---|---|---|---
FoodLMM(Yin et al. 2023) | 38.96 | 54.04 | 69.84 | 54.88 | 51.3 | 53.8
FoodLMM+MoE | 39.31 | 53.6 | 67.14 | 52.99 | 49.59 | 52.53
RoDE (ours) | 38.45 | 52.47 | 67.1 | 53.92 | 47.84 | 51.96

Table 3: Nutrition estimation results on Nutrtion5k dataset. The evaluation
metric employed is the percentage Mean Absolute Error (pMAE).

### Experiment Setup

Our approach commenced by utilizing the pre-trained weights of the LLaVA-
Lightning-7B-v1-1 (Liu et al. 2023a) model, subsequently applying SFT on the
Uni-Food and Nutrition5k datasets (Thames et al. 2021). In our RoDE model, we
employ a heterogeneous array of experts with LoRA ranks of [2, 4, 8, 16]. Our
primary baseline is FoodLMM (Yin et al. 2023), which to our knowledge is the
only versatile LMM specialized in food-related tasks. Additionally, we compare
with FoodLMM+MoE, which substitutes the plain LoRA module with a Mixture of
Experts (MoE) module. Each MoE module consists of four LoRAs with a rank of 8.
The routing mechanism for the MoE module is a linear layer followed by a
softmax operation, as detailed in Section Mixture of Experts (MoE)..
Importantly, for simplicity, we omit the specialized token tags and additional
heads used by FoodLMM specifically for nutrition estimation, and instead
directly convert the output tokens from the LMM into text to generate
nutritional predictions. For the standardization of the ingredient vocabulary,
we adopt the same clustering protocol used for ingredients in InverseCooking
(Salvador et al. 2019).

Tasks and Evaluation Metrics. The tasks and corresponding metrics in our
experiments are as follows. Ingredient Recognition: The objective here is to
identify and list the ingredients present in a dish shown in an image. We
assess performance using Intersection over Union (IoU) as the metric. Recipe
Generation: The goal of this task is to generate cooking instructions for the
dish captured in an image. We measure performance using the commonly used text
generation metrics in NLP, SacreBLEU (Post 2018) and Rouge-L (Lin 2004).
Nutrition Estimation: This task involves estimating a range of nutritional
parameters from a dish’s image, including the total food mass, total energy,
total fat, total carbohydrates, and total protein content. Energy is measured
in kilocalories, while the remaining parameters are quantified in grams. We
employ the mean absolute error as a percent of the respective mean for that
field (pMAE) (Thames et al. 2021) for the metric of nutrition estimation.

Figure 6: Sparsity Heatmap Comparison. This heatmap visualizes the disparity in task allocation sparsity between routers using the softmax strategy and those employing our proposed linear rectification method. The y-axis denotes the middle transformer blocks, while the x-axis represents the output of the routers.  Settings | LoRA rank | Routing strategy | MoE | SacreBLEU ($\uparrow$ ) | Rouge-L ($\uparrow$ ) | Ingredient IoU ($\uparrow$ ) | pMAE ($\downarrow$ )
---|---|---|---|---|---|---|---
Var1 | 8 | - | $\times$ | 7.59 | 36.44 | 22.59 | 53.11
Var2 | 32 | - | $\times$ | 8.39 | 36.68 | 22.7 | 54.14
Var3 | 8, 8, 8, 8 | Top-1 | ✓ | 8.28 | 37.2 | 24.08 | 53.53
Var4 | 8, 8, 8, 8 | Softmax | ✓ | 8.7 | 38.5 | 24.54 | 53.66
Table 4: Ablation Study on MoE. The integration of MoE can enhance the LMM, not solely due to the introduction of additional LoRA parameters Settings | LoRA rank | Routing strategy | SacreBLEU ($\uparrow$ ) | Rouge-L ($\uparrow$ ) | Ingredient IoU ($\uparrow$ ) | pMAE ($\downarrow$ )
---|---|---|---|---|---|---
Var1 | 8, 8, 8, 8 | Top-1 | 8.28 | 37.2 | 24.08 | 53.53
Var2 | 8, 8, 8, 8 | Softmax | 8.7 | 38.5 | 24.54 | 53.66
Var3 | 8, 8, 8, 8 | LR | 9.48 | 40.03 | 26.7 | 52.75
Var4 | 2, 4, 6, 8 | Softmax | 8.66 | 38.15 | 24.54 | 53.22
Var5 | 2, 4, 6, 8 | LR | 9.4 | 39.67 | 26.76 | 52.63
Table 5: Comparison of Three Routing Strategies. Our proposed Linear Rectified
Router demonstrates superior performance on both homogeneous and heterogeneous
combinations of experts.

Implementation Details. In this study, we utilize LLaVA-7B (Liu et al. 2023a)
as the base LMM, wherein a CLIP ViT-L model serves as the vision encoder, and
Llama 7B (Touvron et al. 2023) serves as the Language Model. For the CLIP
ViT-L encoder, the input image resolution is set at 336x336, which is then
divided into 14 patches. Subsequently, a two-layer MLP is used to transform
these image patches into 576 tokens. For efficiency, the LoRA module is added
to the query and key projection layer. Unless specifically stated, the rank of
LoRA is set to 8. Hyperparameter $\alpha$ is set to 16, and the LoRA dropout
rate is set to 0.05.

During the training process of all our experiments, the weights of vision
encoder and Llama are kept constant, with only the added LoRA modules being
trainable. We employ the AdamW optimizer (Loshchilov and Hutter 2017) and use
the WarmupDecayLR learning rate scheduler. The initial learning rate is set to
0.0003, weight decay is set to 0, and we perform 100 warm-up iterations. In
the training stage, we use a batch size of 4, and gradient updates are
computed every 10 batches. The training process is distributed across 4
RTX4090 GPUs. The training epoch is set to 1.

### Performance Comparison

We perform a comparative analysis between our model and two versions of
FoodLMM: a base version equipped with plane LoRA, and an enhanced version,
FoodLMM+MoE, equipped with four 8-rank LoRA and a linear router with softmax.
These methods are evaluated across three tasks: ingredient recognition, recipe
generation, and nutrition estimation. The results are summarized in Table 2.
The table clearly shows that the MoE design enhances the performance of
FoodLMM. By adopting the traditional MoE paradigm, i.e., FoodLMM+MoE, there is
an 8.6% improvement in IoU for ingredients recognition, for recipe generation,
there is a 14.6% increase in SacreBLEU and a 1.1% gain in Rouge-L scores.
However, there is a slight reduction of 1% in nutritional performance.
Overall, the comparison demonstrates the advantages of incorporating MoE.

Furthermore, our model, RoDE, significantly outperforms the traditional MoE
design, achieving the highest scores across all evaluated metrics. In the
ingredient recognition task, RoDE surpasses FoodLMM+MoE by a notable margin of
9.5%. For recipe generation, RoDE shows an 11% higher SacreBLEU score and a
4.2% improvement in Rouge-L score compared to FoodLMM+MoE. In the area of
nutrition estimation, RoDE nearly tops the charts for all nutrient elements
and secures the leading position in terms of average performance. These
experimental results conclusively affirm the superiority of our proposed RoDE
approach in food multi-task learning.

Additionally, we conduct further experiments on Nutrition5k (Thames et al.
2021) dataset, adhering to the methodology established by FoodLMM, where only
image information was utilized for training (without depth information). The
outcomes, as presented in Table 3, unequivocally demonstrate that our methods
consistently surpass both FoodLMM and traditional MoE variations in
performance.

Figure 7: Performance Comparison of Routing Strategies: “Top-1” selects the top-1 expert accorarding router’s score. “Softmax” applies a softmax operation to process the router’s output, while “LR” uses linear rectified router. All the three strategies utilize a consistent LoRA rank setting of [8,8,8,8].  Figure 8: Correlation between the Quantity of Trainable LoRA Parameters and Task Performance Across Five LoRA Settings. For simplicity, the total LoRA ranks in a single layer are quantified as the trainable parameters.  Settings | LoRA rank | Routing strategy | SacreBLEU ($\uparrow$ ) | Rouge-L ($\uparrow$ ) | Ingredient IoU ($\uparrow$ ) | pMAE ($\downarrow$ )
---|---|---|---|---|---|---
Var1 | 5, 5, 5, 5 | LR | 8.4 | 38.04 | 25.02 | 53.42
Var2 | 8, 8, 8, 8 | LR | 9.48 | 40.03 | 26.7 | 52.75
Var3 | 2, 4, 6, 8 | LR | 9.4 | 39.67 | 26.76 | 52.63
Var4 | 2, 4, 8, 16 | LR | 9.66 | 40.11 | 26.86 | 52.58
Table 6: Ablation Study on the Rank Settings of Experts within MoE. The
heterogeneous combination of experts yields superior performance while
utilizing a reduced parameter count.

### Ablation study

In this section, we perform an ablation study to evaluate the contribution and
efficacy of the core components within our framework. For simplicity, we omit
the task names and tag only the metrics.

#### MoE vs. Larger LoRA rank

As illustrated in Table 4, we first compare the standard LoRA (Var1) with its
MoE variant (Var3). The results indicate a superior performance of the MoE
design. To ascertain whether this performance improvement can be attributed to
an increase in the number of trainable LoRA parameters, we augment the LoRA
rank of Var1 to 32 to obtain Var2. A comparison between Var2 and Var4 reveals
the superiority of the MoE design, even though these two variants have an
equivalent number of trainable LoRA parameters. This finding suggests that the
successful integration of MoE is not simply a result of expanding the number
of trainable parameters.

#### Routing Strategy

Then we examine the effects of the routing strategy. Our experiment includes
three routing strategies: 1) similar to (Chen, Jie, and Ma 2024), the top-1
expert is selected, referred to as Top-1; 2) following the approach in (Dou et
al. 2023), we employ a softmax function to normalize the router outputs, which
is denoted as Softmax; 3) we introduce our novel linear rectified router,
which we denote with the abbreviation LR. The results are summarized in Table
5 and visualized in Figure 7. As observed, while the Top-1 allocation MoE
outperforms the standard LoRA (Var1), it is surpassed by the ”soft” allocation
strategies (i.e., Var2 and Var3). This indicates the efficiency of employing
an ensemble of experts to address a single task. Moreover, the comparison
between the LR and Softmax routers (i.e.,  Var2 and Var3, Var4 and Var5)
suggests that the LR routing strategy is superior. These results further
underscore the effectiveness of our linear rectification routing strategy.

In addition, we visualize the heatmaps of router outputs for both the Softmax
and LR routing strategies across several middle transformer blocks. Figure 6
illustrates this comparative analysis, clearly indicating that our proposed
Linear Rectified Router achieves a higher degree of sparsity in task
allocation.

#### Rank configuration of LoRA

We conduct an ablation study to evaluate the impact of different LoRA rank
configurations. The results, as presented in Table 6, span multiple expert
configurations, including: 1) Var1 \- Four LoRAs, each with a uniform rank of
5. 2) Var2 \- Four LoRAs, each with a uniform rank of 8. 3) Var3 \- A
heterogeneous set of four LoRAs with ranks of [2, 4, 6, 7]. 4) Var4 \- Another
diverse set of four LoRAs with a distinct rank composition from Var3,
specifically [2, 4, 8, 16]. The analysis of Var1 and Var2 suggests that
configurations with higher-ranked LoRA experts are more effective. However,
despite having the same number of trainable parameters as Var1, Var3
demonstrates superior performance, indicating that a mix of higher and lower-
ranked LoRA experts can also be effective. Moreover, Var3 shows comparable
performance to Var2, with only a slight reduction in SacreBLEU score (by 0.08)
and Rouge-L score (by 0.36), while having 32.5% fewer trainable parameters.
This finding supports the notion that not all LoRA experts require high
capacity to contribute effectively.

By adjusting the ranks in Var3 to create Var4, we observe improved performance
which exceeds that of Var2, despite Var4 having 6.25% fewer trainable
parameters. This outcome implies that a strategic combination of LoRA expert
sizes can effectively support the inclusion of some larger LoRAs, potentially
enhancing their capability to handle complex tasks more efficiently.

### Training Parameter Efficiency

We evaluate various rank settings by visualizing the correlation between the
number of trainable LoRA parameters and their corresponding task performance
on the Uni-Food dataset. Figure 8 illustrates this relationship, with the
y-axis indicating task performance and the x-axis representing the trainable
LoRA parameter count. It is apparent that the MoE architecture incorporates
more trainable LoRA parameters, which substantially boosts performance across
a range of metrics, i.e., recipe SacreBLEU, recipe Rouge-L, and ingredient
IoU. When comparing sets of homogeneous experts to those of heterogeneous
experts (i.e., comparing rank sets [5,5,5,5] to [2,4,6,8]), the introduction
of high rank LoRA can enhance model performance despite when the number of
trainable LoRA parameters is the same. These results suggest that not every
expert requires an extensive parameter set; heterogeneous experts can
effectively coordinate trainable parameters, resulting in better performance
on multi-task learning.

## Conclusion

In this paper, we delve into the broader scope of tasks within the realm of
food studies. We introduce Uni-Food, a comprehensive dataset comprising
classification, ingredient recognition, recipe generation, and nutrition
estimation. Uni-Food serves as a foundational resource empowering a wide
spectrum of food-related research endeavors. The expansion of training tasks
brings about a challenge of task conflict. To mitigate these task conflicts
during SFT on Large LMMs, we introduce the Linear Rectified Mixture of Diverse
Experts (RoDE). RoDE constructs a skill space that experts are sharable, and
arranges a variety of trainable parameters to establish heterogeneous experts
capable of adapting to tasks of different complexities. It employs linear
rectified units to refine linear routers, encouraging the routers to learn
sparse allocation. RoDE is not only GPU memory-efficient but it also
simplifies the optimization process. Our experimental results clearly
demonstrate the effectiveness of our proposed method, highlighting its
efficiency and efficacy in overcoming the multifaceted challenges associated
with the food domain.

## References

  * Achiam et al. (2023) Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ando et al. (2019) Ando, Y.; Ege, T.; Cho, J.; and Yanai, K. 2019.  DepthCalorieCam: A Mobile Application for Volume-Based FoodCalorie Estimation using Depth Cameras.  In _Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management_ , MADiMa ’19, 76–81. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450369169.
  * Bossard, Guillaumin, and Van Gool (2014) Bossard, L.; Guillaumin, M.; and Van Gool, L. 2014.  In _Computer Vision – ECCV 2014_ , 446–461. Cham: Springer International Publishing.
  * Chen and Ngo (2016a) Chen, J.; and Ngo, C.-W. 2016a.  Deep-based ingredient recognition for cooking recipe retrieval.  In _Proceedings of the 24th ACM international conference on Multimedia_ , 32–41.
  * Chen and Ngo (2016b) Chen, J.; and Ngo, C.-w. 2016b.  Deep-based Ingredient Recognition for Cooking Recipe Retrieval.  In _Proceedings of the 24th ACM International Conference on Multimedia_ , MM ’16, 32–41. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450336031.
  * Chen et al. (2020) Chen, J.; Zhu, B.; Ngo, C.-W.; Chua, T.-S.; and Jiang, Y.-G. 2020.  A study of multi-task and region-wise deep learning for food ingredient recognition.  _IEEE Transactions on Image Processing_ , 30: 1514–1526.
  * Chen et al. (2021) Chen, J.; Zhu, B.; Ngo, C.-W.; Chua, T.-S.; and Jiang, Y.-G. 2021.  A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition.  _IEEE Transactions on Image Processing_ , 30: 1514–1526.
  * Chen, Jie, and Ma (2024) Chen, S.; Jie, Z.; and Ma, L. 2024.  Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms.  _arXiv preprint arXiv:2401.16160_.
  * Chhikara et al. (2024) Chhikara, P.; Chaurasia, D.; Jiang, Y.; Masur, O.; and Ilievski, F. 2024.  FIRE: Food Image to REcipe Generation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 8184–8194.
  * Ding et al. (2023) Ding, N.; Qin, Y.; Yang, G.; Wei, F.; Yang, Z.; Su, Y.; Hu, S.; Chen, Y.; Chan, C.-M.; Chen, W.; et al. 2023.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , 5(3): 220–235.
  * Dou et al. (2023) Dou, S.; Zhou, E.; Liu, Y.; Gao, S.; Zhao, J.; Shen, W.; Zhou, Y.; Xi, Z.; Wang, X.; Fan, X.; et al. 2023.  Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment.  _arXiv preprint arXiv:2312.09979_.
  * Gao et al. (2023) Gao, J.; Chen, J.; Fu, H.; and Jiang, Y.-G. 2023.  Dynamic Mixup for Multi-Label Long-Tailed Food Ingredient Recognition.  _IEEE Transactions on Multimedia_ , 25: 4764–4773.
  * Gou et al. (2023) Gou, Y.; Liu, Z.; Chen, K.; Hong, L.; Xu, H.; Li, A.; Yeung, D.-Y.; Kwok, J. T.; and Zhang, Y. 2023.  Mixture of cluster-conditional lora experts for vision-language instruction tuning.  _arXiv preprint arXiv:2312.12379_.
  * Hu et al. (2021) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Huang et al. (2023) Huang, C.; Liu, Q.; Lin, B. Y.; Pang, T.; Du, C.; and Lin, M. 2023.  Lorahub: Efficient cross-task generalization via dynamic lora composition.  _arXiv preprint arXiv:2307.13269_.
  * Kawano and Yanai (2014) Kawano, Y.; and Yanai, K. 2014.  Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation.  In _Proc. of ECCV Workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV)_.
  * Lai et al. (2023) Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2023.  Lisa: Reasoning segmentation via large language model.  _arXiv preprint arXiv:2308.00692_.
  * Lin et al. (2024) Lin, B.; Tang, Z.; Ye, Y.; Cui, J.; Zhu, B.; Jin, P.; Zhang, J.; Ning, M.; and Yuan, L. 2024.  Moe-llava: Mixture of experts for large vision-language models.  _arXiv preprint arXiv:2401.15947_.
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , 74–81.
  * Liu et al. (2023a) Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023a.  Visual Instruction Tuning.
  * Liu et al. (2023b) Liu, Q.; Wu, X.; Zhao, X.; Zhu, Y.; Xu, D.; Tian, F.; and Zheng, Y. 2023b.  Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications.  _arXiv preprint arXiv:2310.18339_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Luo et al. (2023) Luo, M.; Min, W.; Wang, Z.; Song, J.; and Jiang, S. 2023.  Ingredient prediction via context learning network with class-adaptive asymmetric loss.  _IEEE Transactions on Image Processing_.
  * Marin et al. (2019) Marin, J.; Biswas, A.; Ofli, F.; Hynes, N.; Salvador, A.; Aytar, Y.; Weber, I.; and Torralba, A. 2019.  Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images.  _IEEE Trans. Pattern Anal. Mach. Intell._
  * Min et al. (2023) Min, W.; Wang, Z.; Liu, Y.; Luo, M.; Kang, L.; Wei, X.; Wei, X.; and Jiang, S. 2023.  Large scale visual food recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Nair and Hinton (2010) Nair, V.; and Hinton, G. E. 2010.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , 807–814.
  * Pan et al. (2020) Pan, L.-M.; Chen, J.; Wu, J.; Liu, S.; Ngo, C.-W.; Kan, M.-Y.; Jiang, Y.; and Chua, T.-S. 2020.  Multi-modal cooking workflow construction for food recipes.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , 1132–1141.
  * Ponti et al. (2023) Ponti, E. M.; Sordoni, A.; Bengio, Y.; and Reddy, S. 2023.  Combining parameter-efficient modules for task-level generalisation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , 687–702.
  * Post (2018) Post, M. 2018.  A call for clarity in reporting BLEU scores.
  * Qiu et al. (2022) Qiu, J.; Lo, F. P. W.; Sun, Y.; Wang, S.; and Lo, B. 2022.  Mining Discriminative Food Regions for Accurate Food Recognition.  arXiv:2207.03692.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; et al. 2018.  Improving language understanding by generative pre-training.
  * Salvador et al. (2019) Salvador, A.; Drozdzal, M.; Giró-i Nieto, X.; and Romero, A. 2019.  Inverse cooking: Recipe generation from food images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10453–10462.
  * Salvador et al. (2017) Salvador, A.; Hynes, N.; Aytar, Y.; Marin, J.; Ofli, F.; Weber, I.; and Torralba, A. 2017.  Learning cross-modal embeddings for cooking recipes and food images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3020–3028.
  * Thames et al. (2021) Thames, Q.; Karpur, A.; Norris, W.; Xia, F.; Panait, L.; Weyand, T.; and Sim, J. 2021.  Nutrition5k: Towards automatic nutritional understanding of generic food.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8903–8911.
  * Touvron et al. (2023) Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2022) Wang, Z.; Min, W.; Li, Z.; Kang, L.; Wei, X.; Wei, X.; and Jiang, S. 2022.  Ingredient-Guided Region Discovery and Relationship Modeling for Food Category-Ingredient Prediction.  _IEEE Transactions on Image Processing_ , 31: 5214–5226.
  * Yin et al. (2023) Yin, Y.; Qi, H.; Zhu, B.; Chen, J.; Jiang, Y.-G.; and Ngo, C.-W. 2023.  FoodLMM: A Versatile Food Assistant using Large Multi-modal Model.  _arXiv preprint arXiv:2312.14991_.
  * Zhou et al. (2022) Zhou, Y.; Lei, T.; Liu, H.; Du, N.; Huang, Y.; Zhao, V.; Dai, A. M.; Le, Q. V.; Laudon, J.; et al. 2022.  Mixture-of-experts with expert choice routing.  _Advances in Neural Information Processing Systems_ , 35: 7103–7114.
  * Zhu et al. (2019) Zhu, B.; Ngo, C.-W.; Chen, J.; and Hao, Y. 2019.  R2gan: Cross-modal recipe retrieval with generative adversarial network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11477–11486.