# Prompting Medical Large Vision-Language Models to Diagnose Pathologies by
Visual Question Answering

Danfeng Guo<EMAIL_ADDRESS>
Computer Science Department, University of California, Los Angeles, CA, USA
Demetri Terzopoulos<EMAIL_ADDRESS>
Computer Science Department, University of California, Los Angeles, CA, USA
VoxelCloud, Inc., Los Angeles, CA, USA

###### Abstract

Large Vision-Language Models (LVLMs) have achieved significant success in
recent years, and they have been extended to the medical domain. Although
demonstrating satisfactory performance on medical Visual Question Answering
(VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem,
which makes them fail to diagnose complex pathologies. Moreover, they readily
fail to learn minority pathologies due to imbalanced training data. We propose
two prompting strategies for MLVLMs that reduce hallucination and improve VQA
performance. In the first strategy, we provide a detailed explanation of the
queried pathology. In the second strategy, we fine-tune a cheap, weak learner
to achieve high performance on a specific metric, and textually provide its
judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our
methods significantly improve the diagnostic F1 score, with the highest
increase being 0.27. We also demonstrate that our prompting strategies can be
extended to general LVLM domains. Based on POPE metrics, it effectively
suppresses the false negative predictions of existing LVLMs and improves
Recall by approximately 0.07.

Keywords: Medical Visual Question Answering, Large Vision-Language Models,
Prompt Engineering

##  1 Introduction

Research on Large Language Models (LLMs) has yielded astonishing results in
recent years. LLMs with billions of parameters have achieved outstanding
abilities in a wide range of application scenarios (OpenAI, 2022; OpenAI et
al., 2023; Chiang et al., 2023). The success of LLMs has quickly extended into
the Vision-Language (VL) domain. Large Vision-Language Models (LVLMs) are
built upon LLMs by training adapters that project visual features into
features that can be interpreted by LLMs (Li et al., 2023b; Zhang et al.,
2023; Liu et al., 2023b). Visual Question Answering (VQA) is an essential
skill of LVLMs, and VQA accuracy serves as a test metric for most of these
models (Li et al., 2023b; Zhang et al., 2023; Zhu et al., 2023; Liu et al.,
2023b). LVLMs have been pretrained on medical datasets (Li et al., 2023a; Liu
et al., 2023c; Singhal et al., 2023) and they have been tested on medical VQA
tasks (Lau et al., 2018; He et al., 2020). These Medical LVLMs (MLVLMs) have
been able to answer questions regarding the imaging modalities, organs, and
abnormalities depicted by the input medical scans.

Unfortunately, however, “hallucination” has been a major problem for LVLMs.
This refers to the generation of content that is contradictory to the input
images. Hallucination can be measured via VQA. One may ask the model questions
regarding the existence of objects in the input image(s) and the hallucination
level is assessed as the percentage of correctly answered questions. VQA can
also potentially serve for medical image diagnosis. Users pose questions
regarding a pathology and the MLVLM responds based on its analysis of the
medical scans. However, most of the available datasets involve simple
questions such as “what is the modality of this image” and “what is the
organ/tissue in this image”. MLVLMs have yet to be thoroughly evaluated on VQA
accuracy across a broad spectrum of pathologies. Additionally, general VQA
models are usually tested by the commonly known accuracy: the percentage of
correctly answered questions, which is an unsuitable measure for medical VQA.
Medical image classification metrics such as the Precision, Recall, and F1 are
more suitable for the evaluation of medical VQA models. Several strategies
have been explored to enhance the question answering of LLMs/LVLMs, including
chain-of-thought prompting (Zheng et al., 2023), self-consistency (Wang et
al., 2023), and retrieval-based augmentation (Caffagni et al., 2024). All
these methods involve fine-tuning the models, which is expensive. Training-
free methods to improve the VQA accuracy of MLVLMs are desirable.

For MLVLMs, hallucination is exacerbated by imbalanced training data. Many
pathologies are minority categories in medical datasets. Models trained on
large-scale medical data may easily fail to learn the features of less common
pathologies. Addressing data bias typically involves strategies such as
including more data of better quality, but given the scarcity of medical data,
significantly enlarging the dataset may not be feasible. Common remediations
involve re-sampling the data such that the positive and negative cases are
better balanced, but this poses challenges when the data involves multiple
categories of pathology. Additionally, re-sampling may undermine the training
needs of LVLMs, which generally require large quantities of data. These
problems highlight the need of a cost-effective approach to navigate the
problem of minority categories in datasets.

Our study focuses on the VQA abilities of MLVLMs. In particular, we test an
existing MLVLM, LLaVA-Med (Li et al., 2023a) for chest X-ray VQA across 5
categories of pathologies. The results show that the model has low accuracy,
especially on minority pathologies. To enhance its VQA accuracy, we propose
two prompting strategies. The first involves enriching prompts with detailed
explanations of the queried pathology. The explanations include how the
queried pathology is defined and how it appears in images. Our second strategy
involves introducing an auxiliary weak-learner model as another agent. We
train a small image classifier and fine-tune it to identify negative images
accurately. Then, the negative predictions of this classifier are appended to
the prompt as a reference for the MLVLM.

We run our experiments on the MIMIC-CXR-JPG (Goldberger et al., 2000) and
Chexpert (Irvin et al., 2019) datasets. The results show that our prompt
strategies improve the F1 score significantly in most pathology categories
(highest +0.27). We also show that our weak-learner-prompting strategy is
applicable to the general domain. It reduces the false negative predictions of
general domain LVLMs and improves the Recall by around 0.07 according to POPE
metrics (Li et al., 2023c).

To summarize, our contributions include the following:

  1. 1.

We improve the VQA accuracy of MLVLMs by prompting with detailed explanations
of pathologies.

  2. 2.

We introduce a low-cost weak learner model as a reference for LLaVA-Med, and
this effectively reduces the false positive (FP) answers.

  3. 3.

We show that our second prompting strategy can be extended to general domains
to help models adapt to specialized accuracy needs.

Section 2 reviews related work, Section 3 describes our methodology, Section 4
presents our empirical study and its results, and Section 5 draws conclusions
from our research.

##  2 Related Work

#### LVLMs and VQA

LVLMs are built upon LLMs. A pretrained visual encoder extracts the visual
features and an adapter module projects the extracted features to ones that
can be understood by the LLM. Models of this type include those by Liu et al.
(2023b), Zhu et al. (2023), and Zhang et al. (2023). During training, the
visual encoder and the LLM are usually fixed. VQA is an essential skill of
LVLMs. Given an input image, the models should be able to answer questions
correctly regarding that image.

#### Hallucination in LVLM VQA

The hallucination problem usually refers to the LVLM generating a response
that is not consistent with the input image. For VQA, in their generated
answers the models may make mistakes on object presence, location, attributes,
or the mutual relationship between objects. Li et al. (2023c) find that
frequently occurring objects are easily hallucinated by LVLMs, in that they
tend to mention such objects even if it they are absent in the image. Qian et
al. (2024) and Liu et al. (2023a) show that LVLMs sometimes presume the
assumptions in questions are true and easily give wrong answers when asked
about some objects not in the given image.

#### Causes of LVLM VQA Hallucination

Hallucination can result from bias in the training data, missing fine-grained
visual features, and LLM decoding strategies (Liu et al., 2024). For data
bias, the imbalanced distribution of data is an important aspect. When most of
the answers to a question in the training data are “Yes”, the model tends to
answer “Yes” to that question. Missing fine-grained visual features usually
result from the pretraining of the visual encoder. Most LVLMs use the visual
encoder of CLIP trained through contrastive learning. The encoder mainly
focuses on salient features and ignores fine-grained features (Jain et al.,
2023). LVLM decoding strategies mostly choose the next word as the one having
maximum conditional probability given previous text and the input image. This
can lead to hallucination when the model overly relies on the knowledge
learned in its training texts. Other causes include model simplicity and
insufficient attention (Liu et al., 2024).

#### Mitigation of LVLM VQA Hallucination

Strategies to mitigate hallucination in LVLMs mainly fall into two categories:
prompt engineering and model improvement. Regarding the former, Liu et al.
(2023a) leverage visual instructions constructed from the bounding box
information in the input image to prompt LLMs. Zheng et al. (2023) use a chain
of thought scheme to prompt the models to perform step-by-step visual-language
reasoning like humans, which eventually leads to the correct answers. Wang et
al. (2023) generate multiple chains of thought and use the one with the
majority vote as the answer. Caffagni et al. (2024) prompt the model with
explanations of the terms in questions. With regard to the model improvement
strategy for reducing hallucination, Sun et al. (2023b) improve the visual and
text feature alignment through reinforcement learning. Leng et al. (2023)
propose a contrastive decoding strategy to reduce reliance on pretrained
knowledge. Favero et al. (2024) and Zhao et al. (2024) also focus on the
inference stage and propose specialized decoding strategies to mitigate
hallucination. Other strategies for reducing hallucination have been proposed.
For example, Zhou et al. (2024) design a post-processing model to detect
hallucinated objects and rephrase the generated answers, and Sun et al.
(2023a) adapt a reinforcement learning strategy that uses human evaluation of
the hallucination level to improve the model.

#### Assessment of LVLM Hallucination

There are two approaches to assessing hallucination in LVLMs. The first is
VQA. The ground truth information of the input images is leveraged to
construct questions regarding the existence of objects in the images (e.g.,
“Is there a black cat in the image?”), as well as questions about objects
which do not exist in the images. The models are evaluated in terms of the
percentage of correctly answered questions. Metrics of this type include POPE
(Li et al., 2023c), CIEM (Hu et al., 2023), and NOPE (Lovenia et al., 2023).
The second approach is to use pre-designed prompts from which the models
produce various generations that are then evaluated. Examples include CHAIR
(Rohrbach et al., 2018), which counts the hallucinated objects in generated
image captions, and MMHAL-BENCH (Sun et al., 2023b), which uses GPT-4 (OpenAI
et al., 2023) to compare the generations with human answers and determine the
propensity toward hallucination.

#### VQA in MLVLMs

For MLVLMs, given a medical scan, models such as LLaVA-Med (Li et al., 2023a)
and Med-PALM (Liu et al., 2023c) are able to answer questions regarding the
types of modalities, the scanned organs, and medical indicators such as
opacity. They have demonstrated good performance on medical VQA datasets such
as VQA-RAD (Lau et al., 2018), SLAKE (Liu et al., 2021), and Path-VQA (He et
al., 2020). However, most medical questions in existing datasets are simple.
MLVLMs have not been tested on a broader range of complex pathologies.

Figure 1: The structure of common LVLMs.

##  3 Methodology

Figure 1 illustrates the structure of common LVLMs. They are based on a
pretrained unimodal LLM such as Llama (Touvron et al., 2023) and Vicuna
(Chiang et al., 2023). A pretrained visual encoder, such as ViT (Dosovitskiy
et al., 2021) or conventional CNNs, is applied to extract image features that
are projected to the text feature space by an adapter. The projected visual
features are concatenated with the text prompt embeddings and fed to the LLM.
The adapter usually consists of several linear layers with non-linear
activations. The visual encoder and the LLM are usually frozen during
training.

In our work, we choose the pretrained LLaVA-Med (Li et al., 2023a) as our
model, which is a MLVLM built upon LLaVA (Liu et al., 2023b). The model
structure resembles Figure 1. It uses pretrained Vicuna (Chiang et al., 2023)
as the LLM and the pretrained ViT encoder from CLIP (Radford et al., 2021) as
the visual encoder. The adapter is simply a trainable projection matrix. Both
the visual encoder and LLM weights are frozen during training. LLaVA-Med fine-
tunes LLaVA in two steps. First, it fine-tunes LLaVA to generate medical
reports from input medical images. Second, it uses GPT-4 to generate various
questions from the ground truth reports and fine-tunes the model to perform
question answering.

Most MLVLMs are currently trained by medical VQA such that medical diagnosis
can be performed by asking questions related to various pathologies; e.g.,
“Does this image have lung lesion?”. To reduce model hallucination and improve
VQA accuracy, we propose two prompting strategies at the inference stage: (1)
providing the model with detailed explanations about the queried pathologies
and (2) asking the model to consider the inferences of a weak learner.

###  3.1 Prompting With Detailed Explanations

Given imbalanced training data, MLVLMs might not adequately be able to learn
the features of the minority pathologies. To compensate for insufficient
training, we provide a detailed explanation of the queried pathology as a
prompt at the inference stage. The explanation briefly defines the pathology
and lists several key findings in medical images that may indicate its
existence. An example is shown in Figure 2. The model is informed that
Pulmonary Edema is defined as the accumulation of fluid in the lungs. Then
several chest X-ray findings that may suggest its existence are provided. The
model can determine if the given image has Pulmonary Edema by linking the
given findings with the image features.

Prompt templates for a number of pathologies are listed in Section A.

Figure 2: An example of including pathology explanations when prompting an
MLVLM for medical VQA. Figure 3: Example of prompting an MLVLM for medical VQA
using both pathology explanations and reference predictions from a weak
learner.

###  3.2 Prompting With Detailed Explanations and Weak Learners

Data re-sampling is a commonly-used strategy to deal with imbalanced datasets
that are responsible for the tendency of traditional image classification
models to return negative predictions for minority pathologies. Models trained
on re-sampled datasets often exhibit improvements in Precision and Recall
scores; however, this strategy may not be suitable to MLVLMs for two reasons.
First, it is difficult to balance a dataset containing many categories of
pathologies. Second, MLVLMs usually demand much larger datasets and fine-
tuning is also expensive.

One can nevertheless enable MLVLMs to benefit by leveraging small models
trained on re-sampled datasets. Our method resembles multiagent LLM systems,
such as Du et al. (2023), where multiple LLMs debate each other and
hallucination can be corrected by referring to the generated outputs of other
models. Given that traditional image classifiers are smaller, it is feasible
to train multiple small classifiers each of which is trained on re-sampled
datasets of a particular pathology. Those models can be further fine-tuned to
optimize a single aspect, such as fewer false positives (FP) or fewer false
negatives (FN). The classifiers are applied to the medical images and return
preliminary predictions. These predictions are selectively included in the
prompts as references for the MLVLM. Hence, MLVLMs can benefit indirectly from
the nuanced understanding that these specialized models can provide. This
method is meaningful because clinicians usually must balance the trade-off
between overtreatment and undertreatment when making healthcare decisions. For
instance, they may prefer models having a low FP rate if the cost of
overtreatment is higher than that of undertreatment.

An example is shown in Figure 3, which queries about the presence of Edema. We
first provide the model with the detailed explanation of Edema. Then, we use
the weak learner to suppress the FPs. The image is input to an Edema
classifier that has been fine-tuned on a balanced dataset for high sensitivity
and high true negative (TN) rate. If its prediction is negative, we append
after the pathology explanation the prompt “For this image, another agent
thinks the probability of Edema is 0.1”. Instead of using the actual predicted
probability, the probability value is manually chosen because the decision
threshold has been fine-tuned and is no longer 0.5. We do not use a zero
probability value because we do not want the model overly to trust the weak
learner. Although in this example our goal is only to reduce FPs, our strategy
can also be applied to reduce FNs, simply by fine-tuning the classifier for a
high true positive (TP) rate and applying the prompt in the case of positive
predictions.

##  4 Empirical Study

###  4.1 Datasets

Pathology |  $+$Cases
---|---
Atelectasis | 64
Cardiomegaly | 31
Consolidation | 335
Edema | 1,276
Pleural Effusion | 260
Table 1: Positive ($+$) case counts for the 5 test pathologies in the LLaVA-
Med training set.

LLaVA-Med is pretrained on the PMC-15M dataset (Zhang et al., 2024), which
contains image-text pairs of multiple modalities; e.g., CT, MRI, X-ray, etc.
In the first stage, 467,710 image-report pairs were selected for training. In
the second stage, 56,708 question-answer pairs were created from the data of
the first stage to fine-tune the model. Table 1 shows the count of reports in
the LLaVA-Med training data (second stage) that mention one of the five test
pathologies as positive. Relative to the total amount of data, all five
categories are minorities.

| MIMIC-CXR-JPG (5,159) | Chexpert (668)
---|---|---
Category |  $+$Cases |  $-$Cases |  $+$Cases |  $-$Cases
Atelectasis |  1,034 |  4,125 |  178 |  490
Cardiomegaly |  1,258 |  3,901 |  175 |  493
Consolidation |  326 |  4,833 |  35 |  633
Edema |  959 |  4,200 |  85 |  583
Enlarged Cardiomediastinum |  200 |  4,959 |  298 |  370
Fracture |  167 |  4,992 |  6 |  662
Lung Lesion |  202 |  4,957 |  14 |  654
Lung Opacity |  1,561 |  3,598 |  310 |  358
Pleural Effusion |  1,542 |  3,617 |  120 |  548
Pleural Other |  119 |  5,040 |  8 |  660
Pneumonia |  539 |  4,620 |  14 |  654
Pneumothorax |  144 |  5,015 |  10 |  658
Support Devices |  1,457 |  3,702 |  315 |  353
Table 2: Splits of positive ($+$) and negative ($-$) cases (‘uncertain’ is
regarded as negative) for the 13 finding categories in the MIMIC-CXR-JPG and
Chexpert test sets.

To assess the zero-shot performance of the MLVLM, we used the MIMIC-CXR-JPG
(Goldberger et al., 2000) and Chexpert (Irvin et al., 2019) chest X-ray test
sets. They include 5,159 and 668 images, respectively. Neither dataset
overlaps with PMC-15M.

MIMIC-CXR-JPG includes images and medical reports covering 13 categories of
findings: Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged
Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion,
Pneumonia, Pneumothorax, Pleural Other, and Support Devices. The raw reports
are parsed and rough image-level tags are automatically generated by a rule-
based approach (Irvin et al., 2019). Each label contains four values: 1
(positive), 0 (negative), $-1$ (uncertain), and missing. For simplicity, we
treat both uncertain and missing as negative. We also use the MIMIC-CXR-JPG
training set, which contains 227,827 chest X-rays with reports, to train the
weak learner models.

Chexpert covers the same 13 categories as MIMIC-CXR-JPG. However, it does not
include medical reports and has only image-level labels. There is no overlap
between MIMIC-CXR-JPG and Chexpert.

Table 2 shows the split of pathology categories (excluding normal) in the
MIMIC-CXR-JPG and Chexpert test sets. Clearly, almost all pathology categories
are minor classes with much fewer positive than negative occurrences.

For our main testing regimen, we selected the five pathologies in the Chexpert
Competition (Irvin et al., 2019), Atelectasis, Cardiomegaly, Consolidation,
Edema, and Pleural Effusion.

###  4.2 Implementation Details

Prompt |  Template
---|---
PT1 |  “Does this image have {target}?”
PT2 |  “{explanation} Given the information above, does this image have {target}?”
PT3 |  “{explanation} For this image, another agent thinks the probability that it has {target} is {n} percent. Given the information above, does this image have {target}?”
Table 3: The Prompt Templates (PTs). {target} is the pathology cited in the
questions. {explanation} contains a pathology explanation among those listed
in Section A. {n} is the probability associated with the weak learner.

As was mentioned in Section 3, we use the pretrained LLaVA-Med MLVLM without
any further fine-tuning. We convert the classification task into a VQA task by
using the prompt template shown in Row 1 of Table 3, which we name Prompt
Template 1 (PT1). We first run the pretrained LLaVA-Med with PT1. Next, we
incorporate pathology explanations (Row 2 of Table 3), yielding Prompt
Template 2 (PT2). Finally, we integrate the predictions of weak learners into
the prompts (Row 3 of Table 3), resulting in Prompt Template 3 (PT3).

As will be justified by our experiments, our weak learner is designed to
suppress FP predictions. To this end, we use the pretrained ResNet50 (He et
al., 2016). For each pathology, the training dataset was sampled such that the
ratio of positive and negative cases is $2:1$. The model was trained for 10
epochs with a $1e-4$ learning rate. The training process was monitored using
the AUC score and the one with the highest validation AUC was kept. Then, the
decision threshold $d$ was fine-tuned to optimize a weighted sum of
Specificity and Negative Predictive Value (NPV); i.e.,

$d=w_{1}\frac{TN}{TN+FP}+w_{2}\frac{TN}{TN+FN},$ (1)

where weights $w_{1}$ and $w_{2}$ are preset to 0.2 and 0.8, respectively. The
medical images were input to the weak learners to obtain preliminary
predictions for each pathology and only the negative predictions were selected
to craft the PT3 prompts.

The responses returned by LLaVA-Med can take various forms, such as “This
image has Edema”, “Edema is found”, “The fluid in the lung indicates Edema”,
etc. An off-the-shelf Llama-7B (Touvron et al., 2023) serves to summarize long
responses into Yes/No answers such that accuracies could easily be computed.

###  4.3 Results

To demonstrate the efficacy of our prompting strategies, starting from the PT1
baseline, the pathology explanations were provided first (strategy PT2) and
then, based on the results, weak learners were introduced to improve
performance on specific aspects, resulting in strategy PT3.

|  | MIMIC-CXR-JPG | Chexpert
---|---|---|---
Pathology | Metric |  PT1 |  PT2 |  PT1 |  PT2
Atelectasis | Precision |  19.5 |  20.0 |  30.5 |  26.5
Recall |  41.5 |  92.9 |  44.4 |  91.6
F1 |  26.5 |  33.0 |  36.5 |  41.0
Cardiomegaly | Precision |  25.8 |  24.6 |  27.1 |  26.0
Recall |  22.5 |  89.4 |  20.0 |  86.3
F1 |  24.0 |  38.6 |  23.0 |  40.0
Consolidation | Precision |  6.8 |  6.3 |  6.0 |  5.2
Recall |  42.3 |  98.5 |  40.0 |  97.1
F1 |  11.7 |  11.9 |  10.4 |  9.8
Edema | Precision |  19.6 |  18.5 |  11.7 |  13.7
Recall |  36.0 |  72.7 |  29.4 |  76.5
F1 |  25.4 |  29.5 |  16.8 |  23.2
Pleural Effusion | Precision |  30.4 |  30.0 |  22.3 |  17.9
Recall |  42.8 |  92.7 |  49.2 |  90.0
F1 |  35.6 |  45.3 |  30.7 |  29.9
Table 4: LLaVA-Med VQA performance evaluated by Precision, Recall, and F1 (%)
scores of five pathologies on the MIMIC-CXR-JPG and Chexpert test datasets.

#### PT2: Adding Pathology Explanations

Table 4 reports Precision, Recall, and F1 scores of the PT1 and PT2 strategies
on the MIMIC-CXR-JPG and Chexpert test sets.111The AUC and ROC scores commonly
reported in the literature to assess the performances of most medical image
classification models on the MIMIC-CXR-JPG and Chexpert datasets are
unsuitable in our context because MLVLMs output text rather than
probabilities. On MIMIC-CXR-JPG, after adding pathology explanations, the F1
scores increased for detecting Atelectasis, Cardiomegaly, Edema, and Pleural
Effusion, albeit only minimally for Consolidation. On Chexpert, after adding
pathology explanations, the F1 scores for detecting Atelectasis, Cardiomegaly,
and Edema increased, whereas they did not for Consolidation and Pleural
Effusion. The Precision and Recall scores reveal that adding explanations
generally leads to a large increase in Recall, but only minimally influences
Precision. For minority pathologies such as Consolidation whose F1 score is
dominated by low Precision, improving the Recall would not have much effect.
Thus, PT2’s performance bottleneck is Precision.

Pathology | TP | FP | FN
---|---|---|---
Atelectasis | 163 | 453 | 15
Cardiomegaly | 151 | 430 | 24
Consolidation | 28 | 557 | 7
Edema | 65 | 410 | 20
Pleural Effusion | 108 | 495 | 12
Table 5: True positive (TP), false positive (FP), and false negative (FN) counts of LLaVA-Med with the PT2 strategy on the Chexpert test set. Pathology | Metric | PT2 | PT3
---|---|---|---
Atelectasis | Precision | 26.5 | 28.8
Recall | 91.6 | 83.1
F1 | 41.0 | 42.8
Cardiomegaly | Precision | 26.0 | 38.1
Recall | 86.3 | 79.4
F1 | 40.0 | 51.5
Consolidation | Precision | 5.2 | 7.5
Recall | 97.1 | 34.3
F1 | 9.8 | 12.2
Edema | Precision | 13.7 | 36.8
Recall | 76.5 | 50.6
F1 | 23.2 | 42.6
Pleural Effusion | Precision | 17.9 | 25.0
Recall | 90.0 | 85.0
F1 | 29.9 | 38.8
Table 6: Diagnostic accuracies of LLaVA-Med with the PT2 and PT3 strategies on
the Chexpert test set.

#### PT3: Referring to Weak Learners

Going beyond our PT2 strategy, we applied our PT3 strategy to further improve
diagnostic accuracy. Table 5 provides the TP, FP, and FN prediction counts of
LLaVA-Med on the Chexpert test set using the PT2 strategy. Note the large
number of FP cases. Hence, we designed our weak learners to suppress FP
predictions. Table 6 compares the performance on Chexpert before and after
referring to the weak learner. It shows that the F1 prediction accuracy can be
substantially increased by introducing weak learner predictions into the
prompts. The F1 scores of Cardiomegaly, Edema, and Pleural Effusion increase
by 0.115, 0.194 and 0.089, respectively. To further demonstrate the efficacy
of our PT3 strategy, Table 7 compares the FP predictions of the PT2 and PT3
strategies. The reduction of FP cases is noteworthy, especially on Edema, for
which the FP count is reduced by 78.5% (322).

Pathology | PT2 | PT3
---|---|---
Atelectasis | 453 | 365
Cardiomegaly | 430 | 226
Consolidation | 557 | 149
Edema | 410 | 88
Pleural Effusion | 495 | 304
Table 7: FP counts of LLaVA-Med with the PT2 and PT3 strategies on the Chexpert test set. |  | MIMIC-CXR-JPG | Chexpert
---|---|---|---
Pathology | Metric | PT1 | PT2 | PT3 | PT1 | PT2 | PT3
Enlarged Cardiomediastinum | Precision | 4.3 | 3.9 | 5.0 | 49.3 | 44.1 | 49.4
Recall | 15.0 | 89.0 | 53.5 | 12.4 | 85.2 | 59.1
F1 | 6.7 | 7.4 | 9.1 | 19.8 | 58.1 | 53.8
Lung Lesion | Precision | 3.9 | 3.9 | 4.2 | 2.0 | 2.1 | 2.9
Recall | 77.2 | 100.0 | 66.3 | 71.4 | 100.0 | 92.9
F1 | 7.4 | 7.5 | 8.0 | 3.9 | 4.1 | 5.7
Lung Opacity | Precision | 31.4 | 30.4 | 31.9 | 50.0 | 47.2 | 51.4
Recall | 67.6 | 88.8 | 84.1 | 70.3 | 90.7 | 84.8
F1 | 42.8 | 45.3 | 46.2 | 58.5 | 62.1 | 63.3
Pneumonia | Precision | 11.4 | 10.5 | 12.3 | 2.4 | 1.7 | 6.3
Recall | 20.0 | 74.6 | 18.0 | 21.4 | 57.1 | 28.6
F1 | 14.6 | 18.4 | 14.6 | 4.4 | 3.4 | 10.4
Pneumothorax | Precision | 3.0 | 2.6 | 3.6 | 0.0 | 1.7 | 2.0
Recall | 16.7 | 78.5 | 50.7 | 0.0 | 90.0 | 50.0
F1 | 5.1 | 5.1 | 6.8 | 0.0 | 3.3 | 3.8
Table 8: LLaVA-Med VQA performance on the MIMIC-CXR-JPG and Chexpert test sets
for another 5 pathologies.

#### Additional VQA Experiments

Table 8 shows the results of applying the PT1, PT2, and PT3 strategies with
LLaVA-Med on the MIMIC-CXR-JPG and Chexpert datasets across another five
medical findings: Enlarged Cardiomediastinum, Lung Lesion, Lung Opacity,
Pneumonia, and Pneumothorax. Providing pathology explanations (PT2) generally
yields better results over the PT1 baseline, albeit inconsistently.
Introducing weak learner references (PT3) yields only limited increases in
Precision, but large decreases in Recall. Generally, it offers insignificant
improvement. Enlarged Cardiomediastinum, Lung Lesion, Pneumonia, and
Pneumothorax are minor categories and all our experimental settings, including
for the weak learner, fail to learn them. Prompting is apparently unhelpful in
such situations.

Pathology | Radiologist | (Tiu et al., 2022) | PT1 | PT3
---|---|---|---|---
Atelectasis | 69.2 | 64.6 | 26.5 | 41.3
Cardiomegaly | 67.8 | 74.3 | 24.0 | 51.5
Consolidation | 38.5 | 33.3 | 11.7 | 12.2
Edema | 58.3 | 60.2 | 25.4 | 42.6
Pleural Effusion | 73.7 | 70.4 | 35.5 | 46.8
Table 9: F1 scores (%) on 5 pathologies in the Chexpert test set, including
for the radiologist diagnoses, the state-of-the-art benchmark (Tiu et al.,
2022), as well as LLaVA-Med VQA with the PT1 scenario and the PT3 scenario,
which is the best result achieved by applying both our prompting strategies.

| POPE Adversarial | POPE Popular | POPE Random
---|---|---|---
Model | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1
LLaVA | 91.0 | 78.8 | 84.5 | 95.2 | 78.8 | 86.2 | 97.4 | 78.8 | 87.1
with referral | 88.4 | 85.7 | 87.0 | 92.8 | 85.7 | 89.0 | 97.3 | 85.7 | 91.1
MiniGPT-v2 | 88.2 | 77.2 | 82.3 | 92.7 | 77.2 | 84.2 | 97.2 | 77.2 | 86.1
with referral | 86.8 | 84.2 | 85.5 | 91.9 | 84.2 | 87.9 | 97.3 | 84.2 | 90.3

Table 10: Comparison of POPE scores for LVLM models with and without referring
to the predictions of weak learners.

#### SOTA Benchmark

Tiu et al. (2022) report F1 scores for detecting Atelectasis, Cardiomegaly,
Consolidation, Edema, and Pleural Effusion on the Chexpert dataset using their
deep learning model, as well as for the performance of radiologists. Their
work offers a state-of-the-art chest X-ray diagnosis benchmark. Table 9
compares the F1 scores of radiologists, the model of Tiu et al. (2022), and
LLaVA-Med. It shows that LLaVA-Med’s VQA performance of with the baseline PT1
strategy is unsatisfactory, rendering the model far from being deployable in
clinical practice. However, while still underperforming radiologists, our PT3
strategy yields a significant improvement, especially on Atelectasis,
Cardiomegaly, and Edema for which the F1 score increases by approximately 17%
to 21%.

#### Application to General Domain LVLMs

Our prompt strategies can also be applied to general domain LVLMs. We studied
the performance of LLaVA (Liu et al., 2023b) and MiniGPT-v2 (Zhu et al., 2023)
using POPE metrics (Li et al., 2023c), which evaluate the hallucination of
LVLMs by asking questions about the presence of objects. The POPE scores of
LLaVA and MiniGPT-v2 have high Precision and low Recall. Hence, our weak
learner strategy can be used to reduce the FN predictions. We selected an off-
the-shelf Fast-RCNN (Girshick, 2015) as the weak learner, fine-tuned the
detection threshold of bounding box scores to achieve high Recall, and
introduced the positive predictions of the weak learner into the prompts. The
results in Table 10 show that the Recall scores across three POPE categories
increased by around 7% (Precision scores decrease slightly), thus improving
the F1 scores.

##  5 Conclusions and Discussion

We have tested the visual question answering abilities of the LLaVA-Med
medical large vision-language model when applied to the diagnosis of
pathologies. Our results show that the model has unsatisfactory performance
when asked questions regarding the presence of complex pathologies. We
proposed two prompt engineering strategies to improve the visual question
answering accuracy of the model: providing explanations of pathologies and
referring to the predictions of weak learners. The first strategy helps the
model understand minority pathologies that it does not learn well in the
training stage. The second strategy can help improve diagnostic accuracy in
specific ways; e.g., by suppressing false positives. This strategy can also be
applied to LVLMs in other, non-medical domains.

However, our two strategies are not effective on pathologies with extremely
scarce data. For example, providing text explanations for Consolidation,
Fracture, Lung Lesion, Pneumonia, and Pneumothorax may not suffice since the
visual encoder does not adequately learn meaningful visual features. Moreover,
the data may not suffice to adequately train weak learners. A promising
direction for future research would be to devise a strategy for handle these
rare categories. Retrieval Augmented Generation (RAG) could be a potential
solution. In addition to textual explanation of a pathology, for instance,
typical example images can be provided to help the model make diagnostic
decisions.

##  A Pathology Explanations

The explanations of the five pathologies are as follows:

#### Atelectasis:

Atelectasis refers to the partial or complete collapse of a lung or a section
of lung. The features of atelectasis on an X-ray can vary depending on the
cause and extent of the collapse. Some common X-ray features include: 1. The
affected area may appear denser or whiter than normal lung tissue due to the
collapse, leading to increased opacity on the X-ray. 2. The affected portion
of the lung may appear smaller or compressed compared to the surrounding
healthy lung tissue. 3. Atelectasis can cause a shift or displacement of
nearby structures, such as the trachea or heart, toward the affected area. 4.
In obstructive atelectasis (caused by a blockage in the airways), there might
be signs of hyperinflation in the unaffected areas of the lung and a visible
blockage or narrowing in the affected bronchus. 5. Linear or band-like
opacities may be visible, often referred to as plate or band atelectasis,
which can occur due to the collapse of small airways. Given the information
above, does this image have Atelectasis?

#### Cardiomegaly:

Cardiomegaly is enlargement of the heart. The definition is when the
transverse diameter of the cardiac silhouette is greater than or equal to 50%
of the transverse diameter of the chest (increased cardiothoracic ratio) on a
posterior-anterior projection of a chest radiograph or a computed tomography.
Given the information above, does this image have Cardiomegaly?

#### Consolidation:

Consolidation on an X-ray refers to the filling of the lung’s air spaces with
fluid inflammatory exudate, or cellular material. Typical X-ray findings
suggesting consolidation include: 1. Areas of increased density in the lung
tissue, appearing as an opaque or hazy patch on the X-ray. Given the
information above, does this image have Consolidation?

#### Edema:

Pulmonary edema is the accumulation of fluid in the lungs. Some common X-ray
features include: 1. Increased density in the central lung fields resembling
the shape of bat wings. 2. Thin, linear opacities at the lung periphery, often
indicating interstitial edema. 3. Prominent blood vessel markings due to
engorgement from increased pressure in the pulmonary vasculature. Given the
information above, does this image have Edema?

#### Pleural Effusion:

Pleural effusion is the accumulation of fluid in between the parietal and
visceral pleura. Some common X-ray features include: 1. blunting of the
costophrenic/cardiophrenic angle. 2. fluid within the horizontal or oblique
fissures. 3. meniscus is seen. 4. mediastinal shift occurs away from the
effusion.

## References

  * Caffagni et al. (2024) Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.  Wiki-LLaVA: Hierarchical retrieval-augmented generation for multimodal LLMs.  _arXiv Preprint arXiv:2404.15406_ , 2024.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv Preprint arXiv:2305.14325_ , 2023.
  * Favero et al. (2024) Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto.  Multi-modal hallucination control by visual information grounding.  _arXiv Preprint arXiv:2403.14003_ , 2024.
  * Girshick (2015) Ross Girshick.  Fast R-CNN.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1440–1448, 2015.
  * Goldberger et al. (2000) Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * He et al. (2020) Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie.  PathVQA: 30000+ questions for medical visual question answering.  _arXiv Preprint arXiv:2003.10286_ , 2020.
  * Hu et al. (2023) Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun.  CIEM: Contrastive instruction evaluation method for better instruction tuning.  _arXiv Preprint arXiv:2309.02301_ , 2023.
  * Irvin et al. (2019) Jeremy A. Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn L. Ball, Katie S. Shpanskaya, Jayne Seekins, David Andrew Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, C. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and A. Ng.  CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _AAAI Conference on Artificial Intelligence_ , 2019.
  * Jain et al. (2023) Jitesh Jain, Jianwei Yang, and Humphrey Shi.  Vcoder: Versatile vision encoders for multimodal large language models.  _arXiv Preprint arXiv:2312.14233_ , 2023.
  * Lau et al. (2018) Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman.  A dataset of clinically generated visual questions and answers about radiology images.  _Scientific data_ , 5(1):1–10, 2018.
  * Leng et al. (2023) Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing.  Mitigating object hallucinations in large vision-language models through visual contrastive decoding.  _arXiv Preprint arXiv:2311.16922_ , 2023.
  * Li et al. (2023a) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv Preprint arXiv:2306.00890_ , 2023a.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International Conference on Machine Learning_ , 2023b.
  * Li et al. (2023c) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 292–305, Singapore, December 2023c. Association for Computational Linguistics.  .
  * Liu et al. (2021) Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu.  Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , 2021.
  * Liu et al. (2023a) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  In _The Twelfth International Conference on Learning Representations_ , 2023a.
  * Liu et al. (2024) Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng.  A survey on hallucination in large vision-language models.  _arXiv Preprint arXiv:2402.00253_ , 2024.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In _Neural Information Processing Systems_ , 2023b.
  * Liu et al. (2023c) Yinhan Liu, M. Wang, X. Liu, W. Chang, M. Sun, and P. Li.  Large language models encode clinical knowledge.  _Nature_ , 603:589–593, 2023c.  .
  * Lovenia et al. (2023) Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung.  Negative object presence evaluation (NOPE) to measure object hallucination in vision-language models.  _arXiv Preprint arXiv:2310.05338_ , 2023.
  * OpenAI (2022) OpenAI.  ChatGPT blog, 2022.  https://openai.com/blog/chatgpt.
  * OpenAI et al. (2023) OpenAI et al.  GPT-4 technical report.  _arXiv Preprint arXiv:2303.08774_ , 2023.
  * Qian et al. (2024) Yusu Qian, Haotian Zhang, Yinfei Yang, and Zhe Gan.  How easy is it to fool your multimodal LLMs? An empirical analysis on deceptive prompts.  _arXiv Preprint arXiv:2402.13220_ , 2024.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR, 18–24 Jul 2021.
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4035–4045, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  .
  * Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan.  Towards expert-level medical question answering with large language models.  _arXiv Preprint arXiv:2305.09617_ , 2023.
  * Sun et al. (2023a) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell.  Aligning large multimodal models with factually augmented RLHF.  _arXiv Preprint arXiv:2309.14525_ , 2023a.
  * Sun et al. (2023b) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented RLHF.  _arXiv Preprint arXiv:2309.14525_ , 2023b.
  * Tiu et al. (2022) Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar.  Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning.  _Nature Biomedical Engineering_ , 6(12):1399–1406, 2022.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and efficient foundation language models.  _arXiv Preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2023) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2717–2739, Toronto, Canada, July 2023. Association for Computational Linguistics.  .
  * Zhang et al. (2023) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao.  LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv Preprint arXiv:2303.16199_ , 2023.
  * Zhang et al. (2024) Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Matthew P. Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon.  BiomedCLIP: A multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.  _arXiv Preprint arXiv:2303.00915_ , 2024.
  * Zhao et al. (2024) Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu.  Mitigating object hallucination in large vision-language models via classifier-free guidance.  _arXiv Preprint arXiv:2402.08680_ , 2024.
  * Zheng et al. (2023) Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang.  DDCoT: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 5168–5191. Curran Associates, Inc., 2023.
  * Zhou et al. (2024) Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.  Analyzing and mitigating object hallucination in large vision-language models.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models.  _arXiv Preprint arXiv:2304.10592_ , 2023.


# On spherical barycentric coordinates

Abdellatif AITELHAD  Departement of mathematics
Cadi-ayyad university, FSSM, P.O. Box 2390, Marrakech 40 000, Morocco
<EMAIL_ADDRESS>
###### Abstract

This paper describes a novel construction of generalized barycentric
coordinates of points on a sphere with respect to the vertices of a given
spherical polygon that is contained in a common hemisphere. While in the
standard approach such coordinates are derived from their classical planar
counterparts (e.g. Wachspress, or mean value), we instead derive them from 3D
barycentric coordinates of the origin and show that they are endowed with some
useful properties such as edge linearity and Lagrange property. In addition,
we show that spherical mean value coordinates of both approaches coincide
while their corresponding spherical wachspress coordinates are in general
different.

###### keywords:

coordinates, barycentric , spherical

##  1 Introduction

Barycentric coordinates represent a fundamental concept used in major computer
graphics and geometric modeling applications such as mesh parameterization [3]
[14], freeform deformations [15] [17], finite elements [2] and shading [7]
[13].

###  1.1 Generalized barycentric coordinates with respect to arbitrary
polytopes

Generalized barycentric coordinates are an extension of the notion of
barycentric coordinates for simplices, to general polytopes. They being too
large to be discussed in detail here, we briefly review few approaches closely
related to our subject. For more details on generalized barycentric
coordinates see K. Hormann and N. Sukumar [8]. The first generalizations were
proposed by Wachspress [19], U. Pinkall and K. Polthier [12] for convex
polygons and Sibson [16] for scattered sets of points. In 2003, Floater [5]
introduced mean value coordinates that are defined in convex and non-convex
polygons. 3D extensions of Wachspress coordinates Ju et al. [17], Warren et
al. [20] and discrete harmonic coordinates Ju et al. [9], are well defined
within convex polyhedra with triangular faces, while 3D mean value coordinates
are well defined in arbitrary convex or non-convex polyhedra with triangular
faces Floater [4], Ju et al. [18] and extended to arbitrary polyhedra with
polygonal faces Langer et al [10].

######  Definition 1.

Let P be a polytope in $\mathbb{R}^{d}$, with $n$ vertices $v_{1},...,v_{n}$.
The functions $\phi_{i}:\mathbb{R}^{d}{\longrightarrow}\mathbb{R},\;i=1,...,n$
are called barycentric coordinates if they satisfy

  * (a)

Partition of unity: $\sum_{i=1}^{n}\phi_{i}(x)=1,\qquad\forall x\in P.$

  * (b)

Linear precision: $\sum_{i=1}^{n}\phi_{i}(x)\;v_{i}=x,\qquad\forall x\in P.$

The following additional properties are often required:

  * (c)

Non-negativity: $\phi_{i}(x)\geqslant 0,\quad i=1,...,n,\qquad\forall x\in P$.

  * (d)

Lagrange property: $\phi_{i}(v_{j})=\delta_{ij}$
where $\delta_{ij}$ are the Kronecker symbols.

  * (e)

Restriction on facets of the boundary:
For a facet F with vertices $v_{i1},...,v_{im}$, we have

$\sum_{j=1}^{m}\phi_{ij}(x)\;v_{ij}=x,\qquad\forall x\in F$

and

$\forall j\notin{i_{1},...,i_{m}},\quad\phi_{j}(x)=0,\qquad\forall x\in F.$

  * (f)

Smoothness: The coordinate functions $\phi_{i}$ are $\mathcal{C}^{\infty}$.

###  1.2 Spherical barycentric coordinates

Spherical barycentric coordinates represent another variant of barycentric
coordinates that express a point $x$ inside an arbitrary spherical polygon $P$
as a positive linear combination of $P$’s vertices. They were studied in a
spherical triangle by Möbius [11] (1846) and introduced to computer graphics
by Alfeld et al [1] (1996). These works are limited to triangles on the sphere
or on surfaces like-sphere, where the resulting coordinates are unique because
of the linear independence of the vertices. Next, Ju et al [17] (2005)
extended them to arbitrary convex polygons by appliying Stokes’ theorem to the
dual of a polyhedral cone bounded by rays whose end points are the vertices of
a convex spherical polygon. These coordinates were called ’vector
coordinates’, and are given as ratios of areas of certain dual faces. However,
they are only limited to convex polygons. Later, Langer et al [10] (2006)
developed a new construction of spherical barycentric coordinates of a point x
inside an arbitrary spherical polygon P by using the gnomonic projection into
the tangent plane of the sphere at x. This allowed them to construct 3D Mean
Value barycentric coordinates for arbitrary, closed polygonal meshes. In all
of these constructions, the linear precision property is preserved at the cost
of sacrificing the partition of unity property. However, research in this very
promising field remains very limited. In this work, we preserve the linear
precision property with the resulting sacrifices of partition of unity. The
relaxed property proposed by Alfred et al [1]

$\sum_{i=1}^{n}\phi_{i}(x)\geqslant 1,\qquad\forall x\in P$ (1.1)

is rather a consequence of the linear precision property than a condition,
indeed

$x=\sum_{i=1}^{n}\phi_{i}(x)\;v_{i}$

hence

$1=\parallel
x\parallel=\parallel\sum_{i=1}^{n}\phi_{i}(x)\;v_{i}\parallel\leqslant\sum_{i=1}^{n}\parallel\phi_{i}(x)\;v_{i}\parallel=\sum_{i=1}^{n}\phi_{i}(x)$

##  2 Construction

Our goal in this section is to find barycentric coordinates with respect to
spherical polygons that lie in some hemisphere.
A spherical polygon has the same definition as the planar one except that its
edges are geodesics (arcs of great circles) connecting the vertices.

######  Definition 2.

Let $P$ be a spherical polygon on the unit sphere centred at $0$, with
vertices $v_{1},v_{2},...,v_{n}$, which are ordered anti-clockwise, viewed
from outside the sphere. We call any positive values $\psi_{i},\;i=1,...,n$
spherical barycentric coordinates, if they satisfy

$\sum_{i=1}^{n}\psi_{i}(x)\;v_{i}=x,\qquad\forall x\in P.$

Let $P$ be a spherical polygon on the unit sphere centered at $0$, with
vertices $v_{1},v_{2},...,v_{n}$, cyclically indexed ($v_{i+n}=v_{i}$), and
$x$ be an interior point of $P$. We consider the (non-spherical) polyhedron
$Q=[v_{1},v_{2},...,v_{n},x,-x]$, bounded by the triangular faces
$[x,v_{i},v_{i+1}]$ and $[-x,v_{i},v_{i+1}]$, $i=1,...,n$ (see figure 1).

Figure 1: The spherical polygon $P$ and the polyhedron $Q$

Now we state the following theorem

######  Theorem 1.

Spherical barycentric coordinates, for points inside the polygon P, are given
by

$\psi_{i}(x)=\frac{\phi_{i}(0)}{\phi_{n+2}(0)-\phi_{n+1}(0)}.$ (2.1)

and on the boundary, by $\psi_{i}(v_{j})=\delta_{ij}$ and

$\left\\{\begin{array}[]{lll}\psi_{j}(x)=\dfrac{\phi_{j}(0)}{\phi_{n+2}(0)}&\mbox{
}\\\ \psi_{j+1}(x)=\dfrac{\phi_{j+1}(0)}{\phi_{n+2}(0)}&\mbox{}\\\
\psi_{k}(x)=0&\mbox{for}&k\neq j,j+1\end{array}\right.$ (2.2)

where $\phi_{i},\;i=1,...,n+2$ are any well known 3D barycentric coordinates
defined on $Q$
Furthermore, the $\psi_{i}^{\prime}s$ are linear on the edges of $P$.

###### Proof.

  1. 1.

The origine $0$ lies in the interior of $Q$ ($0\in[x,-x]$) and it can be
written as a linear combination of the vertices $v_{1},v_{2},...,v_{n},x,-x$
as follows

$\sum_{i=1}^{n}\phi_{i}(0)\;v_{i}+\phi_{n+1}(0)\;x+\phi_{n+2}(0)\;(-x)=0$
(2.3)

Hence

$\sum_{i=1}^{n}\phi_{i}(0)\;v_{i}=(\phi_{n+2}(0)-\phi_{n+1}(0))\;x.$ (2.4)

The point $p=\sum_{i=1}^{n}\phi_{i}(0)\;v_{i}$ on the left-hand side of (2.4)
belongs to the polyhedral cone $P^{\prime}$ of the vertices $v_{i},i=1,...,n$.
This implies that $\phi_{n+2}(0)-\phi_{n+1}(0)\geq 0$, since the point
$q=(\phi_{n+2}(0)-\phi_{n+1}(0))\;x$ on the right-hand side would otherwise be
outside of $P^{\prime}$. We claim that $\phi_{n+2}(0)-\phi_{n+1}(0)\neq 0$. On
the contrary, suppose that $\phi_{n+2}(0)-\phi_{n+1}(0)=0$. Then we would have

$p=\sum_{i=1}^{n}\phi_{i}(0)\;v_{i}=0$

and therefore $\phi_{i}(0)=0,i=1,...,n$. Indeed, suppose there is a
$k\in\left\\{1,...,n\right\\}$ such that $\phi_{k}(0)\neq 0$, then we would
have $-v_{k}=\sum_{i=1,i\neq k}^{n}\dfrac{\phi_{i}(0)}{\phi_{k}(0)}\;v_{i}$,
but the point $q=\sum_{i=1,i\neq
k}^{n}\dfrac{\phi_{i}(0)}{\phi_{k}(0)}\;v_{i}$ lies in the polyhedral cone
$P^{\prime\prime}$ of the vertices $v_{1},,...,v_{k-1},v_{k+1},...,v_{n}$,
while $-v_{k}$ is outside of $P^{\prime\prime}$. A contradiction. Now we
conclude from the partition of unity property that
$\phi_{n+2}(0)=\phi_{n+1}(0)=\dfrac{1}{2}$. The restriction on facets of the
boundary proprety (e) shows that this is only possible if the edge $[x,-x]$
coincides with an edge of $Q$, but this would imply that $x$ coincides with a
certain vertex $v_{k}$ of $Q$ and therefore we would have
$0=\phi_{k}(0)=\phi_{n+1}(0)=\dfrac{1}{2}$. A contradiction.
Since x is in the interior of P, equation (2.4) gives

$x=\sum_{i=1}^{n}\frac{\phi_{i}(0)}{\phi_{n+2}(0)-\phi_{n+1}(0)}\;v_{i}$

From $\phi_{i}(0)\geqslant 0$ and $\phi_{n+2}(0)-\phi_{n+1}(0)>0$ we conclude
that $\psi_{i}(x)\geqslant 0.$ We now compute these coordinates on the
boundary using properties (d) and (e), and show that they are linear on each
edge and satisfy Lagrange property.

  2. 2.

On the edges
If a point $x$ inside $P$ approaches the arc $e_{j}$, then $0$ approaches the
interior of the face (triangle) $[-x,v_{j},v_{j+1}]$ of $Q$. The restriction
on facets of the boundary proprety (e) shows that in the limit

$\phi_{i}(0)\;=0,\;\>for\;i\neq j,j+1,n+2$ (2.5)

and therefore $\psi_{i}(x)=\dfrac{\phi_{i}(0)}{\phi_{n+2}(0)}$ for $i=j,j+1$
$\quad(\phi_{n+1}(0)=0\;see\ equation~{}\eqref{moneq6})$
where $\phi_{i}(0),\;i=1,...,n+2$ are the continuous extensions to the
boundary of the 3D barycentric coordinates used in equation (2.1). Therefore,
equation equation (2.1) becomes

$\displaystyle x$ $\displaystyle=$
$\displaystyle\dfrac{\phi_{j}(0)}{\phi_{n+2}(0)}\;v_{j}+\dfrac{\phi_{j+1}(0)}{\phi_{n+2}(0)}\;v_{j+1}$
(2.6) $\displaystyle=$
$\displaystyle\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1}$ (2.7)

where $\psi_{j}(x)=\dfrac{\phi_{j}(0)}{\phi_{n+2}(0)}$ and
$\psi_{j+1}(x)=\dfrac{\phi_{j+1}(0)}{\phi_{n+2}(0)}$.
Spherical barycentric coordinates on the edge $e_{j}$ are therefore given by

$\left\\{\begin{array}[]{lll}\psi_{j}(x)=\dfrac{\phi_{j}(0)}{\phi_{n+2}(0)}&\mbox{
}\\\ \psi_{j+1}(x)=\dfrac{\phi_{j+1}(0)}{\phi_{n+2}(0)}&\mbox{}\\\
\psi_{k}(x)=0&\mbox{for}&k\neq j,j+1\end{array}\right.$

  3. 3.

Lagrange property
Equation $(2.7)$ yields

$\displaystyle\lim_{x\rightarrow v_{i}}\;x=\displaystyle\lim_{x\rightarrow
v_{i}}\;\psi_{i}(x)\;v_{i}+\psi_{i+1}(x)\;v_{i+1}$

hence

$v_{i}=\displaystyle\lim_{x\rightarrow
v_{i}}\;\psi_{i}(x)\;v_{i}+\displaystyle\lim_{x\rightarrow
v_{i}}\;\psi_{i+1}(x)\;v_{i+1}$

i.e.

$\left(1-\displaystyle\lim_{x\rightarrow
v_{i}}\;\psi_{i}(x)\right)\;v_{i}=\displaystyle\lim_{x\rightarrow
v_{i}}\;\psi_{i+1}(x)\;v_{i+1}$

but this means that $v_{i}$ and $v_{i+1}$ are collinear. A contradiction.
So we must have $1-\displaystyle\lim_{x\rightarrow v_{i}}\;\psi_{i}(x)=0$ and
$\displaystyle\lim_{x\rightarrow v_{i}}\;\psi_{i+1}(x)=0.$
Finally, the fact that $\psi_{j}(x)=0$ for $j\neq i,i+1$, (see equation (2.4))
completes the proof.

  4. 4.

Linearity on the edges
We have $\forall
x\in[v_{j},v_{j+1}],\;x=\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1}$. To prove
the linearity on the edges, it suffices to verify that

$\psi_{i}\left[\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1}\right]=\psi_{j}(x)\psi_{i}(v_{j})+\psi_{j}(x)\psi_{i}(v_{j+1})$

indeed

     * (a)

for $i\neq j,j+1$, we have $\psi_{i}(x)=0$, hence

$\displaystyle\psi_{j}(x)\psi_{i}(v_{j})+\psi_{j}(x)\psi_{i}(v_{j+1})$
$\displaystyle=\psi_{j}(x)\times 0+\psi_{j}(x)\times
0\qquad(Lagrange\;property\textbf{(d)})$ $\displaystyle=0$
$\displaystyle=\psi_{i}(x)$
$\displaystyle=\psi_{i}\left[\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1}\right]\quad(x=\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1})$

     * (b)

for $i=j$

$\displaystyle\psi_{j}(x)\psi_{i}(v_{j})+\psi_{j}(x)\psi_{i}(v_{j+1})$
$\displaystyle=\psi_{j}(x)\psi_{j}(v_{j})+\psi_{j}(x)\psi_{j}(v_{j+1})\qquad(i=j)$
$\displaystyle=\psi_{j}(x)\times 1+\psi_{j}(x)\times
0\qquad(Lagrange\;property\textbf{(d)})$ $\displaystyle=\psi_{j}(x)$
$\displaystyle=\psi_{i}(x)\qquad(i=j)$
$\displaystyle=\psi_{i}\left[\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1}\right]\quad(x=\psi_{j}(x)\;v_{j}+\psi_{j+1}(x)\;v_{j+1})$

     * (c)

same for $i=j+1.$

∎

######  Remark 1.

The new coordinates are more general than the classical ones in the sens that:

  1. 1.

Unlike in the classical approach, they are well defined inside $P$ without
need of any continuous extension to the special case where the angle
$\theta_{i}$ between $x$ and any vertex of $P$ is half of pi (i.e $\langle
x,v_{i}\rangle=0$)

  2. 2.

The classical approach works only in the case where $\langle x,v_{i}\rangle>0$
for all i, while our approach works perfectly regardless of the signs of the
$\langle x,v_{i}\rangle$

  3. 3.

In the limit case where all vertices $v_{i}$ lying on a great circle $C$, our
approach computes spherical barycentric coordinates of any point $x$ on the
sphere not lying on $C$

###  2.1 Comparison between new and existing coordinates

We adopt the following notation:
NC: The new coordinates introduced above
CC: Spherical coordinates introduced by Langer et al [10]
CF: Spherical coordinates introduced by Floater [6]
MV: Mean value coordinates
WC: Wachspress coordinates

####  2.1.1 Mean value coordinates

We show that NC and CC mean value coordinates coincide in the case where
$\langle x,v_{i}\rangle>0$ for all i.

  * 1.

3D mean value coordinates of [4] are given, for a point x inside the kernel of
a given polyhedron, by

$\psi_{i}(x)=\dfrac{w_{i}(x)}{w}\quad\quad\quad\omega_{i}=\dfrac{1}{\parallel
v_{i}-x\parallel}\;\sum_{T\in T(v_{i})}\;\;\mu_{i,T}$ (2.8)

where $\mu_{i,T}=\dfrac{\beta_{jk}+\beta_{ij}\;\langle
n_{ij},n_{jk}\rangle+\beta_{ki}\;\langle n_{ki},n_{jk}\rangle}{\langle
2e_{i},n_{jk}\rangle}$ and $\beta_{rs}$ is the angle between the two line
segments $[x,v_{r}]$ and $[x,v_{s}]$, $n_{rs}=\dfrac{e_{r}\times
e_{s}}{\parallel e_{r}\times e_{s}\parallel}$,
$e_{i}=\dfrac{v_{i}-x}{\parallel v_{i}-x\parallel}$ and $T(v_{i})$ denotes the
set of faces (triangles) $T$ incident to the vertex $v_{i}$.
Now, we consider the two faces $T1=[v_{i},v_{i+1},x]$ and
$T2=[v_{i},-x,v_{i+1}]$ and compute $\mu_{i,T1}$ and $\mu_{i,T2}$.

$\mu_{i,T1}=\dfrac{\beta_{(i+1)(n+1)}+\beta_{i(i+1)}\left\langle\dfrac{v_{i}\times
v_{i+1}}{\parallel v_{i}\times v_{i+1}\parallel},\dfrac{v_{i+1}\times
x}{\parallel v_{i+1}\times
x\parallel}\right\rangle+\beta_{(n+1)i}\left\langle\dfrac{x\times
v_{i}}{\parallel x\times v_{i}\parallel},\dfrac{v_{i+1}\times x}{\parallel
v_{i+1}\times x\parallel}\right\rangle}{\left\langle
2v_{i},\dfrac{v_{i+1}\times x}{\parallel v_{i+1}\times
x\parallel}\right\rangle}$

and

$\mu_{i,T2}=\dfrac{\beta_{(n+2)(i+1)}+\beta_{i(n+1)}\left\langle\dfrac{v_{i}\times-x}{\parallel
v_{i}\times-x\parallel},\dfrac{-x\times v_{i+1}}{\parallel-x\times
v_{i+1}\parallel}\right\rangle+\beta_{(i+1)i}\left\langle\dfrac{v_{i+1}\times
v_{i}}{\parallel v_{i+1}\times v_{i}\parallel},\dfrac{-x\times
v_{i+1}}{\parallel-x\times v_{i+1}\parallel}\right\rangle}{\left\langle
2v_{i},\dfrac{-x\times v_{i+1}}{\parallel-x\times
v_{i+1}\parallel}\right\rangle}$

hence

$\mu_{i,T1}=\dfrac{\theta_{i+1}+\beta_{i(i+1)}\left\langle\dfrac{v_{i}\times
v_{i+1}}{\parallel v_{i}\times v_{i+1}\parallel},\dfrac{v_{i+1}\times
x}{\parallel v_{i+1}\times
x\parallel}\right\rangle-\theta_{i}\cos\alpha_{i}}{\left\langle
2v_{i},\dfrac{v_{i+1}\times x}{\parallel v_{i+1}\times
x\parallel}\right\rangle}$

and

$\mu_{i,T2}=\dfrac{\pi-\theta_{i+1}-(\pi-\theta_{i})\cos\alpha_{i}-\beta_{i(i+1)}\left\langle\dfrac{v_{i}\times
v_{i+1}}{\parallel v_{i}\times v_{i+1}\parallel},\dfrac{v_{i+1}\times
x}{\parallel v_{i+1}\times x\parallel}\right\rangle}{\left\langle
2v_{i},\dfrac{v_{i+1}\times x}{\parallel v_{i+1}\times
x\parallel}\right\rangle}$

where $\theta_{i}=\widehat{(x,v_{i})}$ and $\alpha_{i}=\widehat{(x\times
v_{i},x\times v_{i+1})}$. Hence

$\mu_{i,T1}+\mu_{i,T2}=\dfrac{\pi(1-\cos\alpha_{i})}{\dfrac{\langle
2v_{i},v_{i+1}\times x\rangle}{\sin\theta_{i}}}$ (2.9)

The term ${V=\langle v_{i},v_{i+1}\times x\rangle}$ in the denominator of
equation (2.9) is the volume of the parallelepiped determined by the vectors
$v_{i},v_{i+1}$ and $x$ and it is given by the formula

$V=\sqrt{1+2\cos\theta_{i}\cos\widehat{(v_{i},v_{i+1})}\cos\theta_{i+1}-{\cos^{2}\theta_{i}}-{\cos^{2}\widehat{(v_{i},v_{i+1})}}-{\cos^{2}\theta_{i+1}}}$
(2.10)

using the identity relating the cross product to the scalar triple product

$\langle a\times b,c\times d\rangle=\langle a,c\rangle\langle
b,d\rangle-\langle a,d\rangle\langle b,c\rangle$ (2.11)

we obtain

$\langle v_{i}\times x,v_{i+1}\times x\rangle=\langle
v_{i},v_{i+1}\rangle-\langle v_{i},x\rangle\langle v_{i+1},x\rangle$

i.e.

$\parallel v_{i}\times x\parallel\parallel v_{i+1}\times
x\parallel\cos\alpha_{i}=\cos\widehat{(v_{i},v_{i+1})}-\cos\theta_{i}\cos\theta_{i+1}$

therefore

$\sin\theta_{i}\sin\theta_{i+1}\cos\alpha_{i}=\cos\widehat{(v_{i},v_{i+1})}-\cos\theta_{i}\cos\theta_{i+1}$

and so

$\cos\widehat{(v_{i},v_{i+1})}=\sin\theta_{i}\sin\theta_{i+1}\cos\alpha_{i}+\cos\theta_{i}\cos\theta_{i+1}$
(2.12)

By inserting this term into equation (2.10) and after a simple calculation we
find

$V=\sin\theta_{i}\sin\theta_{i+1}\sin\alpha_{i}$

Now, we have

$\mu_{i,T1}+\mu_{i,T2}=\dfrac{\pi\;{\sin^{2}\dfrac{\alpha_{i}}{2}}}{\dfrac{\sin\theta_{i}\sin\theta_{i+1}2\sin\dfrac{\alpha_{i}}{2}\cos\dfrac{\alpha_{i}}{2}}{\sin\theta_{i+1}}}$

i.e.

$\mu_{i,T1}+\mu_{i,T2}=\dfrac{\pi\;{\tan\dfrac{\alpha_{i}}{2}}}{2\sin\theta_{i}}$

we do the same for the faces $T3=[v_{i},x,v_{i-1}]$ and
$T4=[v_{i},v_{i-1},-x]$ and we find

$\mu_{i,T3}+\mu_{i,T4}=\dfrac{\pi\;\tan\dfrac{\alpha_{i-1}}{2}}{2\sin\theta_{i}}$

Now, the weight of the origin $0$ with respect to the vertex $v_{i}$ is given
by

$\omega_{i}=\mu_{i,T1}+\mu_{i,T2}+\mu_{i,T3}+\mu_{i,T4}=\dfrac{\pi\;(\tan\dfrac{\alpha_{i}}{2}+\tan\dfrac{\alpha_{i-1}}{2})}{2\sin\theta_{i}}$

we need to compute the weights of the vertices $x$ and $-x$. In the same way
we get

$\mu_{n+2,T2}-\mu_{n+1,T1}=\dfrac{\pi\left\langle\dfrac{v_{i+1}\times
x}{\parallel v_{i+1}\times x\parallel},\dfrac{v_{i}\times v_{i+1}}{\parallel
v_{i}\times v_{i+1}\parallel}\right\rangle+\pi\left\langle\dfrac{x\times
v_{i}}{\parallel x\times v_{i}\parallel},\dfrac{v_{i}\times v_{i+1}}{\parallel
v_{i}\times v_{i+1}\parallel}\right\rangle}{\left\langle
2v_{i},\dfrac{v_{i+1}\times x}{\parallel v_{i}\times
v_{i+1}\parallel}\right\rangle}$

i.e.

$\mu_{n+2,T2}-\mu_{n+1,T1}=\dfrac{\pi\left\langle\dfrac{v_{i+1}\times
x}{\parallel v_{i+1}\times x\parallel},v_{i}\times
v_{i+1}\right\rangle+\pi\left\langle\dfrac{x\times v_{i}}{\parallel x\times
v_{i}\parallel},v_{i}\times v_{i+1}\right\rangle}{\left\langle
2v_{i},v_{i}\times v_{i+1}\right\rangle}.$

By using equations (2.11) and (2.12), we find

$\displaystyle\left\langle\dfrac{v_{i+1}\times x}{\parallel v_{i+1}\times
x\parallel},v_{i}\times v_{i+1}\right\rangle$ $\displaystyle=$
$\displaystyle\dfrac{\cos\theta_{i+1}\cos\widehat{(v_{i},v_{i+1})}-\cos\theta_{i}}{\sin\theta_{i+1}}$
$\displaystyle=$
$\displaystyle\dfrac{\cos\theta_{i+1}\sin\theta_{i}\sin\theta_{i+1}\cos\alpha_{i}-\cos\theta_{i}\cos^{2}\theta_{i+1}}{\sin\theta_{i+1}}$
$\displaystyle=$
$\displaystyle\dfrac{\cos\theta_{i+1}\sin\theta_{i}\sin\theta_{i+1}\cos\alpha_{i}-\cos\theta_{i}\sin^{2}\theta_{i+1}}{\sin\theta_{i+1}}$
$\displaystyle=$
$\displaystyle\cos\theta_{i+1}\sin\theta_{i}\cos\alpha_{i}-\cos\theta_{i}\sin\theta_{i+1}$

and

$\left\langle\dfrac{x\times v_{i}}{\parallel x\times
v_{i}\parallel},v_{i}\times
v_{i+1}\right\rangle=\cos\theta_{i}\sin\theta_{i+1}\cos\alpha_{i}-\cos\theta_{i+1}\sin\theta_{i}$

hence

$\mu_{n+2,T2}-\mu_{n+1,T1}=\dfrac{\pi\cot\theta_{i+1}-\pi\cot\theta_{i+1}\cos\alpha_{i}+\pi\cot\theta_{i}-\pi\cot\theta_{i}\cos\alpha_{i}}{2\sin\alpha_{i}}$

And then

$\displaystyle\mu_{n+2,T2}-\mu_{n+1,T1}$ $\displaystyle=$
$\displaystyle\dfrac{\pi(\cot\theta_{i+1}+\cot\theta_{i})(1-\cos\alpha_{i})}{2\sin\alpha_{i}}$
$\displaystyle=$
$\displaystyle\dfrac{\pi}{2}(\cot\theta_{i+1}+\cot\theta_{i})\tan\dfrac{\alpha_{i}}{2}$

now

$\displaystyle w_{n+2}-w_{n+1}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\mu_{n+2,T2}-\mu_{n+1,T1}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\dfrac{\pi}{2}(\cot\theta_{i+1}+\cot\theta_{i})\tan\dfrac{\alpha_{i}}{2}$
$\displaystyle=$
$\displaystyle\dfrac{\pi}{2}\left(\sum_{i=1}^{n}\cot\theta_{i}\tan\dfrac{\alpha_{i}}{2}+\sum_{i=1}^{n}\cot\theta_{i+1}\tan\dfrac{\alpha_{i}}{2}\right)$

by setting $k=i+1$ and use the fact that $v_{i+n}=v_{i}$, we get

$\displaystyle w_{n+2}-w_{n+1}$ $\displaystyle=$
$\displaystyle\dfrac{\pi}{2}\left(\sum_{i=1}^{n}\cot\theta_{i}\tan\dfrac{\alpha_{i}}{2}+\sum_{k=2}^{n+1}\cot\theta_{k}\tan\dfrac{\alpha_{k-1}}{2}\right)$
$\displaystyle=$
$\displaystyle\dfrac{\pi}{2}\sum_{i=1}^{n}\cot\theta_{i}\left(\tan\dfrac{\alpha_{i}}{2}+\tan\dfrac{\alpha_{i-1}}{2}\right)$

The new coordinates $\psi_{i}(x)$ are therefore given in terms of the
classical coordinates $\lambda_{i}(x)$ by

$\displaystyle\psi(x)=\frac{\phi_{i}(0)}{\phi_{n+2}(0)-\phi_{n+1}(0)}=\dfrac{w_{i}}{w_{n+2}-w_{n+1}}$
$\displaystyle=$
$\displaystyle\dfrac{\dfrac{\pi\;\left(\tan\dfrac{\alpha_{i}}{2}+\tan\dfrac{\alpha_{i-1}}{2}\right)}{2\sin\theta_{i}}}{\dfrac{\pi}{2}\sum_{i=1}^{n}\cot\theta_{i}\left(\tan\dfrac{\alpha_{i}}{2}+\tan\dfrac{\alpha_{i-1}}{2}\right)}$
$\displaystyle=$ $\displaystyle\lambda_{i}(x)$

  * 2.

figure 2 provides a visual example which confirms that both coordinates
coincide.

Figure 2: Contour lines of the MV with respect to the vertex $v_{1}$ for
values of $\psi_{1}$ in $[0.09,0.10]$, $[0.11,0.12]$, $[0.17,0.18]$,
$[0.23,0.24]$, $[0.30,0.29]$ and $[0.35,0.36]$

####  2.1.2 Wachspress coordinates

Figure 3: Contour lines of the WC with respect to the vertex $v_{1}$ for
values of $\psi_{1}$ in $[0.09,0.10]$, $[0.11,0.12]$, $[0.17,0.18]$,
$[0.23,0.24]$, $[0.30,0.29]$ and $[0.35,0.36]$

Figure 3 (respectively 4), provides contour lines of WC for NC and CC
(respectively NC and CF). The coordinates seem to differ: we intend to give
the proof of the general case in the future.

Figure 4: Contour lines of the WCF with respect to the vertex $v_{1}$ for
values of $\psi_{1}$ in $[0.09,0.10]$, $[0.11,0.12]$, $[0.17,0.18]$,
$[0.23,0.24]$, $[0.30,0.29]$ and $[0.35,0.36]$

##  3 Conclusion

The new approach gives us a direct relationship between spherical barycentric
coordinates and 3D barycentric coordinates via the origin $0$. Their resulting
coordinates are more general than the classical ones.

## References

  * [1] P. Alfred, M. Neamtu, L. L. Schumaker: _Bernstein-Bézier polynomials on spheres and sphere-like surfaces_ Comput. Aided Geom. Des. 13, 4 (1996), 333–349.
  * [2] M. Arroyo, M. Ortiz: _Local maximum-entropy approximation schemes: a seamless bridge between finite elements and meshfree methods_ Int. J. Numer. Meth. Engng 65, 13 (2006), 2167–2202.
  * [3] M. Desbrun, M. Meyer, P. Alliez: _Intrinsic parameterizations of surface meshes_ Computer Graphics Forum 21 (2002), 209–218.
  * [4] M. S. Floate, G Kos, M. Reimers: _Mean value coordinates in 3D_ Comp.Aided.Geom.design 22(2005),623 631
  * [5] M. S. Floater: _Mean value coordinates_ Computer Aided Geometric Design 20, 1 (2003), 19–27. 1, 2, 5, 6
  * [6] M. S. Floater: _Generalized barycentric coordinates and applications_ Acta Numerica (2016), pp. 001– doi:10.1017/S09624929c
  * [7] H. Gouraud: _Continuous shading of curved surfaces_ IEEE Trans. Computers C-20, 6 (1971), 623–629.
  * [8] K. Hormann, N. Sukumar: _Generalized Barycentric Coordinates in Computer Graphics and Computational Mechanics_ Taylor and Francis, CRC Press, 2017.
  * [9] T. Ju, P. Liepa, and J. Warren: _A general geometric construction of coordinates in a convex simplicial polytope_ Computer Aided Geometric Design, 24(3):161–178, 2007.
  * [10] T. Langer, A. Balayev, H P. Seidel: _Spherical Barycentric Coordinates_ Eurographics Symposium on Geometry Processing (2006) Konrad Polthier, Alla Sheffer (Editors).
  * [11] A F. Mobius: _Der barycentrische Calcul._. Johann Ambrosius Barth, Leipzig, 1827.
  * [12] U. Pinkall and K. Polthier: _Computing discrete minimal surfaces and their conjugates_ Experimental Mathematics, 2(1):15–36, 1993.
  * [13] B. T. Phong: _Illumination for computer generated pictures_ Communications of ACM 18, 6 (1975), 311–317.
  * [14] J. Schreiner, A. Asirvatham, E. Praun, H. Hoppe: _Inter-surface mapping_ ACM Trans. Graph. 23, 3(2004), 870–877.
  * [15] T. W. Sederberg, S. R. Parry: _Freeform deformation of solid geometric models_ In Computer Graphics, Proceedings of ACM SIGGRAPH (1986), pp. 151–160.
  * [16] R. Sibson: _A vector identity for the Dirichlet tessellation_ Math. Proc. Cambridge Phil. Soc. 87 (1980).
  * [17] J. Tao, S. Schaefer, J. Warren, M. Desbrun: _A geometric construction of coordinates for convex polyhedra using polar duals_ In Proceedings of the Symposium on Geometry Processing (2005), pp. 181–186.
  * [18] Tao Ju, Scott Schaefer, and Joe Warren: _Mean value coordinates for closed triangular meshes_ In ACM SIGGRAPH 2005 Papers. ACM, 561–566
  * [19] _A Rational Finite Element Basis_ E. L Wachspress: A Rational Finite Element Ba- vol. 114. Academic Press, New York, 1975.
  * [20] Joe Warren, Scott Schaefer, Anil N Hirani, and Mathieu Desbrun: _Barycentric coordinates for convex sets_ Advances in computational mathematics 27, 3 (2007), 319–338