# Towards the Use of Slice-based Cohesion Metrics with Learning Analytics to
Assess Programming Skills

Max Kesselbacher  Department of Informatics Didactics
University of Klagenfurt
Klagenfurt, Austria
<EMAIL_ADDRESS>Andreas Bollin  Department of Informatics Didactics
University of Klagenfurt
Klagenfurt, Austria
<EMAIL_ADDRESS>
###### Abstract

In programming education, it makes a difference whether you are dealing with
beginners or advanced students. As our future students will become even more
tech-savvy, it is necessary to assess programming skills appropriately and
quickly to protect them from boredom and optimally support the learning
process. In this work, we advocate for the use of slice-based cohesion metrics
to assess the process of program construction in a learning analytics setting.
We argue that semantically related parts during program construction are an
essential part of programming skills. Therefore, we propose using cohesion
metrics on the level of variables to identify programmers’ trains of thought
based on the cohesion of semantically related parts during program
construction.

###### Index Terms:

programming education, assessment and feedback, static slicing, learning
analytics

##  I Introduction

In his WiPSCE’16 keynote [1, p. 14], Raymond Lister stressed that, in his
(neo-Piagetian) view111In the neo-Piagetian view of Lister, learning to
program occurs in four main stages of overlapping waves, with the ability to
mentally execute, i.e. trace, program code as a cornerstone of novice
programmers’ progression., quite some steps are necessary to master writing
one’s own first programs, and curricula and teachers are not adequately
prepared for facilitating the learning process. The situation seems to
continue at the university level, where high failure rates are observed in
programming courses [2], or are at least compounded by faulty teaching
strategies [3].

Even if one is not a supporter of (neo-)Piagetian ideas, to develop the
programming skills of our future software engineers in the best possible way,
it is essential to understand the learning processes, to know more about the
strategies experts follow when constructing their program, and to learn how
they are mapping their trains of thought to running code. Early work by Weiser
[4], Burnstein et al. [5], Broad and Filer [6] and also the application in the
formal methods domain [7] already showed that programmers think in structures
detectable by slices. This makes slice-based program analysis a promising
avenue towards our aim, namely identifying code construction patterns and
strategies that can then be used to support novice users in learning to
program.

Slicing techniques are usually applied on finished programs or successive
program versions stored in software version control systems. With the
emergence of learning analytics (LA) [8, 9], there is a great potential to
improve individual assessment and feedback, even during program construction.
This strongly affects the next generation of software engineers: with
heterogenous previous programming experiences and diversity in the possible
tools for learning, they need to be supported individually; a prospect made
possible with LA.

We argue that the granularity of current slice metrics does not fit our
established aims: assessing the method-level cohesion [10] alone is too coarse
for individual feedback, while statement-level cohesion [11] is too fine-
grained to reconstruct a programmer’s train of thought.

We advocate the use of slice-based cohesion metrics on the level of variables
and showcase its use by analyzing two Java implementations of the same example
problem (implemented by an undergraduate student and by a professional
programmer). We demonstrate that trains of thought can be identified in the
program construction sequence. Our data sets and tools are publicly available
[12].

##  II Related Work

Static program slicing has been introduced by Weiser [13], and he showed that
experienced programmers tend to think in program slices when reasoning about
code, for example during debugging [4]. The computation of slice-based
cohesion metrics, based on slice profiles, has been introduced by Ott and
Thuss [10]. Their metrics (Min/Max-Coverage, Overlap, Tightness, Parallelism)
make it possible to quantify the cohesion of a method (on method-level). These
metrics have been empirically studied by Meyers and Binkley [14], providing
baseline measures, establishing that the metrics can identify deteriorating
software quality as a result of changes, and investigating the inter-metric
relations for sets of metrics that provide a distinct viewpoint on program
code. Krinke proposed an adaptation of the cohesion metrics on statement-
level, measuring cohesive statements to help maintainers identify parts
responsible for low module cohesion that should be restructured [11]. For
programming education, static slicing has been used as an instructional method
to improve students’ programming skills [15] but is not widely considered.

Slice-based cohesion metrics have been adapted to formal specifications [16],
and Bollin has shown that high cohesion relates to single trains of thought,
while low cohesion and small module slice intersections relate to multiple
trains [7].

LA enables researchers to analyze educational data on a completely new scale
[8, 9]. In programming, this includes recording program versions that are not
considered finished, enabling a process-oriented analysis. Applications for
text-based programming include large-scale investigations of programming
mistakes in the BlueJ programming environment [17], clustering of program
versions of novice programmers solving a specific problem [18], investigations
of a state model for programming behaviour [19], and learning curve analysis
for programming concepts [20].

We propose to use slice-based cohesion metrics in a LA setting for text-based
programming education, following the ideas and findings of Bollin [7], to
improve individual assessment and feedback for learning programmers. To the
best of our knowledge, there is no previous work that applies slice-based
cohesion metrics in a LA setting.

##  III Showcase Study: Cohesion on Variable-Level

###  III-A Participants and Example Problem

The example problem used for this showcase study is given in Figure 1 (a). We
recruited participants from two populations ($17$ undergraduate and graduate
students from a Software Engineering project management course at the
University of Klagenfurt, and $9$ professional programmers from an Austrian
software development company) to implement the example problem in Java. We use
this example problem specifically because there are multiple ways to solve the
problem: syntactically (loops and conditional branching may be used but are
not required) and semantically (computing the output numbers directly from the
array or computing one number from the binary array and the second one from
the first number).

With the example problem’s openness, it is possible to test participants of
varying programming skills while also providing potential differences to be
assessed with a variety of metrics. In our showcase study, we use slice-based
cohesion metrics to differentiate ways to solve the example problem.

We selected two implementations to showcase the use of slice-based cohesion
metrics on variable-level in a LA setting: one implementation by an
undergraduate programmer showing two trains of thought with little cohesion
between parts (Section III-C), and one implementation by a professional
programmer that features a single train of thought and high cohesion (Section
III-D). Both implementations are fully functional and correctly implemented.
We do not claim experimental validity by the selection of the implementations.

###  III-B Collection and Computation of Cohesion Metrics

The implementation is recorded with an IDE-based LA approach [9], recording
changes on keystroke granularity [8] (each keystroke and the resulting program
version). The data collection is part of an ongoing project with a baseline
collection of keystroke granularity [12]. We convert the data into a sequence
of consecutive, compilable program versions.

The participants used the IDE IntelliJ to implement the example problem. The
implementation was recorded with an IDE plugin. The consecutive program
versions are stored on a headless data collection endpoint. Access to the data
is possible at our data collection repository [12]. We adapted the slice-based
cohesion metrics of Ott and Thuss [10] to compute the Coverage for each local
variable and parameter of a method, and implemented an automatic computation
of slice profiles and slice-based cohesion metrics for variables as an
adaptation to the Java slicer JRazor [21]. The IDE plugin, the data collection
server and the Java slicer are developed and hosted at our department and
available as open source222Source code available at: https://gitlab-
iid.aau.at/seqtrex.

The metrics computation is based on unions of forward and backward slices,
originally called metric slices [10]. We are focusing on semantics, simply
calling them union slices. Our computation of union slices is similar to those
of metric slices: the backward slice is computed from the last reference of
the slicing variable, the forward slice is computed from the definitions of
the slicing variable that are included in the backward slice. However, we only
compute the forward slice from the first definition. This way, multiple union
slices can be computed for a slicing variable: iteratively computing pairs of
backward and forward slices for references and definitions of the slicing
variable not covered so far, until all of them are included in at least one
union slice. Different cohesive parts of a method can be found, each captured
by a union slice. Our adaptation also makes the computation of the pairwise
cohesion of all variables possible, which is not showcased here333Additional
information can be found in the Ph.D. thesis of the first author, currently as
work in progress: https://www.aau.at/wp-
content/uploads/2021/03/Kesselbacher_Thesis.pdf. Coverage is computed as
follows:

###### Definition 1

Coverage($M$,$v$). Coverage, for method $M$ and slice profile $SP$ for
variable $v$, is the average of the variable union slice lengths divided by
the method length.

$Coverage(M,v)=\frac{1}{|SP_{v}|}\sum_{i=1}^{|SP_{v}|}{\frac{|UnionSlice(v)_{i}|}{|M|}}$

1 | public class
---|---
| ConvertBinary $\\{$
2 | // Converts the input
3 | // array of 8 bits, and
4 | // prints decimal
5 | // and hexadecimal.
6 | // Example input:
7 | // convertBinArr(
7 | // $\\{$1,1,0,0,0,0,1,1$\\}$)
8 | // Example output:
9 | // 195 / C3
10 | i void convertBinArr(
| ii int[] bN)$\\{\\}\\}$
(a) (a)

1 | void convertBinArr(
---|---
| iiint[] bN)$\\{$
2 | i double dec=0;
3 | i String hex="";
4 | i double hD1=0;
5 | i double hD2=0;
6 | i for(int i=0;
| iii i$<$bN.length;i++)$\\{$
7 | ii if(bN[i]$==$1)$\\{$
8 | iii dec+=Math.pow(2,i);
9 | iii if(i$<=$3)
10 | iiii hD2+=Math.pow(2,i);
11 | iii if(i$>$3)
12 | iiii hD1+=Math.pow(2,i-4);$\\}$$\\}$
13 | i hex=hexString(hD1)
| ii +hexString(hD2);
14 | i out.println(dec);
15 | i out.println(hex);$\\}$
(b) (b)

1 | void convertBinArr(
---|---
| iiiint[] bN)$\\{$
2 | i int sum=0;
3 | i int mul=1;
4 | i for(int i=0;
| iii i$<$bN.length;i++)$\\{$
5 | ii sum+=mul*bN[i];
6 | ii mul*=2; $\\}$
7 | i out.println(sum);
8 | i out.println(
| ii convertIntToStr(sum/16)
| ii +convertIntToStr(sum%16)
| ii );$\\}$
(c) (c)

(d) (d) (e) (e)

Figure 1: The top row shows the example problem (a) and end versions
implemented by an undergraduate student (b) and a professional programmer (c).
The bottom row shows the evolution of Coverage for selected variables and the
relative method slice intersection for all consecutive, compilable program
versions (undergraduate (d), professional (e)). Each point on the x-axis
represents a compilable program version during program construction, starting
with the first compilable program change (e.g. variable declaration) and
ending with the program versions shown in the top row. The final number on the
x-axis represents the number of consecutive, compilable program versions. The
y-axis represents the Coverage and method intersection values.

###  III-C Showcase I: Undergraduate Programmer

The undergraduate programmer implements two computational parts to solve the
example program (Figure 1 (b), lines 6-12). They compute the decimal sum (dec)
and the two hexadecimal digits (hD1 / hD2) separately, controlled by the loop
variable i. After the loop, they invoke Integer.toHexString(...) (line 13, in
pseudo-code) to compute the hexadecimal output, and print both results.

The separated method concerns can be observed in the evolution of Coverage on
variable-level (Figure 1 (d)). With the implementation of the second
computation part (the addition of the variable hD), the coverage of the
control variable i remains high while the coverage of the data variables dec
and hD are only moderately high. Towards the end, the undergraduate replaces
hD with hD1/hD2, which further decreases the coverage of dec. Both data
variables have a coverage greater than $0.5$, but the relative method slice
intersection is low during the whole method construction and even $0$ for most
program versions. This means that, for most program versions, there is no
method statement that is included in all union slices. Even when removing line
3 (which causes the empty intersection), the relative method slice
intersection is only $0.27$.

###  III-D Showcase II: Professional Programmer

The professional programmer follows a different approach to solve the example
program. They only compute the decimal sum (sum) in their loop (Figure 1 (c),
lines 4-6). To handle the hexadecimal output, they implement the method
intToStr (not shown for brevity) to compute the hexadecimal digit
corresponding to the integer parameter, and feed both half-bytes to this
method (line 8).

The evolution of Coverage on variable-level (Figure 1 (e)) shows that the
professional programmer eventually achieves a high Coverage for all variables,
in contrast to the undergraduate’s program construction sequence. During
program construction, the relative method slice intersection decreases
whenever a new variable is introduced (sum / mul / char2). Subsequently, they
are integrated or refactored (in the case of char2, which was a temporary
variable to hold half of the hexadecimal result), which results in a high
coverage for all variables as well as a high relative method slice
intersection. The relative method slice intersection of the end version is
$\frac{8}{9}=0.\dot{8}$ \- all method nodes other than the definition of sum
in line 2 is included in every union slice. Note that the loop corresponds to
three nodes in the method PDG.

##  IV Discussion

###  IV-A Identifying Trains of Thought in the Implementations

Bollin argues that single trains of thought translate to slice intersections
covering all predicates of cohesive Z specifications. Different thoughts
result in smaller intersections [7]. By analogy, single trains of thought
should result in cohesive method implementations and high relative method
slice intersections. In this paper, we propose the notion of a cohesive method
having a relative method slice intersection $>0.5$, with the rationale that,
above this threshold, all variable union slices should have method statements
in common.

Following this notion, we can establish that the undergraduate implements
multiple trains of thought: the variable-level Coverage metrics show a
reasonably high cohesion greater than $0.5$, the slice intersection never
exceeds $0.5$ and shows that there are no cohesive method parts across all
variables. It is therefore important to look at both variable-level Coverage
as well as method slice intersection measures.

The professional programmer integrates all variables into a cohesive method
implementation, following a single train of thought. The slice intersection
exceeds $0.5$ multiple times during the program construction, including most
method parts in all union slices. The variable-level Coverage metrics also
indicate high method cohesion.

###  IV-B Limits of Cohesion Metrics and Threats to Validity

Applying the slice-based cohesion metrics on variable-level revealed some
limits. For small methods, typically used in the training of novice
programmers, implementations will often result in a single union slice per
variable that covers the better part of the method. Moreover, primitive input
parameters are usually not re-defined, which makes iteratively computed union
slices less important for those parameters. For non-primitive input
parameters, the approach is still worthwhile.

Moreover, threats to validity need to be addressed. First, the uncontrolled
openness of the example problem can be seen as a threat to validity. There are
various ways to implement a solution, which are not inherently tied to the
level of programming skills. However, for this pre-study, this uncontrolled
openness is a strength to collect different implementations.

Second, the recruitment of participants was uncontrolled regarding their
programming skills. Undergraduate and graduate students of the curricula of
Applied Computer Science and Management of Information Systems have
participated in the study. This leads to uncontrolled variance in the
students’ programming experience. The professional programmers reported a mean
programming experience of $17.29\pm 9.62$ years. We mitigate this threat by
only reporting a selected case study.

A third threat to validity arises from the adaptation to the Java slicer
JRazor [21] to compute the slice-based cohesion metrics, which was not
validated elsewhere. To mitigate this threat, we double-checked the reported
metric results by hand.

###  IV-C Use of the Cohesion Metrics for Education and Training

The outlined IDE-based LA approach facilitates real-time assessment and
feedback of program construction sequences. We see different use cases to
employ the reported cohesion metrics and visualizations in education and
training. First is the direct use by educators, which provides an additional,
automatically processed, aspect during the assessment of student-written
programs and supports the educators to give in-depth feedback regarding the
program construction as a process. This is especially valuable when assessing
long methods.

Second is the use of IDE-based interventions [9] to augment the information
accessible to students during programming, including variable-level cohesion
reports on demand, or cohesion information in the programming view for each
variable.

By incorporating the topic of cohesion in the process of learning to program,
students benefit from experiencing program structures that lead to low and
high cohesion, respectively. Programming tasks with target cohesion values can
be designed so that students practice the creation of highly cohesive programs
- starting from program code that is functional but non-cohesive. Such tasks
teach students the design principle of single responsibilities in a
quantifiable manner. Students are thereby empowered to develop a holistic view
of programming: not writing single source code statements, but constructing
semantically meaningful program parts.

In light of the neo-Piagetian learning process model established by Lister, we
suggest that students need to operate on the third, concrete operational,
stage (being capable of ’a purposeful approach to writing code’ [1]) in order
to benefit most from the cohesion information. However, they can thereby be
supported in their process of developing expert programming skills towards the
last, formal operational, stage.

##  V Conclusion and Future Work

To better and faster prepare our students for future challenges, we aim to
improve the individual assessment and feedback of programming skills by
incorporating slice-based cohesion metrics into IDE-based LA research. In this
contribution, we report on a pre-study to showcase the use of the slice-based
cohesion metric Coverage on variable-level.

We showcase the metric, investigating two Java implementations of an example
problem (conversion of a binary input array to decimal and hexadecimal
numbers), done by an undergraduate and a professional programmer. Different
trains of thought during program construction can be identified by measuring
the intersection of all variable union slices. Our data sets and tools can be
openly accessed (see Section III-B).

We plan to conduct a systematic evaluation of the relation between programming
skills and slice-based cohesion metrics in the future. A tabulated
experimental design can be formed by controlling the programming skills to
form homogeneous experimental cohorts, while separating implementations by
strategic implementation approaches.

##  VI Author Profile

Max Kesselbacher is a University Assistant at the Department of Informatics
Didactics at the University of Klagenfurt. He completed graduate studies in
computer science as a teaching profession as well as computer science with a
focus on software engineering and now aims to improve the education of
computer science and software engineering in particular. He contributes to
informatics workshops, organizes coding contests, and operates a platform to
investigate the lasting effects of interventions. In his Ph.D. thesis he
focuses on improving the acquisition of programming skills by employing a
learning analytics approach, measuring program construction, and making
individual support and feedback possible.

Andreas Bollin is a Full Professor at the University of Klagenfurt and head of
the Department of Informatics Didactics. He worked on numerous projects
dealing with computer science education and different types of new media in
education. He authored and co-authored over 70 international peer-reviewed
publications, combining informatics didactics, serious games, programming,
formal methods, and software engineering. He heads a non-profit center for
teaching computing science to the public (the ”Informatikwerkstatt”) with more
than 14.000 attendees in the past four years. The focus of his research
includes computing education in primary and secondary education, educational
and serious games, computational thinking, competency, maturity models and
gender/personality aspects in teaching.

## References

  * [1] R. Lister, “Toward a developmental epistemology of computer programming,” in _Proceedings of the 11th Workshop in Primary and Secondary Computing Education_ , ser. WiPSCE ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 5–16. [Online]. Available: https://doi.org/10.1145/2978249.2978251
  * [2] R. Bornat, S. Dehnadi, and Simon, “Mental models, consistency and programming aptitude,” in _Proceedings of the Tenth Conference on Australasian Computing Education - Volume 78_ , ser. ACE ’08. AUS: Australian Computer Society, Inc., 2008, p. 53–61.
  * [3] A. Luxton-Reilly, “Learning to program is easy,” in _ITiCSE ’16_. New York, USA: ACM, 2016, pp. 284–289.
  * [4] M. Weiser, “Programmers use slices when debugging,” _Communications of the ACM_ , vol. 25, no. 7, pp. 446–452, 1982.
  * [5] I. Burnstein and F. Saner, “Applying fuzzy reasoning to the program understanding,” in _Proceedings of SEKE’98, Tenth International Conference on Software Engineering and Knowledge Engineering_ , June 1998, pp. 394–401.
  * [6] A. Broad and N. Filer, “Applying case-based reasoning to code understanding and generation,” in _Proceedings of the Fourth United Kingdom Case-Based Reasoning Workshop (UKCBR4)_ , University of Salford, Salford, England, September 1999, pp. 35–48.
  * [7] A. Bollin, “Metrics for quantifying evolutionary changes in z specifications,” _Journal of Software: Evolution and Process_ , vol. 25, no. 9, pp. 1027–1059, 2013. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1596
  * [8] P. Ihantola, A. Vihavainen, A. Ahadi, M. Butler, J. Börstler, S. H. Edwards, E. Isohanni, A. Korhonen, A. Petersen, K. Rivers, M. Rubio, J. Sheard, B. Skupas, J. Spacco, C. Szabo, and D. Toll, “Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies,” _ITiCSE WGR’16_ , pp. 41–63, 2015.
  * [9] C. D. Hundhausen, D. M. Olivares, and A. S. Carter, “IDE-Based Learning Analytics for Computing Education: A Process Model, Critical Review, and Research Agenda,” _ACM Trans. Comput. Educ_ , vol. 17, no. 26, pp. 1–26, 2017.
  * [10] L. M. Ott and J. J. Thuss, “Slice based metrics for estimating cohesion,” _Proceedings - 1st METRIC 1993_ , pp. 71–81, 1993.
  * [11] J. Krinke, “Statement-level cohesion metrics and their visualization,” _SCAM 2007 - Proceedings 7th IEEE International Working Conference on Source Code Analysis and Manipulation_ , pp. 37–46, 2007.
  * [12] M. Kesselbacher, K. Wiltschnig, and A. Bollin, “Block-based learning analytics repository and dashboard: Towards an interface between researcher and educator,” in _Proceedings of the 15th Workshop on Primary and Secondary Computing Education_ , ser. WiPSCE ’20. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3421590.3421662
  * [13] M. Weiser, “Program slicing,” _Proceedings - International Conference on Software Engineering_ , pp. 439–449, 1981.
  * [14] T. M. Meyers and D. Binkley, “An empirical study of slice-based cohesion and coupling metrics,” _ACM Transactions on Software Engineering and Methodology_ , vol. 17, no. 1, pp. 1–27, 2007.
  * [15] K. L. Eranki and K. M. Moudgalya, “Program Slicing Technique : A novel approach to improve programming skills in novice learners,” _SIGITE 2016 - Proceedings of the 17th Annual Conference on Information Technology Education_ , pp. 160–165, 2016.
  * [16] A. Bollin, “Slice-based Formal Specification Measures–Mapping Coupling and Cohesion Measures to Formal Z,” _NFM 2010 - Second NASA Formal Methods Symposium_ , p. 24, 2010.
  * [17] N. C. Brown, A. Altadmri, S. Sentance, and M. Kölling, “Blackbox, five years on: An evaluation of a large-scale programming data collection project,” _Proceedings of the 2018 ACM Conference ICER_ , pp. 196–204, 2018\.
  * [18] P. Blikstein, M. Worsley, C. Piech, M. Sahami, S. Cooper, and D. Koller, “Programming Pluralism: Using Learning Analytics to detect patterns in the learning of computer programming,” _Journal of the Learning Sciences_ , vol. 23, no. 4, pp. 561–599, 2014.
  * [19] A. S. Carter, C. D. Hundhausen, and O. Adesope, “The Normalized Programming State Model : Predicting Student Performance in Computing Courses Based on Programming Behavior,” _ICER ’15_ , pp. 141–149, 2015.
  * [20] K. Rivers, E. Harpstead, and K. Koedinger, “Learning Curve Analysis for Programming,” in _Proceedings ICER ’16_ , 2016, pp. 143–151.
  * [21] A. Rama, “Slicing von objektorientierten java programmen,” Master’s thesis, University of Klagenfurt, 2013.


# Cheap-fake Detection with LLM using Prompt Engineering ††thanks: This work
is partially supported by the Sichuan Provincial Key Laboratory of Intelligent
Terminals (Grant Number: SCITLAB-20016). Corresponding authors are Xiaohong
Liu<EMAIL_ADDRESS>and Wenyi Wang (wangwenyi@uestc.edu.cn).

Guangyang Wu  University of Electronic Science
and Technology of China
Chengdu, China  Weijie Wu  University of Electronic Science
and Technology of China
Chengdu, China  Xiaohong Liu  Shanghai Jiao Tong University
Shanghai, China  Kele Xu  National University of Defense Technology
Changsha, China  Tianjiao Wan  National University of Defense Technology
Changsha, China  Wenyi Wang  University of Electronic Science
and Technology of China
Chengdu, China

###### Abstract

The misuse of real photographs with conflicting image captions in news items
is an example of the out-of-context (OOC) misuse of media. In order to detect
OOC media, individuals must determine the accuracy of the statement and
evaluate whether the triplet ( i.e., the image and two captions) relates to
the same event. This paper presents a novel learnable approach for detecting
OOC media in ICME’23 Grand Challenge on Detecting Cheapfakes. The proposed
method is based on the COSMOS structure, which assesses the coherence between
an image and captions, as well as between two captions. We enhance the
baseline algorithm by incorporating a Large Language Model (LLM), GPT3.5, as a
feature extractor. Specifically, we propose an innovative approach to feature
extraction utilizing prompt engineering to develop a robust and reliable
feature extractor with GPT3.5 model. The proposed method captures the
correlation between two captions and effectively integrates this module into
the COSMOS baseline model, which allows for a deeper understanding of the
relationship between captions. By incorporating this module, we demonstrate
the potential for significant improvements in cheap-fakes detection
performance. The proposed methodology holds promising implications for various
applications such as natural language processing, image captioning, and text-
to-image synthesis. Docker for submission is available at
https://hub.docker.com/repository/docker/mulns/ acmmmcheapfakes.

###### Index Terms:

Large Language Models, Prompt Engineering, Cheap-fakes

##  I Introduction

The emergence of social media has dramatically altered the process by which
people access information, resulting in a significant upswing in available
information, hastening the spread of fake news and other types of misleading
data. On social media, two principal forms of deceptive information regularly
encountered are deepfakes and cheapfakes [1]. While ”deepfakes” refer to
videos that have been doctored using machine learning or other AI-based
methods to create or combine human faces and bodies, ”cheap fakes” are a far
less costly variant of doctored videos generated through readily available
software tools such as Photoshop, PremierePro, among others. Cheapfakes are
often created through manipulating image captions, image editing or by
adjusting video speed. Due to their ease of creation, cheapfakes are perceived
to be more ubiquitous and damaging than deepfakes. The out-of-context use of
images, where unaltered photos are put to new and deceptive contexts, is one
of the main reasons cheapfakes pose such a danger. This phenomenon occurs when
an image is sourced from various locations with contradictory or conflicting
captions. The detection of misinformation based on out-of-context images is
particularly arduous since the visual content is unchanged, and the misleading
or incorrect information is only conveyed through the image-text combination.

The COSMOS baseline framework [1] employs a two-step method to address the
problem of detecting cheap-fake images. The first step involves an image-text
matching module that evaluates the coherence between an image and its caption.
The second step utilizes an out-of-context (OOC) detection module to predict
the final outcome. The approach relies on semantic textual similarity (STS)
scores to determine whether an image-caption pair is OOC or not (NOOC). S-BERT
is used to calculate the semantic similarity between two captions, where the
input is a pair of captions and the output is a similarity score in the range
from 0 to 1. If the similarity score is less than the pre-defined threshold,
the triplet is predicted as out-of-context.

However, there are certain scenarios where the STS scores may not perform
well. For instance, if two captions are contradictory, the STS model may
produce a high score if the ratio of similar words is high. Conversely, a pair
of entailment captions may have a low STS score if one caption is much more
detailed than the other. Therefore, a more comprehensive evaluation of the
relationship between two captions is critical. Tran et al. [2] proposed using
a Natural Language Inference (NLI) model to determine whether the given
”hypothesis” and ”premise” logically follow (entailment) or unfollow
(contradiction) or are undetermined (neutral) with each other. However, the
performance of this model is still limited and may produce unreliable results
for challenging cases. In recent years, Large Language Models (LLMs) have
emerged, which possess enhanced semantic comprehension abilities compared to
conventional BERT-based models. This development has motivated the use of LLM
as a powerful tool for evaluating the coherence between two captions. To
enhance detection accuracy in cases where previous methods may not be
effective, we leverage the GPT3.5 model. This model has shown remarkable
results in various NLP tasks and can provide a more comprehensive evaluation
of the relationship between two captions. By utilizing the GPT3.5 model, we
can address the limitations of previous methods and enhance the overall
performance of the COSMOS framework. Nevertheless, two key challenges must be
addressed. Firstly, the full parameters of GPT3.5 is not currently openly
accessible, and its usage is limited to the OpenAI API. Secondly, the adaptive
nature of GPT3.5 and its frequent updates can result in dynamic and
potentially unstable outcomes.

This study presents an innovative approach to feature extraction utilizing
prompt engineering to develop a robust and reliable feature extractor. The
proposed method captures the correlation between two captions and effectively
integrates this module into the COSMOS baseline model. Our study emphasizes
the significance of prompt engineering in feature extraction, which allows for
a deeper understanding of the relationship between captions. By incorporating
this method into the baseline model, we demonstrate the potential for
significant improvements in cheap-fakes detection performance. The proposed
methodology holds promising implications for various applications such as
natural language processing, image captioning, and text-to-image synthesis.

Figure 1: Overall framework of our method.

##  II Related work

###  II-A Cheapfakes Detection

The baseline method, COSMOS [1], utilizes image-text matching and a BERT-based
module to detect cheap-fakes. Akgul et al. [1] have generated a dataset of
200,000 images with 450,000 textual captions to train the image-text matching
model. The COSMOS use a heuristic pipeline to determine whether an image-
caption triplet is out-of-context (OOC). In the ”ACMMM 2022 Grand Challenge on
Detecting Cheapfakes”, La et al. [3] proposed to use image captioning method
to enhance accuracy [3]. They convert the image to caption using a image
captioning model, and extract the features with the RoBERTa model. Afterwards,
they use an ANN or SVM for binary classifying according to features of three
captions. Moreover, La et al. [4] proposed a Visual Semantic Reasoning (VSRN)
method to enhance the image-text matching module. They use a pre-trained
DeBERTa model to obtain additional semantic information between two captions.
Tran et al. [2] proposed using a Natural Language Inference (NLI) model to
determine whether the given ”hypothesis” and ”premise” logically follow
(entailment) or unfollow (contradiction) or are undetermined (neutral) with
each other. Furthermore, they use online-search to address the hard cases for
image-caption matching.

###  II-B Fact-Checking

Fact-checking is a vital task aimed at evaluating the veracity of statements
made by prominent individuals, including politicians, pundits, and other
public figures [5, 6]. In recent years, researchers have proposed various
techniques to address the problem of fact-checking. Shi et al. [7] proposed a
discriminative path-based approach for fact-checking in knowledge graphs. The
method incorporates connectivity, type information, and predicate interactions
to identify true statements accurately. Jin et al. [8] introduced a novel
Recurrent Neural Network (RNN) with an attention mechanism (att-RNN) to
integrate multimodal features for rumor detection effectively. Ferreira et al.
[9] introduced the Emergent dataset, which is a digital journalism project
aimed at debunking rumors. This dataset has been used to develop effective
stance detection techniques.

##  III Methods

This section presents a novel approach for detecting cheapfakes that
incorporates both textual and visual information using a learnable multimodal
method. We build upon the baseline algorithm of the challenge [10], COSMOS,
which is a two-step approach that evaluates the coherence between an image and
two captions, followed by the coherence between the two captions themselves.
Our proposed method enhances the baseline by introducing a feature extractor,
built using a Large Language Model (LLM) called GPT3.5, to improve detection
accuracy. The method is designed to solve the first task of the ICME’23 Grand
Challenge [10] on Detecting Cheapfakes, which focuses on detecting whether an
image-caption triplet is in or out of context. Figure 1 depicts an overview of
our proposed method.

###  III-A Coherence between Image and Captions

Our method utilize the Image-text matching module, which was introduced in a
prior work referred to as COSMOS [1]. This module is designed to take as input
an image-caption pair, and to generate a bounding box with an image-caption
coherence score. A higher score value indicates a greater level of coherence
between the input image and caption. The bounding box produced by this module
frames the region that corresponds to the maximum coherence score.

The Image-text matching module operates by first taking the caption as input,
which is then processed by the Universal Sentence Encoder (USE) to produce a
512-dimensional vector. This vector is then further processed by the Text
Encoder to obtain a 300-dimensional vector. Concurrently, the image is
inputted into a pre-trained Mask-RCNN to generate up to 10 object bounding
boxes. For each object, the Object Encoder is used to produce a
300-dimensional embedding vector. To evaluate the coherence between the image
and caption, the intersection-over-union (IoU) score is calculated through a
dot product of embedding vectors. If the value of the IoU score is lower than
a pre-defined threshold, the triplet is predicted as NOOC. In contrast, if the
IoU score value is greater than the threshold, further estimation is carried
out for the captions in the triplet. In this work, we set the threshold value
to 0.25, which is determined based on prior studies [2].

###  III-B Coherence between captions

Once the threshold is met, the proposed method evaluates the textual semantic
relationship between the two captions in the triplet for further
classification. In addition to the similarity of the two sentences estimated
by BERT-based models, we introduce a feature vector that represents the
coherence of the two sentences. Specifically, we first estimate a similarity
vector $[S_{base},S_{large}]$ using SBERT and BERT-large models for each pair
of captions following [1], which represents the semantic similarity between
them.

Next, we utilize the GPT3.5 model (also denoted as ChatGPT) to generate a
discriminated vector $[c_{1},c_{2},\dots,c_{6}]$ that represents the semantic
relation between the captions regarding various features, including the
probability of being out of context, the consistency of subject matters, the
consistency of broader context, coherence, information completeness, and
semantic similarity. In order to guide the GPT3.5 model to conclude these
features, we carefully designed prompts as shown below:

“Given two sentences, I am going to ask you six questions. You should provide
a final answer in a python list of length 6 where each component is a rate
value (integer ranging from 0 to 9).

  * •

The first question: Determine whether these two sentences are out of context.
Rate your judgment by an integer number ranging from 0 to 9, where 9 refers to
being completely out of context, and 0 refers to being completely in context.

  * •

The second question: Determine whether the subject matters of these two
sentences are the same. Rate your judgment by an integer number ranging from 0
to 9, where 9 indicates that the subject matters are completely different, and
0 indicates that the subject matters are completely the same

  * •

The third question: Determine whether the broader context of these two
sentences refer to are the same. Rate your judgment by an integer number
ranging from 0 to 9, where 9 indicates that the broader context is completely
different, and 0 indicates that the broader context is completely the same

  * •

The fourth question: Determine whether these two sentences cohere together.
Please rate your judgment by an integer number ranging from 0 to 9, where 9
indicates that the two sentences are not coherent at all, and 0 indicates that
the two sentences are highly coherent

  * •

The fifth question: Determine whether any information is missing that could
help to explain the relationship between the two sentences. Please rate your
judgment by an integer number ranging from 0 to 9, where 9 indicates that
important information is missing, and 0 indicates that there is no information
missing.

  * •

The sixth question: Determine the semantic similarity between the two
sentences. Semantic similarity should be rated by an integer number ranges
from 0 to 9, where 0 refers to semantically identical, and 9 refers to
completely semantic different.

The two sentences are [CAPTION1, CAPTION2]. You should output the python list
only without explanations.”

To produce a stable output, we leverage the OpenAI API to implement the
feature extractor using the pre-trained model ‘gpt-3.5-turbo-0301’.
Specifically, this model is a snapshot of ‘gpt-3.5-turbo’ from March 1st 2023.
Unlike ‘gpt-3.5-turbo’, this model will not receive updates, and will only be
supported for a three month period ending on June 1st 2023. Furthermore, the
temperature value also effect the randomness of the model. Specifically, the
higher temperature make the output more random, while lower values will make
it more focused. Therefore, we set the temperature to 0 to obtain
deterministic results. Once the similarity vector $[s_{base},s_{large}]$ and
the GPT vector $[c_{1},\dots,c_{6}]$ are computed for each input sample, they
are passed as input to an ensemble classifier, AdaBoost [11]. The AdaBoost
classifier is a meta-estimator that begins by fitting a classifier on the
original dataset and then fits additional copies of the classifier on the same
dataset but where the weights of incorrectly classified instances are adjusted
such that subsequent classifiers focus more on difficult cases. This
classifier utilizes the extracted features to generate predictions of either 0
for NOOC or 1 for OOC.

##  IV Experiments

In this section, we describe the training details and the testing results
compared with previous methods.

###  IV-A Training Details

To train the binary classifier for predicting the OOC or NOOC label based on
the feature vector, we partitioned the public testing dataset of the ICME’23
Grand Challenge on Detecting Cheapfakes into a training set (50%) and a
testing set (50%). The public testing dataset comprised 1000 samples, each of
which consisted of an image and two captions as inputs, along with the
corresponding OOC or NOOC labels. Due to the limited number of samples
available for training, we adopted a set of simple classifiers to prevent
overfitting while achieving superior performance. We trained the classifiers
using the 5-fold cross-validation method.

###  IV-B Experimental Results

Table I presents a comparison of the results obtained from various
classifiers, including Support Vector Machine (SVM), Random Forest (RF), and
AdaBoost, with those obtained from previous methods. The second column of the
table indicates the accuracy on the testing dataset (i.e., half of the public
test dataset), while the third column represents the accuracy on the entire
public test dataset. The GPT+AdaBoost classifier was selected as the final
solution since it achieved the highest score on the testing dataset and
demonstrated better generalization ability than GPT+RF.

TABLE I: Comparisons with other methods. We emphasize the best accuracy with bold text. Methods | Acc. (test) | Acc. (full)
---|---|---
COSMOS [1] | 0.839 | 0.821
Tran et al. [2] | - | 0.891
La et al. [3] | - | 0.867
La et al. [4] | - | 0.760
GPT+SVM (ours) | 0.852 | 0.867
GPT+RF (ours) | 0.858 | 0.929
GPT+AdaBoost (ours) | 0.894 | 0.888

##  V Conclusion

In conclusion, our proposed method uses the COSMOS structure to evaluate the
coherence between an image and two captions, and between two captions. The
method employs first estimate the image-caption coherence represented by an
IoU value, and S-BERT and BERT-large models to estimate a similarity vector.
Additionally, we use the GPT-3.5 model to generate a discriminative vector
that represents the semantic relation between the captions based on a set of
carefully designed features. The use of these methods and models results in an
accurate and stable representation of the coherence between image-caption
triplets, and improve the baseline model by large margin.

## References

  * [1] Shivangi Aneja, Chris Bregler, and Matthias Nießner,  “COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning,”  arXiv e-prints, p. arXiv:2101.06278, Jan. 2021.
  * [2] Quang-Tien Tran, Thanh-Phuc Tran, Minh-Son Dao, Tuan-Vinh La, Anh-Duy Tran, and Duc Tien Dang Nguyen,  “A textual-visual-entailment-based unsupervised algorithm for cheapfake detection,”  in Proceedings of the 30th ACM International Conference on Multimedia, New York, NY, USA, 2022, MM ’22, p. 7145–7149, Association for Computing Machinery.
  * [3] Tuan-Vinh La, Quang-Tien Tran, Thanh-Phuc Tran, Anh-Duy Tran, Duc-Tien Dang-Nguyen, and Minh-Son Dao,  “Multimodal cheapfakes detection by utilizing image captioning for global context,”  in Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval, New York, NY, USA, 2022, ICDAR ’22, p. 9–16, Association for Computing Machinery.
  * [4] Tuan-Vinh La, Minh-Son Dao, Quang-Tien Tran, Thanh-Phuc Tran, Anh-Duy Tran, and Duc-Tien Dang-Nguyen,  “A combination of visual-semantic reasoning and text entailment-based boosting algorithm for cheapfake detection,”  in Proceedings of the 30th ACM International Conference on Multimedia, New York, NY, USA, 2022, MM ’22, p. 7140–7144, Association for Computing Machinery.
  * [5] Andreas Vlachos and Sebastian Riedel,  “Fact checking: Task definition and dataset construction,”  in Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, Baltimore, MD, USA, June 2014, pp. 18–22, Association for Computational Linguistics.
  * [6] Ray Oshikawa, Jing Qian, and William Yang Wang,  “A Survey on Natural Language Processing for Fake News Detection,”  arXiv e-prints, p. arXiv:1811.00770, Nov. 2018.
  * [7] Baoxu Shi and Tim Weninger,  “Discriminative predicate path mining for fact checking in knowledge graphs,”  Know.-Based Syst., vol. 104, no. C, pp. 123–133, jul 2016.
  * [8] Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo,  “Multimodal fusion with recurrent neural networks for rumor detection on microblogs,”  in Proceedings of the 25th ACM International Conference on Multimedia, New York, NY, USA, 2017, MM ’17, p. 795–816, Association for Computing Machinery.
  * [9] William Ferreira and Andreas Vlachos,  “Emergent: a novel data-set for stance classification,”  in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, June 2016, pp. 1163–1168, Association for Computational Linguistics.
  * [10] Duc-Tien Dang-Nguyen, Sohail Ahmed Khan, Cise Midoglu, Michael Riegler, Pål Halvorsen, and Minh-Son Dao,  “Grand Challenge On Detecting Cheapfakes,”  arXiv e-prints, p. arXiv:2304.01328, Apr. 2023.
  * [11] Yoav Freund and Robert E. Schapire,  “A decision-theoretic generalization of on-line learning and an application to boosting,”  J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119–139, 1997.