# Wikipedia’s Balancing Act: A Tool for Collective Intelligence or Mass
Surveillance?

Simon Liu
BCom, LLB
The University of Sydney

(12 December 2022)

###### Abstract

Wikipedia has evolved beyond its original function as an online encyclopedia
in an increasingly complex data-driven society. The social platform is met
with a balancing act between collective intelligence and mass surveillance;
processes need to be developed to protect individuals and the community from
government mass surveillance without sacrificing the important contributions
made through prohibited anonymous communication software. Case studies are
provided from NSA government surveillance practices, the anti-SOPA legislation
movement, and research that covers Wikipedia’s involvement with participatory
journalism, disinformation, self-censorship, and the use of Tor. This paper
proposes that a common ground can be developed between individuals, public and
private institutions through future research in socio-cultural anthropology
and policy frameworks around data retention and government accountability.
Wikipedia is used as an example within the US intelligence community as a
complex organisation that can adapt to changes through its iterative nature,
which draws insight into how policy frameworks can be future-proofed. Finally,
this paper is a wake-up call to individuals, private institutions, and
governments to remain vigilant about the storage and use of personal
information as a result of contributing to online communities.

##  1 Introduction

Educational websites that exist today not only function as sources of
information but also as social platforms. In particular, Wikipedia’s unique
open-editing model invites anyone to contribute information to Wikipedia pages
and articles regardless of their expertise, creating a community-generated
social space similar to Facebook, Instagram, Reddit, and Twitter (Di Lauro &
Johinke, 2017). Founded in 2001, Wikipedia as an online encyclopedia has
played an integral role in disrupting the landscape of traditional social,
political, legal, and economic institutions through its collective
intelligence, influencing the anti-SOPA legislation movement in 2012 and
engaging in participatory journalism (Greenstein & Zhu, 2012; Pasquale, 2015;
Konieczny, 2014; Lih, 2004). However, with great power comes great
responsibility. As one of the most visited and widely cited websites in the
world with over 1.9 edits made per second, Wikipedia’s high search engine
ranking has the potential to amplify disinformation, prevalent in ongoing news
events like natural disasters (Wikipedia, 2020; Rosenzweig, 2006; Sáez-
Trumper, 2019). In fact, studies have shown that humans are worse when
compared to bots at distinguishing between legitimate and hoax articles
(Kumar, West, & Leskovec, 2016). The impact of collective intelligence on
society will be examined, presented from the perspectives of all types of
stakeholders.

In an increasingly data-driven society, the question of organisational trust
on our personal data needs to be explored: what kind of personal information
is being collected? In the context of Wikipedia, bots are used as editorial
oversight and control, contributing about 20% of all edits (Geiger, 2017).
These data-driven algorithms that parse through information exists across all
organisations, including government intelligence communities as exposed by NSA
whistleblower Edward Snowden, who leaked government surveillance practices
such as the involvement of technology firms with PRISM, the mass data
collection system which expressly identified Wikipedia as a target of
surveillance (Wales & Tretikov, 2015). Public awareness of increased mass
surveillance amounted to a ‘chilling effects’ situation — “that certain state
acts may chill or deter people from exercising their freedoms or engaging in
legal [online] activities” (Penney, 2016). Penney’s chilling effects studies
on Wikipedia contributors of controversial or sensitive topics explain a sharp
decline in the number of edits of terrorism-related articles post-Snowden
leaks, which is attributable to: the uncertainty surrounding the vague and
ambiguous application of the law; or alternatively, to avoid being labelled or
tracked by the government as non-conformists or criminals (Schauer, 1978;
Solove, 2006). Thus, the use of Tor, an anonymous communication software, will
be explored to understand the motivation behind Wikipedia contributors masking
their digital footprint and whether Tor users are more likely to edit articles
considered to be controversial or sensitive (Forte, Andalibi, & Greenstadt,
2017; Tran, Champion, Forte, Hill, & Greenstadt, 2020).

Finally, what does the future hold for Wikipedia? How do we resolve the
tension between collective intelligence and mass surveillance? The Oxford
Dictionary of English definition of “balancing act” (Stevenson, 2010, p. 123)
encapsulates this idea well — Wikipedia needs to perform an action or activity
that requires a delicate balance between different situations or requirements.
Future implications will be discussed with possible solutions, in addition to
a case study by Andrus (2005) on how Wikipedia’s iterative process, if
replicated within the intelligence community, can overcome the increasingly
complex and non-linear national security environment.

##  2 Collective Intelligence

###  2.1 Why Contribute?

There are many reasons why individuals contribute to online communities, which
include both intrinsic and extrinsic motivation. From generalised reciprocity
to social recognition, users will continue to contribute and learn from others
within a social context (Tedjamulia, Dean, Olsen, & Albrecht, 2005). There are
two factors hypothesised to increase the level of participation: having a
personal goal and commitment to the project. For example, an individual is
more inclined to share their knowledge on a platform that encourages freedom
of speech.

###  2.2 Linus’s Law

Wikipedia’s open-source collaborative platform is a successful example of
collective intelligence in the online community. Notably, Wikipedia articles
have been cited in multiple court cases and judicial opinions around the world
given the platform’s ability to provide up-to-date crowdsourced information
that can be readily relied upon as supplementary material (Wikipedia, 2020;
Rukundo, 2019). Previous citations include geographic information and
definitions of phrases (Miller & Murray, 2010). Linus’s Law explains the
process of collective intelligence; having a large enough number of
contributors will eventually replace the existing inaccurate information with
input that is verifiable and higher in quality (Raymond, 1999; Greenstein &
Zhu, 2012). However, there are practical limitations in this concept. The
attention and quality assurance of each article is not evenly distributed. In
addition, studies have shown that there are inherent cultural and gender bias
on Wikipedia (Hube, 2017). Researchers have observed an increase in fake news
coverage during the 2016 United States presidential campaign in social
communities such as Facebook and Twitter, resulting in ‘filter bubbles’ and
‘echo chambers’ filled with misinformation that reinforces an individual’s
preconceived notion (Guess, Nyhan, & Reifler, 2018). In particular, Wikipedia
functions like an amplifier through search engines such as Google, which
captures a snapshot of a Wikipedia article that may be inaccurate at the time,
and the inaccuracies are broadcasted instantly to internet users worldwide
(Rosenzweig, 2006). Another implication of Wikipedia is its effect on
education with its tendency to appear on multiple front page search results.
87.5% of Australian students are reliant on Wikipedia as an information
source, however, survey data of the students has demonstrated its use as
supplementary material, rather than as a primary source (Selwyn & Gorard,
2016).

###  2.3 Neutral Point of View

Wikipedia’s neutral point of view policy plays a significant role in
preventing disinformation online. As a defining principle of Wikipedia, it is
a guideline that all contributors should follow in good faith — to write
fairly and reliably to provide a complete representation of the topic, and
without editorial bias, presenting all points of views proportionally (Cap,
2012). Wikipedia’s neutral point of view policy stands to the test of time
since the creation of the platform, and has continued to function as a
cornerstone of the Wikipedia ideology. It is understood that the barriers
around editorial bias are “not firmly structural, but rather socially
constructed”, which demonstrates that a reduction in disinformation on
Wikipedia over time can be attributed to an increasing emphasis on community
participation to moderate and prevent disinformation on online platforms
(Livingstone, 2010).

###  2.4 Disinformation

Despite Wikipedia’s role in tackling disinformation, it is still necessary to
consider how disinformation could be easily spread through Wikipedia. For
example, ongoing news events like natural disasters can be exploited due to
the lack of up-to-date information and therefore, disinformation can appear
across the search results of search engines due to its high page ranking
(Sáez-Trumper, 2019). This led to a spread of disinformation as seen in
Croatia Wikipedia, where certain extremist or fascist events have been
represented more positively than in its English counterpart (Vrandečić, 2019).
Human experiments have also demonstrated the poor ability to distinguish
between legitimate and hoax Wikipedia articles with a 66% detection rate,
attributable to human bias of perceiving well-formatted articles as
legitimate. An example is the “Bicholim conflict”, an article that achieved a
Good Article status for half a decade despite being a hoax article (Kumar et
al., 2016).

###  2.5 Anti-SOPA Legislation Movement

The rise of user-generated content websites has resulted in participatory
journalism, operating similar to traditional media companies in the process of
gathering, filtering, and visualising content to be presented in a digestible
format (Lih, 2004). Wikipedia’s unique open-editing model is faster and more
efficient than traditional media companies, for example, the unbureaucratic
process between reading ongoing news events and editing the article provides
live updates to readers worldwide (Stein, 2013). These websites have also
participated in online activism, which held the power to influence
international policy such as the anti-SOPA legislation movement in 2012
(Konieczny, 2014). The “Stop Online Piracy Act” Bill 2011 (US) was a
controversial legislation, if enacted, could impose liability on online
communities for the actions of its users if alleged of any online copyright
infringement, bringing about a chilling effect of site-wide censorship that
can be easily exploited on a publicly editable platform such as Wikipedia.
Wikipedia contributors played a role in gaining international protest
movement; a majority vote on the Wikipedia:SOPA poll supported Wikipedia’s
24-hour page blackout joined by Google and other websites, and shortly
afterwards, the bill was postponed indefinitely (Wikipedia, 2020). A survey
concluded that Wikipedia contributors are motivated by ‘free speech’: a desire
to collectively contribute, derived from a free-information ideology of
democratic values (Nov, 2007). Yet, the guidelines of Wikipedia is
paradoxical. The proud platform that encourages open discussion and
collaboration amongst its users discourages the use of the term ‘terrorist’,
which was also adopted by news organisation Reuters in an apology resulting
from the aftermath of the September 11 attacks (Wikipedia, 2020; Glocer &
Linnebank, 2001). This demonstrates Wikipedia’s evolution of what it was
originally as an online encyclopedia, into a social, political, legal, and
economic conglomerate.

###  2.6 Future of Collective Intelligence

It has been demonstrated that an unregulated ‘free market’ of contributions
led to a plethora of high quality legitimate and hoax articles. Wikipedia’s
own guidelines are unable to provide oversight to problems that have been
originally out of the scope of its original function as an online
encyclopedia. If Wikipedia is unable to cope with the demands and consequences
of its contributions, then where does the future of collective intelligence
lead? Faucher, Everett, and Lawson’s (2008) definition of complexity theory
which addresses this question will be explored in the fourth section of this
paper, that asks whether online communities are able to adapt rapidly to
unpredictable future complex environments. Although Wikipedia does not
officially recognise itself as a social networking site, future-proof policies
must be developed around disinformation and its role as a platform. But before
we look at the future implications and possible solutions, Wikipedia has
another force to balance other than collective intelligence.

##  3 Mass Surveillance

###  3.1 Within the “Black Box”

Data-driven algorithms exists across all aspects of our lives, from applying
for a credit card to using voice assistants on our devices. They play an
important but pervasive role in organising, ordering, governing, and
gatekeeping our personal information inside the ‘black box’ which hides the
decision-making source codes that have an extensive impact on society with
minimal accountability (Geiger, 2017; Pasquale, 2015). Currently, researchers
are focused on the question of the future of technologies in transitioning to
open-source software (Chesbrough, 2017; Coelho & Valente, 2017). Literature
papers have overlooked using Wikipedia as a case study which provides
researchers with current insight on the issues surrounding data-driven
algorithms that exist beyond the black box. Either way, our trust in
organisations on our personal data is inevitable and instead, we should ask
ourselves: what kind of personal information is being collected about me and
what purpose is it being used for? Empirical evidence suggests that websites
are increasingly intrusive, personalising our view of the internet and
ultimately, our perspectives in a capitalistic privacy regulation environment
(Cecere & Rochelandet, 2013; Hui & Png, 2006).

###  3.2 Chilling Effects

This leads to the discussion of how a ‘chilling effects’ situation undermines
the collective effort of individuals. Snowden’s leaks from 2013 to 2014 of NSA
‘upstream’ surveillance system was the catalyst for public awareness and a
change in public perceptions around trust and accountability with the
government (Szoldra, 2016). Shortly afterwards, Wikimedia Foundation and other
organisations filed a lawsuit against the NSA and other named state actors
alleging mass surveillance of Wikipedia users, which violated the fundamental
rights of privacy, freedoms of expression and association. Jimmy Wales and
Lila Tretikov, the co-founder of Wikipedia and former executive director of
Wikimedia Foundation respectively, published a joint co-ed contribution on The
New York Times highlighting the NSA’s ability to parse through a user’s
contribution to Wikipedia, thus tracking their digital footprint and creating
a profile based on their personal information (Wales & Tretikov, 2015).
Therefore, it would be reasonable for contributors to be less willingly to
share their information out of fear of persecution, resulting in a domino
effect that disrupts the collective intelligence cycle.

Empirical findings support the chilling effects theory on Wikipedia
contributors post-Snowden leaks; there was a surveillance-related chill that
caused the general trend reversal and sudden drop in the number of edits of
terrorism-related articles (Penney, 2016). The findings also support the
opposite; other topics on Wikipedia were not impacted by the chill as it would
not have been likely to raise privacy concerns. However, there are limitations
as noted by Penney; the research was conducted within a two-year period which
is not determinant of a long-term chilling effects situation. Moreover, the
study was based on English Wikipedia, to which the evidence of chilling
effects may not be applicable in other Wikipedia languages. A further study
conducted by Büchi et al. (2019) has suggested that a subtle and long-term
chilling effect may exist from organisations profiling our information. Other
studies have suggested that a consistent chilling effect is not wide-spread
across different languages, but a wider time-period needs to be considered to
measure its long-term impact (Xie, Johnson, & Gomez, 2019).

###  3.3 Wikipedia and Tor

Open-collaboration projects are trying to address the problem of maintaining
user participation over time, hoping to encourage a diverse range of
contributors to ensure equal representation, quality, and coverage (Forte et
al., 2017; Sydow, Baraniak, & Teisseyre, 2017). Since the public is more aware
about their vulnerabilities when using the internet post-Snowden leaks,
Wikipedia contributors realise that their actions are publicly logged and made
available through public databases (Wikimedia Foundation, 2020). Furthermore,
Wikipedia has blocked the use of Tor, an anonymous communication software,
when editing articles to prevent abuse and vandalism by individuals that are
non-identifiable (Wikipedia, 2020). The Tor IP address blocks were manually
performed by administrators, but soon after, it was replaced with bots which
led to slippages in the system (Tran et al., 2020). Analysis of the Tor IP
addresses have demonstrated that topics considered more controversial and
sensitive (i.e. politics, technology, and religion) were more likely to be
edited by Tor users, and at a higher quality, highlighting the importance and
value of anonymous contributions that have been largely neglected. Additional
research has found that one-time contributions of anonymous users provided the
highest quality contributions out of all users, who tend to be selective with
the topics that they wrote (Lichtenstein & Parker, 2009; Huang, Zhu, Du, &
Lee, 2016). Similar to the limitation of the chilling effect studies on
Wikipedia, the studies were conducted on English Wikipedia, to which the
results may not extend across other Wikipedia languages.

Without getting into the technicalities of proposed machine learning
techniques, onion routing systems such as Tor do not guarantee complete
anonymity (Barker, Hannay, & Bolan, 2010; Mercaldo & Martinelli, 2017). In
order to propose a solution around anonymous editing, it is important to
consider the motivations behind self-censorship. Interviews were conducted to
Wikipedia contributors that did edit and did not edit through Tor; the
findings highlighted that the main concern of contributors was the loss of
privacy and surveillance from governments (Forte et al., 2017). In particular,
contributors that had taken on leadership roles (as time passed by) were more
concerned about their ability to handle privacy threats throughout life
changes such as starting a family or applying for a job or school.

###  3.4 Future of Mass Surveillance

Loss of privacy and mass surveillance are concerns that exist in online
communities. Processes need to be developed to protect individuals and the
community from future chilling effects without sacrificing the important
contributions made through prohibited anonymous communication software. In an
ever-changing social, political, legal, and economic environment, internet
contributors must remain vigilant of their actions, which may have future
negative consequences.

##  4 Future Implications and Possible Solutions

###  4.1 Further Research To Be Conducted

Many studies cited in this paper have not demonstrated its application in
other Wikipedia languages (Hube, 2017; Penney, 2016, Lichtenstein & Parker,
2009; Huang et al., 2016). A better understanding of whether chilling effects
occurs in other languages serves as guidance to combat the ‘thawing’ of
collective intelligence. For example, if the chilling effects situation cannot
be observed in other languages such as Chinese and Japanese as a result of
their collectivist culture, then possible solutions may involve developing a
stronger online community that reinforces notions of group ideology. Likewise,
the number of tor-based edits observed in other languages may provide a better
understanding of which cultures are more susceptible to mass surveillance. We
propose that future research in socio-cultural anthropology may serve as
guidance to balance Wikipedia’s need to maintain its collective intelligence
without deterring individuals from anonymously contributing as a result of
government mass surveillance practices (Gloor et al, 2017).

###  4.2 Complexity Theory

A paper published by Andrus (2005) in the Central Intelligence Agencies’
journal “Studies in Intelligence” draws insight into the complementary nature
of collective intelligence and mass surveillance, and how it can be applied in
a future setting with an example provided within government intelligence
communities. The intelligence community operates in a real-time strategic
environment; the total duration of the intelligence - decision -
implementation cycle ”can be as short as 15 minutes” (Andrus, 2005). Otherwise
known as complexity theory, in an increasingly complex and non-linear national
security environment, intelligence processes must implement iterative
methodologies to incorporate continuous learning and adaptation. Wikipedia as
a complex adaptive organisation is used as a case study to highlight a
flexible and loose management structure in collaboration with complex human-
computer interactions (Faucher et al., 2008). Here, Wikipedia contributors are
granted more autonomy compared to traditional organisations, in which the
cumulative efforts by the community has resulted more efficient and increased
information-sharing, whereby talk pages also serve as a means to discuss and
share different perspectives with other collaborators. To replicate a complex
adaptive system such as Wikipedia, the following question should be
considered: how can a community introduce policies that modify its environment
to adapt to rapid and unpredictable changes? Andrus outlines the six key
elements of a complex adaptive system: self-organisation, emergence,
relationships, feedback, adaptability, and non-linearity (Turnbull & Guthrie,
2019). Applying these concepts to the intelligence community, Andrus theorises
that intelligence officers are able to act more independently and be
knowledgeable through information-sharing, receive more direct feedback
throughout the intelligence - decision - implementation cycle process, and
finally, be made more aware of the bigger picture of the organisation,
reinforcing group ideology.

Later papers have discussed Intellipedia, a Wiki platform introduced in 2006
that is used internally within the Central Intelligence Agency (Werbin, 2011).
Intellipedia has become an integral part of the intelligence community,
improving collaboration and technological competence of intelligence officers
from their bottom-up participation. Real-life successes in North Korea, Iraq,
and Nigeria were recognised in Hickman’s (2010) book. As necessary,
limitations are to be discussed. Further research should take into
consideration of whether dynamic intelligence should replace bureaucratic
intelligence processes (Chomik, 2012). In a broader context, is the future of
complex collective intelligence processes an open-source environment?

###  4.3 The “Balancing Act”

The paradigm of the balance between collective intelligence and mass
surveillance is left unexplored, especially Wikipedia’s role in the process.
Wikipedia will continue to grow socially, politically, legally, and
economically. Likewise, society will observe an increase in self-censorship
from government mass surveillance practices. Towards a better future involves
understanding the complementary nature of the two forces, to develop a policy
framework that imposes stricter data retention laws on users that wish to
remain anonymous, but places the onus on the government to demonstrate its
warranted access to user information (Khatri & Brown, 2010).

##  5 Conclusion and Final Remarks

Wikipedia is introduced in this paper to outline its current and potential
future practices around pertinent issues in today’s society. The two issues
introduced are collective intelligence and mass surveillance. This paper draws
examples from Wikipedia’s involvement with individuals, private institutions,
and governments in order to discuss about how Wikipedia has balanced the two
opposing forces and propose methods to find common ground between the
different stakeholders through policy-making. But as always, there is no safe
harbour. Complex changes are likely to occur in every organisation, that will
have long-lasing detriment internally and externally if not dealt with pre-
emptively.

The reader might perceive the tone of this paper as negative. It is essential
to be pragmatic about the way technologies can be utilised for the benefit of
public and private institutions; it is inevitable that we are increasingly
dependent on the conveniences provided as such. Overall, we should remain
optimistic as a society about what the future of technology holds. Internal
policies of online communities and government legislation will be strictly
regulated in supporting the transition to a more transparent environment with
accountability from all types of stakeholders.

Finally, the best time to act is now. This paper serves as a wake-up call to
individuals, private institutions, and governments to remain vigilant about
the storage and use of personal information as a result of contributing to
online communities. A balancing act is to be performed.

## Acknowledgements

I would like to thank my supervisor, Dr. Frances Di Lauro, for providing me
with the opportunity to partake in this research topic.

## References

Andrus, D. C. (2005). The Wiki and the Blog: Toward a Complex Adaptive
Intelligence Community. Studies in Intelligence, 49(3).

Barker, J., Hannay, P., & Bolan, C. (2010). Using Traffic Analysis to Identify
Tor Usage — A Proposed Study. In Security and Management (pp. 620-623).

Büchi, M., Fosch V. E., Lutz, C., Tamò-Larrieux, A., Velidi, S., & Viljoen, S.
(2019). Chilling Effects of Profiling Activities: Mapping the Issues.
Available at SSRN 3379275.

Cap, C. H. (2012). Towards Content Neutrality in Wiki Systems. Future
Internet, 4(4), 1086-1104.

Cecere, G., & Rochelandet, F. (2013). Privacy intrusiveness and web audiences:
Empirical evidence. Telecommunications Policy, 37(10), 1004-1014.

Chesbrough, H. (2017). The Future of Open Innovation: The future of open
innovation is more extensive, more collaborative, and more engaged with a
wider variety of participants. Research-Technology Management, 60(1), 35-38.

Chomik, A. (2012). Spies Wearing Purple Hats: The use of social computing to
improve information sharing inside the Intelligence Community of the United
States (Master’s Thesis, University of Calgary).

Coelho, J., & Valente, M. T. (2017, August). Why Modern Open Source Projects
Fail. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software
Engineering (pp. 186-196).

Di Lauro, F., & Johinke, R. (2017). Employing Wikipedia for good not evil:
innovative approaches to collaborative writing assessment. Assessment &
Evaluation in Higher Education, 42(3), 478-491.

Faucher, J. B. P., Everett, A. M., & Lawson, R. (2008). A Complex Adaptive
Organization Under the Lens of the LIFE Model: The Case of Wikipedia. In
Fourth Organizational Studies Summer Workshop: “Embracing Complexity:
Advancing Ecological Understanding in Organization Studies”.

Forte, A., Andalibi, N., & Greenstadt, R. (2017, February). Privacy,
Anonymity, and Perceived Risk in Open Collaboration: A study of Tor Users and
Wikipedians. In Proceedings of the 2017 ACM Conference on Computer Supported
Cooperative Work and Social Computing (pp. 1800-1811). Association for
Computing Machinery.

Geiger, R. S. (2017). Beyond opening up the black box: Investigating the role
of algorithmic systems in Wikipedian organizational culture. Big Data &
Society, 4(2), 2053951717730735.

Glocer, T., & Linnebank, G. (2001, October 2). Letter to the editors of
certain US newspapers. Reuters. Retrieved from
https://web.archive.org/web/20011002123400/http://about.reuters.com/statement3.asp

Gloor, P. A., Marcos, J., De Boer, P. M., Fuehres, H., Lo, W., & Nemoto, K.
(2017). Cultural Anthropology through the Lens of Wikipedia. Social Network
Analysis: Interdisciplinary Approaches and Case Studies, 245.

Greenstein, S., & Zhu, F. (2012). Collective Intelligence and Neutral Point of
View: The Case of Wikipedia (No. w18167). National Bureau of Economic
Research.

Guess, A., Nyhan, B., & Reifler, J. (2018). Selective Exposure to
Misinformation: Evidence from the consumption of fake news during the 2016
U.S. presidential campaign. European Research Council, 9(3), 4.

Hickman, G. R. (2010). Leading Change in Multiple Contexts: Concepts and
Practices in Organizational, Community, Political, Social, and Global Change
Settings. Sage Publications.

Huang, L., Zhu, Q., Du, J. T., & Lee, B. (2016). Exploring the dynamic
contribution behavior of editors in wikis based on time series analysis.
Program, 50(1), 41-57.

Hube, C. (2017, April). Bias in Wikipedia. In Proceedings of the 26th
International Conference on World Wide Web Companion (pp. 717-721).

Hui, K. L., & Png, I. P. L. (2006). The Economics of Privacy. In Economics and
Information Systems, Volume 1 (Handbooks in Information Systems) (Chapter 9).

Khatri, V., & Brown, C. V. (2010). Designing Data Governance. Communications
of the ACM, 53(1), 148-152.

Konieczny, P. (2014). The day Wikipedia stood still: Wikipedia’s editors’
participation in the 2012 anti-SOPA protests as a case study of online
organization empowering international and national political opportunity
structures. Current Sociology, 62(7), 994-1016.

Kumar, S., West, R., & Leskovec, J. (2016, April). Disinformation on the Web:
Impact, Characteristics, and Detection of Wikipedia Hoaxes. In Proceedings of
the 25th international conference on World Wide Web (pp. 591-602).
International World Wide Web Conferences Steering Committee.

Lichtenstein, S., & Parker, C. M. (2009). Wikipedia model for collective
intelligence: a review of information quality. International Journal of
Knowledge and Learning, 5(3-4), 254-272.

Lih, A. (2004). Wikipedia as Participatory Journalism: Reliable Sources?
Metrics for evaluating collaborative media as a news resource. Nature, 3(1),
1-31.

Livingstone, R. M. (2010). Let’s Leave the Bias to the Mainstream Media: A
Wikipedia Community Fighting for Information Neutrality. M/C Journal, 13(6).

Mercaldo, F., & Martinelli, F. (2017, September). Tor Traffic Analysis and
Identification. In 2017 AEIT International Annual Conference (pp. 1-6). IEEE.

Miller, J. C., & Murray, H. B. (2010). Wikipedia in Court: When and How Citing
Wikipedia and Other Consensus Websites Is Appropriate. St. John’s Law Review,
84(2), 633.

Nov, O. (2007). What Motivates Wikipedians?. Communications of the ACM,
50(11), 60-64.

Pasquale, F. (2015). The Black Box Society: The Secret Algorithms That Control
Money and Information. Cambridge, Massachusetts; London, England: Harvard
University Press.

Penney, J. W. (2016). Chilling Effects: Online Surveillance and Wikipedia Use.
Berkeley Technology Law Journal, 31(1), 117-182.

Privacy policy. (n.d.). In Wikimedia Foundation. Retrieved August 19, 2020,
from https://foundation.wikimedia.org/wiki/Privacy_policy

Raymond, E. (1999). The Cathedral and the Bazaar. Knowledge, Technology &
Policy, 12(3), 23-49.

Rosenzweig, R. (2006). Can History Be Open Source? Wikipedia and the Future of
the Past. The Journal of American History, 93(1), 117-146.

Rukundo, S. (2019). Wikipedia in the Courts: An examination of the citation of
Wikipedia in judicial opinions in Uganda. Computer Law & Security Review,
35(5), 105316.

Sáez-Trumper, D. (2019). Online Disinformation and the Role of Wikipedia.
ArXiv, abs/1910.12596.

Schauer, F. (1978). Fear, Risk and the First Amendment: Unraveling the
Chilling Effect. Boston University Law Review, 58(5), 685-732.

Selwyn, N., & Gorard, S. (2016). Students’ use of Wikipedia as an academic
resource — Patterns of use and perceptions of usefulness. The Internet and
Higher Education, 28, 28-34.

Solove, D. J. (2006). A Taxonomy of Privacy. University of Pennsylvania Law
Review, 154(3), 477.

Stein, L. (2013). Policy and Participation on Social Media: The Cases of
YouTube, Facebook, and Wikipedia. Communication, Culture & Critique, 6(3),
353-371.

Stevenson, A. (Ed.). (2010). Oxford Dictionary of English. Oxford University
Press, USA.

Stop Online Piracy Act Bill 2011 (US).

Sydow, M., Baraniak, K., & Teisseyre, P. (2017). Diversity of editors and
teams versus quality of cooperative work: experiments on wikipedia. Journal of
Intelligent Information Systems, 48(3), 601-632.

Szoldra, P. (2016, September 16). This is everything Edward Snowden revealed
in just one year of unprecedented top-secret leaks. Business Insider.
Retrieved from https://www.businessinsider.com.au/snowden-leaks-
timeline-2016-9

Tedjamulia, S. J., Dean, D. L., Olsen, D. R., & Albrecht, C. C. (2005,
January). Motivating Content Contributions to Online Communities: Toward a
More Comprehensive Theory. In Proceedings of the 38th Hawaii International
Conference on System Sciences (pp. 193b-193b). IEEE.

Tran, C., Champion, K., Forte, A., Hill, B. M., & Greenstadt, R. (2020, May).
Are anonymity-seekers just like everybody else? An analysis of contributions
to Wikipedia from Tor. In 2020 IEEE Symposium on Security and Privacy (SP)
(pp. 186-202). IEEE.

Turnbull, S., & Guthrie, J. (2019). Simplifying the Management of Complexity:
As Achieved in Nature. Journal of Behavioural Economics and Social Systems,
1(1), 51-73.

Vrandečić, D. (2019). Collaborating on the sum of all knowledge across
languages. Wikipedia @ 20.

Wales, J., & Tretikov, L. (2015, March 10). Stop Spying on Wikipedia Users.
The New York Times. Retrieved from
https://www.nytimes.com/2015/03/10/opinion/stop-spying-on-wikipedia-users.html

Werbin, K. C. (2011). Spookipedia: intelligence, social media and biopolitics.
Media, Culture & Society, 33(8), 1254-1265.

Wikimedia Foundation, et al. v. National Security Agency, et al. (2015).

Wikipedia:Advice to users using Tor. (n.d.). In Wikipedia. Retrieved August
19, 2020, from
https://en.wikipedia.org/wiki/Wikipedia:Advice_to_users_using_Tor

Wikipedia:Manual of Style/Words to watch. (n.d.). In Wikipedia. Retrieved
August 19, 2020, from
https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Words_to_watch

Wikipedia:SOPA initiative. (n.d.). In Wikipedia. Retrieved August 19, 2020,
from https://en.wikipedia.org/wiki/Wikipedia:SOPA_initiative

Wikipedia:Statistics. (n.d.). In Wikipedia. Retrieved August 19, 2020, from
https://en.wikipedia.org/wiki/Wikipedia:Statistics

Wikipedia:Wikipedia as a court source. (n.d.). In Wikipedia. Retrieved August
19, 2020, from
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_as_a_court_source

Wikipedia:Wikipedia in judicial opinions. (n.d.). In Wikipedia. Retrieved
August 19, 2020, from
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_in_judicial_opinions

Xie, C. X., Johnson, I., & Gomez, A. (2019, May). Detecting and Gauging Impact
on Wikipedia Page Views. In Companion Proceedings of The 2019 World Wide Web
Conference (pp. 1254-1261).


– Under Review 2023 Full Paper – MIDL 2023 Leo Milecki1${}^{\text}{({\char
0\relax})}$<EMAIL_ADDRESS>
1 MICS, CentraleSupelec, Paris-Saclay University, Inria Saclay, France and
Vicky Kalogeiton2<EMAIL_ADDRESS>
2 LIX, École Polytechnique, CNRS, Institut Polytechnique de Paris, France and
Sylvain Bodard3,6<EMAIL_ADDRESS>
3 Department of Adult Radiology, Necker Hospital, Paris University, France and
Dany Anglicheau4,6<EMAIL_ADDRESS>
4 Department of Nephrology and Kidney Transplantation, Necker Hospital, Paris
University, France and Jean-Michel Correas3,6<EMAIL_ADDRESS>and
Marc-Olivier Timsit5,6<EMAIL_ADDRESS>
5 Department of Urology, HEGP, Necker Hospital, Paris University, France
6 UFR Médecine, Paris-Cité University, France and Maria Vakalopoulou1
<EMAIL_ADDRESS>

# MEDIMP: Medical Images and Prompts for renal transplant representation
learning

###### Abstract

Renal transplantation emerges as the most effective solution for end-stage
renal disease. Occurring from complex causes, a substantial risk of transplant
chronic dysfunction persists and may lead to graft loss. Medical imaging plays
a substantial role in renal transplant monitoring in clinical practice.
However, graft supervision is multi-disciplinary, notably joining nephrology,
urology, and radiology, while identifying robust biomarkers from such high-
dimensional and complex data for prognosis is challenging. In this work,
taking inspiration from the recent success of Large Language Models (LLMs), we
propose MEDIMP – Medical Images and Prompts – a model to learn meaningful
multi-modal representations of renal transplant Dynamic Contrast-Enhanced
Magnetic Resonance Imaging (DCE MRI) by incorporating structural
clinicobiological data after translating them into text prompts. MEDIMP is
based on contrastive learning from joint text-image paired embeddings to
perform this challenging task. Moreover, we propose a framework that generates
medical prompts using automatic textual data augmentations from LLMs. Our goal
is to learn meaningful manifolds of renal transplant DCE MRI, interesting for
the prognosis of the transplant or patient status (2, 3, and 4 years after the
transplant), fully exploiting the available multi-modal data in the most
efficient way. Extensive experiments and comparisons with other renal
transplant representation learning methods with limited data prove the
effectiveness of MEDIMP in a relevant clinical setting, giving new directions
toward medical prompts. Our code is available at
https://github.com/leomlck/MEDIMP.

###### keywords:

Contrastive Learning, Natural Language Processing, LLM, MRI, Renal
transplantation, Medical Prompts.

††editors: Under Review for MIDL 2023

##  1 Introduction

End-stage renal disease is characterized by an irremediable reduction in
kidney function, and renal replacement therapy is required to save the
patient’s life. Being more cost-effective than long-term dialysis and highly
improving quality of life, renal transplantation emerges as the most effective
solution [Suthanthiran and Strom (1994)]. However, a substantial risk of
transplant chronic dysfunction persists and may lead to graft loss or patient
death [Hariharan et al. (2021)]. In clinical practice, the graft health status
is primarily indicated by calculating the glomerular filtration rate (GFR)
from the creatinine level resulting from blood tests. For further
examinations, medical imaging plays a substantial role, and diverse imaging
modalities have been investigated to monitor renal transplants [Sharfuddin
(2013)].

Learning powerful representations of medical imaging is of utmost importance,
given the usual small size and limited annotations available. In such a
setting, learning is performed in two stages. In the first stage, different
self-supervised or weakly-supervised learning methods [Taleb et al. (2020),
Krishnan et al. (2022)] are used on the available imaging datasets, applying
different types of learning, such as contrastive or adversarial learning
[Sowrirajan et al. (2021), Azizi et al. (2021), Boyd et al. (2021)]. Such
representations are then frozen or fine-tuned for different downstream tasks,
for which the amount of information is not enough for fully supervised
learning. Such pretrainings could provide better representations and
outperform ImageNet pretrained networks when applied to medical imaging.
However, they may produce suboptimal representations for the downstream tasks
that merely capture spurious correlations [Arjovsky et al. (2019)]. More
particularly, for renal transplantation, Milecki et al. (2022) proposed
weakly-supervised tasks from clinical information to learn rich
representations of DCE MRI using a single continuous attribute, confirming
that combining imaging and clinical data leads to powerful biomarkers for
prognosis.

Recent advances in Natural Language Processing (NLP) make textual data a
potent candidate for designing weakly-supervised tasks to train computer
vision models. Multiview contrastive learning Bachman et al. (2019) has been
investigated to take advantage of jointly training an image and text encoder
in Zhang et al. (2020); Radford et al. (2021); Jia et al. (2021); Müller et
al. (2022). For natural images, Radford et al. (2021) produced robust
representations using 400 million (image, text) pairs, reporting competitive
results on several downstream tasks on unseen datasets compared to fully
supervised baselines. Zhang et al. (2020) used chest X-rays and pathology
descriptions from radiology experts’ diagnoses. Müller et al. (2022) extended
this joint image-text representation learning for localized tasks like
semantic segmentation or object detection. All these studies consider 2D
images, and the medical ones used the MIMIC-CXR database, the largest dataset
containing paired medical images and radiology reports. However, such data
curation is arduous, being highly time-consuming for medical experts, while
such reports contain mainly information about the corresponding medical
imaging exam, and they do not focus on other comorbidities.

Therefore, we propose to go one more step forward by generating
representations using paired imaging and clinicobiological attributes in a
relevant clinical setup with limited data. We explore recent NLP advances in
Large Language Models (LLMs). In particular, ChatGPT OpenAI (2022), a 175B
parameters model, offers a powerful tool to produce textual data.
Specifically, textual data allow advantages as opposed to tabular data for
medical applications: (1) Contextual information: Textual data contains rich
contextual information helping to better understand the underlying patterns in
the data; (2) Better representation: text can provide a more expressive
representation of the information contained in the data, leading to improved
performances as our model better capture the complexities of the
clinicobiological data. (3) Transferability and Interpretability: text is
often more transferable across domains than tabular data. Moreover, text is
more interpretable by humans, which is valuable for validating and
understanding the decisions made by the proposed method.

Specifically, we introduce MEDIMP (MEDical IMages and Prompts). This method
learns relevant DCE MRI representations of renal transplants using contrastive
learning from images and medical prompts pairs. The learned manifold enabled
us to outperform state-of-the-art methods in the challenging task of kidney
function prediction $2$, $3$, and $4$ years post-transplantation from $4$ DCE
MRI follow-up exams. Our contributions are: (i) We propose a semi-automatic
medical prompt generation from tabular data; to the best of our knowledge,
this is the first work to propose such an approach for augmenting medical
textual data; (ii) We extend existing approaches that combine text and imaging
data by integrating 3D medical inputs and fine-tuning strategies; our approach
allows for using pretrained NLP models on a limited amount of textual data.

##  2 Method

fig:overview

Figure 1:  Overview of our proposed method MEDIMP – Medical Images and
Prompts. 1\. Medical prompts are generated from clinicobiological data using
predefined templates of sentences, given as inputs to Large Language Models to
produce augmented text data. 2. The medical prompts are used to learn multi-
modal representations of renal transplants DCE MRI using contrastive learning
from image-text pairs.

Our multi-modal representations are based on contrastive learning, coupling
imaging, and text embeddings. Our text is based on attributes that are easily
accessible, widely used in clinical practice, and supplementary to imaging
data. Our goal is to use the learned manifold of renal transplant DCE MRI for
the prognosis of the transplant or patient status after $2,3$, and $4$ years
post-transplantation. An overview of the method is presented in fig:overview.

###  2.1 Contrastive learning from joint text-image pairs

The first component of MEDIMP is pretraining an image encoder $E_{\text{I}}$,
and a text encoder $E_{\text{T}}$, following a contrastive learning scheme
using image-text pairs. Let us denote $(x_{i},x_{t})\in\mathbb{R}^{B\times
N_{x}\times N_{y}\times N_{z}}\times\mathbb{R}^{B\times T}$ a batch of $B$
corresponding pairs of an 3D MRI volume $x_{ib}$ and a tokenized text $x_{tb}$
for $b\in[\\![1,B]\\!]$. Both encoders transform $x_{i}$ and $x_{t}$, in
$f_{i}=E_{\text{I}}(x_{i})$ and $f_{t}=E_{\text{T}}(x_{t})$ respectively, to a
batch of $D$ dimensional embeddings. Both encoders are jointly trained to
maximize the cosine similarity between the $B$ pairs of image and text
embeddings by optimizing the two following losses:

$\mathcal{L}_{i\rightarrow
t}=\sum_{b=1}^{B}-\log{{e^{\text{cos}(f_{ib},f_{tb})/\tau}}\over{\sum_{k=1}^{B}e^{\text{cos}(f_{ib},f_{tk})/\tau}}}\quad,$
(1)

where $\text{cos}(\cdot,\cdot)$ is the cosine similarity function and
$\tau\in\mathbb{R}^{+}$ a learned temperature parameter. Such loss was first
proposed as the InfoNCE loss van den Oord et al. (2018) to maximize a lower
bound on mutual information and is widely used in recent uni-modal contrastive
learning frameworks Chen et al. (2020). $\mathcal{L}_{i\rightarrow t}$
enforces the image embeddings to align to the text embeddings and is,
therefore, asymmetric. Similarly, we define $\mathcal{L}_{t\rightarrow i}$:

$\mathcal{L}_{t\rightarrow
i}=\sum_{b=1}^{B}-\log{{e^{\text{cos}(f_{tb},f_{ib})/\tau}}\over{\sum_{k=1}^{B}e^{\text{cos}(f_{tb},f_{ik})/\tau}}}\quad.$
(2)

The total loss is obtained by averaging eq:1 and eq:2, denoted as
$\mathcal{L}_{\text{contrastive}}$. $\mathcal{L}_{\text{contrastive}}$ learns
a multi-modal feature space by jointly optimizing $E_{\text{I}}$ and
$E_{\text{T}}$ to maximize the cosine similarity of the embeddings $f_{i}$ and
$f_{t}$ between the $B$ true pairs per batch and minimizing the cosine
similarity of the $B^{2}-B$ false pairs.

###  2.2 Medical prompts from structural clinical and biological data

To exploit image-text pairing with contrastive learning, as well as the
expression and encoding capability of recent NLP model advances, such as Large
Language Models (LLMs) Brown et al. (2020); OpenAI (2022); Radford et al.
(2021). We propose a framework to generate textual data from structural
clinicobiological data that describe variables used in clinical practice and
linked to the graft survival. The process is displayed on the left side of
fig:overview. First, medical experts guided us to set thresholds to categorize
continuous variables into text labels such as “low”, “high”, “stable”, and
“unstable” and to produce one template sentence per variable of interest,
e.g., “the GFR of the patient is very low at the first-year follow-up exam”.
However, templates offer only one way of expressing the information of the
variables. Indeed, the richness of language vocabulary can provide a variety
of descriptions for the same information, such as “During the first-year
follow-up visit, the transplant patient’s GFR is found to be very low.”, or
“The transplant patient’s GFR is assessed as very low at the date follow-up
examination.”, thus generating descriptive text to train the proposed
contrastive scheme. This richness was leveraged by recent advances in LLMs at
training; hence, they offer robust NLP tools. Specifically, we use the
dialogue LLM ChatGPT OpenAI (2022) to produce $N=10$ textual data
augmentations for each template sentence. All generated prompts are reported
in Appendix A.

###  2.3 Implementation details

The image encoder followed a 3D ResNet50 architecture initialized with CLIP
Radford et al. (2021) weights, a model pretrained on a dataset of 400 million
(image, text) pairs collected from the internet. We extended the attention-
based pooling layer of CLIP to 3D, as well as we duplicated the weights to 3D
in depth to match our data dimension. For the text encoder, we used the BERT
Devlin et al. (2019) architecture initialized with the Bio+Clinical BERT
Alsentzer et al. (2019) model pretrained on the MIMIC clinical notes Johnson
et al. (2016). The first $11$ layers of Bio+Clinical BERT were frozen, fine-
tuning only the last layer of the transformer with our contrastive task.
Appendix C discusses the ablation of fine-tuning more layers for our task. The
temperature parameter $\tau$ was initialized to $0.07$, incorporated into the
model as a learnable parameter, and clipped to prevent scaling the logits by
more than $100$, following the recommendations of CLIP trainings.

For the image data, we used data augmentation with the sequential application
with a $0.5$ probability of horizontal flipping, random affine transformation,
random Gaussian blur ($\sigma\in[0,0.5]$), random Gaussian noise
($\sigma\in[0,0.05]$), and random contrast perturbation
($\log\gamma\in[-0.3,0.3]$), using TorchIO python library Pérez-García et al.
(2021).

In our experiments, we use the Adam Kingma and Ba (2015) optimizer with
decoupled weight decay regularization Loshchilov and Hutter (2017) of $0.02$
with a starting learning rate of $5e^{-5}$ following a cosine schedule and
preceded by a linear warm-up of $40$ epochs. The batch size was set to $88$
and the model trained for $200$ epochs with mixed-precision on $4$ NVIDIA
Tesla V100 GPU using Pytorch Paszke et al. (2019).

##  3 Data

Our study was approved by the Institutional Review Board, which waived the
need for patients’ consent. The data cohort corresponds to study reference ID-
RCB: 2012-A01070-43 and ClinicalTrials.gov identifier: NCT02201537. All the
data used in this study are anonymized. Our imaging data are based on DCE MRI
series collected from $105$ subjects (split as $72/5$ training/validation, and
$28$ test). Each subject underwent up to $4$ follow-up exams, taking place
approximately $15$ days, $30$ days, $3$ months, and $1$ year post-
transplantation. The DCE MRI volumes sized $512\times 512\times[64-88]$ voxels
included spacing ranging in $[0.78-0.94]\times[0.78-0.94]\times[1.9-2.5]$ mm.
All volumes were cropped around the transplant using an automatic and
unsupervised method for selecting the region of interest and reducing
dimensionality Milecki et al. (2021). Intensity normalization was executed to
each volume independently by applying standard normalization, clipping values
to $[-5,5]$, and rescaling linearly to $[0,1]$.

To provide the clinical and biological data used to generate text annotations,
all $77$ patients in the train set were regularly subjected to blood tests a
few days before the transplantation and several years after to measure the
serum creatinine (Creat) level in $\mu mol.L^{-1}$. The donor’s age variable
and the GFR value at each follow-up exam were also collected. Only the target
labels were available for $28$ test subjects, as defined in Section 4.

##  4 Experiments & Results

fig:tsnes

Figure 2:  t-SNE visualizations of the features of the last layer of MEDIMP
image encoder using the DCE MRI exams. Colormaps are set by the $4$ variables
of interest value used for the medical prompts: (1) Exam (exam date), (2) GFR
($\text{mL.min}^{-1}$, (3) Creat ($\mu mol.L^{-1}$), and (4) donor’s age
(year). Stars symbol display the real data while circles the augmented (aug)
data. D15, D30, M3, and M12 are the 4 exam timestamps.

First, we visualize the obtained representations of DCE MRI from the trained
image encoder using t-SNE decomposition van der Maaten and Hinton (2008).
fig:tsnes shows the different obtained reduced feature spaces by the model
trained on all available clinical and biological variables ($n_{cl}=4$),
adding colormaps related to the different clinical and biological variables of
interest. While the continuous variables were categorized and transformed to
medical prompts, MEDIMP image encoder demonstrates relevant representations
towards (B) the GFR and (C) the Creat. The t-SNE decomposition does not reveal
favorable representations regarding (D) Donor’s Age. On the contrary, the
obtained feature space serves well the (A) Exam date information, where we
retrieve better clustered very early exams (D15, blue) and late exams (M12,
red), due to their respective distance to the transplantation surgery.

#### Downstream task and metrics.

We evaluate MEDIMP on the downstream task of serum creatinine (Creat)
prediction from the imaging features of $4$ follow-up exams using a light
transformer architecture and tailored for missing follow-up exams, proposed by
Milecki et al. (2022). In clinical practice, Creat results from blood tests
and is a primary indicator of kidney function. The binary classification
downstream task is obtained when binarizing the Creat value using a threshold
of $110\mu mol.L^{-1}$ at different prediction dates. The Creat target
prediction value is calculated as the mean over an interval of three months
before and after the prediction dates. To make the task more challenging, we
evaluate the performance of the representations at $2$, $3$, and $4$ years
post-transplantation and also report the mean over the three predictions, for
the $28$ test subjects. The two evaluation metrics used were the F1 score and
the area under the receiver operating characteristic curve (ROC AUC).

#### Ablation.

First, we ablate all information used in our method MEDIMP by adding different
combinations of the clinical and biological measurements in the medical
prompts, i.e., the glomerular filtration rate (GFR) at the exam date, the
number of the patient’s exam (Exam), the creatinine levels variation from the
previous exam (Creat), and the age of the donor (D.A.). For this, we report
the results in the bottom part of tab:tab1 (MEDIMP). We observe that the best
mean scores over the three predictions are obtained using all the available
medical prompts (last row). Moreover, the AUC decreases over the prediction
date, showing the increasing prognosis difficulty with time. Using only the
GFR prompts (row 6), AUC scores are just above random, indicating the need for
more descriptive text.

Table 1:  Comparison of MEDIMP vs the state of the art. We evaluate the
performance of the representations at 2,3,4 years post-transplantation and
report the mean. Ablations in weak annotations added from either the variable
values for the comparison CosEmbLoss pertaining or our proposed generated
textual data are denoted as GFR (GFR value at the follow-up exam), Exam (which
follow-up exam), Creat (creatinine levels variations from the previous exam),
and D.A. (the donor’s age). We report F1 score (F1), and ROC AUC (AUC). Bold,
Underlined indicates the top 1, 2 performing combinations, respectively.

| Weak annotations |  2 years |  3 years |  4 years | Mean
---|---|---|---|---|---
Method |  GFR |  Exam |  Creat |  D.A. | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1
CLIP weights |  |  |  |  |  62.6 |  73.7 |  52.5 |  78.1 |  51.3 |  54.6 |  55.5 |  68.8
CosEmbLoss |  |  |  |  |  76.2 |  86.4 |  77.8 |  70.6 |  67.0 |  77.3 |  73.6 |  78.1
CosEmbLoss |  |  |  |  |  75.5 |  81.1 |  75.6 |  68.8 |  66.1 |  78.1 |  72.4 |  76.0
CosEmbLoss++ |  |  |  |  |  84.4 |  88.9 |  82.5 |  86.4 |  73.9 |  85.7 |  80.3 |  87.0
CosEmbLoss++ |  |  |  |  |  81.6 |  87.8 |  71.3 |  85.1 |  71.3 |  90.2 |  74.7 |  87.7
CosEmbLoss++ |  |  |  |  |  78.2 |  87.0 |  75.0 |  83.3 |  74.8 |  87.0 |  76.0 |  85.8
MEDIMP |  |  |  |  |  56.5 |  83.3 |  51.9 |  79.1 |  49.6 |  90.2 |  52.6 |  84.2
MEDIMP |  |  |  |  |  81.0 |  89.4 |  81.9 |  80.0 |  74.8 |  84.4 |  79.2 |  84.6
MEDIMP |  |  |  |  |  76.9 |  73.2 |  86.3 |  85.7 |  74.8 |  90.2 |  79.3 |  83.0
MEDIMP |  |  |  |  |  72.8 |  86.4 |  71.9 |  81.0 |  71.3 |  71.8 |  72.0 |  79.7
MEDIMP |  |  |  |  |  85.0 |  89.4 |  84.4 |  83.7 |  75.7 |  90.2 |  81.7 |  87.8

#### Comparison to the state of the art.

tab:tab1 reports the results when comparing the proposed MEDIMP with the
previous state-of-the-art Milecki et al. (2022), denoted as CosEmbLoss (row 1
& 2). Note that the main model of CosEmbLoss uses only GFR information. Hence,
it is directly comparable to MEDIMP when using only GFR (6th row)111CosEmbLoss
uses the continuous value on their loss directly. In contrast, we categorize
the variable to generate the prompts, hence using less information than
CosEmbLoss. These experiments reveal that smaller and more compact models such
as the CosEmbLoss perform better than big models when the text information is
not very rich. However, when more variables are integrated, our proposed
methods outperform the CosEmbLoss. For a fair comparison, we also compare
MEDIMP to several baselines with the same level of information. In particular,
we evaluate against three baselines, denoted as the CosEmbLoss++, where we
gradually add the same level of information as in MEDIMP. In practice, we
optimize the same two-stream approach by averaging several cosine embedding
losses based on the number of variables of interest incorporated. We report
these results in rows 3-5 of tab:tab1. We observe that CosEmbLoss++ achieve
their best performance when using a combination of $2$ variables. The best
mean AUC is $80.3\%$ with GFR and D.A., and the best F1 is $87.7\%$ with GFR
and Creat, which are lower than the best MEDIMP performances. Note, the Exam
information was not added to CosEmbLoss++ as this variable is less adapted to
CosEmbLoss. Nevertheless, combining the $3$ variables of interest with
CosEmbLoss++ lowers the performance to $76.0\%$ AUC and $85.8\%$ F1 in Mean.
Overall, MEDIMP with all medical prompts results in the best predictions at
$2$ and $4$ years post-transplantation.

#### Medical Prompt generation.

To demonstrate the relevance of the proposed approach for medical prompt
generation, we compare our main model with two other approaches that produce
text information. The first one is noted as “Manual” and comprises all the
templates indicated by the medical experts, corresponding to only one sentence
per variable of interest. Note that this is the base of our proposed medical
prompting without using the prompt expansion method described in Section 2.2.
The second one uses an existing NLP model (T5 Raffel et al. (2020)) to produce
sentences from structural data. For a fair comparison, we train the T5 model
on the WebNLG 2020 data Gardent et al. (2017) and infer it on our data to
generate text, denoted as “T5 WebNLG”. The results are summarised in tab:tab2,
highlighting the superiority of our method. The “T5 WebNLG” approach offers a
competitive F1 for all the predictions, although the AUC is the lowest except
for the 2 years prediction. We show in Appendix B examples of generated texts
from these three approaches. “Manual” approach lacks diversity in the text
data, and therefore the training process of our proposed approach without text
augmentations is more challenging.

Table 2:  Quantitative evaluation of the proposed method against other text generation methods. All medical prompts were used. We report F1 score (F1), and ROC AUC (AUC). Bold,Underlined indicates the top 1,2 performing combinations, respectively.  | 2 years | 3 years | 4 years | Mean
---|---|---|---|---
Method | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1
MEDIMP |  85.0 |  89.4 |  84.4 |  83.7 |  75.7 |  90.2 |  81.7 |  87.8
Manual |  74.2 |  76.2 |  80.6 |  62.1 |  80.0 |  76.9 |  78.3 |  71.7
T5 WebNLG |  74.8 |  85.7 |  78.8 |  83.3 |  74.8 |  85.7 |  76.1 |  84.9

##  5 Discussion

Our experiments have shown improvements in the representation learning
capabilities of deep image encoders for renal transplantation MRI compared to
the previous state-of-the-art approach for the specific application of renal
transplant function forecasting. MEDIMP aimed at enhancing representation
learning approaches using external data, by leveraging the power of deep NLP
models, such as LLMs, and by introducing a novel process to incorporate
relevant clinical and biological medical information. We deem that such an
approach crossing modalities in medical research would highly improve the
capacity to learn and understand complex medical or biological phenomena.
However, some limitations of the proposed framework remain to be analyzed. (1)
First, although limited data is part of the challenges this study aims to
tackle, supplementary test data would indubitably support validating our
method. To our knowledge, no public medical imaging dataset offers
simultaneously longitudinal imaging, biological, as well as clinical data for
each patient for prognosis at different times. Nevertheless, our proposed
framework could be easily translated to similar datasets having imaging data
and any type of tabular data. (2) Second, this work constitutes a first
attempt to generate medical prompts as text information from a few clinical
and biological variables of interest, which are very important to apprehend
complex medical concepts. Thus, we would seek to extend this work to be able
to exploit more variables to guide the training of the image encoders. Using
our proposed framework this can be easily implemented since only a few
templates will need to be defined to automatically generate text
augmentations. (3) Finally, the study of the development and comprehension of
LLMs is recent. We expect such NLP models to increase the emphasis on using
textual data. However, the main drawback of these models is their reliability,
in the sense that inaccurate generation might be challenging to detect. In
this work, we use ChatGPT to enable relevant and robust text augmentations. No
information from the sensitive clinical data was leaked into the prompts,
which is an essential process in the medical domain. One could think of
several ways to further use such models as powerful tools, for instance,
through prompt engineering tuned for medical cases.

##  6 Conclusion

We have introduced MEDIMP, an approach to learn powerful manifolds of renal
transplant DCE MRI data toward transplant function forecasting in the context
of renal transplantation monitoring. This approach provides an elegant way to
incorporate clinical or biological information into the learning process of
feature extraction of medical imaging data. The presented representation
learning strategy enabled us to outperform the state of the art in the
challenging task of creatinine prediction. These promising results advocate
using textual data from emerging LLMs to assist in training robust medical
imaging models.

This work was performed using HPC resources from the
http://mesocentre.centralesupelec.fr/“Mésocentre” computing center of
CentraleSupelec and École Normale Supérieure Paris-Saclay supported by CNRS
and Région Île-de-France, as well as it was granted access to the HPC
resources of IDRIS under the allocation 2022-AD011013541 made by GENCI.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical BERT embeddings.  In _Clinical Natural Language Processing Workshop (ClinicalNLP)_ , pages 72–78. Association for Computational Linguistics, 2019\.
  * Arjovsky et al. (2019) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  Eprint http://arxiv.org/abs/1907.02893arXiv:1907.02893, 2019\.
  * Azizi et al. (2021) S. Azizi, B. Mustafa, F. Ryan, Z. Beaver, J. Freyberg, J. Deaton, A. Loh, A. Karthikesalingam, S. Kornblith, T. Chen, V. Natarajan, and M. Norouzi.  Big self-supervised models advance medical image classification.  In _International Conference on Computer Vision (ICCV)_ , pages 3458–3468. IEEE, 2021.
  * Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32. Curran Associates, Inc., 2019.
  * Boyd et al. (2021) Joseph Boyd, Mykola Liashuha, Eric Deutsch, Nikos Paragios, Stergios Christodoulidis, and Maria Vakalopoulou.  Self-supervised representation learning using visual field expansion on digital pathology.  In _International Conference on Computer Vision (ICCV)_ , pages 639–647. IEEE, 2021.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning (ICML)_. PMLR, 2020\.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , pages 4171–4186. Association for Computational Linguistics, 2019.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini.  The WebNLG challenge: Generating text from RDF data.  In _International Conference on Natural Language Generation (INLG)_ , pages 124–133. Association for Computational Linguistics, 2017.
  * Hariharan et al. (2021) Sundaram Hariharan, Ajay K. Israni, and Gabriel Danovitch.  Long-term survival after kidney transplantation.  _The New England Journal of Medicine_ , 385(8):729–743, 2021.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning (ICML)_. PMLR, 2021\.
  * Johnson et al. (2016) A. E. Johnson, T. J. Pollard, L. Shen, L. W. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3:160035, May 2016.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Krishnan et al. (2022) R. Krishnan, P. Rajpurkar, and E. J. Topol.  Self-supervised learning in medicine and healthcare.  _Nature Biomedical Engineering_ , 6(12):1346–1352, Dec 2022.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Lu et al. (2022) Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch.  Frozen pretrained transformers as universal computation engines.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 7628–7636, 2022.
  * Milecki et al. (2021) Leo Milecki, Sylvain Bodard, Jean-Michel Correas, Marc-Olivier Timsit, and Maria Vakalopoulou.  3D unsupervised kidney graft segmentation based on deep learning and multi-sequence MRI.  In _International Symposium on Biomedical Imaging (ISBI)_ , pages 1781–1785. IEEE, 2021.
  * Milecki et al. (2022) Leo Milecki, Vicky Kalogeiton, Sylvain Bodard, Dany Anglicheau, Jean-Michel Correas, Marc-Olivier Timsit, and Maria Vakalopoulou.  Contrastive masked transformers for forecasting renal transplant function.  In _Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , pages 244–254. Springer, 2022.
  * Müller et al. (2022) Philip Müller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert.  Joint learning of localized representations from medical images and reports.  In _European Conference on Computer Vision (ECCV)_ , pages 685–701. Springer, 2022.
  * OpenAI (2022) OpenAI.  Chatgpt: Optimizing language models for dialogue, 2022.  URL https://openai.com/blog/chatgpt/.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32. Curran Associates, Inc., 2019.
  * Pérez-García et al. (2021) Fernando Pérez-García, Rachel Sparks, and Sébastien Ourselin.  Torchio: A python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.  _Computer Methods and Programs in Biomedicine_ , 208:106236, 2021.  ISSN 0169-2607.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , volume 139, pages 8748–8763. PMLR, 2021.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(1), jan 2020\.  ISSN 1532-4435.
  * Sharfuddin (2013) Asif Sharfuddin.  Renal relevant radiology: Imaging in kidney transplantation.  _Clinical journal of the American Society of Nephrology_ , 9, 11 2013\.
  * Sowrirajan et al. (2021) Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, and Pranav Rajpurkar.  MoCo-CXR: MoCo pretraining improves representation and transferability of chest X-ray models.  In _Medical Imaging with Deep Learning, 7-9 July 2021, Lübeck, Germany_ , volume 143 of _Proceedings of Machine Learning Research_ , pages 728–744. PMLR, 2021.
  * Suthanthiran and Strom (1994) Manikkam Suthanthiran and Terry B. Strom.  Renal transplantation.  _The New England Journal of Medicine_ , 331(6):365–376, 1994.
  * Taleb et al. (2020) Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3D self-supervised methods for medical imaging.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 18158–18172. Curran Associates, Inc., 2020.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  Eprint http://arxiv.org/abs/1807.03748arXiv:1807.03748, 2018\.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 30. Curran Associates, Inc., 2017.
  * Zhang et al. (2020) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz.  Contrastive learning of medical visual representations from paired images and text.  Eprint http://arxiv.org/abs/2010.00747arXiv:2010.00747, 2020\.

##  Appendix A ChatGPT prompts

The main conversation with the dialogue LLM ChatGPT OpenAI (2022) to generate
our medical prompts is the following:

  * •

Input 1: “I want to find synonyms or text augmentations of full sentences in
the context renal transplantation. Are you familiar with these concepts ?”

  * Output 1: “Yes, I am familiar with the concept of finding synonyms or text augmentations of full sentences, also known as text augmentation or data augmentation. This is a technique used to artificially increase the size and diversity of a dataset by applying various natural language processing (NLP) techniques such as synonym replacement, random insertion, deletion, and substitution of words. In the context of renal transplantation, it could be used to expand the number of examples of medical reports or other texts related to renal transplantation by generating similar but slightly different versions of the original text.”

  * •

Input 2: “The full sentence template is “The age of the donor is {age}. The
glomerular filtration rate (GFR) of the patient is {gfr} at {date} follow-up
exam. And the creatinine levels variation were {adj}” where {age}, {gfr},
{date}, {adj} can be changed by me but not by you, do you understand ?”

  * Output 2: “Yes, I understand. You have provided a sentence template with placeholders for age, glomerular filtration rate (GFR), date, and an adjective that can be changed by you. And you are looking for text augmentation to generate similar but slightly different versions of the original text with the help of natural language processing(NLP) techniques such as synonym replacement, random insertion, deletion, and substitution of words.”

  * •

Input 3: “Provide full sentence synonyms of the sentences I provided, you are
allowed to give more context information and not especially respect the exact
number of sentences.”

  * Output 3: “Here are some examples of synonyms or text augmentations of the sentence template you provided:

    1. 1.

”The age of the kidney donor is {age}. The patient’s glomerular filtration
rate was measured to be {gfr} at the {date} follow-up appointment. The
creatinine levels showed {adj} variation.”

    2. 2.

”The individual who provided the kidney for transplant was {age} years old.
The recipient’s GFR was determined to be {gfr} during the {date} check-up.
Creatinine levels fluctuated {adj}.”

    3. 3.

”At the time of transplant, the donor was {age} years of age. The patient’s
GFR was evaluated as {gfr} at the {date} follow-up examination. Creatinine
levels demonstrated {adj} changes.”

    4. 4.

”The age of the individual who donated the kidney was {age}. The GFR of the
transplant recipient was {gfr} at the {date} examination. There were {adj}
variations in creatinine levels.”

    5. 5.

”The kidney donor was {age} years old. The GFR of the transplant patient was
measured as {gfr} at the {date} follow-up visit. The creatinine levels showed
{adj} fluctuations.”

    6. 6.

”The donor’s age was {age} at the time of transplant. The patient’s glomerular
filtration rate was found to be {gfr} during the {date} appointment.
Creatinine levels were {adj}.”

Please note that the sentences are not exact synonyms, but variations that
convey similar information, and that it’s possible to generate a lot more
variations by changing the words, phrases or structures of the sentences.”

  * •

Input 4: “Give ten more synonym examples of the provided sentence and you can
generate more variations as you proposed.”

  * Output 4: “Sure, here are ten more examples of text augmentations of the sentence template you provided:

    1. 1.

“The kidney donor was {age} years of age. The patient’s GFR was found to be
{gfr} at the {date} check-up. Creatinine levels displayed {adj} fluctuations.”

    2. 2.

“At the time of transplant, the age of the kidney donor was {age}. The
patient’s glomerular filtration rate was {gfr} at the {date} follow-up.
Creatinine levels varied {adj}.”

    3. 3.

“The individual who provided the kidney for transplant was {age} years old.
The GFR of the recipient was determined to be {gfr} during the {date}
examination. There were {adj} changes in creatinine levels.”

    4. 4.

“The age of the kidney donor was {age}. The GFR of the transplant recipient
was {gfr} at the {date} follow-up visit. Creatinine levels fluctuated {adj}.”

    5. 5.

“The individual who donated the kidney was {age} years of age. The patient’s
glomerular filtration rate was measured as {gfr} at the {date} check-up. The
creatinine levels demonstrated {adj} variations.”

    6. 6.

“At the time of transplant, the donor’s age was {age}. The GFR of the patient
was evaluated as {gfr} during the {date} follow-up. The creatinine levels
showed {adj} fluctuations.”

    7. 7.

“The kidney donor was {age} years old at the time of transplant. The patient’s
glomerular filtration rate was {gfr} at the {date} examination. Creatinine
levels varied {adj}.”

    8. 8.

“The individual who provided the kidney for transplant was {age} years of age.
The recipient’s GFR was determined to be {gfr} at the {date} follow-up
appointment. The creatinine levels displayed {adj} changes.”

    9. 9.

“At the time of transplant, the age of the kidney donor was {age}. The GFR of
the transplant patient was {gfr} during the {date} check-up. Creatinine levels
fluctuated {adj}.”

    10. 10.

“The individual who donated the kidney was {age} years old. The glomerular
filtration rate of the transplant recipient was {gfr} at the {date}
examination. The creatinine levels showed {adj} variation.”

Please note that the above sentences are not exact synonyms, but variations
that convey similar information, and that there are many more variations that
can be generated by changing the words, phrases or structures of the
sentences.”

##  Appendix B Textual data generation

We compare the proposed approach for medical prompt generation with two other
approaches that produce text annotations. The first one is noted as “Manual”
and comprises all the templates indicated by the medical experts,
corresponding to only one sentence per variable of interest. Note that this is
the base of our proposed medical prompting without using the LLM augmentation
method. The second one uses an existing NLP model (T5 Raffel et al. (2020)) to
produce sentences from structural data. We train the T5 model on the WebNLG
2020 data Gardent et al. (2017) and infer it on our data to generate text,
denoted as “T5 WebNLG”. We observe that the “Manual” approach lacks diversity
in the textual data, as no text augmentations are performed for this
straightforward process. “T5 WebNLG” offers some more variability in words
used, but the structure of the sentences remains similar and straightforward.
Moreover, some incorrect generations occur, e.g., “The age of the donor” is
replaced by “The age of the patient”. Such errors introduce anomalies in the
data, a highly sensitive issue in such a medical context.

  * •

“Manual” – from one sentence template:
“The age of the donor is low. The glomerular filtration rate (GFR) of the
patient is high at one month follow-up exam. And the creatinine levels
variation were stable.”;

  * •

“T5 WebNLG” – pretraining a model to generate textual data from structural
data:

    * -

correct generation example: “The age of the donor is high. The glomerular
filtration rate is medium. The creatinine levels of a patient are unstable.”;

    * -

incorrect generation example: “The age of the patient was low. The glomerular
filtration rate of GFR is an extrem low rate. The creatinine levels of a
patient are unstable.”

  * •

MEDIMP: see Appendix A.

##  Appendix C Freezing the text encoder

For the text encoder, we used the BERT Devlin et al. (2019) architecture
initialized with the Bio+Clinical BERT Alsentzer et al. (2019) model
pretrained on the MIMIC clinical notes. BERT is based on the transformer
Vaswani et al. (2017) architecture and composed of $12$ transformer blocks.
Our main models were obtained by freezing the first $11$ layers of the
Bio+Clinical BERT model, fine-tuning only the last layer of the transformer
with our contrastive task.

Benefiting from a dataset of 400 million (image, text) pairs collected from
the internet, Radford et al. (2021) trained both their image and text encoder
from scratch. While we used the same initialization as Zhang et al. (2020),
they froze their text encoder’s first half ($6$ layers). In recent NLP work,
Lu et al. (2022) suggested only finetune normalization layers (LN) in the
transformer blocks, without finetuning of the self-attention and feedforward
layers of the residual blocks. tab:tab3 reports the ablation results using
this latest strategy, denoted as not LN, and gradually freezing the $6$, $9$,
and $11$ first layers of the text encoder. The ablation was evaluated using
two sets of weak annotations from our proposed method, first the GFR and date,
denoted as MEDIMP A and second the GFR, Exam, and Donor’s Age, denoted as
MEDIMP B. We observe that freezing the first 11 layers gives us the best
performances, and this is the strategy we used for our paper.

Table 3:  Ablation results on the way of freezing the text encoder. We report
F1 score (F1), and ROC AUC (AUC). Bold, Underlined indicates the top 1, 2
performing combinations, respectively.

|  | 2 years | 3 years | 4 years | Mean
---|---|---|---|---|---
Method |  Freezing $E_{\textbf{T}}$ | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1
MEDIMP A |  First 11 |  81.1 |  89.4 |  81.9 |  80.0 |  74.8 |  84.4 |  79.2 |  84.6
MEDIMP A |  First 9 |  74.8 |  75.7 |  81.9 |  81.0 |  76.5 |  68.6 |  77.7 |  75.1
MEDIMP A |  First 6 |  74.2 |  74.4 |  70.0 |  82.1 |  83.5 |  68.6 |  75.9 |  75.0
MEDIMP A |  not LN |  73.5 |  70.6 |  77.5 |  80.0 |  73.0 |  71.8 |  74.7 |  74.1
MEDIMP B |  First 11 |  76.9 |  73.2 |  86.3 |  85.7 |  74.8 |  90.2 |  79.3 |  83.0
MEDIMP B |  First 9 |  83.7 |  64.5 |  78.1 |  82.9 |  75.7 |  64.7 |  79.2 |  70.7
MEDIMP B |  First 6 |  75.5 |  70.6 |  84.4 |  80.0 |  84.4 |  82.9 |  81.4 |  77.8
MEDIMP B |  not LN |  66.7 |  81.0 |  79.4 |  80.9 |  60.9 |  85.7 |  69.0 |  82.5

##  Appendix D Cross-validation results

We evaluated our representations on the downstream task of kidney function
prediction 2 years post-transplantation proposed by Milecki et al. (2022). We
performed the task on two more prediction dates, namely 3 and 4 years post-
transplantation, to better highlight the significance of our proposed
approach. Nevertheless, following the instruction of Milecki et al. (2022), we
also performed 10-fold cross-validation on the training set. We report below
those cross-validation results (ROC AUC, F1 as mean $\pm$ standard deviation)
for our validation set, for the best combination (denoted with ⋆) of
CosEmbLoss Milecki et al. (2022), CosEmbLoss++, and our proposed MEDIMP, for
the three different tasks. While demonstrating similar cross-validation
results over the mean of the three prediction tasks, MEDIMP enables to obtain
lower variation on the validation sets.

Table 4:  Cross-validation results. We report F1 score (F1), and ROC AUC (AUC)
as mean $\pm$ standard deviation for the best combinations of CosEmbLoss,
CosEmbLoss++ and our proposed MEDIMP, denoted with ⋆.

Method | 2 years | 3 years | 4 years | Mean
---|---|---|---|---
Validation set | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1
$\text{CosEmbLoss}^{\star}$ |  $93.3\pm 12.0$ |  $86.4\pm 12.2$ |  $81.7\pm 15.9$ |  $74.4\pm 23.3$ |  $84.5\pm 16.5$ |  $64.6\pm 26.9$ |  $86.5\pm 14.8$ |  $75.1\pm 20.8$
$\text{CosEmbLoss++}^{\star}$ |  $91.7\pm 14.6$ |  $88.8\pm 10.4$ |  $84.1\pm 14.3$ |  $71.4\pm 20.5$ |  $83.3\pm 13.9$ |  $69.1\pm 27.3$ |  $86.4\pm 14.3$ |  $76.4\pm 19.4$
$\text{MEDIMP}^{\star}$ |  $89.3\pm 11.4$ |  $80.1\pm 11.8$ |  $87.5\pm 4.1$ |  $81.7\pm 6.5$ |  $81.4\pm 13.9$ |  $71.9\pm 29.6$ |  $86.1\pm 9.1$ |  $77.9\pm 16.0$