Materials science in the era of large language models: a perspective

1]Ge Lei
These authors contributed equally.
1, 2]Ronan Docherty †
1]Samuel J. Cooper Corresponding<EMAIL_ADDRESS>
[1]Dyson School of Design Engineering, Imperial College London, London SW7 2DB
[2]Department of Materials, Imperial College London, London SW7 2DB

Lei et al.
LLMs in materials science

Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex code generation to heuristic finding for combinatorial problems. In this paper we offer a perspective on their applicability to materials science research, arguing their ability to handle ambiguous requirements across a range of tasks and disciplines mean they could be a powerful tool to aid researchers. We qualitatively examine basic LLM theory, connecting it to relevant properties and techniques in the literature before providing two case studies that demonstrate their use in task automation and knowledge extraction at-scale.
At their current stage of development, we argue LLMs should be viewed less as oracles of novel insight, and more as tireless workers that can accelerate and unify exploration across domains.
It is our hope that this paper can familiarise material science researchers with the concepts needed to leverage these tools in their own research.

§ INTRODUCTION

Materials science as a discipline sits at the intersection of physics, chemistry, and often biology, and therefore requires a broad range of both skills and knowledge. A single project can cover multiple length scales, requiring various literature reviews, hypothesis generation and project planning before any experiments take place. Laboratory work can require elaborate synthesis and sample preparation routes, typically followed by a wide variety of characterization techniques. Acquired data must be processed and then analysed, either by fitting to models, comparing to simulations, or calculating uncertainties. Theoreticians must understand and leverage a variety of computational techniques from Density Functional Theory, to Computational Fluid Dynamics, and more recently to deep learning. This may require knowing multiple programming languages, as well as having the skills to deploy code across multiple environments, like high-performance clusters or cloud services.

The rapid advancement of Artificial Intelligence (AI) - neural-network based deep-learning in particular - over the recent decade has been driven by increasingly powerful hardware and increasingly massive datasets [1].
The culmination of this advancement is the Large Language Model (LLM), a transformer [2] based neural network with billions of learnable parameters trained on as large a corpus of text as possible[3].
Various LLMs exist, like OpenAI's GPT-4 [4], Google's Gemini [5], Meta's LLaMA 2 [6], and Anthropic's Claude 3 [7]. They are mostly the product of large companies with the financial and computational resources to train them, though some open source models exist [8, 9].
Despite their simple training objective of reproducing human-like text[10], the combination of broad training data and deep network has resulted in impressive emergent capabilities and applicability to different domains and problems[11].

LLMs naturally have a strong apparent understanding of the structure of natural language, being able to translate, transpose, generate, and answer questions based on texts. They are sometimes able to (or appear able to) perform reasoning and extract patterns from textual and numerical data [12, 13], extending their use beyond just language-based applications.
This combination makes them competent programmers[14], but also effective managers or co-ordinators in complex tasks[15].
Whilst they perform best in workflows with a strong, LLM-independent feedback signal[16] they are capable of automating processes in ambiguous scenarios through trial-and-error.
Compared to say a Convolutional Neural Network (CNN), the transformer architecture is more amenable to multi-modality, able to combine and process encodings of text and images [17]. This multi-modality massively expands the range of problems to which LLMs can be applied [4, 5].

Like other computer programs, but unlike human scientists, LLMs are inexhaustible - able to run all day, every day, which is useful not just in automated digital discovery workflows, but also for setups like automated laboratories or pilot lines [18, 19].
They are typically more flexible and adaptable than traditional computer programs, making them more effective when run continuously.
The ability to process instructions in natural language, retrieve domain knowledge, generate code and co-ordinate systems, paired with their tireless operation and immunity to boredom make LLMs appealing tools to a materials science researcher. If used judiciously they could speed up materials discovery and perform large scale analyses previously impractical for even the largest teams of researchers.

The development of the computer revolutionised information processing and research - we argue that domain-grounded LLMs will produce another step-change in materials science.
In this paper we explore the potential role of LLMs in material science, starting with a qualitative examination of the theory underpinning transformers and LLMs in Section <ref>.

Next in Section <ref> we discuss the capabilities of modern LLMs and LLM-based workflows across a variety of domains and how they might be applied to materials science.

Section <ref> two case-studies which use LLMs in materials science workflows. The first case study uses LLMs to automate tasks during 3D microstructure analysis and the second uses LLMs to extract labels for micrographs from papers using abstracts and figure captions to create a new dataset. Finally in Section <ref> we examine the issues and challenges around using LLMs in research, including hallucinations, cost, and depth of understanding.

§ LLM THEORY: FROM ATTENTION TO CHATGPT

A multi-scale diagram of an LLM. (a) shows an attention map for an example sentence, note how `Law' is strongly linked to its pronoun `its'. (b) shows a transformer encoder layer, made up of an attention layer and (fully-connected) feed-forward layer. Multiple of these encoder layers with associated decoder layers form an LLM in (c), which is pretrained in an self-supervised manner on a large text corpus. This LLM is fine-tuned to ensure its responses better match human preferences without diverging too much from the original model via RLHF, as shown in (d).  Figures (a), (b) adapted from [2] and (c), (d) adapted from [20].

§.§ Attention and transformers

Attention (or self-attention), originally used for sequence modelling in recurrent neural networks[21], is a mechanism designed to force a neural network to consider the rest of the elements in a sequence (the `context') in its representation of the current element. For a sentence that is represented as a sequence of tokens (efficient vector representations of words in terms of common sub-parts like prefixes) like  the dog chased its own tail, attention would place emphasis on the dog token in its representation of the its token - the consideration of context allow it to model noun-pronoun relations. An example attention map for a sentence is shown in Figure <ref>. A more thorough description is available in the Supplementary in Section <ref>.

Transfomers were introduced by Vaswani et. al. in 2017[2] as a neural network architecture that only used self-attention for sequence modelling. The removal of recurrent layers meant less sequential operations were needed, meaning training could be parallelized even for a single training example (like a long sentence). The use of attention in place of convolutions meant shorter distances for information propagation across a sequence, making it easier to learn long-range connections.

Despite being the most efficient way to include the whole context of a sequence of $n$ tokens in a single layer, computing the interaction of every token with every other token means attention is an $O(n^2)$ operation. This limits the total `context length' of the input sequence based on the amount of (GPU) memory. The quadratic scaling is the major downside of transformers and researchers are looking to mitigate this with techniques like windowed attention[22] or moving to linear state-space models like Mamba[23], though these approaches lose global context.

Another consequence of attention is that there is no implicit ordering of tokens in the network - this information must be added in the form of a `positional embedding` to the vector representation of each token in the sequence. The simplest way of doing this is word-order, i.e, which number the token is in the sequence, though other embeddings like sinusoidal or learned embeddings are also used[24].
An embedding is just a vector representation of a quantity in a new subspace - this can be as simple as one-hot encoding showing the presence of a feature or as complicated as a set of features learned by a deep CNN.

§.§ Pretraining and language modelling

Supervised training is updating the weights of a neural network to minimize the loss between the labels predicted by a model, $\hat{y}$ and the labels from the dataset $y$ for a given input $x$. As an example, the $x$ could be a photo of a dog and $y$ could be a label from a human saying dog. In  training, the human labels $y$ are replaced with some transformation of the input $y = f(x)$.

Ideally during self-supervised training the network learns strong representations of the data and can be fine-tuned or paired with another network on labelled data for specific tasks. This has two advantages - firstly that it reduces the amount of human labour needed to label the inputs, $x$, and secondly that it is believed to produce more robust representations[25] than supervised learning, due to the lack of `shortcuts' available.
An example of a `shortcut' is learning to predict a dog by detecting a lead, or detecting a polar bear based on ice in the background - learning these might mean ignoring more relevant and generalisable features[26].

Transformers are parallelizable so scale well with added data and compute, and can easily learn long-range connections [2].
Self-supervised learning requires little or no human input - massive text datasets can be collected through automated web-scraping[27] - and generates strong learned representations.
This combination makes transformers prime candidates for self-supervised learning on large text datasets to create multi-purpose language models.

One of the first works to apply self-supervised learning to large text datasets with transformers was Radford et. al. in 2018 [10], where a transformer was pre-trained on 7,000 unpublished books before being fine-tuned on tasks like question-answering and classification. It was pre-trained using next-token prediction and operated autoregressively, i.e it predicted next-token probabilities for all tokens in its vocabulary, selected the highest one, added it to the input and predicted the new next token. This was called generative pre-training, and the model was called a Generative Pretrained Transformer (GPT).

GPT's pre-training was left-to-right causal language modelling where the sequence had to be masked to prevent the transformer seeing future tokens (specifically the current token of interest) and predicting that. An alternative approach is masked language modelling, where only the current token of interest in the sequence is masked and the rest is part of the context - this is bidirectional and future context can be considered. This was the approach used for Google's BERT in 2018 [28]. The bidirectional language modelling meant BERT had higher performance on benchmarks but meant it could not be autoregressive/generative - a key factor in ChatGPT's later popularity.

§.§ Aligning outputs via RLHF

Always selecting the most probable token during autoregression leads to coherent and deterministic results, but can limit the ability of the model to be `creative'. Picking tokens in proportion to their probability is a simple strategy for more diverse text, and a common way to parameterise the distribution and therefore control text generation is `temperature'[29]. Temperature is a scalar term introduced before the softmax function that generates per-token probabilities, with a large temperature increasing the probability of previously rare tokens and making more common tokens less likely[30, 31]. A low temperature does the opposite, and a temperature of 0 is used to refer to most likely token selection.

`Prompting' is a consequence of the autoregressive learning objective of LLMs - a user's prompt is given to the LLM as a sequence and the LLM generates the most likely subsequent tokens. The model must be fine-tuned to act in a true question/answer or chatbot style[3]. The notion of prompting has found success in other domains, like Meta's promptable `Segment Anything Model'[32].

In 2022 OpenAI published a paper on InstructGPT[33], a pre-trained model which was then trained on a dataset of prompts to desired responses and finally fine-tuned via RLHF[34].
RLHF, shown in Figure <ref>, contains two LLMs - a frozen LLM and the LLM to fine-tune. A prompt is fed to both, and the fine-tuned LLM's response is fed to a reward model (a NN trained to emulate human preferences) to generate a reward score. A second term is added to the reward based on the KL divergence between the frozen and fine-tuned LLM to prevent model drift. This reward is fed into a reinforcement learning policy, like Proximal Policy Optimization (PPO)[35] to update the weights.

InstructGPT had significantly fewer parameters than GPT-3 but outperformed it, signalling the power of reinforcement learning in aligning a model's outputs with human preferences. Despite this impressive performance it is worth noting that at no point during the pre-training, training or fine-tuning are models explicitly trained to minimize factual errors or to reason - saying the sky is green goes against human preference and would therefore be penalised, but most labellers would be unaware if the model had confused ferro- and ferrimagnets if the text was otherwise coherent.

§ CAPABILITIES OF LLMS IN RESEARCH

Machine learning has seen widespread application in materials science, from characterization[36, 37] to property prediction[38, 39, 40] to materials discovery and design[41, 42, 43].
They have mostly been applied in well-structured tasks with strong supervision (i.e, fully labelled single-task datasets)[44].
Frequently this has involved researchers developing models trained only on their data for their problems, causing poor generalisation to new materials or processing conditions[45].

LLMs, by virtue of the the size of the number of parameters and the scale of their training data, have strong natural language skills and emergent properties that make them promising candidates for processing more unstructured and varied data[44].
In this section we explore some of these emergent capabilities, examine how researchers have used them in various disciplines and consider them in a materials science context.

§.§ LLM properties: intrinsic and emergent

Diagram of LLM capabilities explored in Section <ref> and potential materials-science related applications. These emergent capabilities can be combined with each other and integrated into traditional pipelines (genetic algorithms, databases, etc.) to form the different applications.

§.§.§ Optimizing responses with prompt engineering

Prompt engineering refers to optimizing the prompt the user gives to an LLM in order to produce a `better' answer or response. This can involve making the response more or less precise, conform to specific rules or schema changing the level of the explanation, i.e, explain like I'm five years old....
Choosing the correct prompt is more of an art than a science, but some work has been done on creating and testing various principles[46, 47] or using an LLM to optimise prompts for another LLM[48]. It has been found that it is generally better to be precise, structured (using paragraphs, whitespace and question/answer blocks) and explain to the LLM what `role' it should act in.
Example prompts for our case studies, which utilise these principles, are in Supplementary Sections <ref> and <ref>.

§.§.§ In-Context Learning (ICL) and property prediction

One important emergent property that is useful to optimize a prompt is in-context learning, or few-shot learning[3], where a few example input/output pairs are provided to the model in the prompt. This is useful for familiarising the model with unknown concepts.
Why it works is still a topic of debate, and it is a counter-intuitive phenomenon. Min et. al.[49] found that during in-context learning, randomly swapping output labels in examples the (introducing wrong information into the context) only slightly decreased the accuracy of LLMs when predicting related new samples. This suggested showing the model the structure and input/output distributions were more important than demonstrating the underlying logical mapping.

However, subsequent research[12] studied this effect with larger LLMs and found that as the number of model parameters increased, the adverse effects of wrong information in examples became more pronounced.  As the percentage of swapped labels increased, the accuracy on unswapped examples dropped below $50\%$ (the random baseline), implying the larger LLMs learnt the reverse logical mapping from the (swapped) in-context examples, rather than just the problem structure.

Google's Gemini 1.5 Pro[50] recently demonstrated its advanced in-context learning capabilities by accurately translating English into Kalamang, a language spoken by fewer than 200 people. Initially, the model struggled with translations due to a lack of Kalamang training data. However, after processing 250k tokens of linguistic documentation on Kalamang without undergoing a traditional training regimen, it achieved near-human levels of translation accuracy.

As well as various natural language and mathematical problems[13], ICL has been used for both quantitative and qualitative material property prediction [51, 52, 53, 54, 44]. However, Microsoft AI4Research noted that despite good qualitative predictions, the quantitative predictions of LLMs were lacking[54].

§.§.§ Error correction via Chain-of-Thought (CoT) reasoning & self-reflection

Chain-of-Thought reasoning involves asking the LLM to break a problem into a series of steps in order to improve accuracy. This can be included in the prompt in a phrase like think through the problem step-by-step or as part of an ICL prompt where the example given is broken into a series of steps. CoT has been found to improve performance on tasks that require complex reasoning[55, 56].

Various explanations for this improved performance have been suggested, including that requesting the model think step-by-step increases the length of the sequence. Recall that during autoregression the whole sequence including the current output is fed into the model to generate the next token - adding more tokens gives the model more context and thus more `space' to compute with, as more text means more interactions in the attention layer.
Another possibility is that longer sequences reduces the space of likely sequences to those that contain the correct answer; if the model has repeated John has 4 apples multiple times as part of its explanation the probability of outputting future tokens that use (directly or indirectly) a different number of apples is reduced compared to directly outputting the answer.

Self-reflection is a consequence of ICL and CoT and involves giving an LLM an evaluation of its previous prompt in a new prompt, this can be pointing out errors or a broader evaluation. This can be from a human[55], a program (i.e, a stack trace)[57], another LLM[58] or even itself[59]. Self-reflection improves performance, potentially for the same reasons as ICL and CoT, but also because correcting a wrong output may be a simpler task than generating the correct output de novo.

§.§.§ Pre-existing and fine-tuned materials domain knowledge

LLMs are trained on large corpuses of text that contain facts about the world, including large datasets of scientific papers[60]. Common facts will be repeated many times across these texts, making it statistically likely that an LLM will reproduce them when prompted to. Microsoft AI4Science found that In biology and materials design, GPT-4 possesses extensive domain knowledge, which they evaluated by asking domain experts to rate outputs about various drug molecules, general materials design principles, mathematical concepts like PDEs and more[54].

The ability to act as an oracle for common shallow information across many domains[61] is useful in a multi-disciplinary field like materials science, but the regression to the mean encouraged by the pre-training task and autoregression can limit the usefulness of LLMs for deep information recall.

One way of overcoming this is fine-tuning on domain-specific knowledge. This domain specific knowledge can be collected traditional web-scraping or using ML models[62] and then used to fine-tune a language model like BERT[28]. Models like SciBERT[63] outperformed BERT and other state-of-the-art models for tasks like text classification or Named Entity Recognition (NER). MatSciBERT[64] took this process a step further and fine-tuned SciBERT on materials science specific data to outperform SciBERT on materials science text tasks.

Full fine-tuning of any large ML model is expensive and risks `catastrophic forgetting'[65], where a model loses information from its general (pre)training during the domain-specific fine-tuning. One way to alleviate both the cost and catastrophic forgetting problem is Parameter-Efficient-Fine-Tuning (PEFT), where only a small subset of the model's parameters are updated. Examples PEFT schemes include LORA[66], adapters[67] and prompt/prefix-tuning methods[68, 69].

§.§.§ Comprehensive programming skills

Large quantities of text exist online (and therefore in LLM training sets) about programming: discussions, help forums and source code, and the move towards approachable, high-level programming languages like Python means source code is increasingly similar to natural language.
These two facts mean LLMs are proficient at generating, modifying, correcting and summarizing code in a variety of languages for a variety of tasks[14, 70].

Programming is ubiquitous in modern science, from data processing, analysis, visualization, simulations, instrument interfaces, etc. and the ability to write reasonable code across all these different tasks is obviously useful for researchers. LLMs have been shown to be proficient in these tasks in a materials science context[54]. The ability to code in different contexts is also fundamental for many of the workflows explored in Section <ref>.

§.§.§ Multi-modality - enriching materials characterization

The tokenization, positional embedding and attention mechanics of transformers are heavily flexible and therefore capable of jointly modelling different modalities and tasks[71].
A prominent example of this is OpenAI's CLIP (Contrastive Language–Image Pre-training)[17], where a model is trained to maximize the similarity of text and image representations for text-image pairs collected from the internet. The success of CLIP and other multimodal representations[72] has led to the rise of Vision Language Models (VLMs) like GPT-4[4], LLaVa[73] and Gemini[5] which can use information from text and images to aid in the generation and processing of both.

Joint text-image reasoning has the potential to be a useful analysis tool when combined with existing datasets of materials images and descriptions - consider searching the literature for  microstructures that display similar features, defects or artefacts to yours, with potential answers from related papers signposted.

Images are not the only mode of data that transformers can learn to use with text. There are examples using videos via 3D CNN embeddings[74], speech/audio using spectrograms[75] and even graphs via Graph Neural Network (GNN) embeddings[76]. Notably, OpenAI's Sora[77] has extended this versatility further by generating high-fidelity videos, demonstrating the application of transformers beyond static images to dynamic, temporal data.

Finding suitable embeddings for the wide range of characterization techniques that exists in materials science (CLIP for micrographs, GNNs for crystallographic information from XRD, 1D CNNs/LTSMs for spectral data) and fine-tuning a transformer or LLM with them could be a promising direction for injecting domain-specific knowledge or priors.

§.§ Resulting workflows

These properties are flexible and composable, meaning they can be combined in a wide range of potential workflows in various domains, including materials science[44]. Below are a few examples of such workflows, and though they are split into separate sections there are strong links and similarities between them. The key commonality is letting LLMs act as high-level managers whilst other, more robust systems perform low-level tasks.

§.§.§ RAG: generation from custom datasets

Retrieval Augmented Generation (RAG) involves performing a lookup into a traditional database and using the retrieved information as part of a prompt to an LLM in order to achieve better or more accurate generation[78].
The lookup is usually based on some function of a user request - one common way to match the semantics of a user's search to a database is `vector search', where embeddings of every item in a database are pre-computed using a language model like BERT[28] and the ones with the highest similarity (usually cosine similarity) with the embedding of the user's request are returned[79].

RAG has several benefits to LLM workflows[80]: firstly, hallucinations are reduced as models only need to process existing information in a prompt rather than generate (or fabricate) it. It is more interpretable as the retrieved documents can be linked back to to confirm the results. Finally, these databases can be updated simply by computing the embeddings for the new items - without RAG the LLM would need to be retrained or fine-tuned to add the new information.

The utility afforded by RAG is clear - many companies are trying to use or sell it [79], and it is a feature in GPT-4 [4]. It is not hard to see how LLMs paired with a vector database of materials papers using, say, MatSciBERT's[64] text embeddings could prove useful in research. Indeed, some have already used RAG alongside knowledge graphs for materials design[81].

§.§.§ Tool-using and making for analysis pipelines

Diagram of the FunSearch[16] evolutionary workflow, where an LLM is prompted with a problem specification and best example heuristics from the previous iteration and tasked with combining them to generate better candidate heuristics to solve a problem. These new heuristics are evaluated, stored in database and the process repeated. This process was able to discover a new upper bound for the largest cap set in 8 dimensions. Taken from [16].

LLMs can be trained to use use tools like search engines[82], translations, mathematics plugins, etc., which is useful in situations where they typically underperform like arithmetic [83]. This can be achieved through ICL and `prompt managers' by providing details of the tools and situations in which to use them to the LLM and running the generated code or API (Application-Programmer Interface) calls [84, 85].

Another more involved approach used by Toolformer[83] was to use ICL to make LLMs annotate an existing language dataset with API calls for a variety of tools where it deemed them useful. They then fine-tuned the model on that data, including a loss term to indicate when the API call improved the accuracy of the generation. This approach has the benefit of not relying on prompts, which can crowd the limited context window and sometimes be ignored by the LLM.

LLMs can generate code and as such are able to produce their own tools. The LLMs As Tool Makers (LATM)[86] framework used LLMs to generate tools which then be used by other LLMs. They noted that tools were harder to make than use, so had a more powerful LLM (GPT-4) generate the tools, tests and documentation and a weaker LLM (GPT-3.5) use the tools.

A tool-making and using LLM with a human-in-the-loop could be useful for materials science problems where the workflows and requirements are varied (in terms of data types, desired analyses or post-processing) like in image processing. This could be further combined with RAG on relevant papers for domain knowledge engagement and a database of generated tools to obviate the prompt context window limit. Progress has been made on that front, including ChatGPT integration for the ImageJ macro language inside ImageJ itself [87].

§.§.§ Task integration: the future of automated labwork?

Various papers have shown that LLMs can act effectively as managers of various sub-components, like other software tools or even other LLMs (called `agents'). This tends to involve feeding the outputs of these tools or LLMs as a prompt into the manager LLM.

One fun example of co-ordination is `Generative Agents: Interactive Simulacra of Human Behavior'[88], where LLMs acted as villagers in a sandbox with a set of possible actions and locations. They performed inter-agent communication and had a recursively summarised memory of events fed into their prompt to maintain consistency.

Maintaining a memory external to the LLM (i.e, in a text file) has been explored by studies like MemGPT[89] which aimed to emulate modern Operating System memory management to allow LLMs to perform tasks like large document summarization and multi-session chats. To achieve this they had a traditional scheduler with events for document uploads and timers, and allowed the LLM a set of functions to call in response including send message, read, write, and send interrupts.

`Coscientist'[18] used LLMs as a coordinator to design, plan and execute chemical research. It can call web search APIs, generate and execute Python code, search documentation, interact with and write code for physical hardware. Despite the need for manual intervention to execute the experiment, it is a promising example of how LLMs can orchestrate various research and lab tasks.

Much effort is being made to integrate LLMs with robotics[90] as task planners[91], reasoning agents[92] or as part of a broader vision-language-action multimodal model[93].
Advancements in grounded robotics and embodied AI will further development of automated labwork, improving all-in-one workflows like Coscientist. However, it is worth noting the margin for error (and hallucinations) is much smaller in labs, where a wide variety of hazardous chemicals and processes are handled frequently.

§.§.§ Optimization loops and `flow engineering'

Another useful LLM-based workflow is meta-optimization, where instead of generating an optimal solution to a problem, an LLM generates the code to produce the optimal solution. `Eureka'[94] used an LLM to generate reward functions for reinforcement learning applied to robotics simulations. They used a genetic algorithm, where the best generated reward functions and their summary statistics were included in a prompt to allow the LLM to `reflect' and then synthesise a new, better set of reward functions. The framework outperformed expert-written reward functions on a large majority of tasks.

Deepmind's FunSearch[16] followed a similar approach, using LLMs to generate heuristics for approximating solutions to mathematical problems like the cap set or online bin packing problem. They also used a genetic algorithm framework, asking the LLM to combine aspects of best-performing heuristic programs to generate new ones. Like Eureka, this relied on a combination of ICL, CoT and a feedback signal from an external program - in Eureka's case this was RL simulations using the reward functions which tracked quantities like time upright and for FunSearch this was small validation programs which evaluated how well the heuristic performed (i.e, if the cap set was valid and how large it was).

The FunSearch process found a new upper bound for the largest cap set in 8 dimensions, exceeding previous upper bounds found by human mathematicians. Despite this success, this was not a triumph of artificial mathematical understanding - a review of FunSearch noted it was remarkable for the shallowness of the mathematical understanding that it seems to exhibit[95] - instead it was proof of the power of LLMs inside an evolutionary framework.

The LLM in FunSearch did not need to always be correct - the strong feedback signal from the deterministic evaluators ensured mathematical correctness. This is therefore a good model for reconciling the LLM's occasional hallucinations with the need for scientific accuracy.
Based on the results, it seems the key contribution of the LLM was to reduce the search space of the genetic algorithm from all possible functions to all plausible functions, hugely increasing convergence time and final performance.

A recent meta-optimization coding paper is AlphaCodium [96], which used a multi-step framework combining reflection on a given specification, human-written tests and LLM-generated tests. The emphasis on tests was because they noticed it was easier to generate useful unit tests (which could then improve future generation) than the correct code. They called this process `flow-engineering' and improved pass accuracy on challenging code problems from 19% with just GPT-4 to 44% with GPT-4 as part of the AlphaCodium flow.
A useful feature of all these meta-optimization loops is that they tend to be LLM-agnostic i.e, GPT-4 would work equally as well as LLAMA or FALCON.

An important aspect of FunSearch (and other LLM meta-optimizations) was that the programs it generated were interpretable by humans. By examination of the program that generated the new upper bound, the researchers found a new symmetry in the cap set problem. This human-in-the-loop approach to optimization and discovery is appealing in the natural sciences - one could imagine tasking an LLM evolutionary framework to find new functionals in DFT or approximating solutions to physically-relevant combinatorial problems like the max-cut problem[97].

§ LLM WORKFLOWS IN MATERIALS SCIENCE: TWO CASE STUDIES

§.§ Case study 1: automated 3D microstructure analysis

Diagram of MicroGPT's workflow, beginning with dataset collection and filtering. This is followed by tool making and using to extract metrics from the data - this can be an existing tool from its toolkit like tortuosity calculations or created for the specific query.

In Section <ref> we examined the potential of LLMs to make, use, and orchestrate various tools into automated workflows. Typical materials data analysis pipelines require a combination of domain knowledge, statistical understanding, and various programming skills. The programming required is often non-trivial, involving data handling, conversion, simulations, plotting, etc.

LLMs have the potential to reduce the knowledge and skills barriers for these workflows, by offering a natural-language interface to a wide pool of programming knowledge, tool co-ordination, and automation. As an example, we developed MicroGPT - a specialized chat-bot to streamline 3D microstructure analysis. MicroGPT has a variety of functionalities:

* Data Acquisition: MicroGPT can conduct searches for open-source datasets on Zenodo (an interdisciplinary open-access repository) and employ functions to download these datasets using the links available on the respective web pages.

* Filtering: it can retrieve the dataset's metadata, parse it and subsequently refine the data according to the user's (natural language) specifications. Finally, it organize the filtered data into a newly created file directory.

* Integrated simulations: it can apply simulation tools to the 3D microstructures, documenting the simulation outcomes in formats such as CSV. These results can then be automatically uploaded to a cloud provider given an API key.

* Data analysis: it can compare various datasets, collect simulation results and based on user requirements, formulate hypotheses, and provide recommendations.

* Data visualization:  the results of the data analysis can be plotted, either as histograms for distributions of single properties across the dataset or scatter plots to examine the correlations between properties.

* Tool making and reuse: custom tools can be developed based on the user's specifications, stored and reused in later analyses. Over time this will lead to a library of useful and relevant functions that extend MicroGPT's capabilities.

This was achieved using GPT-4's API. Custom functions were defined in terms of their description, arguments and return values (in  format) and input to the GPT using OpenAI's `function calling' so the LLM would call them when appropriate. These were implemented in Python and run client-side. We added more system prompts with explicit instructions to improve stability (see Section <ref> and Supplementary Section <ref>).

To demonstrate these functionalities we used MicroGPT to collect and filter data from MicroLib[98], a collection of plausible, synthetic 3D microstructures generated from DoITPoMS[99] via SliceGAN[100]. It then filtered the structures to only ones related to materials with specific characteristics. Relevant 3D metrics like tortuosity, effective diffusivity, volume fraction, and surface area were calculated using TauFactor 2[101] via a function call.

MicroGPT collated the results, identified a potential outlier, and suggested some materials for further investigation. It successfully correlated metrics such as tortuosity and surface area with desired properties like high flow rates and extensive surface areas for efficient performance.

MicroGPT is a promising example for LLM-assisted analysis workflows, leveraging many of the properties in Section <ref> like natural language understanding, programming skills and chain-of-thought reasoning.
The grounding of MicroGPT using tool like search APIs, RAG, etc. is a future research direction which could both reduce factual errors and enhance domain knowledge engagement for reasoning and hypothesis generation. A detailed example dialog and system prompts are available in the Supplementary Section <ref>.

§.§ Case study 2: labelled microstructure dataset collection

T-SNE plot of the MatSciBERT[64] embeddings of the `material' label assigned by the LLM to each micrograph in the dataset based on the paper abstract and figure caption. Border colour denotes the instrument the micrograph was taken with. Similar materials are grouped together: nanoparticles in the bottom right, energy materials in the middle on the left and quantum dots in the bottom-left corner. Best viewed zoomed in.

There are few large (>1000 entries) micrograph datasets that cover a range of instruments and materials, and even fewer with material-specific labels.
The Cambridge DoITPoMS[99] library contains around 900 labelled micrographs of various materials captured mostly with optical or reflected light microscopy.
Another dataset from Rosella et. al.[102] contains 22,000 SEM images of materials with taxonomic labels.
Biological datasets are larger and better collated[103], contributing to the success of generalist deep-learning approaches like Cellpose [104].

Materials science papers contain many high-quality examples of micrographs taken using a variety of techniques, usually with descriptive captions and abstracts.
Traditional string-matching approaches like regex may be capable of detecting whether a given figure contains a micrograph and extracting the instrument used to take it from the caption, but detecting which material is present is generally not possible.
The problem is further complicated if the figure contains multiple sub-figures like plots or diagrams alongside the micrograph, which occurs frequently.

LLMs and VLMs offer solutions to both these problems, displaying strong natural language skills and ability to consider wider contexts like paper abstracts and therefore enabling large-scale automated micrograph collection and labelling from the literature. Some work using GPT-4V for extracting information from a paper's figures exists, for example analysing graphs (PXRD plots, TGA curves, etc.) in reticular chemistry papers[105] by treating each page in the  as an image.

We began by scraping paper metadata (title, authors, abstracts, links, etc.) from arXiv and chemrXiv that matched the query `microscopy' via their APIs.
For each paper we then downloaded the , ran the `pdffigures2.0' figure and caption extractor[106] and saved the image-caption pairs alongside the metadata.
We further extracted the subfigures for each figure by detecting connected components surrounded by whitespace and removing small (less than 200$^{2}$ px) results.

A two step screening process was used, first we fed captions and abstract to a text-only LLM (GPT3.5 or 4) to determine if a micrograph was present, what instrument was used and what material the micrograph was of.
Next we prompted a VLM (GPT4-V) with the specific subfigure, its parent figure, caption and abstract to work out if that specific subfigure was a micrograph and again what instrument was used and what material was imaged.

Evaluation of micrograph detection performance of GPT-4 supplied with figure caption and paper abstract, including a confusion matrix in (a) and statistics in (b). GPT4's performance is strong across the board, with good sensitivity, specificity and accuracy for instrument and material labels.

After running this process on 382 papers (a subset of the 14,000 scraped) we collected 842 micrographs, each with an instrument and material label - a link to the dataset is available in Section <ref>. Figure <ref> shows a visualization of the dataset, where micrographs are grouped based on how similar the MatSciBERT[64] embeddings of their labels are.
The LLM-generated labels were compared to human labels recorded with a custom GUI (developed for this case study) for each figure and subfigure to work out the accuracy of the process.

During the case study we evaluated the performance of various setups, including using GPT3.5 or 4 and whether we prompted the LLM with abstract or not.
GPT4 far outperformed 3.5, and using the abstract led to a minor improvement over not. See Figure <ref> in Supplementary Section <ref> for details. The performance of GPT-4 with abstract is shown in Figure <ref>, with a sensitivity and specificity above 90% for micrograph detection, and material and instrument accuracy above 80%.

We found that LLMs were competent labellers, sometimes matching human labels almost exactly. The success is mostly attributable to the fact that the task could be done with no materials-science specific knowledge due to how well-structured scientific captions are. The text-only LLMs make mistakes when the caption mentions `image' without showing a micrograph, i.e, in a plot of statistics taken from an SEM image. The VLM did not have this problem, and there were no false positives after the second step (though this may be because the first step was already a strong filter), this is discussed further in Supplementary Section <ref>.

More details on the setup, including the system prompt, can be found in Section <ref> of the Supplementary. The code needed to reproduce the results or run on more specific queries is available in Section <ref>. In the future we intend to apply this automated approach to a much wider dataset, with the hopes of creating a varied micrograph dataset for computer vision applications.

§ ISSUES AND CHALLENGES

There are naturally a few problems with integrating LLMs into materials science workflows, the most prominent and concerning being that of hallucination or confabulation. Huang et. al.[107] provide a taxonomy of hallucination types, separating hallucinations into two main types: factual and `faithfulness'. Factual hallucinations involve being wrong or fabricating facts, and `faithfulness' hallucinations involve ignoring user provided instructions or information or making logical errors.

Various causes of hallucinations have been suggested[107], including (but not limited to) pre-training on incorrect or duplicated data, randomness from output sampling and a `capability misalignment' between the demands made by RLHF fine-tuning and the model's capabilities - LLMs may been trained to hallucinate in some cases.

These fixes for hallucinations exist mostly at the dataset or training level, which is difficult for all but the largest research groups to manage. As noted in Section <ref>, RAG is a good way to mitigate factual hallucinations[107], as manipulating existing data is easier than recall, and it can supply a model with information from outside its training set. Chain-of-thought reasoning can also sometimes mitigate logical hallucinations[107], though asking a model to correct itself requires knowing the output was wrong in the first place, reducing the value-add of LLMs.

As well as contributing towards hallucinations, data duplication (alongside autoregression and the pre-training objective) can also contribute to an LLM's tendency to output towards a generic or modal answer. This is not just a problem if asking about uncommon materials or analysis techniques but also if using LLMs to explore a hypothesis space, design principles or automate experiments. The risk of using LLMs in research is that we reinforce existing biases and overlook unconventional approaches not well-represented in the training data.

There are practical issues to implementing LLMs in materials research. The models are expensive to run if using a cloud provider like OpenAI's API, or if run locally require powerful GPUs with at least 8GB of VRAM (which are also expensive).
Quantizing these models (storing their weights with less floating-point precision) can ameliorate this, at the cost of slightly diminished accuracy.
For research groups or companies dealing with sensitive or proprietary data there are privacy issues around uploading data to cloud-based LLMs - running local models is a good workaround but requires more know-how.

§ CONCLUSION

To conclude, we have explored the basic theory behind LLMs, linking their industrial-scale self-supervised pre-training and reinforcement learning-based fine-tuning to their impressive natural language skills. We then examined existing workflows using LLMs, indicating areas where they have been or could be applied to materials science research. Finally we demonstrated two example workflows using LLMs, one for 3D microstructure data analysis co-ordination and another for the automated collection of LLM-labelled micrographs from the literature.

We believe the versatility and emergent properties of LLMs will make them strong tools in an increasingly automated, connected and data-driven research environment. This is doubly true for materials science which must cover a broad range of length-scales, materials and topics.

At their current stage of development, LLMs are promising tools for accelerating research and exploration, acting as tireless interdisciplinary workers. They must, however, be used with full understanding of their drawbacks - not as oracles or generators of new, deep insights but in workflows that are robust to and that minimise hallucinations. There is an old saying: fire is a good servant, but a bad master.

§ CODE AVAILABILITY

The code needed to run the micrograph scraping, extraction and LLM labelling (including the resulting dataset) is available at <https://github.com/tldr-group/micrograph_extractor> with an MIT license agreement.

The code to run Microgpt is available at <https://github.com/tldr-group/Microgpt>

§ ACKNOWLEDGEMENTS

This work was supported by funding from Lee Family Scholarships (received by GL), and funding from the the EPRSC and SFI Centre for Doctoral Training in Advanced Characterisation of Materials (EP/S023259/1 received by RD).

The authors would like to thank other members of the TLDR group for discussions and feedback, specifically Isaac Squires who suggested using LLMs to collate a labelled micrograph dataset.

Thank you to arXiv for use of its open access interoperability.

§ COMPETING INTERESTS

The authors declare no competing interests.

§ REFERENCES

[1]
C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” arXiv preprint arXiv:1707.02968, 2017.

[2]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.

[3]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv preprint arXiv:2005.14165, 2020.

[4]
OpenAI, “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.

[5]
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, and A. Hauth, “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.

[6]
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, and S. Bhosale, “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.

[7]
Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” 2024.

[8]
G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only,” arXiv preprint arXiv: 2306.01116, 2023.

[9]
D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. A. Smith, and H. Hajishirzi, “OLMo: Accelerating the Science of Language Models,” arXiv preprint arXiv: 2402.00838, 2024.

[10]
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018.

[11]
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent Abilities of Large Language Models,” arXiv preprint arXiv:2206.07682, 2022.

[12]
J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou, and T. Ma, “Larger language models do in-context learning differently,” arXiv preprint arXiv:2303.03846, 2023.

[13]
A. S. et. al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” 2023.

[14]
M.-F. Wong, S. Guo, C.-N. Hang, S.-W. Ho, and C.-W. Tan, “Natural language generation and understanding of big code for ai-assisted programming: A review,” Entropy, vol. 25, no. 6, p. 888, 2023.

[15]
H. Yang, S. Yue, and Y. He, “Auto-gpt for online decision making: Benchmarks and additional opinions,” arXiv preprint arXiv:2306.02224, 2023.

[16]
B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,” Nature, vol. 625, pp. 468–475, Jan. 2024.

[17]
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, and J. Clark, “Learning transferable visual models from natural language supervision,” in International conference on machine learning, pp. 8748–8763, PMLR, 2021.

[18]
D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes, “Autonomous chemical research with large language models,” Nature, vol. 624, no. 7992, pp. 570–578, 2023.

[19]
N. J. Szymanski, B. Rendy, Y. Fei, R. E. Kumar, T. He, D. Milsted, M. J. McDermott, M. Gallant, E. D. Cubuk, A. Merchant, H. Kim, A. Jain, C. J. Bartel, K. Persson, Y. Zeng, and G. Ceder, “An autonomous laboratory for the accelerated synthesis of novel materials,” Nature, vol. 624, no. 7990, pp. 86–91, 2023.

[20]
N. Lambert, L. Castricato, L. von Werra, and A. Havrilla, “Illustrating reinforcement learning from human feedback (rlhf),” 2022.

[21]
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2016.

[22]
I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020.

[23]
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023.

[24]
P. Dufter, M. Schmitt, and H. Schütze, “Position information in transformers: An overview,” arXiv preprint arXiv:2102.11090, 2021.

[25]
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” arXiv preprint arXiv:2104.14294, 2021.

[26]
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, p. 665–673, Nov. 2020.

[27]
“Common crawl dataset.”

[28]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2019.

[29]
D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A Learning Algorithm for Boltzmann Machines*,” Cognitive Science, vol. 9, no. 1, pp. 147–169, 1985.

[30]
J. Ficler and Y. Goldberg, “Controlling linguistic style aspects in neural language generation,” arXiv preprint arXiv:21707.02633, 2017.

[31]
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” 2020.

[32]
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment Anything,” arXiv preprint arXiv:2304.02643, 2023.

[33]
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” arXiv preprint arXiv: 2203.02155, 2022.

[34]
P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” arXiv preprint arXiv: 1706.03741, 2023.

[35]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.

[36]
D. P. Finegan, I. Squires, A. Dahari, S. Kench, K. L. Jungjohann, and S. J. Cooper, “Machine-learning-driven advanced characterization of battery electrodes,” ACS Energy Letters, vol. 7, no. 12, pp. 4368–4378, 2022.

[37]
M. H. Rafiei, W. H. Khushefati, R. Demirboga, and H. Adeli, “Neural network, machine learning, and evolutionary approaches for concrete material characterization,” ACI Materials Journal, vol. 113, no. 6, 2016.

[38]
E. Champa-Bujaico, P. García-Díaz, and A. M. Díez-Pascual, “Machine learning for property prediction and optimization of polymeric nanocomposites: a state-of-the-art,” International Journal of Molecular Sciences, vol. 23, no. 18, p. 10712, 2022.

[39]
D. Chen, K. Gao, D. D. Nguyen, X. Chen, Y. Jiang, G.-W. Wei, and F. Pan, “Algebraic graph-assisted bidirectional transformers for molecular property prediction,” Nature communications, vol. 12, no. 1, p. 3521, 2021.

[40]
Q. Zhou, S. Lu, Y. Wu, and J. Wang, “Property-oriented material design based on a data-driven machine learning technique,” The journal of physical chemistry letters, vol. 11, no. 10, pp. 3920–3927, 2020.

[41]
R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik, “Automatic chemical design using a data-driven continuous representation of molecules,” ACS central science, vol. 4, no. 2, pp. 268–276, 2018.

[42]
A. Merchant, S. Batzner, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk, “Scaling deep learning for materials discovery,” Nature, pp. 1–6, 2023.

[43]
C. Zeni, R. Pinsler, D. Zügner, A. Fowler, M. Horton, X. Fu, S. Shysheya, J. Crabbé, L. Sun, J. Smith, B. Nguyen, H. Schulz, S. Lewis, C.-W. Huang, Z. Lu, Y. Zhou, H. Yang, H. Hao, J. Li, R. Tomioka, and T. Xie, “Mattergen: a generative model for inorganic materials design,” preprint arXiv:2312.03687, 2024.

[44]
K. M. Jablonka, Q. Ai, A. Al-Feghali, S. Badhwar, J. D. Bocarsly, A. M. Bran, S. Bringuier, L. C. Brinson, K. Choudhary, and D. Circi, “14 examples of how llms can transform materials science and chemistry: a reflection on a large language model hackathon,” Digital Discovery, vol. 2, no. 5, pp. 1233–1250, 2023.

[45]
A. Goetz, A. R. Durmaz, M. Müller, A. Thomas, D. Britz, P. Kerfriden, and C. Eberl, “Addressing materials’ microstructure diversity using transfer learning,” npj Computational Materials, vol. 8, p. 27, Feb. 2022.

[46]
J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023.

[47]
S. M. Bsharat, A. Myrzakhan, and Z. Shen, “Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4,” arXiv preprint arXiv:2312.16171, 2024.

[48]
C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, “Large language models as optimizers,” arXiv preprint arXiv:2309.03409, 2023.

[49]
S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,” arXiv preprint arXiv:2202.12837, 2022.

[50]
G. Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024.

[51]
S. Balaji, R. Magar, and Y. Jadhav, “Gpt-molberta: Gpt molecular features language model for molecular property prediction,” arXiv preprint arXiv:2310.03030, 2023.

[52]
A. N. Rubungo, C. Arnold, B. P. Rand, and A. B. Dieng, “Llm-prop: Predicting physical and electronic properties of crystalline solids from their text descriptions,” arXiv preprint arXiv:2310.14029, 2023.

[53]
S. J. Yang, S. Li, S. Venugopalan, V. Tshitoyan, M. Aykol, A. Merchant, E. D. Cubuk, and G. Cheon, “Accurate prediction of experimental band gaps from large language model-based data extraction,” arXiv preprint arXiv:2311.13778, 2023.

[54]
M. R. AI4Science and M. A. Quantum, “The impact of large language models on scientific discovery: a preliminary study using gpt-4,” arXiv preprint arXiv:2311.07361, 2023.

[55]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022.

[56]
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022.

[57]
N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” arXiv preprint arXiv:2303.11366, 2023.

[58]
V. Nair, E. Schumacher, G. Tso, and A. Kannan, “Dera: enhancing large language model completions with dialog-enabled resolving agents,” arXiv preprint arXiv:2303.17071, 2023.

[59]
J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han, “Large language models can self-improve,” arXiv preprint arXiv:2210.11610, 2022.

[60]
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027, 2020.

[61]
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?,” arXiv preprint arXiv:1909.01066, 2019.

[62]
F. Kuniyoshi, J. Ozawa, and M. Miwa, “Analyzing research trends in inorganic materials literature using nlp,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 319–334, Springer, 2021.

[63]
I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,” arXiv preprint arXiv:1903.10676, 2019.

[64]
T. Gupta, M. Zaki, N. A. Krishnan, and Mausam, “Matscibert: A materials domain language model for text mining and information extraction,” npj Computational Materials, vol. 8, no. 1, p. 102, 2022.

[65]
Y. Lin, L. Tan, H. Lin, Z. Zheng, R. Pi, J. Zhang, S. Diao, H. Wang, H. Zhao, and Y. Yao, “Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models,” arXiv preprint arXiv:2309.06256, 2023.

[66]
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.

[67]
Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria, “Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,” arXiv preprint arXiv:2304.01933, 2023.

[68]
B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691, 2021.

[69]
X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190, 2021.

[70]
L. Moussiades and G. Zografos, “Openai's gpt4 as coding assistant,” arXiv preprint arXiv:2309.12732, 2023.

[71]
P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” arXiv preprint arXiv:2206.06488, 2023.

[72]
J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” arXiv preprint arXiv:2201.12086, 2022.

[73]
H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” 2023.

[74]
C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” arXiv preprint arXiv:1904.01766, 2019.

[75]
A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” arXiv preprint arXiv:2107.00135, 2022.

[76]
R. Cai, J. Yuan, B. Xu, and Z. Hao, “Sadga: Structure-aware dual graph aggregation network for text-to-sql,” arXiv preprint arXiv:2111.00653, 2022.

[77]
OpenAI, “Sora: Creating video from text.” <https://openai.com/sora>, 2024.

[78]
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, and T. Rocktäschel, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.

[79]
J. Lin, R. Pradeep, T. Teofili, and J. Xian, “Vector search with openai embeddings: Lucene is all you need,” arXiv preprint arXiv:2308.14963, 2023.

[80]
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, Q. Guo, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2024.

[81]
M. J. Buehler, “Generative Retrieval-Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials Design,” ACS Engineering Au, Jan. 2024.
Publisher: American Chemical Society.

[82]
R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, and W. Saunders, “Webgpt: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021.

[83]
T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” arXiv preprint arXiv:2302.04761, 2023.

[84]
C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual chatgpt: Talking, drawing and editing with visual foundation models,” arXiv preprint arXiv:2303.04671, 2023.

[85]
D. Surís, S. Menon, and C. Vondrick, “Vipergpt: Visual inference via python execution for reasoning,” arXiv preprint arXiv:2303.08128, 2023.

[86]
T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou, “Large language models as tool makers,” arXiv preprint arXiv:2305.17126, 2023.

[87]
R. Haase.

[88]
J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1–22, 2023.

[89]
C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez, “Memgpt: Towards llms as operating systems,” arXiv preprint arXiv:2310.08560, 2023.

[90]
Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, S. Zhao, Y. Q. Chong, C. Wang, K. Sycara, M. Johnson-Roberson, D. Batra, X. Wang, S. Scherer, Z. Kira, F. Xia, and Y. Bisk, “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023.

[91]
Y. Kant, A. Ramachandran, S. Yenamandra, I. Gilitschenski, D. Batra, A. Szot, and H. Agrawal, “Housekeep: Tidying virtual households using commonsense reasoning,” arXiv preprint arXiv:2205.10712, 2022.

[92]
D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023.

[93]
S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang, “Language-driven representation learning for robotics,” arXiv preprint arXiv:2302.12766, 2023.

[94]
Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” arXiv preprint arXiv:2310.12931, 2023.

[95]
E. Davis, “Using a large language model to generate program mutations for a genetic algorithm to search for solutions to combinatorial problems: Review of (Romera-Paredes et al., 2023).,” Jan. 2024.

[96]
T. Ridnik, D. Kredo, and I. Friedman, “Code generation with alphacodium: From prompt engineering to flow engineering,” arXiv preprint arXiv:2401.08500, 2024.

[97]
A. Coja-Oghlan, P. Loick, B. F. Mezei, and G. B. Sorkin, “The ising antiferromagnet and max cut on random regular graphs,” arXiv preprint arXiv:2009.10483, 2020.

[98]
S. Kench, I. Squires, A. Dahari, and S. J. Cooper, “Microlib: A library of 3d microstructures generated from 2d micrographs using slicegan,” Scientific Data, vol. 9, no. 1, p. 645, 2022.

[99]
J. Eliot, “Doitpoms micrograph library,” 2000.

[100]
S. Kench and S. J. Cooper, “Generating three-dimensional structures from a two-dimensional slice with generative adversarial network-based dimensionality expansion,” Nature Machine Intelligence, vol. 3, no. 4, pp. 299–305, 2021.

[101]
S. Kench, I. Squires, and S. Cooper, “Taufactor 2: A gpu accelerated python tool for microstructural analysis,” Journal of Open Source Software, vol. 8, no. 88, p. 5358, 2023.

[102]
R. Aversa, M. H. Modarres, S. Cozzini, R. Ciancio, and A. Chiusole, “The first annotated set of scanning electron microscopy images for nanoscience,” Scientific Data, vol. 5, p. 180172, Aug. 2018.

[103]
E. Williams, J. Moore, S. W. Li, G. Rustici, A. Tarkowska, A. Chessel, S. Leo, B. Antal, R. K. Ferguson, U. Sarkans, A. Brazma, R. E. Carazo Salas, and J. R. Swedlow, “Image Data Resource: a bioimage data integration and publication platform,” Nature Methods, vol. 14, pp. 775–781, Aug. 2017.

[104]
M. Pachitariu and C. Stringer, “Cellpose 2.0: how to train your own model,” Nature Methods, Nov. 2022.

[105]
Z. Zheng, Z. He, O. Khattab, N. Rampal, M. A. Zaharia, C. Borgs, J. T. Chayes, and O. M. Yaghi, “Image and data mining in reticular chemistry powered by GPT-4V,” Digital Discovery, pp. –, 2024.
Publisher: RSC.

[106]
C. Clark and S. Divvala, “PDFFigures 2.0: Mining figures from research papers,” in 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL), pp. 143–152, 2016.

[107]
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232, 2023.

[108]
M. Phuong and M. Hutter, “Formal algorithms for transformers,” arXiv preprint arXiv:2207.09238, 2022.

[109]
B. Rohrer.

[110]
C. Huyen, “Sampling for Text Generation.”

[111]
M. Freitag and Y. Al-Onaizan, “Beam Search Strategies for Neural Machine Translation,” in Proceedings of the First Workshop on Neural Machine Translation, Association for Computational Linguistics, 2017.

§ SUPPLEMENTARY INFORMATION

outer arc=0pt,

§ EXTENDED LLM THEORY

§.§ Attention

Attention is implemented with the matrix multiplication [2]

\begin{equation}
\text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V
\end{equation}

where Q, K, V are all linear projections of the input sequence of tokens $X$ with learnable weight matrices $W_{Q}$, $W_{k}$ and $W_{v}$ and $\sqrt{d_{k}}$ is a scaling factor to prevent the dot product getting too large.
Vaswani et. al.[2] describe it as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
$QK^{T}$ describes how important each token in the context is at predicting or representing the value of every other token[108].
This can be interpreted as a differentiable lookup table[109] or learned message passing, with $QK^{T}$ determining the similarity of the input to the (learned) keys of this table and V being the (learned) values. The values in $V$ are then weighted by this similarity score and passed to the next layer.

The authors also proposed multi-head attention[2], where different heads perform the attention calculation on the same input sequence (again in parallel), which improved the model's ability to focus on multiple positions in a sequence. Without the multiple heads, averaging over different training examples where different positions in the sequence contain the relevant pattern (noun-pronoun, verb-subject, etc.) would have diminished this ability.

§.§ Transformer network architectures

The original transformer network[2] had encoder-decoder architecture, with the encoder formed of $N$ identical encoder layers and the decoder formed of $N$ identical decoder layers. Each encoder layer has multi-headed attention applied to an embedding (vector representation of the tokens in a new subspace) of the input sequence or the previous layer's output, followed by a feed-forward linear layer. The decoder layer has an additional multi-headed attention layer at the start, and `cross-attention' with the output of $n^{th}$ encoder layer. An example encoder layer is show in Figure <ref>.

The original transformer network was an encoder-decoder, which worked well for sequence-to-sequence translation, but not all transformers or LLMs are encoder-decoders. The GPT family [10, 3, 4] are decoder-only, which work better for auto-regressive token generation.

§.§ LLM output sampling

Various sampling and post-processing strategies exist, which can be combined with temperature and each other. `Top-k sampling' chooses the best k-possible tokens and only uses them in the softmax, reducing the computational overhead. `Top-p sampling' chooses from tokens to sample from by summing the $n$ most probable tokens until their cumulative probability is greater than some cutoff $p$ [110].

`Test-time sampling' generates multiple outputs for the same prompt and uses some heuristic like total sequence probability to decide which to return to the user [110]. This is effective but increases the total compute cost. Other ways of improving results (at the cost of added compute) are search strategies, where various next-token options are explored in parallel. A notable example of this strategy is `beam search'[111].

LLM outputs can be forced to conform to a given format or scheme (like JSON) via `constraint sampling', which involves filtering output token logits to a list of valid tokens for the scheme and selecting the most probable (i.e, a JSON should end in a }) [110]. This form of sampling could prove incredibly useful for writing programs or robotics routines or outputting niche data formats (common in research) that must fit some existing rules. OpenAI have already added a `JSON mode' to their API for such situations.

§ CASE STUDY 1: AUTOMATED 3D MICROSTRUCTURE ANALYSIS

§.§ Full Prompts

Below are the system prompts for MicroGPT

Prompt 1: MicroGPT system prompt

instructions_microgpt = """ You are an assistant to analyze microstructure. Remember:

1. You can invoke tools for analysing tomographic data.
For image analysing, please ensure to call the function once for each path name provided.

Typically, the number of times the function needs to be invoked corresponds directly to the number of path names you have.

2. After writing the code, always use a function, create_and_execute_python_file, to upload and execute it.

3. If the user ask for anlysis the all images in a specific folder, please use data_analysis function. If use ask for analysis an image, please use other function.

4. If the user ask to filter data in a dataset, eg. try to find iron related 3D images in a specific directory, please use data_filter function.

5. If the user requests to reuse a tool that is included in a Python file, please employ the 'tool_reuse' function"""

Prompt 2: simulation function

delimiter = "####"

system_message = f"""
Follow these steps to answer the customer queries.

Step 1:{delimiter} First, determine if the user is asking a question about analyzing 3D images in a specific directory. If the user is asking about analyzing 3D images in a specific directory, call a function to extract the filenames of images in the directory.

Step 2:{delimiter} Next, call a simulation function to analyze the images.

Step 3:{delimiter} Finally, store all the data in a CSV file.

Use the following format:

Step 1:{delimiter} <step 1 reasoning>

Step 2:{delimiter} <step 2 reasoning>

Step 3:{delimiter} <step 3 reasoning>

Make sure to include {delimiter} to separate every step.

Prompt 3: data filtering function

delimiter = "####"

system_message = f"""
Follow these steps to answer the customer queries.

Step 1:{delimiter} First, confirm whether the user is asking you to filter the data in the database based on their criteria.         Determine the directory of the database.

Step 2:{delimiter} If the user is asking about                 filter 3D images in a dataset, you have a function called find_json, you can use it to unfold zip file, find the meta data of the dataset in the file and extract the metadata

Step 3:{delimiter} Now that you have the metadata, which is in a JSON format, focus on the description and keywords within the metadata. Filter all the data in the database that aligns with the user's criteria.

Use the following format:

Step 1:{delimiter} <step 1 reasoning>

Step 2:{delimiter} <step 2 reasoning>

Step 3:{delimiter} <step 3 reasoning>

Make sure to include {delimiter} to separate every step.

Prompt 4: tool reuse function

delimiter = "####"

system_message = f"""
Follow these steps to answer the customer queries.

Step 1:{delimiter} First, confirm whether the user wants to reuse the tool created before, by modifing the code of a file according to his needs and then run it. If yes, call the read_file function to read this file, specifying the file path clearly

Step 2:{delimiter} Now that you have the code, make modifications according to the user's requirements.

Step 3:{delimiter} Call the create_and_execute_python_file function to save and overwrite the original code, keeping the file name unchanged,clearly specifying the file path to be saved.

Use the following format:

Step 1:{delimiter} <step 1 reasoning>

Step 2:{delimiter} <step 2 reasoning>

Step 3:{delimiter} <step 3 reasoning>

Make sure to include {delimiter} to separate every step.

§.§ An example interaction

The following is a conversation between the user and MicroGPT regarding the Microlib Dataset. These dialogues   the following capabilities of MicroGPT: data collection, custom tool creation and reuse, data filtering, data simulation, data analysis, data visualization, etc.

Data Collection

User: Can you search for the Microlib online, which is a dataset of 3D microstructures?

Assistant: The Microlib dataset of 3D microstructures is available on Zenodo. Here are some of the relevant records:

1. [Zenodo Record 7118559](https://zenodo.org/records/7118559)

2. [Zenodo Record 4047774](https://zenodo.org/records/4047774)

3. [Zenodo Record 7867413](https://zenodo.org/records/7867413)

4. [Zenodo Record 8112442](https://zenodo.org/records/8112442)

5. [Zenodo Record 7660542](https://zenodo.org/records/7660542)

6. [Zenodo Record 7043464](https://zenodo.org/records/7043464)

7. [Zenodo Record 5011434](https://zenodo.org/records/5011434)

8. [Zenodo Record 5090057](https://zenodo.org/records/5090057)

9. [Zenodo Record 7702179](https://zenodo.org/records/7702179)

10. [Zenodo Record 7820168](https://zenodo.org/records/7820168)

You can visit these links to find more information about each dataset and access the data.

User: Download the first one zenodo record 7118559'

Assistant: Do you want to download the file from

microlibDataset.zip?download=1? (yes/no): yes

File downloaded: microlibDataset.zip

The dataset from Zenodo record 7118559 has been downloaded successfully as `microlibDataset.zip`. If you would like to proceed with extracting and organizing the files, or any other action with this dataset, please let me know!

Custom Tool Creation and Reuse

User: Please write and execute a script to unzip the file './microlibDataset.zip

Assistant: Python script saved as created_script.py
Do you want to execute the script? (yes/no):yes

I have written and executed a Python script to unzip the file 'microlibDataset.zip'. The contents have been successfully extracted. If you need to work with these files or have any other requests, please let me know!

User: Please unzip the file named './Test_tool_reuse.zip' using the tool in './created_script.py

Step 1:#### To unzip the file './Test_tool_reuse.zip', we first need to read the existing Python script located at './created_script.py'. We will use the read_file function to read the content of this file. This will allow us to understand the current functionality of the script and modify it to include the code necessary for unzipping the file.

Step 2:#### After reading the file, we need to modify the code to add the functionality to unzip files. We will import the 'zipfile' module and use the 'ZipFile' class along with the 'extractall' method to unzip the file. The code modification will include handling the path of the zip file and the destination directory where the contents will be extracted.

Step 3:#### Once the code has been modified to include the unzip functionality, we will call the create_and_execute_python_file function to save and overwrite the original code in './created_script.py'. This will update the script with the new functionality, and then we will execute the script to perform the unzipping operation.

========================Start solving the problem!=================

Python script saved as created_script.py
Do you want to execute the script? (yes/no): yes
The files have been extracted to: ./Test_tool_reuse

The script has been updated to unzip the file 'Test_tool_reuse.zip', and the execution of the script was successful. The files have been extracted to a directory named 'Test_tool_reuse'

Data Filter

User: In the 'microlibDataset.zip' file, can you find all the 3D images related to cast iron?

Step 1:#### The user is asking to filter the data in the database based on their criteria, which is to find all the 3D images related to cast iron. The directory of the database is indicated as 'microlibDataset.zip'.

Step 2:#### Since the user is asking to filter 3D images in a dataset, I will use the function find_json to unfold the 'microlibDataset.zip' file, find the metadata of the dataset in the file, and extract the metadata.

Step 3:#### Now that I have the content of metadata in json-format, I will focus on the description and keywords in the metadata, and filter all the data in the database which align with the user's criteria, which in this case is "cast iron".

========================Start solving the problem!=================

Found file: data_anns.json. Is this the metadata file? (yes/no):yes

JSON file contains items with the following fields:

Enter the fields you want to use to filter your data (separated by commas):name,keyword

Based on the metadata extracted from the 'microlibDataset.zip' file, we can filter all the data related to "cast iron." Here are the entries that match the criteria:

- Microstructure 066: Keywords including "cast iron"
- Microstructure 068: Keywords including "cast iron"
- Microstructure 072: Keywords including "cast iron"
- Microstructure 360: Keywords including "cast iron"
- Microstructure 365: Keywords including "white cast iron"
- Microstructure 368: Keywords including "white cast iron"
- Microstructure 370: Keywords including "white cast iron"
- Microstructure 372: Keywords including "white cast iron"
- Microstructure 376: Keywords including "white cast iron"
- Microstructure 378: Keywords including "grey cast iron"
- Microstructure 381: Keywords including "grey cast iron"
- Microstructure 387: Keywords including "grey cast iron"
- Microstructure 393: Keywords including "cast iron"
- Microstructure 396: Keywords including "cast iron"
- Microstructure 406: Keywords including "cast iron"
- Microstructure 782: Keywords including "cast iron"
- Microstructure 784: Keywords including "cast iron"
- Microstructure 797: Keywords including "cast iron"

These entries are filtered out because they contain the keyword "cast iron" or a variation thereof, such as "white cast iron" or "grey cast iron." If you need more information about these microstructures or if you need to extract specific files related to these entries, please let me know.

Please input an exmaple of data filename: microstructure001.tif

Based on the example filename 'microstructure001.tif' provided by the user, we can deduce that the data filenames in the dataset follow the pattern 'microstructureXYZ.tif', where 'XYZ' represents a three-digit number, covering all the ones in the dataset.

Files with the pattern mentioned above have been extracted successfully. However, since I do not have direct access to the file system to list all the filenames, I will now list the filenames based on the metadata provided earlier that are related to "cast iron":

1. microstructure066.tif
2. microstructure068.tif
3. microstructure072.tif
4. microstructure360.tif
5. microstructure365.tif
6. microstructure368.tif
7. microstructure370.tif
8. microstructure372.tif
9. microstructure376.tif
10. microstructure378.tif
11. microstructure381.tif
12. microstructure387.tif
13. microstructure393.tif
14. microstructure396.tif
15. microstructure406.tif
16. microstructure782.tif
17. microstructure784.tif
18. microstructure797.tif

These files are stored in a data folder and are ready for any further analysis or review that you may need. If you'd like to proceed with any specific analysis or have other requests, please let me know!

Data Simulation

User: Could you analyze the 3D images in the './data' folder to determine their tortuosity, diffusion factor, volume fraction, and surface area?

Step 1: #### The user is asking to analyse 3D images in a specific directory ('./data'). Therefore, I will call a function to extract the filenames of images in that directory.

Step 2: #### Now that we have the filenames, I will call a simulation function to analyse the images for tortuosity, diffusion factor, volume fraction, and surface area.

Step 3: #### After the analysis is complete, I will store all the data into a CSV file.

=================Start solving the problem!================

The image paths from the specified directory ('./data') are as follows:

- './data/microstructure368.tif'
- './data/microstructure378.tif'
- './data/microstructure782.tif'
- './data/microstructure370.tif'
- './data/microstructure365.tif'
- './data/microstructure387.tif'
- './data/microstructure797.tif'
- './data/microstructure360.tif'
- './data/microstructure376.tif'
- './data/microstructure396.tif'
- './data/microstructure372.tif'
- './data/microstructure406.tif'
- './data/microstructure066.tif'
- './data/microstructure072.tif'
- './data/microstructure784.tif'
- './data/microstructure393.tif'
- './data/microstructure068.tif'
- './data/microstructure381.tif'

The simulation results for the analysis of the 3D images are as follows:

1. Microstructure368.tif:
- Effective Diffusivity: 0.4152
- Tau (Tortuosity Factor): 1.8422
- Volume Fraction: [0.2351, 0.7649]
- Surface Area: 0.0919

2. Microstructure378.tif:
- Effective Diffusivity: 0.3918
- Tau: 1.9773
- Volume Fraction: [0.2252, 0.7748]
- Surface Area: 0.1074

3. Microstructure782.tif:
- Effective Diffusivity: 0.3223
- Tau: 1.7827
- Volume Fraction: [0.4255, 0.5745]
- Surface Area: 0.0396

4. Microstructure370.tif:
- Effective Diffusivity: 0.0464
- Tau: 4.2417
- Volume Fraction: [0.8031, 0.1969]
- Surface Area: 0.0838

... (summary continues for each image)

One of the images, Microstructure797.tif, encountered an issue and could not be processed.

The data from the analysis has been successfully stored into a CSV file. You can download it using the following link:

[Download microstructure_analysis_results.csv]

Cast iron related microstructure's simulation results in Microlib

Microstructure     Effective Diffusivity     Tortuosity     Surface Area     2cmVolume Fraction
Phase 1     2cmVolume Fraction
Phase 2

Microstructure     Effective Diffusivity     Tortuosity     Surface Area     2cmVolume Fraction
Phase 1     2cmVolume Fraction
Phase 2

late after line=
before reading=`#=12,

Data Analysis

User: Review and compare these data to identify any data points that are conspicuously unreasonable or significantly divergent from the rest.

Assistant: Based on the analysis of the dataset for outliers, we've identified the following:

1. **microstructure396.tif**: The effective diffusivity of 0.6924 is identified as an outlier, as it is significantly higher than what we would expect compared to other data points.

2. **microstructure372.tif**: This microstructure has an extraordinarily high tortuosity (Tau) value of 85.388, which is significantly higher than any other data point, indicating that this value is an outlier and warrants further investigation.

3. **microstructure784.tif**: The surface area value of 0.1768 is marked an outlier on the higher end, meaning it is significantly larger than most other surface area values, possibly indicating a highly complex interface between phases.

The rest of the data points appear reasonable, with volume fractions summing close to 1 as they should. It's important to remember that while the statistical method used here highlights significant deviations from the mean, it cannot conclusively determine whether a value is truly unreasonable without additional context regarding the materials and processes involved. These identified outliers should be reviewed carefully with consideration of the experimental conditions or possibly re-measured if experimental error is suspected.

User: I believe that only the data for microstructure372 is incorrect. Given that there is an issue with this data, please disregard microstructure 372 when addressing the following question. Analyze which microstructure possesses the highest tortuosity.

Assistant: After correctly excluding the data for microstructure372, the microstructure with the highest tortuosity is microstructure360, which has a tortuosity (Tau) value of 5.9349.

User: Please compare microstructure 393, 368, and 365

Assistant: Here is a comparison of the specified microstructures based on their properties:

### Effective Diffusivity (D_eff):

- **microstructure393.tif**: Has a moderate effective diffusivity of around 0.148.

- **microstructure368.tif**: Exhibits the highest effective diffusivity among the three, at about 0.415.

- **microstructure365.tif**: Close to microstructure393.tif, with a value of around 0.171.

### Tortuosity (Tau):

- **microstructure393.tif**: Has the highest tortuosity with a value of approximately 2.725.

- **microstructure368.tif**: Has the lowest tortuosity, with a value close to 1.842.

- **microstructure365.tif**: Has a tortuosity value around 2.091.

### Surface Area:

- **microstructure393.tif**: Has a moderate surface area valued at approximately 0.061.

- **microstructure368.tif**: Has the highest surface area with a value of around 0.092.

- **microstructure365.tif**: Has the lowest surface area, near 0.043.

### Volume Fraction Phase 1:

- **microstructure393.tif**: Has a reasonably high volume fraction phase 1 of approximately 0.597.

- **microstructure368.tif**: Has the lowest volume fraction phase 1 of about 0.235.

- **microstructure365.tif**: Has the highest volume fraction phase 1, around 0.643.

### Volume Fraction Phase 2 (complementary to Phase 1):

- **microstructure393.tif**: Has the lowest volume fraction phase 2 of about 0.403.

- **microstructure368.tif**: Has the highest volume fraction phase 2 of approximately 0.765.

- **microstructure365.tif**: Has a moderate volume fraction phase 2, around 0.357.

Overall, **microstructure368.tif** stands out due to its high effective diffusivity and surface area, which are beneficial for a catalyst carrier that facilitates reaction and filtration. However, **microstructure365.tif** also has desirable properties, such as the highest volume fraction phase 1, which might imply a more solid structure possibly beneficial for mechanical stability in a catalyst support. **Microstructure393.tif** appears to be a middle ground with moderate properties across the board.

Data Visualization

User: Disregard the unreasonable data, can you generate some figures to create visualizations for the data? Histograms for each numerical column to understand the distribution of values. Scatter plots to explore relationships between pairs of numerical variables (e.g., Effective Diffusivity vs. Tortuosity)

Assistant: Here are the visualizations for the data:

### Histograms for Each Numerical Column
- The histograms show the distribution of values for each numerical variable: Effective Diffusivity, Tortuosity, Surface Area, Volume Fraction Phase 1, and Volume Fraction Phase 2.

### Scatter Plots for Relationships Between Pairs of Numerical Variables
- The scatter plots explore relationships between pairs of numerical variables, such as:

1. Effective Diffusivity vs. Tortuosity

2. Effective Diffusivity vs. Surface Area

3. Effective Diffusivity vs. Volume Fraction Phase 1

4. Tortuosity vs. Surface Area

5. Tortuosity vs. Volume Fraction Phase 1

6. Surface Area vs. Volume Fraction Phase 1

These figures provide insights into how the various microstructural properties are distributed and how they may relate to one another within the dataset.

Histograms of effective diffusivity, tortuosity, surface area, and volume fraction of cast-iron microstructures in Microlib as returned by MicroGPT.

Scatter plots exploring the relationships between pairs of numerical variables of cast-iron microstructures in Microlib as returned by MicroGPT.

§ CASE STUDY 2: AUTOMATED LABELLED MICROSTRUCTURE DATASET COLLECTION

§.§ Scraping and figure extraction

The metadata (title, abstract, authors, DOI, etc.) of 14,000 papers matching the search query  were scraped from arXiv and chemrXiv via their APIs. A subset of 500 of these were downloaded from chemrXiv and the figures and their matching captions extracted with `pdffigures2.0'[106]. This worked well but the tool sometimes failed to extract the whole figure or caption.

Figures were split into subfigures by detecting whitespace pixels (pixels whose intensity $\approx 255$) and thresholding. The binary image was dilated to remove small gaps. The connected components of the non-whitspace pixels were computed and bounding boxes generated - these were then saved as subfigures. Small figures like sub-figure letter captions, were discarded. This process works under the assumption that figures use around 5px whitespace to separate subfigures, which tended to be true in most cases.

§.§ LLM labelling

We chose to use GPT3.5 and GPT4 as the LLM and VLM in the study, primarily for convenience and performance. Given the relatively uncomplicated nature of the task, a local LLM would be preferable for running this workflow at-scale. The prompts used for each step are in Section <ref>.

The caption and abstract text-only GPT4 worked well for detecting and categorization of figures (see Section <ref>), but in order to detect if a specific subfigure was a micrograph a vision model was needed.

§.§ Evaluation

A custom labelling GUI was written to label figures and subfigures in papers in terms of whether they had a micrograph present, what material was shown and what instrument was used to capture the micrograph. It also allowed comparison of the human labels to the LLM labels for evaluation. We chose to do labelling before evaluation (i.e, a two-step process) rather than just reviewing the LLM answers in order to avoid bias.

For the VLM evaluation, we noted the lack of false positives by examining the figures in the produced dataset and resulting T-SNE plot, and not via the custom GUI. This method did not allow us to report the full confusion matrix.

§.§ Performance and issues

Confusion matrices of micrograph detection for (a) GPT3.5 with caption & abstract, (b) GPT4 with caption & abstract,  (c) GPT3.5 with caption & no abstract and (d) GPT4 with caption & no abstract. GPT4 with abstract performs the best, with the highest sensitivity and specificity.

Confusion matrices of micrograph detection for two string matching (regex) approaches. (a) is the simple regex scheme, where a figure is labelled as containing a micrograph if it contains the phrase `image' or `micrograph'. (b) is the greedy scheme, where a figure is labelled as containing a micrograph if it matches the simple scheme or contains the name of a microscopy technique like `SEM' or `AFM'. The simple approach works well though has less true positives than LLM-based results where the caption is more ambiguous. As expected, the greedy approach has more true positives but far more false positives.

§.§ Prompts

Prompt 1: LLM micrograph screening utilizing only the captions of figures as input

system_message ="""

You are an expert on micrographs, images captured using microscopes such as SEM, TEM, or AFM, etc.

Given caption of a figure from an academic paper, answer the question:

Is there a micrograph in this figure? Respond with 'true' or 'false'

IMPORTANT: The anwer should only contain pure JSON format only with the fields "isMicrograph"

Here's an example of output:

"isMicrograph": "true or false"

Prompt 2: LLM micrograph screening utilizing the captions of figures and paper abstract as input

system_message ="""

You are an expert materials scientist. You study micrographs, which are images taken using a microscope.

Based on an academic paper's abstract and a specific figure's caption, provide answers in JSON format to the following questions:

1. Is there a micrograph in this specific figure? Answer with 'true' or 'false'.

2. If a micrograph is present, list the techniques used in this figure (e.g., SEM, TEM, Optical Microscopy). Note that techniques mentioned in the abstract might not be used in this figure.

3. If a micrograph is present, list the the full name of materials depicted in the micrographs e.g., 'Lithium Nickel-Manganese-Cobalt (NMC) 811 cathode' or 'Insulin aggregates'.

4. Are there any noteworthy details about the micrograph, such as unique processing conditions or observed anomalies, in a series of brief phrases (e.g., ['heat-treated', 'cracked', 'sintered'])?

Here's an example of how the JSON output should look with micrograph present:

"isMicrograph": "true",

"instrument": "Technique",

"material": "Description",

"comments": ["comment1", "comment2", "comment3"]

Here's an example of how the JSON output should look without micrograph present:

"isMicrograph": "false"

IMPORTANT: The answer should only contain pure JSON data matching the fields provided in the examples.

user_message = f""" The abstract is: {abstract_escaped}, and the captions are: {captions_escaped}"""

Prompt 3: VLM micrograph recognition with a single sub-figure as input

user_message_1 = """

You are an expert materials scientist specializing in micrographs.

Typically, these are greyscale and might include annotations like scale bars or arrows, although not always.

Your task:Given a subfigure image from a research paper

1. check if it is a micrograph or another type of figure, such as a graph or diagram

2. check if it includes sub-subfigures.

If the image is solely a micrograph without any sub-subfigures, you should respond with 'TRUE';

otherwise,  respond with 'FALSE'.

Here is an example of output:

"Reason: The image is a micrograph with 2 sub-subfigures.

Answer: FALSE"

Prompt 4: VLM micrograph recognition with a main figure and a sub-figure as input

system_message = """

You are an expert materials scientist working on micrographs. The first image is a main image is taken from a research paper. The second image is a subfigure cropped from the main image. It is might be a micrograph.

Focus on the abstract of the paper, captions, and the content of these images. Answer the questions below:

1.Is the cropped image a micrograph? Answer with a single 'true' or 'false'.

2.What technique (e.g., SEM, TEM) was used to create the micrograph in the cropped image? Provide a brief answer, such as 'SEM' or 'Optical Microscopy'.

3.What material is shown in the micrograph? Provide the full name e.g., 'Lithium Nickel-Manganese-Cobalt (NMC) 811 cathode' or 'Insulin aggregates'.

4.If there are any interesting things about the micrograph, like specific processing conditions or anomalies, put these in a list of single phrases (e.g ['heat-treated, 'cracked', 'sintered']).

5.Which part of the caption of the mainfigure does this subfigure correspond to? Extract the caption of the subfigure, and its label e.g (a), (b) if possible.

Here's an example of the JSON output format:

"isMicrograph": True,

"instrument": "Technique",

"material": "Description",

"comments": ["comment1", "comment2", "comment3"]

"subfigure_caption": "(label): caption of subfigure"

IMPORTANT: The answer should only contain pure JSON data matching the fields provided in the examples.

user_message = f"The abstract is: {abstract_escaped}, and the captions are: {captions_escaped}"