Now that I have summarized many of the major arguments for increased use of audio journals, both for groups primarily focused on speech sampling and groups primarily focused on digital phenotyping, I will spend the rest of the section expanding on the dynamics of these fields at present and how audio journals ought to fit into the future. This will also include a discussion of the importance of software tools like my pipeline in that future, as well as a more detailed set of arguments on the value of autobiographical patient perspectives. The latter point is not necessary to support the high level claim of the chapter on the severely underrated status of daily audio diaries, but it is an important point for psychiatry more broadly in my opinion, and it remains very relevant to the journal format. Ultimately, by carefully characterizing the great value of audio diaries, I will have further motivated the contributions of this chapter, along with the upcoming discussion section on future directions for this datatype at multiple scientific timescales (<ref>).

§.§.§ Clinical and computational advantages make for the best of both worlds

Audio journals provide scientific advantages across a number of domains, yet this ironically might be the reason they've remained relatively ignored. As will be reviewed below in section <ref>, the issue has apparently impacted audio journals on an organizational level in the early stages of AMPSCZ. With such a large collaborative project it is expected that responsibilities be partitioned and certain subgroups interact only rarely, but because audio journals don't fit neatly in a single box, they have not fit neatly under a single team's responsibilities either. Journals are simultaneously a speech sampling datatype and a phone app datatype, and as is consistent with prior psychology research, people are less likely to do something that they think might be someone else's responsibility.

This is probably not the only factor limiting progress with audio diaries in AMPSCZ, however it is clean representation of a more abstract problem that I feel currently pervades relevant research projects. Many traditional speech sampling groups have transitioned to the digital age through the use of interview or task recordings for analysis – formats that have a more immediate connection to prior psychiatry literature, but lack a number of the advantages that digital psychiatry can provide. It is especially problematic when this involves application of recent data science/machine learning techniques that are not appropriate for the sample sizes they are able to obtain with their chosen datatypes.

On the other hand, many emerging digital psychiatry groups have put the majority of their focus on the datatypes that can generate the largest amount of data, and with a highly quantitative lens. Despite many such studies already collecting EMA surveys to use as one label source for symptom severity, the audio diaries that can often be collected by the same app don't seem to receive much consideration – meanwhile these diaries are a potential multidirectional threat, from objective and quantitative vocal feature extraction all the way to subjective and qualitative report of clinically relevant phenomena. They could serve as inputs or labels, and they may be seen in terms of their potential for advanced machine learning applications like transformers or in terms of their potential for very human driven psychiatric analysis. If much better smart watches are available in 5 years or if industry data processing tools soundly beat anything developed by academic science at e.g. labeling a particular movement pattern of note, the digital phenotyping datasets and papers in progress right now might not age well. An audio journal dataset or a truly multifaceted audio journal study is much less likely to age poorly. Intermediate results can be extremely important, but again when thinking for the field as a whole, it is important for generational processes that some contributions are more timeless.

Proper collection and particularly analysis of daily audio diaries can indeed be more resource intensive than is warranted for some digital phenotyping studies, and it can also be well beyond the scope of rigorously designed interview or task recording studies that focus on addressing scientific questions directly related to the respective formats. I see little reason to believe such decisions are being made in a principled manner at present though, which is what has motivated much of this section: I aim to present detailed arguments for the use of audio journals that address both perspectives, such that future research plans may hopefully address some of these considerations outright. In doing so, I suspect that a number of hypothetical studies would legitimately increase the role of app-based audio diaries in their investigations. In the upcoming snippets I will address clinical and big data agendas, and provide specific examples of how both inputs and labels of interest could be obtained from diaries in a wide variety of study contexts, with some drawn directly from demonstrated properties of the BLS journal set.

Psychiatrists must not underrate data science tractability nor overrely on existing clinical measures.
In practice, I have witnessed interview recordings prioritized over audio diaries at greater cost on multiple occasions. As such, the following list presents concerns that ought to be considered more often before making such a judgement:

* Interview recordings have been prioritized as a speech sampling method over audio journals to date, likely because they seem less risky from a perspective focused on previous psychiatric literature. However it is very difficult to collect a large number of interview recordings, and those recordings are typically long – which introduces a huge number of potential summary metrics over a small number of data points. Therefore almost all existing literature on the topic has either tested a small handful of already well known features or (more common recently) leaned entirely in to being exploratory with no attempt to make statements on relevant features with any statistical power. It is not sustainable for a whole field to behave this way for an extended time period, in particular as the features tested in this exploratory way completely change with nearly every paper (under the justification that technology advances rapidly and we must keep up).

* Audio journals meanwhile can be collected at lesser cost from a greater number of subjects, and participating subjects will record many more journals providing much better temporal resolution than interview recordings can during the same study period. Overall, audio journal recordings can generate a substantially larger number of data points than interview recordings can, even when the journals are treated as an afterthought. Because each individual journal is much shorter, it is also much more tractable to compare journals over time and to summarize them with a reasonably-sized feature set.

* Self report EMA surveys collected via the same app as audio journal recordings facilitate the collection of patient-provided labels nearly simultaneous with journal submission. While self report surveys have been fairly popular in recent literature, audio journals have not been. Additionally, self-administered scales with extensive prior literature, for example the Beck Depression Inventory (BDI), can be collected using these apps. The timescale and data availability alignment that EMA can be easily designed to have with audio journals is very conducive to quality study design, without requiring high costs or researcher labor burdens. It is odd then that EMA has become quite common to collect, often using an app that could be enabled to administer audio journal prompts too, yet the resulting EMA is used with other datatypes instead. For any proposal with a scientific question involving speech sampling and a study plan involving EMA, prioritization of audio journals should be highly encouraged.

* When interview recordings correspond more closely with clinical scale ratings, they tend to contain a somewhat stunted conversation style that provides less rich content for analysis. These interviews also ask already well established questions, and while such questions obviously do have value in assessing psychiatric illness, it will be very difficult to discover new psychological factors of relevance without considering any datatype that allows more freedom. In my opinion, unconstrained autobiographical reporting from patients should be considered of greater interest as one portion of the academic literature.

* More broadly, if every study uses only established clinical scale ratings (or even worse diagnostic categories) as their labels, often looking at scale totals instead of symptom items at that, it will be extremely difficult for any breakthroughs to occur. As may be a trend throughout modern science, progress will likely be incremental, and incremental in a direction that perhaps cannot ever advance past a certain point due to fundamental limitations on that path.

* Clinical scale ratings can of course still be used to contextualize long term diary collection studies, without introducing the additional cost and complexity of recording, transcribing, and processing the clinical interviews. Collecting interview recordings can still have its uses as well, and indeed there should be some studies that do prioritize these sources. At the same time, study design decisions should follow from scientific questions, not from supposed universal "best practices". Doubly so because the field of digital psychiatry is so new that it is difficult to understand why there even should be such strong trends yet in what datatypes are favored across a wide variety of studies.

Note that interview recordings and audio diaries are compared at length within chapter <ref>, which focuses on interview recording analysis. Examples of good possible reasons for recording interviews instead of or in addition to journal collection are provided there, but ideally future interview studies will address many of the questions in this list to some extent regardless.

Technical formalization must not eschew the value of subjective experiences and qualitative insights.
It is rare to see the aforementioned fishing expedition studies include a careful characterization of the features they have deemed to be useful, despite the fact such a characterization could increase confidence in the stated conclusions by demonstrating a cohesive account of how different features contributed to observed clinical correlations as well as how consistent the features that did not correlate would be with expectations if one were to accept that the positive results were real. This need not be done in a highly algorithmic way to have value, and at times even a few well designed scatter plots would help immensely with assessing how confident one should be in an under-powered reported result. There is an obvious incentive on the authors' part to leave these plots out if they look too noisy, but there is also a cultural phenomenon at play here from what I can tell: rigor has been confounded with quantification, especially quantification that sounds technically complex.

Obviously quantitative methods can be used to test and ultimately make rigorous statements that would otherwise not be possible. However quantitative methods are not inherently rigorous. It is easy for them to be applied incorrectly in a way that reviewers do not notice when working in interdisciplinary fields, and even when applied correctly there are many caveats to keep in mind in interpreting the results. I have seen extremely mathematically savvy people make incorrect starting assumptions due to lack of domain specific knowledge, which led to bad models in spite of theoretically good quantitative methods. One preventative measure against this is to be willing to sanity check yourself in what might seem a "dumb" way. Qualitative review is indeed a fantastic avenue for humans to come up with ideas, not only for future studies, but also for questions that ought to be asked in order to adequately challenge your present conclusions. Furthermore, there is nothing wrong with including qualitative evidence to supplement the strength of your claims, as this enables other scientists to utilize the perspective in forming an opinion on said claims, and to potentially draw new insights from the study results that would not be possible with a paper that reports only traditional statistics.

As mentioned, it is uncommon to find such qualitative characterization even in the supplement of a paper, and thus there is really no way to know if the authors attempted to carefully investigate their data beyond standard "plug and play" modeling. There is not enough collaboration with true machine learning experts in digital psychiatry at present, which exacerbates the issue. But even if there were, I highly doubt it would fully resolve this concern. On the contrary, I have seen incorrect Theorem proofs from highly technical groups make it into top machine learning conferences. A few examples are presented in chapter <ref>, one of which has a very simple counterexample in the form of a 2 unit network; really a counterexample that anyone thinking about the problem from the perspective of the network as opposed to an abstracted-away math problem could have seen.

Additionally, within the context of applying ML to psychiatry, it is not uncommon for technically interesting problems to diverge from clinically practical problems given the current state of the respective fields. Machine learning tools like the facial feature labeling models introduced in chapter <ref> were interesting ML problems when released 5+ years ago, but their use in study of psychiatric illness has been underwhelming to date, with psychiatry groups left in large part alone to figure out how to effectively wield frame by frame facial action unit features from just a double digit number of labeled clinical interviews. The latter is in some sense impossible to cast in terms of a machine learning problem, and regardless most modern ML experts are not focused on learning from small amounts of data, which in practice here would likely involve domain knowledge and careful application of older data science tools.

This brings us to yet another important role for audio journals: they are uniquely well suited for in depth domain specific qualitative analyses that can be paired with feature modeling in a manner that permits widespread interest. Diaries have great potential to better unite the different disciplines making up digital psychiatry, facilitating collaboration that could actually meld some methodological frameworks instead of just passing data back and forth.

The power of combining audio journals with other datatypes.
It should be clear at this point that audio journals are a powerful datatype for future research into psychiatric patient speech, whether that research is very clinically grounded, very data driven, or somewhere in between. The versatility of audio journals does not stop there though, because there are many potential upsides for the use of daily diaries in a wide range of digital psychiatry studies, even those with minimal prior interest in speech.

One major perk of modern digital phenotyping data is the ability to focus analyses at many different timescales. Audio journals have this advantage over interviews to some extent, as journal features can be considered daily, but they can also be summarized weekly, biweekly, or - to align well with many clinical interview protocols - monthly. What makes this even more valuable in the context of multimodal work is the ability to align insights from audio journals with highly detailed measures of the same day from passive sensors.

I will discuss the role of audio journal content in informing clinical ground truth next, but that is only one specific way in which they can provide value. More broadly, they can provide insightful context about a participant's day that can strongly synergize with trends detected in passive sensing. This can be utilized both for validation and for generation of multimodal feature sets that enable greater clinical predictive power than either could alone. Such benefits are not unique to the combination of audio journal and passive sensing, as different passive data streams can supplement each other similarly. However the type of information audio journals supplement with is in large part impossible to obtain with passive objective measures or highly constrained quantifications like EMA.

A few proof of concept examples of the use of audio journals to supplement digital phenotyping data in different ways and at different time scales can be found in chapter <ref>'s case report on an OCD patient undergoing a novel deep brain stimulation treatment. The patient submitted audio diaries very regularly, and on good days would often mention going out for a walk or for a bike ride, while bad days often mentioned not leaving the upstairs. Thus for one method for validation of the collected phone accelerometry data, we checked movement patterns against the specific physical activities (or lackthereof) reported on the days a clear statement was made.

The journal dataset of chapter <ref> was also used to assist in interpretation of observed GPS patterns, as the subject was quite religious and would often mention specific religious commitments or periods of relevant holiday. In the broader dataset, briefer visits to their religious institution were sometimes observed, and they often coincided with a related discussion in the submitted journal. However when the subject spent very long days at a religious institution for a number of days straight immediately after they were unblinded to the stimulation status of the device, there was no religious activity mentioned in any of the diaries that could explain the behavior. Given that the participant would regularly describe their day in detail, this was an interesting omission.

Similarly, the participant in chapter <ref> mentioned taking ADHD medications in their diaries on a regular basis with some periodicity, but started mentioning a different ADHD medication instead for a period in the middle of the study. A medication switch was not clear from clinical records we had available, but this sort of context is extremely relevant for a study involving a neurological intervention along with neural data collection.

These examples all arose from just one case report, a case report that, as it turns out, focused on a dataset containing much less interesting variation than has been observed in a number of BLS subjects to date. This is true not only from the perspective of the richness of audio journal content, as can be seen from the results reported in the thesis, but also from the perspective of passive sensing data analyzed as part of other lab publications drawing from the study [Rahimi-Eichi et al., 2022, Rahimi-Eichi et al., 2021]. Furthermore, the ability to estimate objective properties like vocal speech rate on a daily basis via diaries opens up numerous opportunities for deeper study of phenomena like psychomotor slowing, by creating a dataset of these objective speech features that achieves a temporal resolution actually well-suited for combination with the common passive sensing features of early digital psychiatry.

Ultimately, there is practically an endless number of ways that a diary dataset could end up supplementing a passive sensing one, as journals introduce a flexibility into the subjective self-report data collected for each subject. It is not practical (and at some point scientifically unsound) to collect massive EMA survey responses that cover a wide range of potential labels of interest. Through audio diaries, and alternative filtering approach is taken, where the subject limits what is covered. Unexpected discoveries can still pop up in a way not truly possible for highly constrained study designs, while at the same time the scale of the diary data collected remains very tractable.

Audio journals can supply labels, not just inputs.
As detailed, audio journals can provide inputs for modeling of EMA responses, clinical scores, diagnostic categories, and other common labels used in digital psychiatry. They can be an interesting sole source for input information, or they can synergize with inputs extracted from passive sensing signals. They can also provide interesting context for a case report utilizing such multimodal data sources, and can open up a unique additional avenue for feature validation.

Beyond all that, I believe for certain scientific questions they can stand entirely alone, supplying both inputs and labels – and for other questions they can serve as a primary source of assumed ground truth clinical labels to use with passive sensing derived inputs. As such, they are one of the most versatile digital psychiatry datatypes to date. However, this requires them to be used that way, which again falls into a similar dual threat trap. Most machine learning specialists will not have the clinical expertise necessary to frame audio journal content as a label in a psychiatrically motivated way and will prefer to avoid relatively labor intensive data processing approaches irrespective of scientific potential. Most psychiatrists will be hesitant to depart from existing scales and will not have an informed data science perspective on what audio journal labels might look like.

While such labels would be far from perfect, I feel they have clear scientific uses and at times fairly unique ones. In reviewing examples from throughout this chapter, and in looking at journal content more broadly during the process of executing the research, I found numerous statements that were inarguably pathological, and many more that raised an eyebrow. In my opinion, a trained clinician (or perhaps even an RA) would be very capable of looking at a set of these journals and identifying those days that display a particular symptom of note with high probability. While a generic journal with no signs is not interpretable as a lack of symptoms, the positive signal inherent in those free form recordings where a patient chose to describe symptoms in detail would be highly valuable if it could be predicted, in my opinion.

I would argue that the choice of what to discuss in a given journal is itself a relevant factor of current disease state for many patients. This is particularly true when that choice can be benchmarked against the patient's own baseline recording decisions from a long period of journal submissions. In some ways it might serve as a useful middle ground between the very challenging problem of predicting objectively severe rare events - like hospitalization from a manic episode - and the philosophically incremental problem of predicting biweekly clinical scale ratings or daily EMA symptom checklists. Regardless, this style of self-report enables symptoms to be captured that weren't even being looked for, whether that means nuance in distinguishing symptoms of a targeted disease beyond what EMA can realistically provide or discovering symptoms that were not previously on the radar for that patient.

Given the many issues with current diagnostic labels in psychiatry, having some subset of research take this style of approach seems quite important. It allows for the potential upside of a fishing expedition, but introduces more principle in the sense that real patient experiences can drive the exploration. We need ways to try some things that are not just incremental, but we also can't exactly afford to aimlessly throw shit at the wall forever, with the resources required to build digital psychiatry datasets of even modest size. Outside of psychiatry Einstein magically appearing to cause an intellectual paradigm shift, it is not likely we can avoid this problem through only modern formal scientific mechanisms. Yet listening more to subjective patient experiences in a research context is a plausible way to bridge some of this gap.

One researcher expressed to me that psychiatric science is under a much greater pressure than other fields to avoid appearing unscientific, and so there is a modern fear surrounding certain styles of thinking and discovery. While this may have understandable roots, it is telling that the fear is centered around "appearing" rather than around "being", as is the extent to which the baby seems to have been thrown out with the bathwater. A lot is going to be incorrect when a field is new, so it is infeasible to attempt to enforce universal scientific standards; in order to get to a more established spot, it is critical that early failures are allowed to happen. Additionally, while theoretical philosophizing and instinctual reasoning are certainly not a sufficient methodology for advancing a field of research, I do suspect it is a necessary component that would synergize well with formal scientific processes today if it were no longer defacto stigmatized.

Nevertheless, there are extremely concrete examples of inherently clinical information that was contained in some BLS audio diaries, potential use cases which do not require a leap of faith in psychiatric definitions or an especially qualitative perspective. Take for example the following sentences from a journal submission by 8RC89:

Um, my mother thinks I might be getting hypomanic and she might be right.

If a patient chooses to say they think they are becoming manic in a diary recording, why would that be less trustworthy than a patient submitting a self-report survey response rating their level of mania symptoms as moderate? Why is that less of a clinical ground truth than the patient saying as much in a structured interview? The fact that it was not an explicit response to a question is a double-edged sword for the reasons discussed above, but it does not make sense to so heavily discount the evidence provided by someone choosing to report something like this largely unprompted. It is in many ways less likely to be a false positive, because they are speaking organically in what is experienced in the moment as a private personal expression. I don't see why such a statement should be considered unusable in a direct clinical sense for psychiatry research.

This sort of content is not too difficult to obtain or to track either. With a simple search for "manic" across the BLS journal transcript dataset, I identified 120 submissions across 16 subjects that explicitly discussed feeling manic. With more advanced computational methods, it would be possible to more exhaustively characterize mentions of mania over the study and in a participant-specific context while remaining strictly quantitative. The crucial point here is that this wasn't especially rare. Not all BLS participants experience mania at all, and not all BLS participants regularly submitted quality audio journals, so to find 16 distinct subjects is promising. Moreover, the average number of records using "manic" per participant worked out to $\sim 2.5\%$ of the average number of total records submitted per participant that engaged well with the audio journal collection longitudinally – rare enough to be potentially salient but more common than manic episodes officially documented in the hospital's records.

Notably, the above quote gives further background about the participant and the role of their mother in both their life and in specifically managing their Bipolar disorder. The full diary contains a number of additional thoughts the patient chose to share that day, which can be extra interesting when taken together with their statement about hypomania. This exemplifies the many ways that diaries can be relevant. Fascinating clinical notes could probably be taken on the submissions of 8R without external information sources about the subject, and there are also very explicit mentions of psychopathology that can be directly counted.

The use of extracted features like sentence sentiment has the potential to identify diaries that can be evaluated in a clinically-established way as well. While considering automated sentence sentiment outputs as a form of ground truth may not be appropriate for many studies at present, a reading of the associated journals could certainly identify established signs of psychiatric pathology. The following sentences are just one example of the extreme statements found within many of GFNVM's lowest sentiment journal submissions, per the pipeline:

I hate my brain.

I hate my thoughts.

If someone chooses to make these statements when asked to recount their last 24 hours, how is that not inherently clinically relevant?

Related arguments can be made for a number of foundational passive sensing features; however, there is much room for nuance in the extent to which e.g. decreased physical activity or increased time at home is directly usable as a marker of psychiatric illness severity. Certainly these phenomena could be explained by external factors, and do occur at times in perfectly healthy individuals. While some such features have been found to correlate with e.g. depressive symptom ratings in psychiatric illness patients, and they do have direct links to overall health outcomes, it is not yet clear how they fit into models of psychiatric pathology specifically. This is quite distinct from the sort of statements made in the above quoted GF journal. It would be extremely difficult to find a presently mentally healthy individual, by any clinically accepted standards, who would record a daily free-form audio journal that amounts to a handful of such sentences. If submitted in good faith it is hard to imagine that making these statements would even be definitionally compatible with being psychiatric symptom free.

These examples of course do not address the big inverse limitation – that even subjects who participate a lot may intentionally try to hide symptom severity in the content of what they say, or may have symptoms that cause them to submit journals with less information (or not at all) when more severe. In these cases, the journals (or lackthereof) can still be a powerful input information source, but cannot serve as a label in the way I am proposing. Still, trends over time and relationships between features in the patient daily diaries can be quite interesting without any clear clinical label available.

To be clear, it is absolutely the case that for the field at large to build complete models of journal features in the context of psychiatry, some studies must obtain clinical ground truth labels in a more typical way that is indeed more robust presently (though also not perfect). Likewise, some studies must obtain e.g. neurobiological features in order for mechanistic models to improve. These facts do not mean that all studies must do so however.

In summary, audio journals are highly versatile in the value they can provide to different studies that address different psychiatric questions in different ways. They can simultaneously improve tractability of speech sampling work and enable passive sensing work to generate psychiatric insights that are outside the scope of the most established clinical structures. They can simultaneously be an interesting supplement to nearly any digital psychiatry data collection project and serve as a promising primary data source for hypothesis-driven work. They can even be fascinating as an entirely standalone dataset, to an interdisciplinary audience no less. They are severely underutilized currently in my opinion, not helped in this matter by systemic structures. As such, I will next discuss the role of software infrastructure development in enabling audio diaries, and really all digital psychiatry datatypes, to reach their full potential.

§.§.§ A need for quality software infrastructure

A major theme of chapter <ref> is the importance of good software infrastructure for organizing data, maintaining quality standards, and monitoring progress across groups in a large collaborative data collection project. This discussion occurs in the context of interview recordings and more specifically the infrastructure I developed for interview data collected in the AMPSCZ [NIMH, 2022] project, but the same points apply to audio journals.

Given the lack of existing audio diary analysis literature, as well as the daily data flow expected of the format, good data management infrastructure is even more critical to develop for journal research, if the datatype is to leveraged as it should. While the code released with this chapter was written for processing of studies across the Baker Lab, it can serve as a foundational starting point for a more general open source audio diary processing pipeline that could serve the broader scientific community. Potential future software for the journals being collected but not yet processed by AMPSCZ is in fact covered in this chapter, in section <ref> below – including a list of steps necessary to transform the diary code of this chapter into a central tool like the interview code of chapter <ref>.

Purely in the realm of data management and quality assurance, there are indeed a number of benefits conferred by using software like my pipeline, many of which were demonstrated on the single study scale in the process of collecting the BLS dataset (see section <ref>). These benefits can directly lead to larger datasets of better quality, while simultaneously cutting down on wasteful expenses. Example functionalities of the pipeline are as follows:

* Identifying participants that are intentionally submitting garbage recordings to game the compensation system.

* Automatically filtering out recordings of unacceptable quality prior to spending money on transcriptions or including truly unreliable features in analyses.

* Intervening quickly if subjects have a recurring technical issue with Beiwe (or other recording app).

* Noting common types of quality issues and updating participant instructions accordingly.

* Monitoring and comparing progress across multiple studies in parallel - of particular interest when managing data collection across multiple different locations like in AMPSCZ.

* Flagging suspect transcripts that may arrive to ensure TranscribeMe is maintaining their good quality standards over a long time period.

Note also that due to the particularly sensitive nature of personal audio recordings, it is important that privacy and security standards are upheld when managing these data, another reason that well written core infrastructure for speech sampling research should be a major priority.

Especially as more big data collection projects like AMPSCZ are funded across different disease areas, good centralized software for managing audio journal data would pay rich dividends. Such projects are great for building large datasets that open up additional analysis options not feasible for individual studies; imagine the interesting directions that could be pursued with a dataset of similar longitudinal depth as the 25 or so consistently participating BLS subjects, but constructed from 1000 similar subjects across 40 different international locations rather than 25 at 1. This would lead to a dataset with strong power for many more complex features to be utilized in within and between subjects analyses. It would also enable interesting questions to be asked about generalization across cities, countries, and even languages, while as many data collection related factors as possible were held constant.

However, the practical success of this style of project - and especially the efficiency of repeating similar processes for different disease areas - relies on having good infrastructure in place, for two distinct reasons. The first is that such broad exploratory work has amorphous aims and a variety of researchers that will hopefully one day take advantage of the dataset. Without a clear goal to carefully evaluate against, it becomes necessary to make the clear goal for the collected data to be of the highest quality in all regards possible, thereby allowing for a large number of analysis methods to be employed downstream without issue. The second reason is that in order to recruit many collaborating sites for multimodal psychiatry data collection work at scale, there will need to be many researchers involved in building the dataset who have no prior expertise in speech sampling. Similarly, there will likely be data collection occurring in places with different cultural and legal considerations surrounding speech sampling, not to mention the very obvious relevance of different languages to speech data in particular. Therefore in order to maximize participation as desired without forgoing quality, it is essential to have robust and user friendly software for data management and monitoring.

Because audio diaries have the advantage of being a technically very similar datatype regardless of the disease area considered (more like wrist actigraphy than clinical interview recordings), well architected software would be extremely straightforward to extend to new projects. The question then becomes whether an appropriate initial investment is being made into ensuring such software is written, sooner rather than later. The fact that a single grad student working part time on the project has been responsible for the bulk of the interview recording software infrastructure for AMPSCZ to date, and a plan for the journals has not been fully settled yet almost a year into production data collection, certainly does not bode well.

Unfortunately, this is only the tip of the iceberg. At this time, feature extraction software has not been implemented in any usable form for the AMPSCZ interviews. The arguments on software leading up to this point have focused on dataset management, which is a crucial component but is not sufficient for the project to run smoothly, especially because sharing of raw participant audio recordings is subject to understandably strict guidelines that exclude even most of the researchers involved in data collection and study design from accessing the larger set of them. There are additionally a number of methodological arguments for the availability of open source digital psychiatry processing pipelines that output core sets of validated features from raw inputs. It is unclear why the larger group has not yet come to a conclusion on some sort of base feature set, but it has made clear to me that the need for this sort of software design is much greater than I originally anticipated.

Establishing common foundations for the next era of psychiatric speech sampling research.
An indirect drawback of the rapid rate of progress in machine learning - and most notably here natural language processing - is the difficulty in building a cohesive knowledge base of rigorous applied research results while simultaneously not allowing applied research proposals to fall too far behind present technological capabilities without good reason. There are many factors in play contributing to this issue, including:

* It is challenging for many interdisciplinary groups to properly assess when a new tool is well-suited for their dataset, because that often requires keeping up to date with bleeding edge computational research. At the same time, there is career pressure for many in this capacity to quickly move to the latest tools.

* Exploratory works are much less likely to be followed up on when they utilize methods that will rapidly be replaced and subsequently require their own exploration – thereby destroying much of the possible value of these underpowered characterizations.

* On the opposite end of the spectrum, many established psychiatry groups will not have the technical background from even a software perspective to run more recent methodologies. Therefore much traditional hypothesis-driven work still relies on only older methods with more user friendly software interfaces (e.g. LIWC).

* With minimal established benchmarks for digital psychiatry, there is little consistency in what methods exploratory papers compare against, and there is not a good common language shared across the field that would allow works to build on each other in a sensible manner. This problem is further exacerbated when it comes to the critical task of communication that can bridge groups with clinical versus computational expertise; presently, their aims seem to fly past each other. It might relate as well to the relative paucity of pure machine learning experts attempting to wade into the digital psychiatry space, as opposed to other difficult but currently more tractable real world applied problems.

* Maintaining good practices for collaboration is not only an issue with analysis methods, but also with clear specification of data collection protocols. For example, structured clinical interviews and open ended interviews (discussed more in chapter <ref>) certainly have shared properties of interest, but they still diverge in ways highly relevant to analysis too – yet there are not many cohesive standards in the field for planning nor communicating data collection protocols around relevant data science considerations. This is again a major pain point when it comes to interdisciplinary communication, because psychiatrists do not often have the necessary technical background to identify seemingly small details that are important for software design or machine learning outcomes, while computer scientists do not often have the necessary clinical background to identify when they are abstracting away details of high psychiatric importance or focusing on features that are relatively unimportant. Without a solid framework for communicating on such matters, both groups will remain frustrated.

* Many of the most impressively accurate modern machine learning techniques are at this time difficult to interpret, make very different types of mistakes than classical data science techniques, and have greatly benefited from access to massive datasets for training and evaluation. Medical applications on the other hand have unique considerations in all of these domains, and for psychiatry in particular there is limited availability of clean labels. This is something I will discuss at greater length in the background for chapter <ref>.

* A lot of the above points are compounded by the challenges with data sharing in psychiatry. Due to the sensitive nature of patient data and the lack of agreed upon benchmarks or even foundational feature extraction methods for deidentifying data, published results are generally in broad terms that do not allow future works to make comparisons such as identifying distributional differences in core input properties or included clinical labels. It is therefore especially challenging to build on exploratory results outside of the group that performed them, and groups that publish exploratory works are not always interested in pursuing this style of follow up work.

As noted, many of these concerns apply to digital psychiatry at large. However, most of them are even worse for speech sampling than for other datatypes. Patient speech datasets are generally much smaller than passive sensing datasets, they are much more sensitive when it comes to data sharing considerations, and they involve multiple quite different modalities that warrant different expertises – acoustics, linguistics, and clinically relevant patient subjective reporting.

Of course shifting speech sampling data collection more towards the audio journal format would be a big step towards closing some of the interdisciplinary gap described, but we also very much need more established software tools to fully address concerns. We need these tools across digital psychiatry, but I am most concerned with audio journals in particular, because of their high potential upside and their surprising lack of usage to date.

The pipeline I documented in this chapter already lays some initial groundwork towards addressing such audio diary analysis concerns. This includes pilot feature validation work, careful characterization of output distributions in a relevant dataset, and an exhaustive detailing of ways the pipeline can presently be used and ways it might be built on in the future. Perhaps the most key component of the feature extraction code itself though is simply the choice (and justification) of specific core metrics that others can now more easily run on any audio journals they collect. My hope is that this will serve as a foundational light weight yet relatively modern feature set to bridge results from various future studies with distinct aims.

More importantly for the future of the field, the work in this chapter ought to be followed up on in spirit, whether intentionally or not. There is great need for accessible and well validated software that can provide a common baseline methodology across psychiatric speech sampling research in this increasingly digital era. If lead and maintained by a strong and committed software group, such a code base could become a place where the research community is able to contribute newer features they think have particular promise, and users in turn can help to further validate such features, thereby regularly expanding the common language of the core feature set.

Though it is far from perfect, websites like PapersWithCode have made it easier for groups with different academic backgrounds from around the world to find papers that tackle a particular benchmark machine learning task, and to easily find information on the set of existing popular tasks that attempt to measure a particular type of learning challenge. Not all research does or should fit into such a box, but when a field attempting to utilize machine learning lacks any sort of organization like this it is natural, especially with modern academic incentives, that published research progress resembles a treadmill.

Of course, many machine learning benchmarks can be run by following a simple Python package tutorial, while processing digital psychiatry datatypes in different ways often requires work from scratch. This is exactly the reason that a good open source code base is needed to not only establish baselines, but facilitate them for a variety of researchers. A properly maintained pipeline would greatly reduce the burden of integrating features that showed promise in prior exploratory works into new tests, to combat the problem of perpetual exploration.

Along the same lines of the preliminary visualization functions provided in my pipeline, tools to assist in clinical evaluation and qualitative hypothesis generation can be part of such an open source project as well, helping computationally-inclined researchers to learn more about what clinicians are looking at. Furthermore, when integrating such a pipeline with data collection apps like Beiwe or MindLAMP, collection procedures could be more standardized by default, so that groups will gain a common understanding on important data considerations both clinical and technical. If one does not have a good reason to reinvent the wheel, they shouldn't.

Were such a pipeline to come to fruition, it would also open up the possibility of sharing basic distributional properties across the research community without sharing anything of a sensitive nature or anything that might be considered proprietary by the research group. As state of the art features are improved upon, the core feature set can still remain of strong scientific relevance, as demonstrated throughout this chapter. All research could indeed be improved by sharing a more detailed understanding of such core features, in parallel with development of increasingly complex algorithms on customized datasets.

Developing such software well in the modern academic system will likely carry a high upfront financial cost. However it would pay great dividends to do correctly now, both in terms of true scientific progress and in terms of the cost of data collection of future related studies. These costs are discussed at more length in the context of the large collaborative AMPSCZ data collection project within chapter <ref>. I also review the state of academic software in broader terms in the concluding chapter, along with potential social factors that might be addressable independently of funding available for programming contributions.

Ultimately, if psychiatry wants to really take advantage of technology, it cannot refuse the cost of software development, whether that is paid in terms of a restructuring of the academic credit model or in terms of cold hard cash. You get what you pay for, and bad software requires more maintenance and more monitoring, and it is more difficult to adapt it to new situations or add new features. It is also more difficult for others to use it even if they don't attempt to change anything. A lack of good software now will cause research undertakings to bleed money later, if there is any true belief in a digital future of psychiatry. Then again, a lack of commitment might turn into a self-fulfilling prophecy in the upcoming decade, because it will be challenging to overcome all of the systemic problems mentioned without one.

§.§.§ Leveraging the broad relevance of human story telling

Regardless of how the current brand of interdisciplinary digital psychiatry plays out, it is extremely probable that at some point in the future, some form of digital psychiatry produces truly important results. I expect a frequently collected autobiographical format like the daily audio diary to play a role in such successes, and in particular to help actually bridge the interdisciplinary gap. Journals can be very useful for both "sides" for all the reasons detailed above. There are also interesting philosophical links between journaling and fundamental properties of human cognition - some of which were instrumental in the early days of AI research.

Story understanding in models of human intelligence.
Many believe that story understanding is a key component to human intelligence and communication. A key hypothesis of [Winston, 2011] was that:

The mechanisms that enable humans to tell, understand, and recombine stories separate human intelligence from that of other primates

and that these mechanisms ought to be a source of inspiration for making advances in AI.

As [Winston, 2011] points out, education is full of stories throughout development, from moralizing children's tales to the case reports studied in law and medical school. Indeed, the salient things that people remember and later utilize are often story based. You will get most people angrier with a well-told story of one injustice than statistics about thousands of them. Symbolic reasoning, a pervasive theme in early AI results, can be closely linked with story composition and with the human propensity for creating art [Winston, 2011].

Based on evidence that a variety of types of arts emerged rather suddenly and subsequently spread rapidly $\sim 60,000$ years ago, paleoanthropologists have hypothesized that a large nonlinear effect on intellectual capabilities arose when humans became symbolic, setting them apart from prior humans with close structural similarities as well as other primates like Neanderthals [Tattersall, 1998]. The ability for humans to efficiently share knowledge through symbols and to generalize via their composition, sharing increasingly complex information, enabled a social evolution of sorts perhaps – spread through many forms of story telling.

Story telling perspectives within scientific practice.
In some ways, the human story telling method of learning and teaching can lead to irrational "normal" behavior, and subsequent conclusions that are wholly unscientific. At the same time, scientific results can be reported effectively in a very dry way that would be a stretch to call a story. Still, that does not mean that the story telling perspective should be avoided in science, as it sometimes seems. Human cognitive biases are not universally bad, and can in fact be a source of great insight at times; a source that might become increasingly important to take advantage of as "by the book" thinking can be increasingly automated. They can also be leveraged to improve communication of science, a requirement for good ideas and strong results to actually become used.

This is especially true for interdisciplinary communication, because as much as scientists may want to believe they are substantially more rational than the general public (and maybe they are), it remains difficult to fight human instincts in an alien field. In 1963, Edward Lorenz published "Deterministic non-periodic flow" to a modest response restricted to his own field of meteorology. With minimal addition to the underlying mathematical principles, his work took off to become an entire interdisciplinary field of study - chaos theory - only after he told a silly story about it at the 1972 meeting of the American Association for the Advancement of Science, at the urging of a colleague. The title of that talk of course was "Does the flap of a butterfly's wings in Brazil set off a tornado in Texas?" [Lorenz, 1963, Lorenz, 1972].

The story of Edward Lorenz was originally told to me as an example of the irrational memetic behavior of scientists that apparently transcends generations. It is silly in many respects, and it can lead to some legitimately bad outcomes when bad science is spread in a similar fashion. Yet at the same time, it is not unreasonable to me that the work didn't spread until the story came along. The story made the central theme of the work clear – that there exist deterministic systems for which very small perturbations we cannot reliably measure are capable of producing largely different and thus seemingly random outcomes. It made it clear in a way that would be memorable for the rest of someone's career, in case something relevant ever came up. It also made it clear in a way that could be immediately paralleled (in abstract terms) to many other possible scenarios across a range of fields, regardless of one's prior exposure to nonlinear stability theory, which is a great way to inform hypothesis generation.

The audio journal datatype can tap into this style of field-transcending communication, because it provides source material both for detailed quantitative analyses and for compelling story telling about the results of such analyses in subject(s) of interest. Journals can in fact generate a huge variety of stories because they are a series of stories themselves, which in my opinion has important implications for the future of hypothesis generation in psychiatry. Moreover, the relevance of story telling and story understanding in human intelligence unsurprisingly links back to some psychiatric disease states, making the study of patient driven open ended self-reporting of direct (and not just meta) relevance on this topic.

Story recall abnormalities in psychotic disorders.
Given the hypothesized uniqueness of story telling to human intelligence, Schizophrenia may be of particular note for its potential uniquely human angle. To our knowledge, Schizophrenia is not found in other animals, and in fact many of the genes implicated in the disorder are found in human accelerated regions, the parts of the genome likely associated with human evolution [Insel, 2010, Xu et al., 2015]. If stories are an important part of human nature, and Schizophrenia manifests uniquely in humans, then exploring story telling in Schizophrenia would be a research direction of broad interest. As can be seen throughout this chapter, many audio journals recorded by psychotic disorders patients do take the form of a story, and thus may capture interesting properties of patient story telling over time in an exploratory fashion.

Delusions found in Schizophrenia patients are very often self referential, and commonly contain themes of persecution and thought reading and control [Gold and Hohwy, 2000]. As such, autobiographical story telling like what is captured by audio diaries could be especially interesting. Moreover, general story understanding deficits have also been found in Schizophrenia, as patients will rate unlikely explanations as more likely than controls do, and will have a harder time dismissing contradicted hypotheses than controls do, when presented with basic stories in research tasks [Moritz and Woodward, 2006].

Memory of stories also appears to be impaired in Schizophrenia patients, as demonstrated by performance on story recall tasks where details were poorly remembered and concepts were incorrectly mixed across tasks [Matsui et al., 2007]. [Hoffman et al., 2011] hypothesizes hyperlearning as a key mechanism underlying the story understanding and recall deficits and ultimately the psychotic delusions observed in Schizophrenia patients, based in part on biological evidence of hippocampal over-activation and over-release of dopamine in Schizophrenia, both of which are crucial to learning and reinforcement. Even though patients may not remember many details, the details they do learn may be extended across different stories that should be kept distinct. Hyperlearning in the context of an artificial neural network architecture indeed resulted in the over application of learned concepts in their computational experiments to test this hypothesis [Hoffman et al., 2011].

As diaries enable extended longitudinal data collection, one might consider an analysis of events that are repeatedly mentioned or of how recurring concepts are used by the participant over the duration of the study. A participant-directed autobiographical datatype that can be feasibly obtained relatively frequently for multiple years lends itself to untold potential subjective discoveries about a person, while at the same time presenting opportunities to be connected with a number of potential mechanistic studies in cognitive science and neurobiology, especially when collected in conjunction with other data modalities.

§.§ Summary of pipeline release

Given the salience of the audio journal format and its lack of attention to date, coupled with the general need for common foundational infrastructure in digital psychiatry, there is great potential impact for the development of comprehensive audio journal processing software. In this chapter, I took a large first step towards that end, by both documenting my diary pipeline (section <ref>) and reporting on early scientific results found through the use of the pipeline's outputs (section <ref>). In this discussion section, I will review key takeaways on the capabilities of the pipeline for assisting an audio diary study, from its role in the data collection process through to final statistical analyses. At a high level, the primary functionalities of the code are represented in the following list.

* Data management:

* Recognize newly available recordings as they come in, handling decryption of files where needed and ensuring sensitive files do not remain on the server unencrypted for an extended period.

* Parse Beiwe and subject metadata to assign audio files to the correct study day (real dates removed) with submission timestamp in correct timezone.

* Transfer audio files that satisfy transcription settings to TranscribeMe.

* Track pending transcriptions and pull back completed transcripts when done, converting to CSV format.

* Maintain standardized folder structure and file naming conventions for all outputs.

* Send monitoring emails to study staff with information on newly processed (audio and transcript) files per subject ID upon each run.

* Quality control:

* Compute lightweight quality measures for all recognized audio diaries, with the option to make transcription decisions based on these values.

* Analogously compute basic transcript quality measures on all returned transcripts - introductions to both audio and transcript QC can be found in section <ref> above.

* Send monitoring emails to study staff to notify on relevant QC metrics for newly processed diaries, also optionally integrating with DPDash to easily monitor QC values over time via web app as desired.

* Acoustics feature extraction:

* Run OpenSMILE to obtain low level descriptor acoustic features from the GeMAPS config.

* Run voice activity detection algorithm with Librosa, to subsequently label pause times in the recording and derive diary-level features from that.

* Use pause times to create filtered copies of the low level OpenSMILE features, also computing diary-level summary stats from OpenSMILE outputs.

* Linguistics feature extraction:

* Leverage TranscribeMe notation to count occurrences of disfluency by category in the transcript via simple regular expressions.

* Run VADER sentiment on each transcript sentence to calculate per diary sentiment summary statistics.

* Employ the Google News word2vec embedding model to compute measures of incoherence (within sentence and between) as well as word uncommonness.

* Retrieve basic QC data of potential relevance such as word and sentence counts, optionally add counting of specific keywords.

* Count syllables in each sentence using NLTK, combine with TranscribeMe sentence timestamps to generate an estimate of speaking rate from transcription alone.

* Visualizations to summarize features:

* Create up to date histograms and correlation matrices showing the latest properties of select diary level features across the dataset.

* Create heatmaps showing progression of select features over time in each subject along with clear picture of data missingness.

* Create spectrogram images and sentiment colored word clouds for each individual diary - examples of these and all other visualization functions built into the pipeline by default can be found in section <ref> above.

Note that a major advantage of using the pipeline for feature extraction is not only that it packages together a number of tools of interest, but also that the selection of these tools can serve as a common core of features to be benchmarked against as new methodologies are developed. Within section <ref>, I provided a review of prior validation literature for each applicable tool, and justified its inclusion in the pipeline over other choices. The features included are meant to be feasible for any lab to run and approachable for interpretation, while still maintaining a fairly high quality standard. Thus the pipeline can simultaneously be used by psychiatry groups with minimal technical background as the primary resource for evaluating a number of scientific questions and be used by hybrid groups with a machine learning focus as a comparison point for new techniques – thereby helping to reduce the large amount of unorganized and at times unscientific churn that currently happens with tech exploration in the digital psychiatry literature.

Perhaps more importantly, I have provided preliminary validation results for a number of the included features on our large Bipolar Longitudinal Study (BLS) journal recording dataset, along with a nuanced characterization of the distributions of and relationships between the most key features derived from the pipeline, across BLS and in terms of participant-dependent variability. Because the entire BLS data collection process would be replicable by using the Beiwe app for diary recording and TranscribeMe for high quality professional transcriptions, these results are directly relevant to a number of researchers that may be interested in utilizing audio journals in the future. Additionally, data monitoring and quality assurance pipeline capabilities can improve the final dataset collected over what would be obtained from using Beiwe alone.

To facilitate future pipeline applications, I will next review the results of our work validating the pipeline features (<ref>), and then I will summarize and discuss the feature characterizations and other scientific results obtained thus far from the BLS pipeline outputs (<ref>). Software implementation details will not be included in these discussions, so please see supplemental section <ref> for exhaustive code documentation if needed.

§.§.§ Validation overview

In section <ref>, I explained how the major code design decisions were made and reported on our confidence level in various pipeline outputs based on experiences with our lab data to date. <ref> covered the quality control metrics, <ref> covered the acoustic extracted features, and <ref> covered the linguistic extracted features. Where applicable, I reviewed prior literature on the accuracy and relevance of each tool chosen to be integrated into the pipeline, including comparison to potential alternatives.

I also reported on how well different pipeline outputs aligned with the results of manual review performed by lab members, as well as sanity checking of expected relationships between certain features. In summary, the QC tools provided by the pipeline are not perfect, but are a large improvement over the previously non-existent options for active monitoring of audio journal study submissions. They repeatedly demonstrated the ability to improve our final BLS dataset and save project money in practice, by filtering out bad uploads prior to issuing to TranscribeMe and detecting participants regularly submitting low quality diaries (whether accidental or gaming the compensation).

For statistical analyses, there was strong evidence for the continued use of linguistic disfluency, semantic sentiment, and pause detection features, as well as the various qualitative functions provided by the pipeline to assist in interpretation of journal content. These features were internally consistent with expectations across the sizeable diary feature set, in addition to passing pilot manual review checks for agreement with human-provided ratings. Furthermore, the potential clinical relevance of basic verbosity and diary submission metadata features should not be underrated due to their technical simplicity. Ultimately, the scientific results to be recapped shortly are another very strong testament to the quality of the reviewed pipeline features. There is certainly room for expansion and improvement, but the aim to establish a solid basline has been accomplished.

On the other hand, while the semantic incoherence NLP features and vocal properties extracted from OpenSMILE have been well validated on technical benchmarks in the past, they were more difficult to assess in our hands. Semantic incoherence outputs certainly showed promise, but were also particularly susceptible to some sources of noise, and were difficult to even define expectations for given the general lack of severe psychosis symptoms seen in the BLS dataset. OpenSMILE outputs were not closely investigated in this thesis, as technical details of acoustic signal processing were beyond the scope and the high number of features coupled with their high temporal resolution would make a data-based evaluation a more involved project. Given that GeMAPS OpenSMILE features have been used by many other studies, there should be no problem in using the low level pipeline outputs in a future study, but it is less clear how much value the summary features carry. Regardless, the primary interest with speech sampling here is in linguistic analyses, which will again be a major theme with the interview recordings in chapter <ref>.

Limitations of the pipeline at present are covered in more depth in the documentation of supplemental section <ref>, but for the most part the pilot validation evidence for the included features and other functionalities was strong, providing further support for the use of my code in future works. Still, any study using any open source code should of course perform their own confirmation that measures they intend to use are working as expected. Outside of technical setup discrepancies between research groups and potential undetected bugs in the code, there very well could be certain study populations or specific psychiatry research questions that the features as is would be especially poorly suited for. Given the simplicity of most included features, their utility should for the most part generalize – which is an important additional advantage to this code. Still, some underlying messages remain the same through much of this chapter, including how crucial it is to check for unexpected patterns (whether good or bad) within the details of any research dataset

§.§ Pilot scientific results recap

One major contribution of my work was simply characterizing the primary feature set in a large dataset of journals from patients with psychotic and mood disorders, with a focus on Bipolar disorder. The dataset size in itself was a powerful argument for audio journal speech sampling, with over 10,000 high quality transcripts obtained from across 66 consenting BLS participants. 25 of the subjects submitted over 100 journals with a length above 15 seconds during their time in the study, and some subjects submitted hundreds of journals that were well over 2 minutes and/or continued to submit journals well beyond 2 years of enrollment. As reported along with other participation dynamics in section <ref>, within the first year of enrollment for those 25 participants, there were 2868 data points that had both same day and next day quality transcribed audio of duration $\geq 15$ seconds – so even for tight timescale temporal prediction modeling we obtained a healthy dataset size. Engagement (as measure by word count) did not drop off over the course of the first 2 years of submitted diaries in the main participant set either. This bodes very well for the feasibility of collecting large and rich journal datasets across people and time, and it was clear from recorded content that some subjects truly enjoyed recording the journals.

Scientifically, the distributions (<ref>) and correlation structure (<ref>) of features produced can inform future works on feature selection and model design process, and can ultimately contextualize features that are obtained by other groups and from different study populations. Importantly, there were highly significant distributional differences in select features across all subjects highlighted – including large differences in features like diary word count and mean words per sentence that can impact distributions of many other extracted metrics, and thereby downstream modeling. At the same time, there were also significant distributional differences in downstream features like disfluency usage or sentence sentiment that were not clearly attributable to differences in the considered verbosity measures. These results highlight just how important it is to carefully consider within versus between subject effects. While a large scale study like AMPSCZ may produce a diary dataset of sufficient size to investigate between subject effects clinically, it is difficult to say much about this with the BLS diary set. Not only was the total number of engaged patients relatively small, but also there was a great deal of diversity in diagnosis and other demographics amongst them. Many feature differences between subjects were themselves quite stark and convincingly real, in the sense that the participants are truly different in how their journaling speech affects even relatively simple features over time.

However, there is no way at all to confidently comment on clinical relevance of any between subjects differences in the present work. This was further exacerbated by the lack of control subjects, though there was good heterogeneity of clinical presentation amongst the considered BLS participants. Differences in pause patterns or use of uncommon words or of linguistic disfluency rates (and so on) can occur between individuals for reasons unrelated to pathology, and a large number of subjects would be needed to distinguish. Still, the distributional differences found are critical for a number of reasons. Subject-dependent feature variation might help to identify those metrics that are most salient for each participant's within-subject modeling, before invoking any ground truth labels – though this remains an open question and is probably highly dependent on the feature and the difference. Additionally, relationships between features and pathology within a patient need not have the same direction of effect, something else it may be possible to glean information about from patients' individual distributions. Perhaps most importantly though are the benefits for planning and interpreting models. The shape of the input feature distributions will inform parameter choices, as will the total amount of variance. Correlation between different input features is important to characterize as well to facilitate explainability or possibly reduce the dimensions of input features in an unsupervised manner. This too demonstrated subject-dependent differences in our pilot results, at times in opposite directions.

A second major contribution of my work was to model EMA scores from same day diary features in a smaller subset of participants (to preserve the dataset for others' future analyses), thereby identifying features with grounding in self-reported mood on the diary timescale, both overall and for different individuals (section <ref>). While ultimately simple linear modeling work, this again highlighted a number of important takeaways for future diary work. Additionally, the strong overlap found between diary and EMA response rates in patients engaging with diaries emphasized just how tractable app-based journal modeling can be, even in such a small subset of patients.

The many highly participant-specific effects mentioned above, which extended to patterns in EMA labels of wildly different periodicities and typical self-report severities, demonstrated how important it is that digital psychiatry works properly account for many distinct and interacting explanations in their modeling, or at least show the data carefully enough such that the community can properly judge the origins of significant correlations. For a simple example from this chapter, a linear fit on diary word count alone was able to explain more than $15\%$ of the variance in same-day positive EMA summary score in the selected subset of BLS participants, with off the charts significance. Yet when word count was first normalized to the patient's own baseline via z-score, the relationship went entirely away, dropping to barely $1\%$ variance explained and no significance. It turned out that word count was very different between subjects, as was the distribution of EMA summary scores. The fact that this moderately correlated in such a small total population ($n=7$) does not say much about a real in the clinical sense relationship, and it does not really matter how many diaries each of those individuals submitted.

Conversely, diary word count did have an impressively real clinical relationship for one particular subject (5BT65), with a linear fit to word count alone explaining $24-33\%$ of the variance in their longitudinal EMA summary features and persisting in hold out test set as well checking out with the eye test. Further, it was likely from visualization that a nonlinear and/or rolling beta model would have even better captured their EMA variance. Looking at longer timescale diary feature fluctuations, the correspondence of word count with depressive episode periods in this participant was remarkably strong. Within the same dataset however, most subjects had minimal relationship between EMA summary and same-day journal word count.

Interestingly, one subject (8RC89) had a modest and highly significant linear relationship between EMA and word count, but with higher word count corresponding to more severe self-reported negative EMA symptoms, thereby contrasting with the participant having a highly significant connection between fewer words and more severe self-reported symptoms. This may be explainable by differences in their pathology, as 5B experienced primarily severe depressive episodes, while 8R experienced clear manic episodes at times, and may have been inclined to record longer diaries when e.g. irritable as opposed to a depressed person that may record shorter diaries when they are lacking energy. Indeed looking at feature timecourses in 8R, strong self-reported emotions of both positive and negative valence coincided well with clinical markers for hypomania symptoms. Regardless, it is telling that even in this very pilot exploration examples were easily found that would benefit from hierarchical modeling that includes personalized components for different participants.

Other diary features also contributed significantly to explaining variance in EMA summary scores, with sentiment in particular sharing significance (and unsurprisingly directionality) in relationships with EMA across subjects, even when normalizing features independently within each subject via z-score. This provides strong validation for the use of the VADER sentiment tool in the pipeline. Pause and disfluency-related features demonstrated relevance within some particular subjects as well, and may be even better utilized by a nonlinear model based on properties of their underlying distributions. The features that the scientific analyses showed the least overall promise for were the word2vec-derived ones, though it is worth noting that I focused almost exclusively on participants with heavily mood-related symptoms, so the pipeline incoherence features remain to be properly tested on cases with clear positive psychotic symptoms.

The final major scientific impact of this chapter is through the proof-of-concept case reports of section <ref>, not because of the results directly, but rather because of the simple ideas they consistently demonstrated about the relevance of multi-timescale dynamics and the power of even the simplest qualitative tools to utilize diary content for the improvement of quantitative understanding. Examples include:

* Using word clouds generated over periods of unexpected diary feature changes in 3S to identify topics of relevance - it turned out that a transient disruption to a potential relationship with linguistic disfluencies could be matched with a series of diaries that discussed a temporary attempt to cut back on medication as well as its subsequent failure over an $\sim 1$ week period.

* Automatically accounting the usage rates over time for clinically-relevant words like mentions of mania in the 8R diaries. By narrowing them down in this way, I found 2 entries with compelling statements related to active mania, and when adding those points to the markers of high clinical symptoms, trends in diary and EMA features potentially related to mania became much more in line with expectations.

* Characterizing variance directly at the different timescales and in the different (relevant) datatypes in which it occurred in the 5BT65 dataset, not only identifying striking examples for the utility of EMA in future psychiatry research, but also finding new compelling relationships between diary content and onset of the major mood episodes in 5B. Specific phrasing was found to be highly specific to only the time periods directly before long-term mood episodes identified in 5B, and could flag these days before reliably detectable changes in EMA (in a highly patient-specific manner of course).

The overarching theme across these results is to think carefully about different modeling timescales that could capture different levels of disease-relevant variation, and to take advantage of the iterative process that diaries can enable between content search and statistical analyses.

§.§ Future directions

Throughout the background, validation work, results, and now discussion, I've proposed a variety of broad ideas on ways in which diaries could be used for future digital psychiatry research, as well as suggested a number of specific improvements that could be made directly to the current core pipeline workflow I've described. In this section, I will go into more depth about future applications and new features of particular interest. Broadly, the audio diary pipeline could be directly applied to a wide variety of clinical (or other behavioral) contexts, as free-form journaling prompts can be of interest across a range of disorders and short daily recordings on specific prompts have a wide set of options and use cases that would fit with the current code too. This might include the more traditional clinical use of journaling as a therapeutic intervention.

One major direction suggested by these pilot results is to focus more on fully individualized modeling using these longitudinal data, but considering a larger sample of participants in which the goal is to have a procedure that meta-generalizes. In other words, it is not a prerequisite that any particular relationship is true for a larger set of people, as long as the models generalize well enough over time in the respective subjects and the method for finding those models generalizes to many others. If it indeed would be possible to identify relevant features in an individualized fashion from a larger base feature set using rolling models, one could imagine making patient-specific predictions in real time at relevant points over the course of a study and seeing how accurate they turn out to be. In practice, clinicians would (ideally) update their ideas about a patient over time and may make predictions about e.g. what subtle signs indicate a particular patient might be trending in a particular direction. The fact that research tends to have a single stop point with analysis at the end is a side effect of the very specific model for scientific results publication that is used system-wide. In that model, it is indeed poor practice to check many hypotheses over time, because targeted stopping is an easy way to p-hack. But there are plenty of other ways to quantify accuracy of continual predictions – one might look to literature from the online learning subfield of ML or to financial market model testing practices. Of course, these are largely new problems in the psychiatry research space, because up until recently such longitudinal work was hardly feasible. Unfortunately, analyses of digital methods to date have concentrated too much in the psychiatry research style of evaluation; with the paucity of serious audio journal work, there is hardly anything like this at all in the speech sampling space especially. A greater diversity of methodologies to cover the limitations of each other will be critical to future success.

Of course, if it were possible to learn the behaviors of many participants with a similar overarching method, that could then go a long way in early intervention studies, clinical trial evaluations, and so on. Less obviously (but still critical), it would also be very useful for neurobiology research. A very high level major open question is the extent to which patients with clearly different symptom manifestations in the same disorder are classified incorrectly versus a mechanistically similar underlying pathology causing different behavioral results for non-pathological personal reasons. While it is highly likely that there is quite a bit of bad classification with current disease labels, it is also highly plausible that the same biological mechanism could cause vastly different responses when other non-clinical biology or the outside environment are drastically different. Using individualized longitudinal models, we could have behavioral signal that is both much more appropriate for neural data timescales than traditional scales and much more clinically grounded than a generalized model based on current disease labels could hope to be. This would enable comparing neurobiological signals at times of different model output, regardless of what behavioral features most contributed, to look for shared mechanisms across individuals.

For comparisons across individuals, it will be important for some future diary work to include control subjects, and to overall scale up the number of subjects included. Because BLS included regular ratings of a battery of clinical scales, MRI scans, wrist watch data collection swaps, etc. there was a feasibility limit imposed on the study population size that was much smaller than what could occur with diaries alone. While clinical grounding and multimodal data sources are also important directions, another style of study could simply collect diaries and self-report surveys (that draw from a validated scale rather than a broader swath of bespoke questions) in as large of a population as possible with some diagnosis of interest available, along with a set of controls. This would allow comparisons of temporal dynamics by group to occur in much greater detail, and could provide a bridge between currently popular large population snapshot studies and the hopefully emerging personalized longitudinal modeling work along the lines of what could be achieved with BLS.

A clear more immediate next step of this work would be to continue with the present BLS dataset. It would be ideal to do a deeper dive on variations of ways to clean and summarize the core sentence level feature set produced by the code (and work in additional features like OpenSMILE outputs), including a better understanding of how the features relate to each other within the sentence structure of a diary. It would also be very beneficial to develop methods so that features might have temporal trends quantified in different patients at different timescales, including testing more simple summary techniques like peak finding algorithms or auto-correlations. Given the large amount of participants held out entirely from the EMA modeling and other closer comparisons I did, there are of course many ways the existing work could be extended to the rest of the set, which would enable more formal modeling paradigms that consider both individual and overall fits, in addition to capturing nonlinear relationships that weren't fully fit by the linear modeling in this chapter.

Once initial hypotheses are tested as desired, the BLS dataset could then be used for further exploratory works on entirely new features of potential interest, most notably content-related ones. By developing personalized features derived from topic model style algorithms, as well as performing more qualitative exploration to come up with longer term ideas for content-based feature generation methods, early testing on a variety of ideas could occur, setting up hypotheses for the next lab diary dataset in psychotic and/or mood disorders. Diary content could even be directly compared across different days via modern methods like transformers, something uniquely enabled by this format for patient speech sampling.

Similarly, predictions of future diary features from prior ones and characterization of how feature relationships themselves can change over time in an individual will be quite important topics to eventually explore that BLS could provide a foundation for. Alongside that, there are the many other modalities of data that were collected during BLS and might prove interesting to cross-reference with diary-related discoveries. Ultimately, digital psychiatry is still such an early field and audio diaries are still such a new format that curious exploration by some will be instrumental to formalizing some research processes. Because of the strong size and duration of the BLS journal set and the already well-documented relevant trends observed when selecting only a handful of individuals to look at, it is likely that this could be a very good source for idea generation and method fine tuning as new projects collect data in parallel.

Finally, though there are a huge number of possible benefits of audio journals for psychiatric speech sampling, they are of course not a silver bullet. Studies focusing on other data formats like the clinical interview should of course continue as well, and we should aim to understand how extracted features from these different forms might compare/contrast. The most crucial part is that studies are designed with clear justifications for the data collection and analysis choices they will make - all of which have limitation - in terms of the specific parts of the overall psychiatry search space they wish to tackle. As technology becomes ever more pervasive, digital methods will probably be difficult to entirely separate from psychiatry research eventually, but the specific digital methods employed and their varying design details can have a very wide range to consider many possibilities as appropriate. Thus this highlights the value of early works that lay out many of these details, like what I've done here.

Related characterization work could obviously go much further: not just in considering different disease populations and control datasets, but also to cover many other data collection nuances. It is an open question what sort of participation rates would be achievable in a more severe psychiatric study pool (though note the case report participant of chapter <ref> submitted journals regularly despite severe treatment resistant symptoms), or how demographic and cultural factors might impact diary behaviors. Besides having a large impact on possible scientific value of a given journal dataset, these issues can make it difficult to budget for a study in advance. However, an upside is that the cost of the audio journal study is largely dependent on the participation rates, so as long as poor quality submissions are rejected as the pipeline can do, the cost will be proportional to the completeness of the final dataset. Regardless, it would be very beneficial for the research community to build up a shared knowledge base about diary engagement tendencies in different patient populations, as well as to address questions about how schemes to increase participation mid-study may or may not bias results. The latter question connects to broader technical considerations on handling missingness that remain to be worked out, both because it can be a difficult machine learning problem in general and because in this case missingness may itself be inherently meaningful.

By coupling such works with extensions to feature characterization like described above, future digital psychiatry research would be very well positioned to employ audio journals across those settings where they are likely to be salient, and to do so in a way most likely to address their specific aims. A key piece of building up this knowledge (and creating mew tools for study efficiency like domain-adapted and validated automated transcription tools) will be larger scale collaborative data collection projects, which can facilitate data sharing and subsequently knowledge sharing much better than individual lab projects can. One initiative of immediate relevance then is the AMPSCZ project, as it includes collection of an audio journal datatype and fortunately now has plans to make real use of that datatype.

§.§.§ A note on adapting the pipeline to the AMPSCZ project

The NIMH's Accelerating Medicines Partnership Program - Schizophrenia (AMPSCZ) project is a multimodal big data collection endeavour spanning many research locations internationally, with a focus on young people at clinical high risk for developing Schizophrenia [NIMH, 2022]. This project is a major focus of chapter <ref>, where I will document the data flow and quality assurance pipeline I wrote as infrastructure for the interview recording portion of AMPSCZ. Meanwhile, audio journals are also being collected across many participants, and are discussed at greater length in a scientific planning sense in Appendix <ref>.

As of mid-February 2023, just over 1000 diaries had been successfully submitted by AMPSCZ subjects from across 19 locations. The project is currently in early stages, with only $\sim 2\%$ of planned interviews for the lifetime of AMPSCZ recorded yet (chapter <ref>) – suggesting that the journal dataset is on pace to reach an impressive size of 50,000 recordings. Despite the large amount of data collected to date, there remains no pipeline for data flow or quality monitoring of the diaries, let alone a plan for feature extraction for the project. Only much more recently was a decision made about getting the diaries transcribed, something that resulted from arguments similar to those I've made throughout this chapter along with arguments built on early interview dataset observations (to be covered in chapter <ref>).

This situation only further goes to show that audio journals are a datatype with great scientific value, and moreover a datatype that has been largely neglected and certainly underrated by recent research. We do not yet have any information about the quality of the $> 1000$ recordings collected thus far; not even their typical length nor much about submission trends over subjects and time. There is minimal clarity on leadership for the journal datatype in AMPSCZ, with it awkwardly falling between the self report survey collection and the audio recording collection.

However, given the recent budget shift (resulting from the arguments of Appendix <ref>) to enable gold standard transcripts to be obtained for the AMPSCZ diaries, there is certainly hope for the future of the format, as well as this initiative. The presented arguments could also assist in determining to what extent budgeting for high quality professional transcription for a particular speech sampling research project makes sense.

One future direction for the audio diary pipeline presented in this chapter is therefore adaptation to the AMPSCZ project. While tips for adapting the code to other probable study contexts were provided in supplemental section <ref>, those were focused on usage of the pipeline by individual lab groups. Infrastructure for a large scale international collaborative project can be a much more complicated undertaking. Yet because many of the adaptations needed are analogous to work already implemented for the interview recordings to be described in chapter <ref>, a blending of these two code bases may be more feasible to complete even given that the person to do so will likely have limited bandwidth and/or limited software experience.

To facilitate that process, I have provided a detailed list of the major differences that exist between the current audio journal pipeline and what will be needed for the AMPSCZ project. That can be found in supplemental section <ref>.

§.§ Contributions

In sum, I believe that audio diaries have been severely underutilized in early digital psychiatry research, and especially so in studies that wish to focus on analysis of patient speech. As such, this chapter aims to serve as a thorough account of not only why audio journals should be a priority for the field in the future, but also how audio journals might be best leveraged in practice – including potential considerations from perspectives of study design all the way to final dataset analysis, and supported by a variety of pilot results in a longitudinal Bipolar disorder journal dataset containing over 10,000 submissions from across more than 50 subjects.

My primary contributions from this chapter are as follows:

* Constructed a detailed series of arguments for the scientific utility of the audio journal format in digital psychiatry, synthesizing points from both psychiatry/neuroscience and data science/machine learning, along with highlighting a variety of practical considerations and making direct comparisons to other datatypes that might be considered alternatives.

* Despite the paucity of directly relevant prior research, I also included an extensive literature review on indirectly related topics, as well as a discussion of why digital psychiatry has perhaps underrated audio journals as a datatype to date.

* To further these arguments, I proposed a number of follow up questions of potential scientific interest at the end of the chapter – both those with immediate connection to the audio journal results reported here, and longer term questions that might leverage the unique advantages of diary data.

* Note that a context-specific version of these arguments was later used to make a large impact on the direction of the patient language analysis portion of the NIMH's AMPSCZ project, shifting over $100,000 of transcription budget from the interview protocol to instead build a full diary transcript set. AMPSCZ will be a primary topic in chapter <ref>, and the document I wrote arguing for the change can be found in Appendix <ref>.

* Wrote, tested, documented, and publicly released a pipeline for data management, quality control, feature extraction, and summary visualization of daily audio journal recordings.

* Performed pilot validation of the key included features, and wrote an extended discussion of the pros and cons of these features, justifying their inclusion in the pipeline.

* In particular, the code can not only facilitate future studies in building a quality audio journal research dataset, but it also can serve as a lightweight feature extraction code base that provides a set of core benchmark metrics, to hopefully improve cohesion between future studies in this space.

* I also identified future directions for the pipeline to continue making progress towards these constructive aims.

* Utilized my pipeline to obtain features for scientific analyses from the audio journals that were collected in our Bipolar Longitudinal Study (BLS). One major aim of this endeavour was simply to more deeply characterize properties of the diary features empirically, to serve as an important reference point for future analyses and a blueprint to influence design of future data collection.

* Characterized key feature distributions and correlation structures, identifying a variety of highly significant participant-dependent distributional differences in our dataset. Some differences in factors like word count, speaking rate, duration of pauses, and disfluency use were in line with expectations from the considered participants' clinical profiles.

* More generally the observed feature correlations and distributional differences raised a number of considerations about how to handle modeling in a longitudinal dataset like constructed with audio journals.

* As part of this, I also identified differences in submission patterns across the larger dataset, showing very impressive participation and engagement with the diary format in over $\frac{1}{3}$ of enrolled patients in the multimodal BLS project, as well as isolating when participation dropped off for those who were less engaged. This could facilitate future predictive modeling work, particularly the characterization of missingness distribution over different participant timelines.

* Fit linear models of self-report survey scores based on extracted features from same day diary recordings in a subset of participants, and tested best model fits on hold out data from those same subjects. Through this process I found a number of participant-dependent significant relationships – including with opposite directionalities – in addition to a relevance for sentiment score across considered patients.

* To do so, I characterized responses to emotion-related EMAs in a participant-independent manner across a larger set of subjects, finding that positively-worded and negatively-worded questions had minimal correlations with each other, but strong correlations within group. The distributions were also differently shaped, likely because of the symptom severity that the two types of questions will saturate at.

* When looking at the positive and negative emotion self-report summary scores within the 7 chosen patients, there were obvious participant-specific differences in the mean and variance of each type and the correspondence between the two types of EMA prompts, such that models correlating diary features with EMA directly could perform well based only on telling apart patients, which with $n=7$ is probably not a meaningful distinction.

* For that reason, I fit overall slope models using z-scored versions of the input diary features based on each participants' own mean and variance. When doing so, there was no meaningful relationship between verbosity and positive nor negative EMA across the dataset. At the same time, subject 5B had highly significant relationships with lower word count corresponding to worse symptoms across self-report, explaining $\sim 32\%$ and $\sim 24\%$ of variance with a simple linear fit in the positive and negative summary scores respectively. Meanwhile, subject 8R had a significant relationship between word count and negative EMA mean that explained $\sim 12\%$ of variance, but with higher word count corresponding to worse reported negative emotions.

* Both verbosity models generalized well to hold out sets from random points within the main period and in large part to the end of study hold out set for each participant. There was some degradation of generalization in the beginning of study hold out set however. Looking more closely at 5B, it is apparent that even the training set fit was worse earlier in the study timeline, and a rolling beta model would like be more appropriate to capture the slower changes that occurred in their disease profile over the course of BLS.

* Diary sentiment on the other hand was able to linearly explain $\sim 11\%$ and $\sim 7\%$ of variance in positive and negative emotion EMA responses with a single slope across all of BLS when the sentiment was corrected to the participant's baseline using z-score, which was also highly significant and generalizable to the hold out set. This suggests a consistent role for sentiment across the considered subjects and providing confidence in the VADER sentiment summary output. Sentiment is particularly well suited to this because it is much less likely to have opposite direction effects and much more likely to vary with self-reported emotion on the day level timescale even in typical adults.

* Ultimately, it was clearly important to consider factors specific to within a patient and to take care with temporal dynamics when fitting such longitudinal models, and there is a paucity of psychiatry research to date that use modeling techniques across a number of participants that would avoid smoothing over these interesting signals. Calling attention to the examples uncovered here is important for future work, and further it underscores the importance of diaries in the broader speech sampling research space, because interview recordings cannot feasibly get the longitudinal sample size necessary for a hierarchical style of modeling.

* Explored early n of 1 studies in three participants of especial interest, highlighting interesting properties about how their diary features varied over time and at different scales, both in isolation and as compared to available self-report and clinical rating time courses. By referring to diary content at study points of interest and quantifying occurrences of topics of interest across the diary, I additionally provided proof of concept for some of the powerful ways diaries can be used in a more qualitative sense, to synergize with produced features.

* Participant 5B, diagnosed with Bipolar II, experienced clear depressive episodes clinically during their time in BLS. They also demonstrated some overall improvements in the latter part of their 2 year BLS enrollment. Their EMA scores fluctuated very clearly with clinical episodes, and in the second year the EMA-defined episodes began to fluctuate more rapidly than the $\sim$ once per month clinical ratings could capture, which already demonstrates the power of digital psychiatry.

* Notably, 5B had analogous fluctuations in features like word count and sentiment over the course of their time in BLS, such that those features could have been used to mark the mood episode onsets just as well. Furthermore, there was an uptick in their baseline word count in the second half of the study, and an uptick in some other features that could not be entirely explained by higher word count journals such as verbal filler word usage. Some of these features appeared to spike around the times of transition into and out of an episode.

* 5B's diary content was also highly informative even at a cursory look. They used the phrase "kinda down" only 7 times across all recorded journals, but 4 of these occurrences were right before the first major mood episode and 3 were right before the second one, highly concentrated and preceding a clear signal in the EMA by a couple days.

* Participant 8R, diagnosed with Bipolar I, experienced some depressive periods and some hypomanic episodes during their time in BLS. While their EMA was not as clean cut as 5B, a pattern did emerge with reporting high levels of negative emotions and positive emotions simultaneously at times in the study corresponding to higher mania-related symptoms. In the first year of the study, they discussed a possible hypomanic episode themselves at two distinct times in their diaries, which aligned well with peaks in diary features like speaking rate and rate of disfluency usage, suggesting diary content could help fill in gaps in the YMRS along with EMA trends – thereby increasing confidence in hypotheses about the relevance of some extracted diary features too.

* Participant 3S was also diagnosed with Bipolar I, and submitted many long journals over the course of the study, but did not have a whole lot of clinical variation, which is why the scales stopped being collected much earlier in their enrollment period than other features. Their EMA was also quite noisy, but there were interesting fluctuations as well as sustained over time changes in their extracted diary features. Additionally, there was a noticeable drop in word counts around the time they experienced the highest negative symptom clinical ratings, even though their self-reports indicated lesser symptoms at that same time. Interestingly, by checking journal content around a shorter period of abnormal diary features, a time where 3S attempted to cut back on medications and then quickly changed their mind was uncovered.

These contributions are all important steps towards establishing a future of audio diary analysis in psychiatry, including theoretical arguments for the datatype and pilot demonstration of many of the proposed advantages, as well as tools to assist in replicating our work and identification of a broad range of important considerations for future projects to improve upon them. In the next chapter (<ref>), I will focus on interview recordings as a speech sampling datatype, and introduce the large scale data collection initiative of AMPSCZ. Fortunately, that project has recently reconsidered the value of the audio diary format, and so there will be many exciting opportunities in the future for data at such a scale to be analyzed in a manner that can capture clinically relevant differences both between and within individuals – and perhaps even use the longitudinal fluctuation patterns within each individual as between group predictors. As various interview contexts have their own scientific value too, it will be especially fruitful to see how audio journal and interview recording features from the same person may differ within and across study periods; thus further motivating the relevance of the datatype characterizations I provided across the two chapters.

CHAPTER: AUTOMATED ANALYSIS OF CLINICAL INTERVIEW RECORDINGS[THIS CHAPTER IS CENTERED ON PUBLICLY RELEASING THE SOFTWARE INFRASTRUCTURE I WROTE FOR THE NIMH'S AMPSCZ PROJECT, NOW OPEN SOURCED AT [ENNIS, 2023]. PORTIONS OF THE CHAPTER ALSO DISCUSS RELEVANT WORK FROM A SCHIZOPHRENIA RESEARCH PUBLICATION THAT I CONTRIBUTED TO, LEAD BY [LIEBENTHAL ET AL., 2022]. SEE APPENDIX <REF> FOR DETAILED ATTRIBUTIONS.]

Mirroring chapter <ref> in many ways, this chapter focuses on analysis of clinical interview recordings. While audio diaries have a number of advantages over clinical interviews, clinical interviews also have a number of advantages themselves. Interview datasets are often directly connected to the gold standard clinical scale ratings, as these scores are assigned based on observation of the very interview that was recorded. As such, much of the literature review on acoustics and linguistics already provided in chapter <ref> drew on results from interview recordings, not audio journals, because there is a substantially larger body of existing research on interviews. Further, clinical interviews provide an opportunity to assess social functioning through the interview dialogue, and are much more likely to include recorded video for other modalities of analysis like facial expression dynamics. Given the recent rise of telehealth precipitated by the COVID19 pandemic, collection and standardization of interview recording datasets for psychiatry research has become significantly more feasible.

There is thus a need for open source software tools in this domain, similar to the previously discussed needs for coherent collaboration in the audio journal space. Although there is a much stronger groundwork already available for the use of packages like OpenSMILE in the clinical interview context, there remains a paucity of software options for managing the many multimodal processing tools across an interview dataset, and there is minimal progress on a unifying framework for baseline feature definitions. Because clinical interviews are additionally a more difficult datatype to collect in large samples, success in answering bigger scientific questions with interview recordings will likely require a higher level of data sharing amongst the community. A major aim of this chapter is to release and document my interview processing code base, and ultimately to contribute towards a system for monitoring collection of structured interview recordings across collaborative psychiatry studies.

The chapter will have two major components: the description of and example use cases for my interview data management pipeline (section <ref>), and proof-of-concept scientific results produced using features I extracted from a lab dataset (section <ref>). The former is currently running to facilitate a large, international multi-site study that includes interview recording data (the AMPSCZ project), and the latter is already published [Liebenthal et al., 2022].

Before diving into the details of my contributions and results, I will provide needed context in section <ref> on the many differences to consider between the audio diaries already discussed at length in chapter <ref> and the interview recording format of interest here, as well as the relevant similarities between these datatypes. For key differences, I will also review related prior literature to give background that is not already available in <ref>. To finalize background section <ref>, I will discuss previous psychiatry results connected to the primary hypothesis of [Liebenthal et al., 2022] – about conceptual disorganization symptoms in psychotic illness.

Once background on existing work has been covered, I will introduce the aforementioned AMPSCZ project along with the associated interview data collection methods in section <ref>. At that time, I will also recap the Bipolar Longitudinal Study (BLS) dataset described in chapter <ref>, and provide details on the interview recording methodologies used for that study, as it is the source for the work of [Liebenthal et al., 2022]. That will set the stage for the core sections of the chapter, <ref> and <ref>. Finally, I will close the chapter with a summary of the takeaways, limitations, and overall impact of this work, as well as related future directions of promise, in section <ref>.

§ BACKGROUND

Because of the strong grounding the interview format already has in prior psychiatric practice, it is the area with perhaps the shortest leap between current psychiatry and emerging technologies, and thus has been a popular application of early research. There is a well established evidence base for differences in some acoustic and linguistic properties across many psychiatric illnesses, dating back to older manual scoring studies and further supported by the pilot computational results in this new era of research – background that was covered at length in chapter <ref> (particularly <ref>). With the recent rise of telehealth, expedited by the COVID19 pandemic, collecting a large dataset of interview recordings has become even more feasible, as will be demonstrated with the processing of Zoom interviews in this chapter. Advances generated from such research have strong potential to greatly increase accessibility of present psychiatric resources, in addition to the hopes of uncovering new scientific information.

To further contextualize the interview recording datatype, I will next (<ref>) compare and contrast these recordings with the audio journals previously described in chapter <ref>. Gaps in background material highlighted by the comparison will then be covered, including a discussion of different interview formats and their role in modern psychiatry (<ref>) and a brief review of video processing tools for and results from interview recordings (<ref>). Throughout these sections I will provide a broader overview along with deeper background on subtopics of most relevance to the projects that will be reported on in this chapter. I will close the background section with a discussion of the key aims of my work and how they relate to the need areas identified for future psychiatric interview research (<ref>).

§.§ Comparing structured interview recordings with audio journals

Clinical interview recordings and daily audio diaries share many commonalities. They both lend themselves to a variety of different analysis techniques, whether qualitative or quantitative, and they both contribute objectively measurable information like speech rate as well as subjective introspective information from the patient. In many respects, the features extracted by the pipeline described in chapter <ref> could be immediately applied to the interview context and be used fruitfully by a research investigation. Comparisons of similar features in these two different contexts is in fact an interesting open research question in itself.

Still, there are a number of major differences that affect both many details of how processing of the respective datatypes should be approached  and more broadly what types of questions each datatype can best answer. Key differences include:

* Audio journals are typically an order of magnitude shorter in duration - the maximum length allowed by the BLS diary protocol in chapter <ref> was 4 minutes, while the majority of interview recordings collected in BLS easily exceeded 30 minutes.

* To maintain good participation rates in a journal study, it is likely necessary to keep suggested durations in line with these prior numbers.

* On the other hand, it is possible interview recording durations could decrease on average as remote interviews become an increasingly common occurrence. This will likely depend on study-specific nuances.

* Audio journals are much easier to collect, both for researchers and participants. Thus clinical interviews tend to cover many fewer days in a given period and also may be more difficult to retain participants over a longer study design.

* Short virtual interviews conducted by an automated system (like is already done at some companies for initial fully-automated steps of job screening) could be a middle ground datatype between audio diaries and structured interview recordings in the future.

* However for many structured interview formats of prior interest in psychiatry, there remains a more labor intensive component requiring a longer period of human interaction - making it near impossible to build up a dense day to day dataset like was done with the diaries in chapter <ref>.

* Structured interview time points can be directly tied to well-established clinical outcome measures, as clinical scales are generally scored based on patient interviews. There is no good analog for this in audio journals, with the closest thing being self-report survey responses that are often submitted around the same time as the daily diary submission.

* It is of course possible to fit a model that attempts to predict clinical scale ratings based on diary properties from that and/or preceding days, but such a model would be missing a link to connect patient behavior with how clinicians score symptoms.

* Because of this clear grounding for interview recordings, it is perhaps also considered a less risky approach to the symptom score prediction problem; if clinicians are rating symptoms based on interview content, it would be shocking if extractable features to predict scores didn't exist in these interview recordings.

* I think interview recordings are actually much more risky from an operational/data science perspective, even if they are less risky from a psychiatry one. Regardless, they are a much more common datatype to find in the early computational psychiatry literature, and as such there are more established research protocols and results for interviews.

* Recorded interviews often include video of the participant and sometimes the interviewer along with the audio recording. While it is theoretically possible to include video collection in recording of diaries on the smartphone app, it is more of an added burden to the participant than it is in the interview setting, and it is not included in the typical data collection workflow described in chapter <ref>.

* The video modality of course opens up a host of new potential metrics related to physical movement, and in particular facial affect. Considerations for analysis of interview videos along with relevant prior literature are thus reviewed below in section <ref>.

* Structured interviews are able to assess properties of social functioning that audio journals have no information on.

* Any acoustic and linguistic properties discussed on the journal level in chapter <ref> can be considered in the interview in this additional context - evaluating the connection of these features between interviewer prompts and participant responses over the course of the interview.

* Additional features specific to the interview setting may also be designed to capture properties of human dialogue that are undefined in a monologue setting. A basic example would be measuring the frequency of interruptions, which can be done easily using TranscribeMe's verbatim notation for dialogues.

* Features can be extracted from the video modality to assess social functioning as well. One common prior use case is to quantify gaze direction as an estimation of eye contact behavior, which will be discussed as part of the review in section <ref>. Particularly when video of the interviewer is available aligned with that of the subject, a number of interesting social cues can be derived from the interview video modality.

* The social context may itself change participant behavior in a way that alters the base features that can be compared directly between interview and diary speech. The specifics of the clinical or research setting may also have an impact that is different from a more general social setting. These properties of the clinical interview are neither better nor worse – as was discussed in chapter <ref> there is good reason to believe some participants will be more honest in a journal recording than in an interview conversation, but on the other hand behavior specific to the conversational setting can be of inherent interest.

* App-based daily audio journal prompts are often very open ended, as was the case for the submissions analyzed in chapter <ref>. Interviews on the other hand are usually at least partially structured with a pre-specified list of questions to address, so that one can guarantee certain topics will be covered in each interview time point, and further can cover a fairly large number of them.

* Although it is possible to prompt for a diary recording about a more specific topic (e.g. experienced hallucinations, drug use, etc.), it would be difficult to maintain study engagement while asking for multiple different types of daily journal submissions. It is also unlikely that a participant will have much to say every single day about many of the potential individual topic choices.

* In addition to the questions of clinical interest that are included by design in each interview recording for a particular study, the interview format also allows for follow-up questions to be asked based on what the participant just said. This allows for clarifications to be made as well as for a trained interviewer to uncover information that otherwise wouldn't have been revealed by a patient.

* Ultimately, the information prompted for within an interview can vary greatly based on design, and has quite different considerations than design of journal prompts. As such, various interview structures are reviewed in the next section (<ref>), both to discuss their downstream affects on analysis decisions and contextualize their relationship within existing clinical and research practices.

* It is worth noting that what exactly participants choose to discuss in a free form audio journal format carries interesting information that would be infeasible to obtain in most interview settings, so again these differences are not clear cut advantages or disadvantages for one format over the other.

Given all of that, there is strong argument for collection of both datatypes in a study where possible. Indeed, both the BLS and AMPSCZ projects described in this chapter do include interview recordings and daily audio diaries, as did the OCD case report that is the subject of chapter <ref>.

While I demonstrate the utility of diaries and interviews independently throughout the thesis, at times in capacities that would clearly not be feasible with the other format, ways that their analysis can truly synergize is an important open question that is largely beyond the scope of this thesis.

As the background relevant for audio journals has already been established in chapter <ref> (<ref>), I will now proceed with review of the new topics brought to light here: different ways of structuring an interview for different clinical and research purposes (<ref>), and then analysis methods for video recording data (<ref>).

§.§ Clinical interviews in modern psychiatry

Clinical scales, to be discussed in greater depth in subsection <ref> next, are often used in psychiatry research and especially clinical trials [Wood and Gupta, 2017]. The goal standard scales rated by trained observers are now largely collected through the use of a semi-standardized structured interview conducted by a trained interviewer, though this was not always the case [Williams and Kobak, 2008]. Consistent guidelines for a particular scale will suggest questions to ask in order to assess each item and will also suggest follow-up questions that might be asked depending on patient response and potentially other behaviors. For rating scales, there are often guidelines related directly to the patient response, and for some of these instruments there will also be guidelines for rating observed behaviors outside of the response content. For example, the MADRS is designed to exclude consideration of psychomotor signs, but other scales for rating depression severity do ask trained raters to consider e.g. the extent to which speech was slow [Jain et al., 2007].

Besides the structured clinical interviews associated with particular symptom severity scales, there are a number of other use cases for interviews in psychiatry. Perhaps the most relevant informational interview in actual practice is the Structured Clinical Interview for DSM-5 Disorders (SCID-5), which is a structured interview format somewhat similar to the clinical scales, but intended for clinicians to determine a potential DSM-5 diagnosis at the end of the interview [First et al., 2015]. Some studies use the SCID-5 to confirm participant diagnoses, but it is not a large data source for speech sampling work.

In many cases one might consider psychotherapy sessions as interviews too. Though the primary purpose of "talk therapy" is to be therapeutic, there is still information gathering involved and more generally content that could be quite interesting. In 2019 $\sim 10\%$ of Americans took part in some form of psychotherapy (including both specific evidence-based techniques like cognitive behavioral therapy and other forms of therapy such as modern psychoanalysis) [Terlizzi and Zablotsky, 2019]; the rise of remote therapy options during COVID19 has likely only increased these numbers. At the same time, "psychotherapy" is a very broad term, and most forms of psychotherapy in real practice involve only highly clinician-dependent qualitative information gathering [Lewis et al., 2019]. It is also likely that certain forms of talk therapy contain more sensitive personal conversation than typical clinical interviews, and require a longer term relationship building between therapist and patient. These factors make it difficult for academic science to consider analysis of many types of recorded psychotherapy sessions in research, though in the longer term psychiatry research ought to utilize technology to systematically study techniques that were previously more of an art – and to take care in doing so, as it is easier to throw the baby out with the bathwater than it is to capture nuance in what works and what doesn't.

For research at present, besides recording official clinical interviews, some studies will record free-form interviews specifically for the purposes of analysis, to capture patient speech in dialogue. These interviews could take the form of discussion on a particular prompt, though they are often more like the interview equivalent of our audio journal prompt in chapter <ref>: meant to be participant-driven, eliciting conversation on whatever topic they wish to discuss. It is important for open-ended interviewers to encourage engagement and keep the conversation going as much as possible, without interrupting the participant's line of thinking or unnecessarily directing the conversation topics. So conducting these interviews does require some skill, but it does not require the same sort of formal training and labor burden as clinical interviews do, and while clinical interviews often take an hour or more, it is uncommon for a free-form interview to extend past 30 minutes.