\chapter{Cassandra}\label{ch:cassandra}

\section{Overview}

Apache Cassandra is a Java-based NoSQL database management system. It contains the command-line interface CQL for Cassandra database commands instead of SQL. It has to be recognized that the supported Java version by the open-source project Apache Cassandra is able to be changed with every version. 
That has to be considered during the integration into the \gls{CI/CD} pipeline of the project. There are different multi-arch Dockerfiles for all versions of AdoptOpenJDK for public usage. These can be applied for the base Cassandra image. Afterwards, a multi-arch Docker image for Cassandra, which includes the s390x architecture, can be published. 
A \ref{monitoring} "nagios monitoring check" - usable for containerized environments - is chosen for tests. This one has to be assimilated in the setup. \\ 
This Apache project is using CircleCI for Continuous Integration. 

\section{Installation}\label{Cassandra-Installation}

A Dockerfile\footnote{\url{https://github.com/s390x-container-samples/s390x-cassandra-ci-cd/blob/master/Dockerfile}} for a multi-arch image should be created in this project with the reason for using it for multiple architectures in the future.
Apache Cassandra is a Java-based application. Therefore, the research has contained the search for a multi-arch image with pre-installed Java. The Cassandra version 3.11.6 is supporting Java 8. 
AdoptOpenJDK is maintaining multi-arch images for different Linux distributions with pre-installed AdoptOpenJDK.
In the description of \ref{AdoptOpenJDK-Multi-Arch} "Multi-Arch Image AdoptOpenJDK 8" is explained how the recommended Java version will be chosen for the specific architecture in this base image. 
This base image is integrated with \lstinline!FROM adoptopenjdk:8-jdk-hotspot! on top of the Dockerfile.

Apache Cassandra will be cloned from the Github repository with \\
\lstinline!git clone https://github.com/apache/cassandra.git!. \\
The selected version is set with an argument with the ARG variable. On this way, it is possible to check out other versions with a given argument as \lstinline!--build-arg CASSANDRA_VERSION=3.12.4! in the docker build command. \\
Apache Cassandra is using JNA (Java Native Library)\footnote{\url{https://github.com/java-native-access/jna}} as an extension of Java. JNA provides Java programs easy access to native shared libraries for improved communication with the database system. The ant command is building the application and the JNA subsequently. \\
The command \lstinline!sed -i ' s/Xss256k/Xss32m/'! is setting the size of the frame stack (JVM stack) used by each thread to store local variables. This frame size is reduced to a minimum and stored in both files build.xml and conf/jvm.options.
A soft link (\textbf{ln -s}) will be created to save space in the container. Such a soft link refers to a symbolic path of another file or directory. Following, all files inside of the build directory \path{/root/cassandra} are additionally available under \path{/usr/share/cassandra}. Another benefit is the fast creation of a soft link in comparison to a copy / move command.
\begin{figure}[H]
\centering
\begin{boxedverbatim}
FROM adoptopenjdk:8-jdk-hotspot
ARG CASSANDRA_VERSION=3.11.6
git clone https://github.com/java-native-access/jna.git \
&& cd jna \
&& git checkout 4.2.2 \
&& ant native jar \
# Build and install Apache Cassandra
&& cd $SOURCE_ROOT \
&& git clone https://github.com/apache/cassandra.git \
&& cd cassandra \
&& git checkout cassandra-${CASSANDRA_VERSION} \
&& sed -i ' s/Xss256k/Xss32m/' build.xml conf/jvm.options \
&& ant \
&& rm lib/snappy-java-1.1.1.7.jar \
&& wget -O lib/snappy-java-1.1.2.6.jar https://repo1.maven.org/maven2/org/xerial/snappy/
snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar \
&& rm lib/jna-4.2.2.jar \
&& ln -s $SOURCE_ROOT/cassandra /usr/share/cassandra \
&& rm -rf  $SOURCE_ROOT/jna $SOURCE_ROOT/*.tar.gz  \
&& rm -rf /usr/share/cassandra/test \
\end{boxedverbatim}
 \caption{Cassandra Installation}
    \label{CassandraInstallation}
\end{figure}

Cassandra can be started with \lstinline!CMD ["cassandra", "-R", "-f"]! with \textbf{f} for force and \textbf{R} so start the service as root after setting the environment variables with inside of the Docker container.

\section{Java Optimization}

Apache Cassandra should be allowed to use all available memory of the minimal system effectively.
Default the JVM is configured that only a part of that can be used. These configurations are set in the Java Options.
Afterwards, Apache Cassandra can work with good performance. \\
Every major Java release can differ from another one with such options for Java optimization.

Following Java optimizations are available for Java 8: \\
\begin{itemize}
\item \textbf{-XX:+UseCGroupMemoryLimitForHeap}  \\
This option tells the JVM to look into \path{/sys/fs/cgroup/memory/memory.limit} after the available memory and to use the whole memory if necessary \cite{Flood2017}.
\item \textbf{-XX:ParallelGCThreads=2} \\
Apache Cassandra is referencing  2 required CPUs as a minimum in their documentation \cite{Cassandra}.
This JVM option permits 2 GC (Garbage Collection) threads parallel on different (virtual) processors.
\end{itemize}

All these Java optimization options can be set as an environment variable with \lstinline!ENV JVM_OPTIONS! in the Dockerfile. Then they will be used for every Java process like Apache Cassandra as an example.
As a result, Apache Cassandra can use the whole memory and delivered CPU, which is specified in the \gls{QEMU} options.


\section{Workaround Because of a JVM Issue}\label{jvm-issue}

Apache Cassandra is using OpenJDK 8 for running as default. There are some discussions\footnote{\url{https://lists.apache.org/thread.html/r38f6beaa22e247cb38212f476f5f79efdc6587f83af0397406c06d7c\%40<dev.cassandra.apache.org>}} about supporting OpenJDK 11 at the moment. The Dockerfile mentioned above is using AdoptOpenJDK 8 at the build process is working without any problems on Z systems. There exists a JVM issue during the build process on x86 which is the reason for non-successful builds of Java applications there. That counts for all more complex Java applications.

This case has been tested with AdoptOpenJDK 8 and AdoptOpenJDK 11. The library hsdis-s390.so is not loadable for both Java versions for emulated s390x guests on x86, but during the build process on s390x\footnote{\url{https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=951764}}. This issue does not exist with the version AdoptOpenJDK 14, but Apache Cassandra does not work with this version. \\
This emulation bug of qemu has to be fixed, and the issue has been reported to \gls{KVM} Developers inside of IBM. 

There exists a workaround for the development process for finishing the project. The Docker image is buildable on s390x. Therefore, the Docker image has been built for s390x on s390x with \\  
\lstinline!docker build --squash -t cassandra .!. \\
Afterwards, this image can be saved as a tar archive with \\
\lstinline!docker save cassandra > cassandra.tar!\\ 
on the host. It has to be transferred with rsync to the x86 system then. The image inside of the tar archive can be included to all existing images of \lstinline!docker images! with the command 
\lstinline!docker load --input cassandra.tar!. \\
After this action, the Docker image is usable as it would have been built on this system. 
It can be integrated into \gls{QEMU} and be started without any problems afterwards.

\section{Nagios Monitoring Check for Tests} \label{monitoring}

Nagios is an open-source \gls{monitoring system} to check the status of systems with their services. There are many checks for different services available. One monitoring check\footnote{\url{https://github.com/skyscrapers/monitoring-plugins/blob/master/check_cassandra.pl}} exists for Apache Cassandra. It is testing whether Java would be working, the service Apache Cassandra would be up and running, and whether CQL commands would be possible to execute without any issues. This Perl script is using \textbf{nodetool} for monitoring the JVM and the application. Nodetool is delivered together with Cassandra. It is connecting with the database and can dispense statistics about the Cassandra cluster / host \cite[~p.256]{Carpenter2020}. The result contains the status, information about memory usage and other capacities. This tool can indicate issues resulting in messages via the monitoring plugin in our case.\\
This monitoring check is written in Perl and is an excellent choice for automated tests with error messages in the case if any output would not give an \textbf{OK}. \\
All monitoring plugins in the referenced Github repository are developed additionally for integration into Docker. Following, these scripts do not require any installed nagios system for monitoring.
Summarily, these monitoring plugins can be used for tests inside of containerized environments for testing Docker images.

\newpage
For a better maintenance, \textbf{cassandra\_check.pl} is integrated with a separate Dockerfile with the installation Dockerfile as a base image:
\begin{figure}[H]
\centering
\begin{boxedverbatim}
FROM cassandra-s390x
RUN apt-get update && apt-get install -y \
    perl \
    perl-base \
    libimage-magick-perl     
WORKDIR /bin
COPY cassandra_check.pl /bin/cassandra_check.pl
ENTRYPOINT ["perl", "cassandra_check.pl"]
RUN apt-get remove -y \
    perl \
    libimage-magick-perl\
    perl-base  \
&& apt autoremove -y \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
\end{boxedverbatim}
 \caption{Cassandra Monitoring Check}
    \label{Cassandra-Monitoring}
\end{figure}
The separation of both Dockerfiles profits by the reutilization of the main Dockerfile. The base Docker image can be published in different container registries for public usage without the monitoring check. Accessorily, the build process for the base image of Cassandra and the test with the monitoring check can be split in the \gls{CI/CD} pipeline. The possibility of bug identification is simplified by this process. \\
The test Dockerfile in the monitoring directory\footnote{\url{https://github.com/s390x-container-samples/s390x-cassandra-ci-cd/blob/master/monitoring/Dockerfile}} can be reused for every Cassandra container image based on Ubuntu or Debian. The unique requirement is that the FROM line is matching the name of the tested Cassandra image name.


\section{Run Docker Image in QEMU}\label{RunningDockerImage}

The Docker image will be built with the command  \\
\lstinline!docker build buildx --platform=linux/s390x --squash -t cassandra-s390 .! \\
in the directory with the Cassandra Dockerfile  for the architecture s390x. That is the base Cassandra image.\\
The \verb+--squash+ option is compressing a Docker image and is combining commands in a Dockerfile automatically. The prequisites for building s390x images on x86 are set during the emulation preparation. The command \lstinline!docker images! has to show the registered Docker image with the name cassandra-390 in the local container registry. Afterwards, the monitoring check has to be added. 
The same procedure will be executed in the monitoring directory. The FROM command is referencing the self built Cassandra image with cassandra-s390. With the following command the monitoring check can be attached to the exiting Cassandra image to test it: \\
\verb+docker build buildx --platform=linux/s390x --squash -t cassandra-test-s390x .+ \\
It should be possible to integrate this Docker image into the qemu command. Therefore, a qemu-image will be created with an rounded given size besides of the Docker image in the  \lstinline!docker images! command. So the command \\  
\lstinline!qemu-img create -f raw cassandra-s390x.img 2G!  \\
can be used. This image needs to be converted to any Linux file system because \gls{QEMU} does not know the Docker file system. Therefore, the image is formatted with the command  \lstinline!mkfs.ext4 -F cassandra-s390x.img!. A directory with the name rootfs has to be created and the command for receiving the file system of the Docker image. \\
\lstinline!docker export $(docker create cassandra-test-s390x) | tar -C "rootfs" -xvf -!  \\ 
is exporting the docker image into the directory rootfs. Afterwards, the following command transfers the content of rootfs into the image cassandra.img.

\begin{figure}[H]
\centering
\begin{boxedverbatim}
mkdir /mnt/rootfs
mount -o loop cassandra-s390x.img /mnt/rootfs
cp -r rootfs/* /mnt/rootfs/.
\end{boxedverbatim}
 \caption{Mount Rootfs}
    \label{Mount-Rootfs}
\end{figure}

Now it is possible to run the system with Cassandra: \\
\begin{figure}[H]
\centering
\begin{boxedverbatim}
 /usr/bin/qemu-system-s390x -kernel vmlinuz-4.15.0-20-generic -m 8G \
 -M s390-ccw-virtio -nodefaults -device sclpconsole,chardev=console \
 -parallel none -net none -chardev stdio,id=console,signal=off,mux=on \
 -mon chardev=console -nographic -smp 3 -hda cassandra-s390x.img \ 
 --append 'root=/dev/vda rw console=ttyS0 rdinit=/bin/bash' 
\end{boxedverbatim}
 \caption{Run Cassandra}
    \label{RunCassandra}
\end{figure}

\newpage
\section{Integration into CI/CD}

The community of Apache Cassandra is using CircleCI\footnote{\url{https://circleci.com/}} as a Continuous Integration system.
This \gls{CI/CD} system is connected with Github. The tests are running after every commit in the master repository.
These tests can run in a container or in a virtual machine. These options make it possible to build the Docker image for the architecture s390x and to integrate it into \gls{QEMU} via the \gls{CI/CD} pipeline without additional scripting or workarounds. \\
CircleCI is working with a yaml configuration\footnote{\url{https://circleci.com/docs/2.0/sample-config/}}. 
An enhancement is the opportunity to define shell commands with "command" in the yaml file. Therefore, \ref{LinuxKernel} "Fetching Signed Linux Kernel Image" can be integrated. 
Additionally, dependencies can be created. On this way, it can be assured that the build process for the Docker image is triggered before the file system creation. 
The same counts for the kernel fetching process and the file system creation before the \gls{QEMU} start. \\
This configuration file with the whole test process has got the name \textbf{config.yml}. \\
\textbf{Workflows} defines the order of different process steps with their dependencies with \textbf{- requires}.
The different processes are specified separately above then. 
The advantage of different steps with dependencies is, that every step can be tested and the build process will be aborted after one failed process and this step will be listed with the error message in the \gls{CI/CD} system. \\
The configuration of CircleCI is clearly structured. The workspace can be declared. Running steps can receive names. Shell commands can be executed. Additionally, binaries and other files can be stored as artifacts. \\
The implementation of the job configuration can be found under \ref{CircleCI} "CircleCI configuration for Apache Cassandra". It can only work with a bug fix for the \ref{jvm-issue} "JVM Issue".\\
The system is configured that the version number of Apache Cassandra can be replaced by a new version with \verb+--build-arg CASSANDRA_VERSION=$CASSANDRA_VERSION+ in the docker build command. With that, the checkout of the special Cassandra branch will be executed instead of the configured last one.

