* [165] Abbott BP, et al. (LIGO Scientific Collaboration and Virgo Collaboration). Gw151226: Observation of gravitational waves from a 22-solar-mass binary black hole coalescence. Phys Rev Lett. 2016 Jun;116:241103. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.116.241103.
  * [166] Laser Interferometer Gravitational-Wave Observatory (LIGO). . 2021;Available from: https://www.ligo.caltech.edu.
  * [167] Virgo. . 2021;Available from: https://www.virgo-gw.eu.
  * [168] Armano M, Audley H, Auger G, et al. Sub-femto-$g$ free fall for space-based gravitational wave observatories: Lisa pathfinder results. Phys Rev Lett. 2016 Jun;116:231101. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.116.231101.
  * [169] Amaro-Seoane P, Audley H, Babak S, et al. Laser Interferometer Space Antenna. arXiv:170200786. 2017;Available from: https://arxiv.org/abs/1702.00786.
  * [170] Chiao RY, Speliotopoulos AD. Towards migo, the matter-wave interferometric gravitational-wave observatory, and the intersection of quantum mechanics with general relativity. Journal of Modern Optics. 2004;51(6-7):861–899. Available from: https://www.tandfonline.com/doi/abs/10.1080/09500340408233603.
  * [171] Linet B, Tourrenc P. Phase shift in a gravitational field: possibility of detection by interference. Can J Phys. 1976;54(11):1129–1133.
  * [172] Stodolsky L. Matter and light wave interferometry in gravitational fields. Gen Rel Grav. 1979;11:391.
  * [173] Bordé CJ. Quantum theory of atom-wave beam splitters and application to multidimensional atomic gravito-inertial sensors. General Relativity and Gravitation. 2004;36(3):475–502. Available from: https://doi.org/10.1023/B:GERG.0000010726.64769.6d.
  * [174] Tino GM, Vetrano F. Is it possible to detect gravitational waves with atom interferometers? Classical and Quantum Gravity. 2007 apr;24(9):2167–2178. Available from: https://doi.org/10.1088/0264-9381/24/9/001.
  * [175] Dimopoulos S, Graham PW, Hogan JM, et al. Atomic gravitational wave interferometric sensor. Phys Rev D. 2008 Dec;78:122002. Available from: http://link.aps.org/doi/10.1103/PhysRevD.78.122002.
  * [176] Hogan JM, Johnson DM, Dickerson S, et al. An atomic gravitational wave interferometric sensor in low earth orbit (agis-leo). General Relativity and Gravitation. 2011;43(7):1953–2009. Available from: http://dx.doi.org/10.1007/s10714-011-1182-x.
  * [177] Geiger R. Future gravitational wave detectors based on atom interferometry. World Scientific; 2017. Chapter Chapter 8; p. 285–313. Available from: https://www.worldscientific.com/doi/abs/10.1142/9789813141766_0008.
  * [178] Chaibi W, Geiger R, Canuel B, et al. Low frequency gravitational wave detection with ground-based atom interferometer arrays. Phys Rev D. 2016 Jan;93:021101. Available from: https://link.aps.org/doi/10.1103/PhysRevD.93.021101.
  * [179] Canuel B, Abend S, Amaro-Seoane P, et al. ELGAR - a European laboratory for gravitation and atom-interferometric research. Class Quantum Grav. 2020;Available from: https://iopscience.iop.org/article/10.1088/1361-6382/aba80e/meta.
  * [180] Canuel B, Abend S, Amaro-Seoane P, et al. Technologies for the ELGAR large scale atom interferometer array. arXiv:200704014. 2020;Available from: https://arxiv.org/abs/2007.04014.
  * [181] Hogan JM, Kasevich MA. Atom-interferometric gravitational-wave detection using heterodyne laser links. Phys Rev A. 2016 Sep;94:033632. Available from: http://link.aps.org/doi/10.1103/PhysRevA.94.033632.
  * [182] Graham PW, Hogan JM, Kasevich MA, et al. New method for gravitational wave detection with atomic sensors. Phys Rev Lett. 2013 Apr;110:171102. Available from: http://link.aps.org/doi/10.1103/PhysRevLett.110.171102.
  * [183] Loriani S, Schlippert D, Schubert C, et al. Atomic source selection in space-borne gravitational wave detection. New J Phys. 2019 Jun;21(6):063030. Available from: https://iopscience.iop.org/article/10.1088/1367-2630/ab22d0.
  * [184] Canuel B, Bertoldi A, Amand L, et al. Exploring gravity with the MIGA large scale atom interferometer. Sci Rep. 2018 Sep;8(1):14064. Available from: https://doi.org/10.1038/s41598-018-32165-z.
  * [185] Schubert C, Schlippert D, Abend S, et al. Scalable, symmetric atom interferometer for infrasound gravitational wave detection. arXiv:190901951. 2019;Available from: https://arxiv.org/abs/1909.01951.
  * [186] Yu N, Tinto M. Gravitational wave detection with single-laser atom interferometers. General Relativity and Gravitation. 2011;43(7):1943–1952. Available from: https://doi.org/10.1007/s10714-010-1055-8.
  * [187] Abe M, Adamson P, Borcean M, et al. Matter-wave atomic gradiometer interferometric sensor (MAGIS-100). Quantum Science and Technology. 2021;Available from: http://iopscience.iop.org/article/10.1088/2058-9565/abf719.
  * [188] Badurina L, Bentine E, Blas D, et al. AION: an atom interferometer observatory and network. J Cosmol Astropart Phys. 2020 may;2020(05):011–011. Available from: https://doi.org/10.1088/1475-7516/2020/05/011.
  * [189] Arvanitaki A, Graham PW, Hogan JM, et al. Search for light scalar dark matter with atomic gravitational wave detectors. Phys Rev D. 2018 Apr;97:075020. Available from: https://link.aps.org/doi/10.1103/PhysRevD.97.075020.
  * [190] Graham PW, Hogan JM, Kasevich MA, et al. Mid-band gravitational wave detection with precision atomic sensors. arXiv:171102225. 2017;Available from: https://arxiv.org/abs/1711.02225.
  * [191] Tino GM, Bassi A, Bianco G, et al. SAGE: A proposal for a space atomic gravity explorer. The European Physical Journal D. 2019;73(11):228. Available from: https://doi.org/10.1140/epjd/e2019-100324-6.
  * [192] El-Neaj YA, Alpigiani C, Amairi-Pyka S, et al. AEDGE: Atomic experiment for dark matter and gravity exploration in space. EPJ Quantum Technology. 2020;7:6. Available from: https://doi.org/10.1140/epjqt/s40507-020-0080-0.
  * [193] Hensel T, Loriani S, Schubert C, et al. Inertial sensing with quantum gases: a comparative performance study of condensed versus thermal sources for atom interferometry. Eur Phys J D. 2021 Mar;75(3):108. Available from: https://doi.org/10.1140/epjd/s10053-021-00069-9.
  * [194] Louchet-Chauvet A, Farah T, Bodart Q, et al. The influence of transverse motion within an atomic gravimeter. New J Phys. 2011 Jun;13(6):065025. Available from: https://iopscience.iop.org/article/10.1088/1367-2630/13/6/065025.
  * [195] Schkolnik V, Leykauf B, Hauth M, et al. The effect of wavefront aberrations in atom interferometry. Appl Phys B. 2015 Jun;120(2):311–316. Available from: https://doi.org/10.1007/s00340-015-6138-5.
  * [196] Freier C, Hauth M, Schkolnik V, et al. Mobile quantum gravity sensor with unprecedented stability. J Phys: Conf Ser. 2016 Jun;723:012050. Available from: https://iopscience.iop.org/article/10.1088/1742-6596/723/1/012050/meta.
  * [197] Karcher R, Imanaliev A, Merlet S, et al. Improving the accuracy of atom interferometers with ultracold sources. New J of Phys. 2018 Nov;20(11):113041. Available from: https://iopscience.iop.org/article/10.1088/1367-2630/aaf07d.
  * [198] Baym G, Pethick CJ. Ground-state properties of magnetically trapped Bose-condensed rubidium gas. Phys Rev Lett. 1996 Jan;76:6–9. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.76.6.
  * [199] Cornell EA, Wieman CE. Nobel lecture: Bose-Einstein condensation in a dilute gas, the first 70 years and some recent experiments. Rev Mod Phys. 2002 Aug;74:875–893. Available from: https://link.aps.org/doi/10.1103/RevModPhys.74.875.
  * [200] Ketterle W. Nobel lecture: When atoms behave as waves: Bose-Einstein condensation and the atom laser. Rev Mod Phys. 2002 Nov;74:1131–1151. Available from: https://link.aps.org/doi/10.1103/RevModPhys.74.1131.
  * [201] Dalfovo F, Giorgini S, Pitaevskii LP, et al. Theory of Bose-Einstein condensation in trapped gases. Rev Mod Phys. 1999 Apr;71:463–512. Available from: https://link.aps.org/doi/10.1103/RevModPhys.71.463.
  * [202] Penrose O, Onsager L. Bose-Einstein condensation and liquid helium. Phys Rev. 1956 Nov;104:576–584. Available from: https://link.aps.org/doi/10.1103/PhysRev.104.576.
  * [203] Pitaevskii LP. Vortex lines in an imperfect Bose gas. Sov Phys JETP. 1961 Aug;13(2):451–454.
  * [204] Gross EP. Structure of a quantized vortex in Boson systems. Nuovo Cim. 1961 May;20(3):454–477. Available from: https://doi.org/10.1007/BF02731494.
  * [205] Bogoliubov NN. Lectures on quantum statistics. Vol. 2. New York: Gordon and Breach; 1970.
  * [206] Yukalov VI. Basics of Bose-Einstein condensation. Phys Part Nucl Lett. 2011 May;42(3):460–513. Available from: https://doi.org/10.1134/S1063779611030063.
  * [207] Castin Y, Dum R. Low-temperature Bose-einstein condensates in time-dependent traps: 0beyond the $u(1)$ symmetry-breaking approach. Phys Rev A. 1998 Apr;57:3008–3021. Available from: https://link.aps.org/doi/10.1103/PhysRevA.57.3008.
  * [208] Gardiner CW. Particle-number-conserving Bogoliubov method which demonstrates the validity of the time-dependent Gross-Pitaevskii equation for a highly condensed Bose gas. Phys Rev A. 1997 Aug;56:1414–1423. Available from: https://link.aps.org/doi/10.1103/PhysRevA.56.1414.
  * [209] Debs JE, Altin PA, Barter TH, et al. Cold-atom gravimetry with a Bose-Einstein condensate. Phys Rev A. 2011 Sep;84:033610. Available from: https://link.aps.org/doi/10.1103/PhysRevA.84.033610.
  * [210] Ammann H, Christensen N. Delta kick cooling: A new method for cooling atoms. Phys Rev Lett. 1997 Mar;78:2088–2091. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.78.2088.
  * [211] Morinaga M, Bouchoule I, Karam JC, et al. Manipulation of motional quantum states of neutral atoms. Phys Rev Lett. 1999 Nov;83:4037–4040. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.83.4037.
  * [212] Kovachy T, Hogan JM, Sugarbaker A, et al. Matter wave lensing to picokelvin temperatures. Phys Rev Lett. 2015 Apr;114:143004. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.114.143004.
  * [213] Castin Y, Dum R. Bose-Einstein condensates in time dependent traps. Phys Rev Lett. 1996 Dec;77:5315–5319. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.77.5315.
  * [214] Kagan Y, Surkov EL, Shlyapnikov GV. Evolution of a Bose-condensed gas under variations of the confining potential. Phys Rev A. 1996 Sep;54:R1753–R1756. Available from: https://link.aps.org/doi/10.1103/PhysRevA.54.R1753.
  * [215] Meister M, Arnold S, Moll D, et al. Chapter six - Efficient description of Bose–Einstein condensates in time-dependent rotating traps. In: Arimondo E, Lin CC, Yelin SF, editors. Adv. at., mol., opt. phys. Vol. 66. Elsevier; 2017. p. 375–438. Available from: http://www.sciencedirect.com/science/article/pii/S1049250X17300174.
  * [216] Guéry-Odelin D. Mean-field effects in a trapped gas. Phys Rev A. 2002 Sep;66:033613. Available from: https://link.aps.org/doi/10.1103/PhysRevA.66.033613.
  * [217] Myatt CJ, Burt EA, Ghrist RW, et al. Production of two overlapping Bose-Einstein condensates by sympathetic cooling. Phys Rev Lett. 1997 Jan;78:586–589. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.78.586.
  * [218] Riboli F, Modugno M. Topology of the ground state of two interacting Bose-Einstein condensates. Phys Rev A. 2002 Jun;65:063614. Available from: https://link.aps.org/doi/10.1103/PhysRevA.65.063614.
  * [219] Ho TL, Shenoy VB. Binary mixtures of Bose condensates of alkali atoms. Phys Rev Lett. 1996 Oct;77:3276–3279. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.77.3276.
  * [220] McNamara JM, Jeltes T, Tychkov AS, et al. Degenerate Bose-Fermi mixture of metastable atoms. Phys Rev Lett. 2006 Aug;97:080404. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.97.080404.
  * [221] Ufrecht C, Meister M, Roura A, et al. Comprehensive classification for Bose-Fermi mixtures. New J Phys. 2017 Aug;19(8):085001. Available from: https://iopscience.iop.org/article/10.1088/1367-2630/aa7814/meta.
  * [222] Corgier R, Loriani S, Ahlers H, et al. Interacting quantum mixtures for precision atom interferometry. New J Phys. 2020 Dec;22(12):123008. Available from: https://doi.org/10.1088/1367-2630/abcbc8.
  * [223] Jannin R, Cladé P, Guellati-Khélifa S. Phase shift due to atom-atom interactions in a light-pulse atom interferometer. Phys Rev A. 2015 Jul;92:013616. Available from: https://link.aps.org/doi/10.1103/PhysRevA.92.013616.
  * [224] Stickney JA, Anderson DZ, Zozulya AA. Increasing the coherence time of Bose-Einstein-condensate interferometers with optical control of dynamics. Phys Rev A. 2007 Jun;75:063603. Available from: https://link.aps.org/doi/10.1103/PhysRevA.75.063603.
  * [225] Jamison AO, Plotkin-Swing B, Gupta S. Advances in precision contrast interferometry with Yb Bose-Einstein condensates. Phys Rev A. 2014 Dec;90:063606. Available from: https://link.aps.org/doi/10.1103/PhysRevA.90.063606.
  * [226] Fattori M, D’Errico C, Roati G, et al. Atom interferometry with a weakly interacting Bose-Einstein condensate. Phys Rev Lett. 2008 Feb;100:080405. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.100.080405.
  * [227] Leggett AJ, Sols F. On the concept of spontaneously broken gauge symmetry in condensed matter physics. Found Phys. 1991 Mar;21(3):353–364. Available from: https://doi.org/10.1007/BF01883640.
  * [228] Lewenstein M, You L. Quantum phase diffusion of a Bose-Einstein condensate. Phys Rev Lett. 1996 Oct;77:3489–3493. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.77.3489.
  * [229] Javanainen J, Wilkens M. Phase and phase diffusion of a split Bose-Einstein condensate. Phys Rev Lett. 1997 Jun;78:4675–4678. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.78.4675.
  * [230] Wright EM, Walls DF, Garrison JC. Collapses and revivals of Bose-Einstein condensates formed in small atomic samples. Phys Rev Lett. 1996 Sep;77:2158–2161. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.77.2158.
  * [231] Sinatra A, Castin Y. Binary mixtures of Bose-Einstein condensates: Phase dynamics and spatial dynamics. Eur Phys J D. 2000 Jul;8(3):319. Available from: https://doi.org/10.1007/s100530050042.
  * [232] Roura A. Circumventing Heisenberg’s uncertainty principle in atom interferometry tests of the equivalence principle. Phys Rev Lett. 2017 Apr;118:160401. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.118.160401.
  * [233] Lan SY, Kuan PC, Estey B, et al. Influence of the Coriolis force in atom interferometry. Phys Rev Lett. 2012 Feb;108:090402. Available from: http://link.aps.org/doi/10.1103/PhysRevLett.108.090402.
  * [234] Stockton JK, Takase K, Kasevich MA. Absolute geodetic rotation measurement using atom interferometry. Phys Rev Lett. 2011 Sep;107:133001. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.107.133001.
  * [235] Rosi G. A proposed atom interferometry determination of $G$ at $10^{-5}$ using a cold atomic fountain. Metrologia. 2018;55(1):50–55. Available from: https://doi.org/10.1088/1681-7575/aa8fd8.
  * [236] D’Amico G, Rosi G, Zhan S, et al. Canceling the gravity gradient phase shift in atom interferometry. Phys Rev Lett. 2017;119(25):253201.
  * [237] Overstreet C, Asenbaum P, Kovachy T, et al. Effective inertial frame in an atom interferometric test of the equivalence principle. Phys Rev Lett. 2018 May;120:183604. Available from: https://link.aps.org/doi/10.1103/PhysRevLett.120.183604.
  * [238] Ufrecht C. Generalized gravity-gradient mitigation scheme. Phys Rev A. 2021 Feb;103:023305. Available from: https://link.aps.org/doi/10.1103/PhysRevA.103.023305.
  * [239] Chiow Sw, Williams J, Yu N, et al. Gravity-gradient suppression in spaceborne atomic tests of the equivalence principle. Phys Rev A. 2017 Feb;95:021603. Available from: https://link.aps.org/doi/10.1103/PhysRevA.95.021603.
  * [240] Loriani S, Schubert C, Schlippert D, et al. Resolution of the colocation problem in satellite quantum tests of the universality of free fall. Phys Rev D. 2020 Dec;102:124043. Available from: https://link.aps.org/doi/10.1103/PhysRevD.102.124043.


11institutetext: The University of Texas at Austin, 301 E Dean Keeton St,
Austin, TX 78712 22institutetext: IATE, Univ. Montpellier, CIRAD, INRAE,
Montpellier SupAgro, Montpellier, France

# Investigating the effect of porosity on the soil water retention curve using
the multiphase Lattice Boltzmann Method

Reihaneh Hosseini 11<EMAIL_ADDRESS>Krishna Kumar 11<EMAIL_ADDRESS>Jean-Yves Delenne 22

###### Abstract

The soil water retention curve (SWRC) is the most commonly used relationship
in the study of unsaturated soil. In this paper, the effect of porosity on the
SWRC is investigated by numerically modeling unsaturated soil using the Shan-
Chen multiphase Lattice Boltzmann Method. The shape of simulated SWRCs are
compared against that predicted by the van Genuchten model, demonstrating a
good fit except at low degrees of saturation. The simulated SWRCs show an
increase in the air-entry value as porosity decreases.

##  1 Introduction

The Soil Water Retention Curve (SWRC) correlates the degree of saturation or
water content of unsaturated soil to its matric suction or capillary pressure.
Idealized functional forms of SWRC Brooks1964a ; VanGenuchten1980a  are often
used to estimate mechanical properties of unsaturated soil such as effective
stress and shear strength Khalili1998a ; Lu2006a  as well as hydraulic
properties such as relative permeability Mualem1976 ; FREDLUND1994a . As the
soil undergoes volumetric deformation during shearing, its water retention
behavior changes with the change of porosity. Many researchers have proposed
models that consider the effect of porosity (or void ratio) on SWRC
Aubertin1998a ; Gallipoli2003a ; Tarantino2009a ; Masin2009a ; Zhou2012a . In
this study, we investigate the effect of porosity on the suction
characteristics of unsaturated soil from a micromechanics perspective
Richefeu2016a , by simulating the multiphase system using the Shan-Chen
Lattice Boltzmann Method (LBM).

##  2 Numerical method

An in-house 3D multiphase LBM code is developed for modelling the multiphase
fluid domain of unsaturated granular material. The D3Q19 scheme Qian1992  for
velocity discretization and the BGK collision operator are used. The
interaction between the three phases, i.e. liquid, gas and solids, is
considered by introducing Shan-Chen (SC)-type interaction forces into the
model Shan1993a ; Shan1994 . The SC interaction forces per unit volume are
calculated using

$\boldsymbol{F}_{SC}(\boldsymbol{x})=-\psi(\boldsymbol{x})G\sum_{i}w_{i}\psi(\boldsymbol{x}+\boldsymbol{c_{i}}\Delta
t)\boldsymbol{c_{i}}\Delta t,$ (1)

where $\psi$ is an effective density, $G$ is a parameter that controls the
strength of the interaction (negative for attraction),
$\\{\boldsymbol{c_{i}}\\}$ is the discrete velocity set with $\\{w_{i}\\}$ as
the corresponding weights, and $\Delta t$ is the time step which is set to 1
dimensionless lattice unit (lu). $\psi$ is defined as

$\psi(\rho)=\sqrt{\frac{2}{c_{s}^{2}\Delta t^{2}G}(p(\rho)-\rho c_{s}^{2})},$
(2)

where $c_{s}$ is lattice sound speed equal to $\frac{1}{\sqrt{3}}\frac{\Delta
x}{\Delta t}$ for the D3Q19 model Qian1992 ; He1997  with both $\Delta x$ and
$\Delta t$ set to 1 lu, $\rho$ is density at point x, and $p$ is pressure.
Equation 2 is a rearrangement of the Equation of State (EOS) of the SC-type
multiphase system, which allows incorporating different EOS by simply
redefining $p$ Yuan2006a . The Carnahan-Starling (C-S) EOS is used to increase
the numerical stability for systems with large liquid-gas density ratios:

$p=\rho
RT\frac{1+b\rho/4+(b\rho/4)^{2}-(b\rho/4)^{3}}{(1-b\rho/4)^{3}}-a\rho^{2},$
(3)

with $a=0.4963R^{2}T_{c}^{2}/p_{c}$ and $b=0.18727RT_{c}/p_{c}$ Yuan2006a .
$T_{c}$ is the temperature below which phase separation occurs and $p_{c}$ is
the pressure at which the first and second derivatives of the EOS at $T_{c}$
are zero. The C-S EOS curves for different temperatures are shown in Figure 1.

The developed code is validated by comparing the coexistence densities, the
densities at which the liquid and gas phases coexist in a multiphase system,
from the numerical simulations against the theoretical solution, at the
different $T/T_{c}$ values shown in Figure 1. The density at each node is
initialized to a value between the liquid and gas densities, with a small
random perturbation to allow phase separation, and the system is allowed to
equilibrate. Once a liquid droplet is formed, the average liquid and gas
densities are measured, and the corresponding pressures are calculated using
Equation 3. For this study $a=1$ lu, $b=4$ lu and $R=1$ lu are used Yuan2006a
. Figure 1 shows a good match between the numerical and the theoretical
solutions.

Figure 1: Comparison of the numerical (multiphase LBM simulation) and
theoretical coexistence curves for the C-S EOS at different $T/T_{c}$.

##  3 Model description and simulation

In this study, we use four different grain configurations with initial
porosities ($\eta=V_{void}/V_{total}$) of 0.50, 0.40, 0.30, 0.15
(corresponding to void ratios of 1.00, 0.67, 0.43, 0.18) to investigate the
effect of porosity on the SWRC. These configurations are created by randomly
positioning grains inside a fixed-size domain until the required porosity is
achieved. As the objective of this study is to develop SWRCs at constant
porosities, the grain positions are fixed and grain-grain interactions are not
considered. The granular assembly comprises of grain diameters between 10 lu
and 30 lu, with a uniform distribution. The lattice spacing, $\Delta x$, is
set to 1 lu, which corresponds to a minimum of 10 grid points per grain
diameter. The LBM domain is set to $200\times 200\times 200$ lu3, with
periodic boundaries in every direction to eliminate boundary effects.

In all simulations, the relaxation parameter, $\tau$, for the BGK collision
operator is set to 1 lu. The parameters for the C-S EOS are chosen as $a=1$
lu, $b=4$ lu and $R=1$ lu Yuan2006a , corresponding to $T_{c}=0.0943$ lu,
$p_{c}=0.0044$ lu, and $\rho_{c}=0.13044$ lu. $T/T_{c}$ of 0.7 is selected for
these simulations. $G$ is set to -1 (by using Equation 2 for $\psi$, the
parameter $G$ cancels out when this equation is substituted in Equation 1,
therefore, the magnitude of $G$ is irrelevant and only its sign is of
importance). The density of solids, $\rho_{s}$, in a multiphase simulation
only controls the contact angle between the solids and the liquid and does not
correspond to the physical density of the solids; $\rho_{s}$ closer to the
liquid density, $\rho_{l}$, results in a more hydrophilic surface, whereas
$\rho_{s}$ closer to the gas density, $\rho_{g}$, creates a more hydrophobic
surface Kruger2017 . In these simulations, $\rho_{s}$ is set to
0.98$\rho_{l}$, corresponding to a contact angle of 14∘.

The dimensionless lattice units, lu, used for all the parameters in this study
can be converted to physical units by choosing appropriate conversion factors.
It can be shown that the fluid kinematic viscosity in an LBM simulation is
$\nu=c_{s}^{2}(\tau-0.5)$ in lu Qian1992 ; the conversion factors for length,
$C_{l}$, and time, $C_{t}$, should be chosen such that
$c_{s}^{2}(\tau-0.5)C_{l}^{2}/C_{t}$ corresponds to the correct physical
kinematic viscosity. In addition to this criteria, accuracy and stability
should be taken into account when choosing these conversion factors. For this
qualitative study, all values are simply reported in lu.

In all four configurations of the granular assembly, the density of the fluid
domain is initialized to a value between the liquid and gas coexistence
densities (0.358 lu and 0.009 lu, respectively), with a random perturbation of
$\pm 0.1$, to allow phase separation. After an adequate number of steps, when
the fluid has phase separated and reached equilibrium, liquid is injected into
the system by increasing the density of all fluid nodes by $0.005$ lu every
1000 steps. It is ensured that 1000 steps is enough for the system to reach
equilibrium before the next injection. The SWRC is generated by calculating
the average suction, $\delta p$, and degrees of saturation, $\mathit{S_{r}}$,
at the end of each injection stage. $\delta p$ is defined as the difference
between the average gas pressure and the average liquid pressure in the entire
fluid domain. $\mathit{S_{r}}$ is measured by dividing the total number of
liquid nodes by the total number of fluid nodes (voids). In addition, the
number of liquid clusters at each injection stage is computed using the Depth-
First Search algorithm tarjan1972depth .

##  4 Results and discussion

Figure 2 shows a 3D view of the grain configuration with $\eta$ = 0.40 and the
liquid clusters formed at $\mathit{S_{r}}$ = 0.4. Figure 3 shows snapshots of
2D slices of the unsaturated granular media, indicating that multiphase LBM is
able to capture the different states of liquid clustering: pendular (3a),
funicular (3b), capillary (3c), and droplet states mitarai2006wet . In the
pendular state, pairs of grains are connected by binary liquid bridges (e.g.
dashed-circle zones). In the funicular state, a combination of binary liquid
bridges (e.g. dashed-circle zone) and liquid clusters connecting multiple
grains (e.g. dashed-triangle zone) is present. In the capillary state, the
liquid almost fills the entire pore space between grains, however, its surface
forms concave menisci (e.g. dashed-square zones) exerting a net cohesive force
on the grains. In the droplet state, the liquid immerses the entire domain and
exerts no capillary forces on the grains.

Figure 2: 3D view of the multiphase domain with $\eta$ = 0.40 at
$\mathit{S_{r}}$ = 0.4. Figure 3: Snapshots of 2D slices of the multiphase
domain with $\eta$ = 0.40 at (a) $\mathit{S_{r}}$ = 0.1, (b) $\mathit{S_{r}}$
= 0.4 and (c) $\mathit{S_{r}}$ = 0.7. The dashed-circle zones show examples of
binary liquid bridges, the dashed-triangle zone shows an example of a liquid
cluster connecting multiple grains, and the dashed-square zones show examples
of concave menisci in the capillary state.

The pore-scale evolution of clusters with increase in saturation is
investigate by examining the evolution of number of liquid clusters. Figure 4
shows the number of liquid clusters normalized by the number of disconnected
pores in the soil structure ($N_{cluster}/N_{pores}$) as a function of
$\mathit{S_{r}}$, for different porosities. The circles indicate points at
which snapshots of the simulation are provided in Figure 3. As
$\mathit{S_{r}}$ increases from zero, $N_{cluster}/N_{pores}$ initially
increases rapidly up to a peak value, due to the formation of a large number
of small liquid droplets on the surface of the grains. With further increase
in saturation, these small droplets aggregate together to form liquid films
around the grains and subsequently coalesce to binary bridges between grains,
thereby decreasing the number of individual clusters at a steep rate. The
number of distinct clusters gradually decreases as the system moves from the
pendular state to the funicular state, and finally, to the capillary state.
Eventually, all the pores are filled with liquid, and the cluster count
becomes equal to the number of disconnected pore spaces.

As the porosity decreases, the peak of the curves in Figure 4 shift to the
right, indicating that a higher $\mathit{S_{r}}$ is required for the initial
clusters to connect and form liquid films on the grain surfaces. This is
expected as lower porosity corresponds to a larger number of tiny pore spaces
and a higher surface area of the granular assembly. In addition, in the case
of the lowest porosity, the number of distinct clusters decreases more
gradually throughout the transition from the pendular state to the capillary
state, compared to the other porosity cases.

Figure 4: Evolution of normalized number of clusters as a function of degree
of saturation, for different porosities. The circles indicate points at which
snapshots of the simulation are provided in Figure 3.

The SWRCs simulated for different porosities are shown in Figure 5. The
simulated SWRCs are compared against fitted curves based on the simplified van
Genuchten model VanGenuchten1980a ; Gallipoli2003a

$\mathit{S_{r}}=(\frac{1}{1+(\alpha\delta p)^{n}})^{m},$ (4)

where $\alpha$, $n$ and $m$ are fitting parameters. The fitting parameters for
the curves in this study are reported in Table 1. Parameter $\alpha$ controls
the shift of the curve to left and right and is inversely proportional to the
air-expulsion/air-entry value (AEV). It can be deduced from the shift of the
simulated curves in Figure 5 or the value of $\alpha$ in Table 1, that the AEV
for the simulated SWRCs increases with decreasing porosity. The inverse
relation between the AEV and porosity is consistent with all proposed models
for the SWRC which take void ratio into account Aubertin1998a ; Gallipoli2003a
; Tarantino2009a ; Masin2009a ; Zhou2012a . Parameter $n$ controls the slope
of the curve and is related to the pore size distribution, with higher $n$
corresponding to a more uniform distribution Matlan2014a . Comparing the $n$
parameter for the different curves shows that the curves with the highest and
lowest porosities have steeper slopes at the middle section of the SWRC,
compared to the curves with intermediate porosities. This could be explained
by assuming that the lowest and highest porosity models have uniform small and
large pores, respectively, while the intermediate porosity models have a
combination of small and large pores; however, the verification of this
assumption is left for future studies. Parameter $m$ controls the symmetry of
the curve when $\delta p$ is plotted in log space. As seen in Figure 5, the
maximum suction of the simulated curves is constrained, creating an
unsymmetrical shape for the SWRC at lower porosities. Therefore, $m$ is higher
for lower porosities.

Figure 5: SWRCs from the LBM simulations for different porosities. The dashed lines represent the fitted curves based on the van Genuchten model. Table 1: van Genuchten fitting parameters. $\eta$ | $\alpha$ | $n$ | $m$
---|---|---|---
0.15 | 67 | 6 | 4.50
0.30 | 79 | 4 | 3.47
0.40 | 119 | 4 | 1.66
0.50 | 228 | 5 | 0.75

The SWRC for the simulation with the highest porosity ($\eta$ = 0.50) has the
expected “S” shape and the fitted van Genuchten (vG) curve matches the trend
of the data. For the two simulations with intermediate porosities, the fitted
vG curves match the simulated curves perfectly in the $\mathit{S_{r}}$ range
of 0.1 to about 0.8. However, at higher saturation ($\mathit{S_{r}}$ > 0.8),
the vG curves smoothly transition to zero suction while the simulated curves
continue with the same slope up to $\mathit{S_{r}}$ = 0.96, where the suction
abruptly drops to zero. Furthermore, as the saturation decreases below 0.1 for
these intermediary porosity cases, LBM simulations only indicate a very small
increase in suction, which contrasts the dramatic increase predicted by vG
curves. For the lowest porosity case ($\eta$ = 0.15), the smooth transition to
zero suction at high $\mathit{S_{r}}$ occurs as expected, however, at
$\mathit{S_{r}}$ below 0.2, the suction remains unchanged. Interestingly, all
simulated curves converge to the same value of maximum suction, indicating
that the maximum suction in these multiphase LBM simulations may be
constrained by the modeling parameters. Further research is required to
understand the cause of this behavior.

##  5 Conclusion

The effect of porosity on the Soil Water Retention Curve (SWRC) was studied by
simulating unsaturated soil using the 3D multiphase Shan-Chen-type Lattice
Boltzmann Method. Four different grain configurations with porosities ranging
from 0.15 to 0.50 were used. The four different states of liquid clustering,
namely pendular, funicular, capillary, and droplet states, were observed in
the multiphase simulation for the different porosity cases. The simulations
were able to correctly capture the increase of the air-entry value with
decreasing porosity and the shift of the SWRC towards higher suction. However,
the results indicated that the maximum suction is constrained at low degrees
of saturation and further research is needed to explain this behavior.

## References

  * (1) R.H. Brooks, A.T. Corey, Hydrology Papers 3 (1964)
  * (2) M.T. van Genuchten, Soil Sci. Soc. Am. J. 44, 892 (1980)
  * (3) N. Khalili, M.H. Khabbaz, GÃotechnique 48, 681 (1998)
  * (4) N. Lu, W.J. Likos, J. Geotech. Geoenviron. Eng. 132, 131 (2006)
  * (5) Y. Mualem, Water Resour. Res. 12, 513 (1976)
  * (6) D.G. Fredlund, A. Xing, S. Huang, Can. Geotech. J. 31, 533 (1994)
  * (7) M. Aubertin, J.F. Richard, R.P. Chapuis, Can. Geotech. J. 35, 55 (1998)
  * (8) D. Gallipoli, S.J. Wheeler, M. Karstunen, Géotechnique 53, 105 (2003)
  * (9) A. Tarantino, Géotechnique 59, 751 (2009)
  * (10) D. Masin, Int. J. Numer. Anal. Meth. Geomech. 34, 73 (2009)
  * (11) A.N. Zhou, D. Sheng, J.P. Carter, GÃotechnique 62, 669 (2012)
  * (12) V. Richefeu, F. Radjai, J.Y. Delenne, Comput. Geotech. 80, 353 (2016)
  * (13) Y.H. Qian, D. D’Humières, P. Lallemand, EPL 17, 479 (1992)
  * (14) X. Shan, H. Chen, Phys. Rev. E 47, 1815 (1993)
  * (15) X. Shan, H. Chen, Phys. Rev. E 49, 2941 (1994)
  * (16) X. He, L.S. Luo, Phys. Rev. E 56, 6811 (1997)
  * (17) P. Yuan, L. Schaefer, Phys. Fluids 18, 042101 (2006)
  * (18) T. Krüger, H. Kusumaatmaja, A. Kuzmin, O. Shardt, G. Silva, E.M. Viggen, _The Lattice Boltzmann Method_ (Springer, 2017)
  * (19) R. Tarjan, SIAM J. Comput. 1, 146 (1972)
  * (20) N. Mitarai, F. Nori, Adv. Phys. 55, 1 (2006)
  * (21) S.J. Matlan, M. Mukhlisin, M.R. Taha, Sci. World J. 2014 (2014)


# OwlEyes-Online: A Fully Automated Platform for Detecting and Localizing UI
Display Issues

Yuhui Su  Institute of Software Chinese Academy of Sciences, University of
Chinese Academy of SciencesBeijingChina ,  Zhe Liu  Institute of Software
Chinese Academy of Sciences, University of Chinese Academy of
SciencesBeijingChina ,  Chunyang Chen  Monash UniversityMelbourneAustralia ,
Junjie Wang  Institute of Software Chinese Academy of Sciences, University of
Chinese Academy of SciencesBeijingChina and  Qing Wang  Institute of Software
Chinese Academy of Sciences, University of Chinese Academy of
SciencesBeijingChina

(2021)

###### Abstract.

GUI provides visual bridges between software apps and end users. However, due
to the compatibility of software or hardware, UI display issues such as text
overlap, blurred screen, image missing always occur during GUI rendering on
different devices. Because these UI display issues can be found directly by
human eyes, in this paper, we implement an online UI display issue detection
tool OwlEyes-Online, which provides a simple and easy-to-use platform for
users to realize the automatic detection and localization of UI display
issues. The OwlEyes-Online can automatically run the app and get its
screenshots and XML files, and then detect the existence of issues by
analyzing the screenshots. In addition, OwlEyes-Online can also find the
detailed area of the issue in the given screenshots to further remind
developers. Finally, OwlEyes-Online will automatically generate test reports
with UI display issues detected in app screenshots and send them to users. It
was evaluated and proved to be able to accurately detect UI display issues.
Tool Link: http://www.owleyes.online:7476. Github Link:
https://github.com/franklinbill/owleyes. Demo Video Link:
https://youtu.be/002nHZBxtCY.

UI display, Mobile app, UI testing, Deep learning, Issue detection

††copyright: acmcopyright††price: 15.00††doi:
10.1145/3468264.3473109††journalyear: 2021††submissionid:
fse21demo-p3-p††isbn: 978-1-4503-8562-6/21/08††conference: Proceedings of the
29th ACM Joint European Software Engineering Conference and Symposium on the
Foundations of Software Engineering; August 23–28, 2021; Athens,
Greece††booktitle: Proceedings of the 29th ACM Joint European Software
Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE ’21), August 23–28, 2021, Athens, Greece††ccs: Software
and its engineering

##  1\. Introduction

GUI is widely used among modern mobile apps, making it practical and easy to
use. However, with the development of visual effects of GUI, five categories
of UI display issues (Zhe et al., 2020) such as component occlusion, text
overlap, missing image, null value and blurred screen always occur during the
UI display process, especially on different mobile devices. Detecting those
issues is a hard problem because most of those UI display issues are caused by
many factors, especially for Android, such as different Android OS versions,
device models, and screen resolutions (Wei et al., 2016). Nowadays, some
practical automated testing tools like Monkey (Developers, 2012; Wetzlmaier
and Ramler, 2017), Dynodroid (Machiry et al., 2013) are also widely used in
industry. However, these automated tools can only spot critical crash bugs,
rather than UI display issues that cannot be captured by common tools.
Inspired by the fact that display bugs can be easily spotted by human eyes, we
develop an automated online tool OwlEyes-Online111OwlEyes-Online is named as
our approach is like the owl’s eyes to effectively spot UI display issues. And
our model (nocturnal like an owl) can complement conventional automated GUI
testing (diurnal like an eagle) for ensuring the robustness of the UI., which
provides quick detection and localization of UI display issues from apps or
GUI screenshots.

Figure 1. Overview of OwlEyes-Online

The OwlEyes-Online is a user-friendly web app. Developers can upload GUI
screenshots or apps and receive accurate UI display issue detection results.
When a developer uploads an APK, it will automatically run the app and get its
screenshots, and then we use computer vision technologies to detect the UI
display issues. OwlEyes-Online builds on the CNN to identify the screenshots
with issues and Grad-CAM to localize the regions with UI display issues in the
screenshots for further reminding developers. Finally, it summarizes the
detection and localization results, automatically generates the test report
and sends it to users. Considering that the CNN needs lots of training data,
we adopt a heuristic data generation method to generate the training data.

OwlEyes-Online provides a dashboard for users to upload the screenshots or
apps. After analyzing an uploaded screenshot, it displays detection results in
real-time. As for an app, it automatically generates a test report (issue
screenshots, localization, etc.) and sends the report to the user in an email.

This paper makes the following contributions:

  * •

We implement a CNN based issue detection method and a Grad-CAM based issue
localization method to detect UI display issues from GUI screenshots.

  * •

We develop a fully automated web app. Users only need to upload an APK file,
and OwlEyes-Online will automatically generate test reports and send them to
users. We release the implementation of OwlEyes-Online on Github (Git, 2021).

  * •

An empirical study among professionals proves the value of our UI display
issue detection method and OwlEyes-Online.

##  2\. Our Fully Automated Approach

According to the features of UI display issues, we propose a fully automated
UI display issue detection and localization approach. It mainly includes four
parts, which are heuristic-based data generation, CNN-based issues detection,
Grad CAM-based issues localization, and online inference of GUI issues. As
shown in Figure 1, to improve the accuracy of our model, we use the heuristic-
based data generation method to generate a number of training data. Given an
APK, OwlEyes-Online automatically runs it and collects screenshots. Then the
CNN-based model classifies if they relate to any issues via the visual
understanding. Once an issue is confirmed, our model can further localize its
specific issue position on the UI screenshot by Grad CAM-based model to remind
the developers.

Figure 2. Illustration of our OwlEyes-Online web application.

###  2.1. Heuristic-based Data Generation

Training our proposed CNN for issues detection requires an abundance of
screenshots (He et al., 2016) with UI display issues. However, there is so far
no such type of open dataset, and collecting the related buggy screenshots is
time- and effort-consuming. Therefore, we develop a heuristic-based data
generation method for generating UI screenshots with display issues from bug-
free UI images in Figure 1(a). The data generation is based on the Rico (Deka
et al., 2017) dataset, which contains more than 66K unique screenshots and
their JSON files (i.e., detailed run-time view hierarchy of the screenshot).
With the input screenshot and its associated JSON file, we first localize all
the TextView and ImageView, then randomly chooses a TextView/ImageView
depending on the augmented category. Based on the coordinates and size of the
TextView/ImageView, the algorithm then makes its copy and adjusts its location
or size according to specific rules to generate the screenshots with
corresponding UI display issues.

###  2.2. CNN-based Issues Detection

As the UI display issues can only be spotted via the visual information, we
adopt the convolutional neural network (CNN) (LeCun et al., 1998; Krizhevsky
et al., 2012), which has proven to be effective in image classification and
recognition in computer vision (Simonyan and Zisserman, 2015; Szegedy et al.,
2016; He et al., 2016). Figure 1(b) shows the structure of our model, which
links the convolutional layers, batch normalization layers, pooling layers,
and fully connected layers. Given the input screenshot, we convert it into a
certain image size with fixed width and height as the convolutional layer’s
parameters consist of a set of learnable filters. After the convolutional
layers, the screenshots will be abstracted as a feature graph. In order to
improve the performance and stability of CNN, we add Batch Normalization (BN)
(Ioffe and Szegedy, 2015) layers after the convolutional layer and standardize
the input layer by adjusting and scaling activation. After the BN layer, we
add the Rectified Linear Unit (ReLU) as the activation function of the
network. The last several layers are fully connected neural networks (FC)
which compile the data extracted by previous layers to form the final output.
Finally, we obtain the detection results through softmax (Bishop, 2006).

###  2.3. Grad CAM-based Issues Localization

As shown in Figure 1(c), we adopt the feature visualization method to localize
the detailed position of the issues to remind the developers. We apply the
Grad-CAM model for the localization of UI display issues. Gradient weighted
Class Activation Mapping (Grad-CAM) is a technique for visualizing the regions
of input that are “important” for predictions on CNN-based models (Selvaraju
et al., 2017) . First, a screenshot with the UI display issue is input into
the trained CNN model, and the category supervisor to which the image belongs
is set to 1, while the rest is 0. Then the information is propagated back to
the convolutional feature map of interest to obtain the Grad-CAM positioning.
Through the feedback of global average pooling of the gradient, the weight
$\alpha_{k}^{b}$ of the importance of neurons is obtained. This weight
captures the importance of the feature map $K$ of the target category $b$
(Bug). By performing the weighted combination of the forward activation graph,
we can obtain the class-discriminative localization map. Finally, the point
multiplication with the backpropagation can obtain the Grad-CAM as the result
of issue localization.

###  2.4. Online Inference of GUI Issues

We use 20,000 screenshots generated in section 2.1 to train our issue
detection and localization model. Before the issue detection, we need to
preprocess the APK submitted by the user online. As shown in Figure 1(d), the
user provides an Android APK, and we use the dynamic analysis method to run
the app automatically to obtain the screenshots. In detail, by leveraging the
idea of dynamic app GUI testing (Li et al., 2017; Cai et al., 2020;
Developers, 2012; Su et al., 2017), we adopt an app explorer (Li et al., 2017)
to automatically explore the pages within an application through interacting
with apps using random actions, e.g., clicking, scrolling, and filling in
text. We also provide three testing strategies for users to choose from:
Depth-First-Search (DFS) (Shwail et al., 2013), Breadth-First-Search (BFS)
(Beamer et al., 2012), and random exploration (Developers, 2012).

##  3\. Tool Implementation And Usage

OwlEyes-Online is a web app, which provides a convenient tool for users to
detect and localize the UI display issues in the GUI screenshots.

###  3.1. Web Implementation

OwlEyes-Online can automatically run applications and generate test reports
for users. We customized the deep learning model in PyTorch. The OwlEyes-
Online consists of two parts: running the application automatically, feeding
back the test results in real-time.

Running the app automatically: Figure 2(a) shows an example of our running the
app automatically page. Users can upload the APK or its download link. In
addition, we allow users to customize the exploration strategy, select the
appropriate device, and some personalization settings to provide a more
friendly interactive experience.

The Real-time feedback issue detection results: This page in Figure 2(b) will
give real-time feedback test results while running the application
automatically. On this page, we implement some functions to provide a more
friendly interactive experience, including:

Click to view the localization details: In Figure 2(c), click the screenshot
of the UI display issue to view the localization of it in the screenshot (in
the form of a thermal graph).

Export test report: In Figure 2(d), users fill in e-mail information, and we
will automatically generate test reports and send them to users. The test
report includes the number of issues of the application and the screenshots of
the issue and the XML corresponding to the screenshots.

###  3.2. Model Implementation

Our CNN model is composed of 12 convolutional layers with batch normalization,
6 pooling layers, and 4 full connection layers for classifying UI screenshot
with display issues. The size of a convolutional kernel in the convolutional
layer is 3 $\times$ 3\. We set up the number of convolutional kernels as 16
for convolutional layer 1-4, 32 for convolutional layer 5-6, 64 for
convolutional layer 7-8, and 128 for convolutional layer 9-12. For the pooling
layers, we use the most common-used max-pooling settings (Simard et al.,
2003), i.e., pooling units of size 2 $\times$ 2 applied with a stride
(Simonyan and Zisserman, 2015). We set the number of neurons in each of the
fully connected layers as 4096, 1024, 128, and 2 respectively. For data
preprocessing, we rotate some UI of the horizontal screens to vertical, and
resize the screens to 768 $\times$ 448\. We implement our model based on the
PyTorch (pyt, 2021) framework. The model is trained in an NVIDIA GeForce RTX
2060 GPU (16G memory) with 100 epochs for about 8 hours.

###  3.3. Usage Scenarios

We present several examples to illustrate how developers would interact with
OwlEyes-Online. In some cases, developers collect a large number of
screenshots of applications (such as crowdtesting platform, automatic
testing). However, these automated tools can only spot critical crash bugs,
rather than UI display issues that cannot be captured by common tools.
Developers can upload application screenshots to our OwlEyes-Online directly.
OwlEyes-Online will analyze the screenshots and detect the UI display issue in
the screenshots.

For testing whether UI display issues exist in the application, developers can
directly upload an APK to our OwlEyes-Online , which will automatically
explore the application and detect UI display issues. Developers can also
customize the exploration method and duration and submit the e-mail
information. After the issue detection, OwlEyes-Online will automatically
generate the issue report and send it to the developer’s e-mail. Considering
the network delay, developers can also upload an application’s download link,
and OwlEyes-Online will automatically download the APK in the background for
testing.

##  4\. Evaluation

The goal of our study is to evaluate the usefulness of our platform OwlEyes-
Online in terms of (i) its effectiveness in detecting and localizing UI
display issues, and (ii) the usability of our OwlEyes-Online.

###  4.1. Effectiveness Measurement

Given the effectiveness of our OwlEyes-Online for UI display issue detection,
we conduct experiments on 8K Android mobile GUI collected by one of the
largest crowd-testing platforms (Bai, 2021). This part is also published in
our previous work (Zhe et al., 2020) and and we mainly use evaluation metrics
of precision and recall.

Table 1 shows the performance comparison with the baselines. With OwlEyes-
Online, the precision is 0.85 and the recall is 0.84. We can see that our
proposed OwlEyes-Online is much better than the baselines, i.e., 58% higher in
precision and 17% higher in recall compared with the best baseline, Multilayer
Perceptron (MLP). This further indicates the effectiveness of OwlEyes-Online.
Besides, it also implies that OwlEyes-Online is especially good at hunting for
the buggy screenshots from candidate ones, i.e., significant improvement in
recall.

Table 1. Performance comparison with baselines Method |  Precision |  Recall |  F1-score
---|---|---|---
RF-SIFT |  0.458 |  0.458 |  0.432
RF-SURF |  0.513 |  0.524 |  0.519
RF-ORB |  0.520 |  0.528 |  0.524
MLP |  0.537 |  0.727 |  0.618
OwlEyes-Online |  0.850 |  0.848 |  0.849

###  4.2. Usefulness Measurement

To further assess the usefulness of our approach, we randomly sample 2,000
Android applications from F-Droid (F-d, 2021) and 1,000 applications from
Google Play (Goo, 2021). Note that none of these apps appears in our training
dataset. Among the 3,000 collected applications, 59% (1756/3000) applications
can be successfully run with OwlEyes-Online. For the remaining 1,756
applications, an average of 8 screenshots is obtained for each application. We
then feed those screenshots to our OwlEyes-Online and detect if there are any
display issues. Once a display issue is spotted, we create a bug report by
describing the issue attached with a buggy UI screenshot. Finally, we report
them to the app development team through issue reports or emails. Our OwlEyes-
Online has detected 113 UI display issues, among which 35 have been confirmed
and 29 have been fixed. These fixed or confirmed bug reports further
demonstrate the effectiveness and usefulness of our proposed approach in
detecting UI display issues.

Regarding the user experience of our OwlEyes-Online, we create an online
survey on 20 professional developers, testers, and researchers, all of whom
major in computer science with more than 3 years of app testing or developing
experience. 10 of them are from the industry with practical working
experience222Some testers are from NVIDIA, Citibank, Sony, Baidu, Alibaba,
Three Fast Online, and ByteDance.. We ask them to use our OwlEyes-Online and
ask them about the usefulness of the OwlEyes-Online for their work, as well as
its potential and scalability in the future. In the end, participants fill in
the System Usability Scale (SUS) questionnaire (Brooke et al., 1996) (5-point
Likert scale (Oda et al., 2015) from 1 (strongly disagree) to 5 (strongly
agree)). The questionnaire also asks participants to select the TechLand
system features that they deem most useful or least useful for the tasks.

Figure 3. Average score of SUS results

Figure 3 summarizes the participants’ ratings of the 10 system design and
usability questions in the System Usability Scale questionnaire. The upper
half of figure 3 shows that participants agree or strongly agree that our
system is easy to use and the features of the OwlEyes-Online system are well-
devised. The lower half of figure 3 further confirms the simplicity and
consistency of our OwlEyes-Online system. Furthermore, the average helpfulness
of the OwlEyes-Online system for the tasks is 4.42, which indicates that
participants appreciate the help of the OwlEyes-Online system in the tasks.
All participants indicated that OwlEyes-Online has a good UI display issue
detection effect. Among these professionals, 10 of them are working on app
testing. They think OwlEyes-Online can help them localize the UI display
issues more quickly. 7 Android developers said that our issue localization
model helps them better localize the issue on the UI interface so that they
can better repair the issue later. Among them, 4 developers hope we can
further give the possible repair methods and causes of these issues. The other
3 participants who are studying GUI testing also indicated that they hope we
can analyze the cause of issue in the next stage. They think that using the
visual information of application screenshots is a very helpful and engaging
work.

##  5\. Conclusion

Improving the quality of mobile applications, especially in a proactive way,
is of great value and always encouraged. In this demo, we show OwlEyes-Online,
a fully automated UI display issue detection and localization tool. We use
dynamic analysis to explore the application automatically and get its
screenshots. And users can customize the exploration time, exploration
strategy and so on. Then we can complete the detection and localization of UI
display issues based on CNN and Grad-CAM. Finally, we automatically generate
test reports and send them to users. We evaluate it from two aspects of
detection accuracy and tool practicability. The OwlEyes-Online is proven to be
effective in real-world practice, i.e., 64 confirmed or fixed previously
undetected UI display issues from popular Android apps. It also achieves
boosts of more than 17% and 23% in recall and precision compared with the best
baseline. The evaluation shows that OwlEyes-Online is a good starting point
for UI display issue detection.

In the future, we will further study the root cause of UI display issue.
Finally, according to the issue category, we will devise a set of tools for
recommending patches to developers to fix the UI display issues.

## ACKNOWLEDGMENTS

This work is supported by the National Key Research and Development Program of
China under grant No.2018YFB1403400, National Natural Science Foundation of
China under Grant No. 62072442, No. 62002348.

## References

  * (1)
  * pyt (2021) 2021\.  https://pytorch.org/.
  * F-d (2021) 2021\.  http://f-droid.org/.
  * Goo (2021) 2021\.  http://play.google.com/store/apps/.
  * Bai (2021) 2021\.  Baidu (baidu.com) is the largest Chinese search service provider. Its crowdsourcing test platform (test.baidu.com) is also one of the largest in China.  http://test.baidu.com.
  * Git (2021) 2021\.  Github Link.  https://github.com/franklinbill/owleyes/.
  * Beamer et al. (2012) Scott Beamer, Krste Asanovic, and David Patterson. 2012\.  Direction-optimizing breadth-first search. In _SC’12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis_. IEEE, 1–10.  https://doi.org/10.1109/SC.2012.50
  * Bishop (2006) Christopher M Bishop. 2006\.  _Pattern recognition and machine learning_.  springer.
  * Brooke et al. (1996) John Brooke et al. 1996\.  SUS-A quick and dirty usability scale.  _Usability evaluation in industry_ 189, 194 (1996), 4–7.  https://doi.org/10.1201/9781498710411-35
  * Cai et al. (2020) Tianqin Cai, Zhao Zhang, and Ping Yang. 2020.  Fastbot: A Multi-Agent Model-Based Test Generation System Beijing Bytedance Network Technology Co., Ltd.. In _Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test_. 93–96.  https://doi.org/10.1145/3387903.3389308
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017\.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual Symposium on User Interface Software and Technology_ _(UIST ’17)_.  https://doi.org/10.1145/3126594.3126651
  * Developers (2012) Android Developers. 2012\.  Ui/application exerciser monkey.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun. 2016\.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE Computer Society, Los Alamitos, CA, USA, 770–778.  https://doi.org/10.1109/CVPR.2016.90
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ _(JMLR Workshop and Conference Proceedings, Vol. 37)_. 448–456.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.  https://doi.org/10.1145/3065386
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.  https://doi.org/10.1109/5.726791
  * Li et al. (2017) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017\.  Droidbot: a lightweight ui-guided test input generator for android. In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_. IEEE, 23–26.  https://doi.org/10.1109/ICSE-C.2017.8
  * Machiry et al. (2013) Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013\.  Dynodroid: An Input Generation System for Android Apps. In _Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering_ (Saint Petersburg, Russia) _(ESEC/FSE 2013)_. Association for Computing Machinery, New York, NY, USA, 224–234.  https://doi.org/10.1145/2491411.2491450
  * Oda et al. (2015) Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015.  Learning to generate pseudo-code from source code using statistical machine translation (t). In _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 574–584.  https://doi.org/10.1109/ASE.2015.36
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. In _The IEEE International Conference on Computer Vision (ICCV)_.  https://doi.org/10.1109/ICCV.2017.74
  * Shwail et al. (2013) Safaa H Shwail, Alia Karim, and Scott Turner. 2013\.  Probabilistic multi robot path planning in dynamic environments: A comparison between A* and DFS.  _International Journal of Computer Applications_ 975 (2013), 8887\.
  * Simard et al. (2003) Patrice Y. Simard, Dave Steinkraus, and John C. Platt. 2003\.  Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In _Proceedings of the Seventh International Conference on Document Analysis and Recognition - Volume 2_ _(ICDAR ’03)_. IEEE Computer Society, USA, 958.  https://doi.org/10.1109/ICDAR.2003.1227801
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).
  * Su et al. (2017) Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017.  Guided, stochastic model-based GUI testing of Android apps. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. ACM, 245–256.  https://doi.org/10.1145/3106237.3106298
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2818–2826.  https://doi.org/10.1109/CVPR.2016.308
  * Wei et al. (2016) Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2016.  Taming Android fragmentation: Characterizing and detecting compatibility issues for Android apps. In _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering_. 226–237.  https://doi.org/10.1145/2970276.2970312
  * Wetzlmaier and Ramler (2017) Thomas Wetzlmaier and Rudolf Ramler. 2017.  Hybrid Monkey Testing: Enhancing Automated GUI Tests with Random Test Generation. In _Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing_ (Paderborn, Germany) _(A-TEST 2017)_. Association for Computing Machinery, New York, NY, USA, 5–10.  https://doi.org/10.1145/3121245.3121247
  * Zhe et al. (2020) Liu Zhe, Chen Chunyang, Wang Junjie, Huang Yuekai, Hu Jun, and Wang Qing. 2020\.  Owl Eyes: Spotting UI Display Issues via Visual Understanding. In _2020 35rd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE.  https://doi.org/10.1145/3324884.3416547